Search in sources :

Example 21 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project hadoop-cos by tencentyun.

the class CosNativeFileSystemStore method retrieve.

/**
 * @param key The key is the object name that is being retrieved from the
 *            cos bucket.
 * @return This method returns null if the key is not found.
 * @throws IOException Retrieve the specified cos key with a specified byte range start failed.
 */
@Override
public InputStream retrieve(String key, long byteRangeStart) throws IOException {
    try {
        LOG.debug("Retrieve the cos key: {}, byte range start: {}.", key, byteRangeStart);
        long fileSize = getFileLength(key);
        long byteRangeEnd = fileSize - 1;
        GetObjectRequest getObjectRequest = new GetObjectRequest(this.bucketName, key);
        if (this.trafficLimit >= 0) {
            getObjectRequest.setTrafficLimit(this.trafficLimit);
        }
        this.setEncryptionMetadata(getObjectRequest, new ObjectMetadata());
        if (byteRangeEnd >= byteRangeStart) {
            getObjectRequest.setRange(byteRangeStart, fileSize - 1);
        }
        COSObject cosObject = (COSObject) callCOSClientWithRetry(getObjectRequest);
        return cosObject.getObjectContent();
    } catch (Exception e) {
        String errMsg = String.format("Retrieving key [%s] with byteRangeStart [%d] " + "occurs an exception: %s.", key, byteRangeStart, e);
        handleException(new Exception(errMsg), key);
        // never will get here
        return null;
    }
}
Also used : COSObject(com.qcloud.cos.model.COSObject) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) ObjectMetadata(com.qcloud.cos.model.ObjectMetadata) ResponseNotCompleteException(com.qcloud.cos.exception.ResponseNotCompleteException) CosServiceException(com.qcloud.cos.exception.CosServiceException) CosClientException(com.qcloud.cos.exception.CosClientException) IOException(java.io.IOException)

Example 22 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project hadoop-cos by tencentyun.

the class CosNativeFileSystemStore method retrieveBlock.

@Override
public InputStream retrieveBlock(String key, long byteRangeStart, long byteRangeEnd) throws IOException {
    LOG.debug("Retrieve the cos key: {}, byte range start: {}, byte range end: {}.", key, byteRangeStart, byteRangeEnd);
    try {
        GetObjectRequest request = new GetObjectRequest(this.bucketName, key);
        request.setRange(byteRangeStart, byteRangeEnd);
        if (this.trafficLimit >= 0) {
            request.setTrafficLimit(this.trafficLimit);
        }
        this.setEncryptionMetadata(request, new ObjectMetadata());
        COSObject cosObject = (COSObject) this.callCOSClientWithRetry(request);
        return cosObject.getObjectContent();
    } catch (CosServiceException e) {
        String errMsg = String.format("Retrieving the key %s with the byteRangeStart [%d] " + "occurs an exception: %s.", key, byteRangeStart, e);
        handleException(new Exception(errMsg), key);
    } catch (CosClientException e) {
        String errMsg = String.format("Retrieving key %s with the byteRangeStart [%d] and the byteRangeEnd [%d] " + "occurs an exception: %s.", key, byteRangeStart, byteRangeEnd, e);
        handleException(new Exception(errMsg), key);
    }
    return null;
}
Also used : CosServiceException(com.qcloud.cos.exception.CosServiceException) COSObject(com.qcloud.cos.model.COSObject) CosClientException(com.qcloud.cos.exception.CosClientException) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) ObjectMetadata(com.qcloud.cos.model.ObjectMetadata) ResponseNotCompleteException(com.qcloud.cos.exception.ResponseNotCompleteException) CosServiceException(com.qcloud.cos.exception.CosServiceException) CosClientException(com.qcloud.cos.exception.CosClientException) IOException(java.io.IOException)

Example 23 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project hadoop-cos by tencentyun.

the class CosNativeFileSystemStore method retrieve.

/**
 * retrieve cos key
 *
 * @param key The key is the object name that is being retrieved from the
 *            cos bucket.
 * @return This method returns null if the key is not found.
 * @throws IOException Retrieve the specified cos key failed.
 */
@Override
public InputStream retrieve(String key) throws IOException {
    LOG.debug("Retrieve the key: {}.", key);
    GetObjectRequest getObjectRequest = new GetObjectRequest(this.bucketName, key);
    if (this.trafficLimit >= 0) {
        getObjectRequest.setTrafficLimit(this.trafficLimit);
    }
    this.setEncryptionMetadata(getObjectRequest, new ObjectMetadata());
    try {
        COSObject cosObject = (COSObject) callCOSClientWithRetry(getObjectRequest);
        return cosObject.getObjectContent();
    } catch (Exception e) {
        String errMsg = String.format("Retrieving the cos key [%s] occurs an exception: %s", key, e);
        handleException(new Exception(errMsg), key);
    }
    // never will get here
    return null;
}
Also used : COSObject(com.qcloud.cos.model.COSObject) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) ObjectMetadata(com.qcloud.cos.model.ObjectMetadata) ResponseNotCompleteException(com.qcloud.cos.exception.ResponseNotCompleteException) CosServiceException(com.qcloud.cos.exception.CosServiceException) CosClientException(com.qcloud.cos.exception.CosClientException) IOException(java.io.IOException)

Example 24 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project hadoop-cos by tencentyun.

the class CosNativeFileSystemStore method callCOSClientWithRetry.

// posix bucket mkdir if the middle part exist will return the 500 error,
// and the rename if the dst exist will return the 500 status too,
// which make the related 5** retry useless. mo improve the resp info to filter.
private <X> Object callCOSClientWithRetry(X request) throws CosServiceException, IOException {
    String sdkMethod = "";
    int retryIndex = 1;
    int l5ErrorCodeRetryIndex = 1;
    while (true) {
        try {
            if (request instanceof PutObjectRequest) {
                sdkMethod = "putObject";
                if (((PutObjectRequest) request).getInputStream().markSupported()) {
                    ((PutObjectRequest) request).getInputStream().mark((int) ((PutObjectRequest) request).getMetadata().getContentLength());
                }
                return this.cosClient.putObject((PutObjectRequest) request);
            } else if (request instanceof UploadPartRequest) {
                sdkMethod = "uploadPart";
                if (((UploadPartRequest) request).getInputStream().markSupported()) {
                    ((UploadPartRequest) request).getInputStream().mark((int) ((UploadPartRequest) request).getPartSize());
                }
                return this.cosClient.uploadPart((UploadPartRequest) request);
            } else if (request instanceof CopyPartRequest) {
                sdkMethod = "copyPartRequest";
                return this.cosClient.copyPart((CopyPartRequest) request);
            } else if (request instanceof HeadBucketRequest) {
                // use for checking bucket type
                sdkMethod = "headBucket";
                return this.cosClient.headBucket((HeadBucketRequest) request);
            } else if (request instanceof RenameRequest) {
                sdkMethod = "rename";
                this.cosClient.rename((RenameRequest) request);
                return new Object();
            } else if (request instanceof GetObjectMetadataRequest) {
                sdkMethod = "queryObjectMeta";
                return this.cosClient.getObjectMetadata((GetObjectMetadataRequest) request);
            } else if (request instanceof DeleteObjectRequest) {
                sdkMethod = "deleteObject";
                this.cosClient.deleteObject((DeleteObjectRequest) request);
                return new Object();
            } else if (request instanceof CopyObjectRequest) {
                sdkMethod = "copyFile";
                return this.cosClient.copyObject((CopyObjectRequest) request);
            } else if (request instanceof GetObjectRequest) {
                sdkMethod = "getObject";
                return this.cosClient.getObject((GetObjectRequest) request);
            } else if (request instanceof ListObjectsRequest) {
                sdkMethod = "listObjects";
                return this.cosClient.listObjects((ListObjectsRequest) request);
            } else if (request instanceof InitiateMultipartUploadRequest) {
                sdkMethod = "initiateMultipartUpload";
                return this.cosClient.initiateMultipartUpload((InitiateMultipartUploadRequest) request);
            } else if (request instanceof CompleteMultipartUploadRequest) {
                sdkMethod = "completeMultipartUpload";
                return this.cosClient.completeMultipartUpload((CompleteMultipartUploadRequest) request);
            } else if (request instanceof AbortMultipartUploadRequest) {
                sdkMethod = "abortMultipartUpload";
                this.cosClient.abortMultipartUpload((AbortMultipartUploadRequest) request);
                return new Object();
            } else {
                throw new IOException("no such method");
            }
        } catch (ResponseNotCompleteException nce) {
            if (this.completeMPUCheckEnabled && request instanceof CompleteMultipartUploadRequest) {
                String key = ((CompleteMultipartUploadRequest) request).getKey();
                FileMetadata fileMetadata = this.queryObjectMetadata(key);
                if (null == fileMetadata) {
                    // if file not exist must throw the exception.
                    handleException(nce, key);
                }
                LOG.warn("Complete mpu resp not complete key [{}]", key);
                // todo: some other double check after cgi unified the ctime of mpu.
                return new CompleteMultipartUploadResult();
            } else {
                throw new IOException(nce);
            }
        } catch (CosServiceException cse) {
            String errMsg = String.format("all cos sdk failed, retryIndex: [%d / %d], " + "call method: %s, exception: %s", retryIndex, this.maxRetryTimes, sdkMethod, cse);
            int statusCode = cse.getStatusCode();
            String errorCode = cse.getErrorCode();
            LOG.debug("fail to retry statusCode {}, errorCode {}", statusCode, errorCode);
            // 对5xx错误进行重试
            if (request instanceof CopyObjectRequest && hasErrorCode(statusCode, errorCode)) {
                if (retryIndex <= this.maxRetryTimes) {
                    LOG.info(errMsg, cse);
                    ++retryIndex;
                } else {
                    LOG.error(errMsg, cse);
                    throw new IOException(errMsg);
                }
            } else if (request instanceof CompleteMultipartUploadRequest && hasErrorCode(statusCode, errorCode)) {
                // complete mpu error code might be in body when status code is 200
                // double check to head object only works in big data job case which key is not same.
                String key = ((CompleteMultipartUploadRequest) request).getKey();
                FileMetadata fileMetadata = this.queryObjectMetadata(key);
                if (null != fileMetadata) {
                    // if file exist direct return.
                    LOG.info("complete mpu error in body, error code {}, but key {} already exist, length {}", errorCode, key, fileMetadata.getLength());
                    return new CompleteMultipartUploadResult();
                }
                // here same like the copy request not setting the interval sleep for now
                if (retryIndex <= this.maxRetryTimes) {
                    LOG.info(errMsg, cse);
                    ++retryIndex;
                } else {
                    LOG.error(errMsg, cse);
                    throw new IOException(errMsg);
                }
            } else if (statusCode / 100 == 5) {
                if (retryIndex <= this.maxRetryTimes) {
                    if (statusCode == 503) {
                        if (useL5Id) {
                            if (l5ErrorCodeRetryIndex >= this.l5UpdateMaxRetryTimes) {
                                // L5上报,进行重试
                                l5EndpointResolver.updateRouteResult(-1);
                                l5ErrorCodeRetryIndex = 1;
                            } else {
                                l5ErrorCodeRetryIndex = l5ErrorCodeRetryIndex + 1;
                            }
                        }
                    }
                    LOG.info(errMsg, cse);
                    long sleepLeast = retryIndex * 300L;
                    long sleepBound = retryIndex * 500L;
                    try {
                        if (request instanceof PutObjectRequest) {
                            LOG.info("Try to reset the put object request input stream.");
                            if (((PutObjectRequest) request).getInputStream().markSupported()) {
                                ((PutObjectRequest) request).getInputStream().reset();
                            } else {
                                LOG.error("The put object request input stream can not be reset, so it can not be" + " retried.");
                                throw cse;
                            }
                        }
                        if (request instanceof UploadPartRequest) {
                            LOG.info("Try to reset the upload part request input stream.");
                            if (((UploadPartRequest) request).getInputStream().markSupported()) {
                                ((UploadPartRequest) request).getInputStream().reset();
                            } else {
                                LOG.error("The upload part request input stream can not be reset, so it can not " + "be retried" + ".");
                                throw cse;
                            }
                        }
                        // if direct retry may occur 403 not found the upload id.
                        if (request instanceof CompleteMultipartUploadRequest && statusCode == 503) {
                            String key = ((CompleteMultipartUploadRequest) request).getKey();
                            FileMetadata fileMetadata = this.queryObjectMetadata(key);
                            if (null != fileMetadata) {
                                // if file exist direct return.
                                LOG.info("complete mpu error might access time out, " + "but key {} already exist, length {}", key, fileMetadata.getLength());
                                return new CompleteMultipartUploadResult();
                            }
                        }
                        Thread.sleep(ThreadLocalRandom.current().nextLong(sleepLeast, sleepBound));
                        ++retryIndex;
                    } catch (InterruptedException e) {
                        throw new IOException(e.toString());
                    }
                } else {
                    LOG.error(errMsg, cse);
                    throw new IOException(errMsg);
                }
            } else {
                throw cse;
            }
        } catch (Exception e) {
            throw new IOException(e);
        }
    }
}
Also used : HeadBucketRequest(com.qcloud.cos.model.HeadBucketRequest) ResponseNotCompleteException(com.qcloud.cos.exception.ResponseNotCompleteException) InitiateMultipartUploadRequest(com.qcloud.cos.model.InitiateMultipartUploadRequest) UploadPartRequest(com.qcloud.cos.model.UploadPartRequest) AbortMultipartUploadRequest(com.qcloud.cos.model.AbortMultipartUploadRequest) IOException(java.io.IOException) CompleteMultipartUploadResult(com.qcloud.cos.model.CompleteMultipartUploadResult) ResponseNotCompleteException(com.qcloud.cos.exception.ResponseNotCompleteException) CosServiceException(com.qcloud.cos.exception.CosServiceException) CosClientException(com.qcloud.cos.exception.CosClientException) IOException(java.io.IOException) DeleteObjectRequest(com.qcloud.cos.model.DeleteObjectRequest) CopyPartRequest(com.qcloud.cos.model.CopyPartRequest) ListObjectsRequest(com.qcloud.cos.model.ListObjectsRequest) CopyObjectRequest(com.qcloud.cos.model.CopyObjectRequest) GetObjectMetadataRequest(com.qcloud.cos.model.GetObjectMetadataRequest) CosServiceException(com.qcloud.cos.exception.CosServiceException) RenameRequest(com.qcloud.cos.model.RenameRequest) COSObject(com.qcloud.cos.model.COSObject) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) PutObjectRequest(com.qcloud.cos.model.PutObjectRequest) CompleteMultipartUploadRequest(com.qcloud.cos.model.CompleteMultipartUploadRequest)

Example 25 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class PutGetDelTest method testGetObjectIfNoneMatchRightEtag.

@Test
public void testGetObjectIfNoneMatchRightEtag() throws IOException {
    if (!judgeUserInfoValid()) {
        return;
    }
    File localFile = buildTestFile(1024);
    String key = "ut/" + localFile.getName();
    cosclient.putObject(bucket, key, localFile);
    try {
        String fileEtag = Md5Utils.md5Hex(localFile);
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);
        List<String> eTagList = new ArrayList<>();
        eTagList.add("\"" + fileEtag + "\"");
        getObjectRequest.setNonmatchingETagConstraints(eTagList);
        COSObject cosObject = cosclient.getObject(getObjectRequest);
        assertNull(cosObject);
    } catch (CosServiceException cse) {
        fail(cse.toString());
    } finally {
        cosclient.deleteObject(bucket, key);
        assertTrue(localFile.delete());
    }
}
Also used : CosServiceException(com.qcloud.cos.exception.CosServiceException) COSObject(com.qcloud.cos.model.COSObject) ArrayList(java.util.ArrayList) File(java.io.File) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) Test(org.junit.Test)

Aggregations

GetObjectRequest (com.qcloud.cos.model.GetObjectRequest)36 File (java.io.File)26 CosServiceException (com.qcloud.cos.exception.CosServiceException)16 ObjectMetadata (com.qcloud.cos.model.ObjectMetadata)14 COSObject (com.qcloud.cos.model.COSObject)12 CosClientException (com.qcloud.cos.exception.CosClientException)10 PutObjectRequest (com.qcloud.cos.model.PutObjectRequest)10 Test (org.junit.Test)9 TransferManager (com.qcloud.cos.transfer.TransferManager)8 BasicCOSCredentials (com.qcloud.cos.auth.BasicCOSCredentials)7 COSCredentials (com.qcloud.cos.auth.COSCredentials)7 Download (com.qcloud.cos.transfer.Download)7 COSClient (com.qcloud.cos.COSClient)6 ClientConfig (com.qcloud.cos.ClientConfig)6 Region (com.qcloud.cos.region.Region)6 IOException (java.io.IOException)6 ArrayList (java.util.ArrayList)6 CopyObjectRequest (com.qcloud.cos.model.CopyObjectRequest)5 MultipleFileDownload (com.qcloud.cos.transfer.MultipleFileDownload)5 ExecutorService (java.util.concurrent.ExecutorService)5