Search in sources :

Example 31 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class TransferManager method downloadDirectory.

/**
 * Downloads all objects in the virtual directory designated by the keyPrefix given to the
 * destination directory given. All virtual subdirectories will be downloaded recursively.
 *
 * @param bucketName The bucket containing the virtual directory
 * @param keyPrefix The key prefix for the virtual directory, or null for the entire bucket. All
 *        subdirectories will be downloaded recursively.
 * @param destinationDirectory The directory to place downloaded files. Subdirectories will be
 *        created as necessary.
 */
public MultipleFileDownload downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory) {
    if (keyPrefix == null)
        keyPrefix = "";
    List<COSObjectSummary> objectSummaries = new LinkedList<COSObjectSummary>();
    Stack<String> commonPrefixes = new Stack<String>();
    commonPrefixes.add(keyPrefix);
    long totalSize = 0;
    // This is a depth-first search.
    do {
        String prefix = commonPrefixes.pop();
        ObjectListing listObjectsResponse = null;
        do {
            if (listObjectsResponse == null) {
                ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName).withDelimiter(DEFAULT_DELIMITER).withPrefix(prefix);
                listObjectsResponse = cos.listObjects(listObjectsRequest);
            } else {
                listObjectsResponse = cos.listNextBatchOfObjects(listObjectsResponse);
            }
            for (COSObjectSummary s : listObjectsResponse.getObjectSummaries()) {
                // name.
                if (!s.getKey().equals(prefix) && !listObjectsResponse.getCommonPrefixes().contains(s.getKey() + DEFAULT_DELIMITER)) {
                    objectSummaries.add(s);
                    totalSize += s.getSize();
                } else {
                    log.debug("Skipping download for object " + s.getKey() + " since it is also a virtual directory");
                }
            }
            commonPrefixes.addAll(listObjectsResponse.getCommonPrefixes());
        } while (listObjectsResponse.isTruncated());
    } while (!commonPrefixes.isEmpty());
    /* This is the hook for adding additional progress listeners */
    ProgressListenerChain additionalListeners = new ProgressListenerChain();
    TransferProgress transferProgress = new TransferProgress();
    transferProgress.setTotalBytesToTransfer(totalSize);
    /*
         * Bind additional progress listeners to this MultipleFileTransferProgressUpdatingListener
         * to receive ByteTransferred events from each single-file download implementation.
         */
    ProgressListener listener = new MultipleFileTransferProgressUpdatingListener(transferProgress, additionalListeners);
    List<DownloadImpl> downloads = new ArrayList<DownloadImpl>();
    String description = "Downloading from " + bucketName + "/" + keyPrefix;
    final MultipleFileDownloadImpl multipleFileDownload = new MultipleFileDownloadImpl(description, transferProgress, additionalListeners, keyPrefix, bucketName, downloads);
    multipleFileDownload.setMonitor(new MultipleFileTransferMonitor(multipleFileDownload, downloads));
    final CountDownLatch latch = new CountDownLatch(1);
    MultipleFileTransferStateChangeListener transferListener = new MultipleFileTransferStateChangeListener(latch, multipleFileDownload);
    for (COSObjectSummary summary : objectSummaries) {
        // TODO: non-standard delimiters
        File f = new File(destinationDirectory, summary.getKey());
        File parentFile = f.getParentFile();
        if (parentFile == null || !parentFile.exists() && !parentFile.mkdirs()) {
            throw new RuntimeException("Couldn't create parent directories for " + f.getAbsolutePath());
        }
        // All the single-file downloads share the same
        // MultipleFileTransferProgressUpdatingListener and
        // MultipleFileTransferStateChangeListener
        downloads.add((DownloadImpl) doDownload(new GetObjectRequest(summary.getBucketName(), summary.getKey()).<GetObjectRequest>withGeneralProgressListener(listener), f, transferListener, null, false));
    }
    if (downloads.isEmpty()) {
        multipleFileDownload.setState(TransferState.Completed);
        return multipleFileDownload;
    }
    // Notify all state changes waiting for the downloads to all be queued
    // to wake up and continue.
    latch.countDown();
    return multipleFileDownload;
}
Also used : MultipleFileTransferProgressUpdatingListener(com.qcloud.cos.event.MultipleFileTransferProgressUpdatingListener) COSObjectSummary(com.qcloud.cos.model.COSObjectSummary) ArrayList(java.util.ArrayList) ObjectListing(com.qcloud.cos.model.ObjectListing) CountDownLatch(java.util.concurrent.CountDownLatch) LinkedList(java.util.LinkedList) Stack(java.util.Stack) ListObjectsRequest(com.qcloud.cos.model.ListObjectsRequest) MultipleFileTransferStateChangeListener(com.qcloud.cos.event.MultipleFileTransferStateChangeListener) ProgressListener(com.qcloud.cos.event.ProgressListener) COSProgressListener(com.qcloud.cos.event.COSProgressListener) COSProgressListenerChain(com.qcloud.cos.event.COSProgressListenerChain) ProgressListenerChain(com.qcloud.cos.event.ProgressListenerChain) RandomAccessFile(java.io.RandomAccessFile) File(java.io.File) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest)

Example 32 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class AbstractCOSEncryptionClientTest method testTransferManagerUploadDownBigFile.

@Test
public void testTransferManagerUploadDownBigFile() throws IOException, CosServiceException, CosClientException, InterruptedException {
    if (!judgeUserInfoValid()) {
        return;
    }
    TransferManager transferManager = new TransferManager(cosclient);
    File localFile = buildTestFile(1024 * 1024 * 10L);
    File downFile = new File(localFile.getAbsolutePath() + ".down");
    String key = "ut/" + localFile.getName();
    String destKey = key + ".copy";
    try {
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, localFile);
        ObjectMetadata objectMetadata = new ObjectMetadata();
        putObjectRequest.setMetadata(objectMetadata);
        Upload upload = transferManager.upload(putObjectRequest);
        UploadResult uploadResult = upload.waitForUploadResult();
        assertTrue(uploadResult.getETag().contains("-"));
        assertNotNull(uploadResult.getRequestId());
        assertNotNull(uploadResult.getDateStr());
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);
        Download download = transferManager.download(getObjectRequest, downFile);
        download.waitForCompletion();
        // check file
        assertEquals(Md5Utils.md5Hex(localFile), Md5Utils.md5Hex(downFile));
        CopyObjectRequest copyObjectRequest = new CopyObjectRequest(bucket, key, bucket, destKey);
        Copy copy = transferManager.copy(copyObjectRequest, cosclient, null);
        copy.waitForCompletion();
    } finally {
        // delete file on cos
        clearObject(key);
        // delete file on cos
        clearObject(destKey);
        if (localFile.exists()) {
            assertTrue(localFile.delete());
        }
        if (downFile.exists()) {
            assertTrue(downFile.delete());
        }
    }
}
Also used : TransferManager(com.qcloud.cos.transfer.TransferManager) CopyObjectRequest(com.qcloud.cos.model.CopyObjectRequest) Copy(com.qcloud.cos.transfer.Copy) Upload(com.qcloud.cos.transfer.Upload) UploadResult(com.qcloud.cos.model.UploadResult) File(java.io.File) ObjectMetadata(com.qcloud.cos.model.ObjectMetadata) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) Download(com.qcloud.cos.transfer.Download) PutObjectRequest(com.qcloud.cos.model.PutObjectRequest) Test(org.junit.Test)

Example 33 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class TransferManagerDemo method pauseDownloadFileAndResume.

// 将文件下载到本地(中途暂停并继续开始)
public static void pauseDownloadFileAndResume() {
    // 1 初始化用户身份信息(secretId, secretKey)
    COSCredentials cred = new BasicCOSCredentials("AKIDXXXXXXXX", "1A2Z3YYYYYYYYYY");
    // 2 设置bucket的区域, COS地域的简称请参照 https://www.qcloud.com/document/product/436/6224
    ClientConfig clientConfig = new ClientConfig(new Region("ap-beijing-1"));
    // 3 生成cos客户端
    COSClient cosclient = new COSClient(cred, clientConfig);
    // bucket名需包含appid
    String bucketName = "mybucket-1251668577";
    ExecutorService threadPool = Executors.newFixedThreadPool(32);
    // 传入一个threadpool, 若不传入线程池, 默认TransferManager中会生成一个单线程的线程池。
    TransferManager transferManager = new TransferManager(cosclient, threadPool);
    String key = "aaa/bbb.txt";
    File downloadFile = new File("src/test/resources/download.txt");
    GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, key);
    try {
        // 返回一个异步结果copy, 可同步的调用waitForCompletion等待download结束, 成功返回void, 失败抛出异常.
        Download download = transferManager.download(getObjectRequest, downloadFile);
        Thread.sleep(5000L);
        PersistableDownload persistableDownload = download.pause();
        download = transferManager.resumeDownload(persistableDownload);
        showTransferProgress(download);
        download.waitForCompletion();
    } catch (CosServiceException e) {
        e.printStackTrace();
    } catch (CosClientException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    transferManager.shutdownNow();
    cosclient.shutdown();
}
Also used : TransferManager(com.qcloud.cos.transfer.TransferManager) COSCredentials(com.qcloud.cos.auth.COSCredentials) BasicCOSCredentials(com.qcloud.cos.auth.BasicCOSCredentials) BasicCOSCredentials(com.qcloud.cos.auth.BasicCOSCredentials) CosClientException(com.qcloud.cos.exception.CosClientException) COSClient(com.qcloud.cos.COSClient) PersistableDownload(com.qcloud.cos.transfer.PersistableDownload) CosServiceException(com.qcloud.cos.exception.CosServiceException) ExecutorService(java.util.concurrent.ExecutorService) Region(com.qcloud.cos.region.Region) ClientConfig(com.qcloud.cos.ClientConfig) File(java.io.File) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) PersistableDownload(com.qcloud.cos.transfer.PersistableDownload) MultipleFileDownload(com.qcloud.cos.transfer.MultipleFileDownload) Download(com.qcloud.cos.transfer.Download)

Example 34 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class TransferManagerDemo method downLoadFile.

// 将文件下载到本地
public static void downLoadFile() {
    // 1 初始化用户身份信息(secretId, secretKey)
    COSCredentials cred = new BasicCOSCredentials("AKIDXXXXXXXX", "1A2Z3YYYYYYYYYY");
    // 2 设置bucket的区域, COS地域的简称请参照 https://www.qcloud.com/document/product/436/6224
    ClientConfig clientConfig = new ClientConfig(new Region("ap-beijing-1"));
    // 3 生成cos客户端
    COSClient cosclient = new COSClient(cred, clientConfig);
    // bucket名需包含appid
    String bucketName = "mybucket-1251668577";
    ExecutorService threadPool = Executors.newFixedThreadPool(32);
    // 传入一个threadpool, 若不传入线程池, 默认TransferManager中会生成一个单线程的线程池。
    TransferManager transferManager = new TransferManager(cosclient, threadPool);
    String key = "aaa/bbb.txt";
    File downloadFile = new File("src/test/resources/download.txt");
    GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, key);
    try {
        // 返回一个异步结果copy, 可同步的调用waitForCompletion等待download结束, 成功返回void, 失败抛出异常.
        Download download = transferManager.download(getObjectRequest, downloadFile);
        download.waitForCompletion();
    } catch (CosServiceException e) {
        e.printStackTrace();
    } catch (CosClientException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    transferManager.shutdownNow();
    cosclient.shutdown();
}
Also used : TransferManager(com.qcloud.cos.transfer.TransferManager) COSCredentials(com.qcloud.cos.auth.COSCredentials) BasicCOSCredentials(com.qcloud.cos.auth.BasicCOSCredentials) BasicCOSCredentials(com.qcloud.cos.auth.BasicCOSCredentials) CosClientException(com.qcloud.cos.exception.CosClientException) COSClient(com.qcloud.cos.COSClient) CosServiceException(com.qcloud.cos.exception.CosServiceException) ExecutorService(java.util.concurrent.ExecutorService) Region(com.qcloud.cos.region.Region) ClientConfig(com.qcloud.cos.ClientConfig) File(java.io.File) GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) PersistableDownload(com.qcloud.cos.transfer.PersistableDownload) MultipleFileDownload(com.qcloud.cos.transfer.MultipleFileDownload) Download(com.qcloud.cos.transfer.Download)

Example 35 with GetObjectRequest

use of com.qcloud.cos.model.GetObjectRequest in project cos-java-sdk-v5 by tencentyun.

the class BasicImageProcessing method imageCroppingDemo.

public static void imageCroppingDemo(COSClient cosClient) {
    String bucketName = "examplebucket-1250000000";
    String key = "qrcode.png";
    GetObjectRequest getObj = new GetObjectRequest(bucketName, key);
    // 宽高缩放50%
    String rule = "imageMogr2/iradius/150";
    getObj.putCustomQueryParameter(rule, null);
    cosClient.getObject(getObj, new File("qrcode-cropping.png"));
}
Also used : GetObjectRequest(com.qcloud.cos.model.GetObjectRequest) File(java.io.File)

Aggregations

GetObjectRequest (com.qcloud.cos.model.GetObjectRequest)36 File (java.io.File)26 CosServiceException (com.qcloud.cos.exception.CosServiceException)16 ObjectMetadata (com.qcloud.cos.model.ObjectMetadata)14 COSObject (com.qcloud.cos.model.COSObject)12 CosClientException (com.qcloud.cos.exception.CosClientException)10 PutObjectRequest (com.qcloud.cos.model.PutObjectRequest)10 Test (org.junit.Test)9 TransferManager (com.qcloud.cos.transfer.TransferManager)8 BasicCOSCredentials (com.qcloud.cos.auth.BasicCOSCredentials)7 COSCredentials (com.qcloud.cos.auth.COSCredentials)7 Download (com.qcloud.cos.transfer.Download)7 COSClient (com.qcloud.cos.COSClient)6 ClientConfig (com.qcloud.cos.ClientConfig)6 Region (com.qcloud.cos.region.Region)6 IOException (java.io.IOException)6 ArrayList (java.util.ArrayList)6 CopyObjectRequest (com.qcloud.cos.model.CopyObjectRequest)5 MultipleFileDownload (com.qcloud.cos.transfer.MultipleFileDownload)5 ExecutorService (java.util.concurrent.ExecutorService)5