Search in sources :

Example 1 with HeadObjectResponse

use of software.amazon.awssdk.services.s3.model.HeadObjectResponse in project beam by apache.

the class S3FileSystem method multipartCopy.

@VisibleForTesting
CompleteMultipartUploadResponse multipartCopy(S3ResourceId sourcePath, S3ResourceId destinationPath, HeadObjectResponse sourceObjectHead) throws SdkServiceException {
    CreateMultipartUploadRequest initiateUploadRequest = CreateMultipartUploadRequest.builder().bucket(destinationPath.getBucket()).key(destinationPath.getKey()).storageClass(config.getS3StorageClass()).metadata(sourceObjectHead.metadata()).serverSideEncryption(config.getSSEAlgorithm()).ssekmsKeyId(config.getSSEKMSKeyId()).sseCustomerKey(config.getSSECustomerKey().getKey()).sseCustomerAlgorithm(config.getSSECustomerKey().getAlgorithm()).build();
    CreateMultipartUploadResponse createMultipartUploadResponse = s3Client.get().createMultipartUpload(initiateUploadRequest);
    final String uploadId = createMultipartUploadResponse.uploadId();
    List<CompletedPart> completedParts = new ArrayList<>();
    final long objectSize = sourceObjectHead.contentLength();
    CopyPartResult copyPartResult;
    CompletedPart completedPart;
    // without using S3FileSystem.copy in the future
    if (objectSize == 0) {
        final UploadPartCopyRequest uploadPartCopyRequest = UploadPartCopyRequest.builder().bucket(sourcePath.getBucket()).key(sourcePath.getKey()).copySource(sourcePath.getBucket() + "/" + sourcePath.getKey()).uploadId(uploadId).partNumber(1).sseCustomerKey(config.getSSECustomerKey().getKey()).sseCustomerAlgorithm(config.getSSECustomerKey().getAlgorithm()).copySourceSSECustomerKey(config.getSSECustomerKey().getKey()).copySourceSSECustomerAlgorithm(config.getSSECustomerKey().getAlgorithm()).build();
        copyPartResult = s3Client.get().uploadPartCopy(uploadPartCopyRequest).copyPartResult();
        completedPart = CompletedPart.builder().partNumber(1).eTag(copyPartResult.eTag()).build();
        completedParts.add(completedPart);
    } else {
        long bytePosition = 0;
        Integer uploadBufferSizeBytes = config.getS3UploadBufferSizeBytes();
        // Amazon parts are 1-indexed, not zero-indexed.
        for (int partNumber = 1; bytePosition < objectSize; partNumber++) {
            final UploadPartCopyRequest uploadPartCopyRequest = UploadPartCopyRequest.builder().bucket(sourcePath.getBucket()).key(sourcePath.getKey()).copySource(destinationPath.getBucket() + "/" + sourcePath.getKey()).uploadId(uploadId).partNumber(partNumber).copySourceRange(String.format("bytes=%s-%s", bytePosition, Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1))).sseCustomerKey(config.getSSECustomerKey().getKey()).sseCustomerAlgorithm(config.getSSECustomerKey().getAlgorithm()).copySourceSSECustomerKey(config.getSSECustomerKey().getKey()).copySourceSSECustomerAlgorithm(config.getSSECustomerKey().getAlgorithm()).build();
            copyPartResult = s3Client.get().uploadPartCopy(uploadPartCopyRequest).copyPartResult();
            completedPart = CompletedPart.builder().partNumber(1).eTag(copyPartResult.eTag()).build();
            completedParts.add(completedPart);
            bytePosition += uploadBufferSizeBytes;
        }
    }
    CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(completedParts).build();
    CompleteMultipartUploadRequest completeUploadRequest = CompleteMultipartUploadRequest.builder().bucket(destinationPath.getBucket()).key(destinationPath.getKey()).uploadId(uploadId).multipartUpload(completedMultipartUpload).build();
    return s3Client.get().completeMultipartUpload(completeUploadRequest);
}
Also used : CompletedPart(software.amazon.awssdk.services.s3.model.CompletedPart) ArrayList(java.util.ArrayList) CreateMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest) CreateMultipartUploadResponse(software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse) UploadPartCopyRequest(software.amazon.awssdk.services.s3.model.UploadPartCopyRequest) CopyPartResult(software.amazon.awssdk.services.s3.model.CopyPartResult) CompletedMultipartUpload(software.amazon.awssdk.services.s3.model.CompletedMultipartUpload) CompleteMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)

Example 2 with HeadObjectResponse

use of software.amazon.awssdk.services.s3.model.HeadObjectResponse in project beam by apache.

the class S3FileSystemTest method matchNonGlobNotReadSeekEfficient.

@Test
public void matchNonGlobNotReadSeekEfficient() {
    S3FileSystem s3FileSystem = buildMockedS3FileSystem(s3Config("s3"));
    S3ResourceId path = S3ResourceId.fromUri("s3://testbucket/testdirectory/filethatexists");
    long lastModifiedMillis = 1540000000000L;
    HeadObjectResponse headObjectResponse = HeadObjectResponse.builder().contentLength(100L).lastModified(Instant.ofEpochMilli(lastModifiedMillis)).contentEncoding("gzip").build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(HeadObjectRequest.builder().bucket(path.getBucket()).key(path.getKey()).build())))).thenReturn(headObjectResponse);
    MatchResult result = s3FileSystem.matchNonGlobPath(path);
    assertThat(result, MatchResultMatcher.create(ImmutableList.of(MatchResult.Metadata.builder().setSizeBytes(100).setLastModifiedMillis(lastModifiedMillis).setResourceId(path).setIsReadSeekEfficient(false).build())));
}
Also used : HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) MatchResult(org.apache.beam.sdk.io.fs.MatchResult) S3TestUtils.buildMockedS3FileSystem(org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem) Test(org.junit.Test)

Example 3 with HeadObjectResponse

use of software.amazon.awssdk.services.s3.model.HeadObjectResponse in project beam by apache.

the class S3FileSystemTest method testAtomicCopy.

private void testAtomicCopy(S3FileSystem s3FileSystem, SSECustomerKey sseCustomerKey) throws IOException {
    S3ResourceId sourcePath = S3ResourceId.fromUri(s3FileSystem.getScheme() + "://bucket/from");
    S3ResourceId destinationPath = S3ResourceId.fromUri(s3FileSystem.getScheme() + "://bucket/to");
    CopyObjectResponse.Builder builder = CopyObjectResponse.builder();
    String sseCustomerKeyMd5 = toMd5(sseCustomerKey);
    if (sseCustomerKeyMd5 != null) {
        builder.sseCustomerKeyMD5(sseCustomerKeyMd5);
    }
    CopyObjectResponse copyObjectResponse = builder.build();
    CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder().copySource(sourcePath.getBucket() + "/" + sourcePath.getKey()).destinationBucket(destinationPath.getBucket()).destinationBucket(destinationPath.getKey()).sseCustomerKey(sseCustomerKey.getKey()).copySourceSSECustomerAlgorithm(sseCustomerKey.getAlgorithm()).build();
    when(s3FileSystem.getS3Client().copyObject(any(CopyObjectRequest.class))).thenReturn(copyObjectResponse);
    assertEquals(sseCustomerKeyMd5, s3FileSystem.getS3Client().copyObject(copyObjectRequest).sseCustomerKeyMD5());
    HeadObjectResponse headObjectResponse = HeadObjectResponse.builder().build();
    s3FileSystem.atomicCopy(sourcePath, destinationPath, headObjectResponse);
    verify(s3FileSystem.getS3Client(), times(2)).copyObject(any(CopyObjectRequest.class));
}
Also used : CopyObjectRequest(software.amazon.awssdk.services.s3.model.CopyObjectRequest) HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) CopyObjectResponse(software.amazon.awssdk.services.s3.model.CopyObjectResponse)

Example 4 with HeadObjectResponse

use of software.amazon.awssdk.services.s3.model.HeadObjectResponse in project beam by apache.

the class S3FileSystemTest method matchNonGlobNullContentEncodingWithOptions.

@Test
public void matchNonGlobNullContentEncodingWithOptions() {
    S3FileSystem s3FileSystem = buildMockedS3FileSystem(s3Options());
    S3ResourceId path = S3ResourceId.fromUri("s3://testbucket/testdirectory/filethatexists");
    long lastModifiedMillis = 1540000000000L;
    HeadObjectResponse headObjectResponse = HeadObjectResponse.builder().contentLength(100L).lastModified(Instant.ofEpochMilli(lastModifiedMillis)).contentEncoding(null).build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(HeadObjectRequest.builder().bucket(path.getBucket()).key(path.getKey()).build())))).thenReturn(headObjectResponse);
    MatchResult result = s3FileSystem.matchNonGlobPath(path);
    assertThat(result, MatchResultMatcher.create(ImmutableList.of(MatchResult.Metadata.builder().setSizeBytes(100).setLastModifiedMillis(lastModifiedMillis).setResourceId(path).setIsReadSeekEfficient(true).build())));
}
Also used : HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) MatchResult(org.apache.beam.sdk.io.fs.MatchResult) S3TestUtils.buildMockedS3FileSystem(org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem) Test(org.junit.Test)

Example 5 with HeadObjectResponse

use of software.amazon.awssdk.services.s3.model.HeadObjectResponse in project beam by apache.

the class S3FileSystemTest method testMultipartCopy.

private void testMultipartCopy(S3FileSystem s3FileSystem, SSECustomerKey sseCustomerKey, long s3UploadBufferSizeBytes) throws IOException {
    S3ResourceId sourcePath = S3ResourceId.fromUri(s3FileSystem.getScheme() + "://bucket/from");
    S3ResourceId destinationPath = S3ResourceId.fromUri(s3FileSystem.getScheme() + "://bucket/to");
    CreateMultipartUploadResponse.Builder builder = CreateMultipartUploadResponse.builder().uploadId("upload-id");
    String sseCustomerKeyMd5 = toMd5(sseCustomerKey);
    if (sseCustomerKeyMd5 != null) {
        builder.sseCustomerKeyMD5(sseCustomerKeyMd5);
    }
    CreateMultipartUploadResponse createMultipartUploadResponse = builder.build();
    when(s3FileSystem.getS3Client().createMultipartUpload(any(CreateMultipartUploadRequest.class))).thenReturn(createMultipartUploadResponse);
    assertEquals(sseCustomerKeyMd5, s3FileSystem.getS3Client().createMultipartUpload(CreateMultipartUploadRequest.builder().bucket(destinationPath.getBucket()).key(destinationPath.getKey()).build()).sseCustomerKeyMD5());
    HeadObjectResponse.Builder headObjectResponseBuilder = HeadObjectResponse.builder().contentLength((long) (s3UploadBufferSizeBytes * 1.5)).contentEncoding("read-seek-efficient");
    if (sseCustomerKeyMd5 != null) {
        headObjectResponseBuilder.sseCustomerKeyMD5(sseCustomerKeyMd5);
    }
    HeadObjectResponse headObjectResponse = headObjectResponseBuilder.build();
    assertGetObjectHead(s3FileSystem, createObjectHeadRequest(sourcePath, sseCustomerKey), sseCustomerKeyMd5, headObjectResponse);
    CopyPartResult copyPartResult1 = CopyPartResult.builder().eTag("etag-1").build();
    CopyPartResult copyPartResult2 = CopyPartResult.builder().eTag("etag-2").build();
    UploadPartCopyResponse.Builder uploadPartCopyResponseBuilder1 = UploadPartCopyResponse.builder().copyPartResult(copyPartResult1);
    UploadPartCopyResponse.Builder uploadPartCopyResponseBuilder2 = UploadPartCopyResponse.builder().copyPartResult(copyPartResult2);
    if (sseCustomerKeyMd5 != null) {
        uploadPartCopyResponseBuilder1.sseCustomerKeyMD5(sseCustomerKeyMd5);
        uploadPartCopyResponseBuilder2.sseCustomerKeyMD5(sseCustomerKeyMd5);
    }
    UploadPartCopyResponse uploadPartCopyResponse1 = uploadPartCopyResponseBuilder1.build();
    UploadPartCopyResponse uploadPartCopyResponse2 = uploadPartCopyResponseBuilder2.build();
    UploadPartCopyRequest uploadPartCopyRequest = UploadPartCopyRequest.builder().sseCustomerKey(sseCustomerKey.getKey()).build();
    when(s3FileSystem.getS3Client().uploadPartCopy(any(UploadPartCopyRequest.class))).thenReturn(uploadPartCopyResponse1).thenReturn(uploadPartCopyResponse2);
    assertEquals(sseCustomerKeyMd5, s3FileSystem.getS3Client().uploadPartCopy(uploadPartCopyRequest).sseCustomerKeyMD5());
    s3FileSystem.multipartCopy(sourcePath, destinationPath, headObjectResponse);
    verify(s3FileSystem.getS3Client(), times(1)).completeMultipartUpload(any(CompleteMultipartUploadRequest.class));
}
Also used : HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) UploadPartCopyResponse(software.amazon.awssdk.services.s3.model.UploadPartCopyResponse) CreateMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest) UploadPartCopyRequest(software.amazon.awssdk.services.s3.model.UploadPartCopyRequest) CreateMultipartUploadResponse(software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse) CopyPartResult(software.amazon.awssdk.services.s3.model.CopyPartResult) CompleteMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest)

Aggregations

HeadObjectResponse (software.amazon.awssdk.services.s3.model.HeadObjectResponse)12 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem)9 Test (org.junit.Test)9 MatchResult (org.apache.beam.sdk.io.fs.MatchResult)6 HeadObjectRequest (software.amazon.awssdk.services.s3.model.HeadObjectRequest)3 ListObjectsV2Response (software.amazon.awssdk.services.s3.model.ListObjectsV2Response)3 S3Object (software.amazon.awssdk.services.s3.model.S3Object)3 CompleteMultipartUploadRequest (software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest)2 CopyObjectRequest (software.amazon.awssdk.services.s3.model.CopyObjectRequest)2 CopyPartResult (software.amazon.awssdk.services.s3.model.CopyPartResult)2 CreateMultipartUploadRequest (software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest)2 CreateMultipartUploadResponse (software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse)2 ListObjectsV2Request (software.amazon.awssdk.services.s3.model.ListObjectsV2Request)2 UploadPartCopyRequest (software.amazon.awssdk.services.s3.model.UploadPartCopyRequest)2 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 VisibleForTesting (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)1 SdkServiceException (software.amazon.awssdk.core.exception.SdkServiceException)1 CompletedMultipartUpload (software.amazon.awssdk.services.s3.model.CompletedMultipartUpload)1