Search in sources :

Example 1 with S3Options

use of org.apache.beam.sdk.io.aws2.options.S3Options in project beam by apache.

the class S3FileSystemTest method matchNonGlobNullContentEncodingWithOptions.

@Test
public void matchNonGlobNullContentEncodingWithOptions() {
    S3FileSystem s3FileSystem = buildMockedS3FileSystem(s3Options());
    S3ResourceId path = S3ResourceId.fromUri("s3://testbucket/testdirectory/filethatexists");
    long lastModifiedMillis = 1540000000000L;
    HeadObjectResponse headObjectResponse = HeadObjectResponse.builder().contentLength(100L).lastModified(Instant.ofEpochMilli(lastModifiedMillis)).contentEncoding(null).build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(HeadObjectRequest.builder().bucket(path.getBucket()).key(path.getKey()).build())))).thenReturn(headObjectResponse);
    MatchResult result = s3FileSystem.matchNonGlobPath(path);
    assertThat(result, MatchResultMatcher.create(ImmutableList.of(MatchResult.Metadata.builder().setSizeBytes(100).setLastModifiedMillis(lastModifiedMillis).setResourceId(path).setIsReadSeekEfficient(true).build())));
}
Also used : HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) MatchResult(org.apache.beam.sdk.io.fs.MatchResult) S3TestUtils.buildMockedS3FileSystem(org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem) Test(org.junit.Test)

Example 2 with S3Options

use of org.apache.beam.sdk.io.aws2.options.S3Options in project beam by apache.

the class S3FileSystemTest method deleteThousandsOfObjectsInMultipleBucketsWithS3Options.

@Test
public void deleteThousandsOfObjectsInMultipleBucketsWithS3Options() throws IOException {
    S3FileSystem s3FileSystem = buildMockedS3FileSystem(s3Options());
    List<String> buckets = ImmutableList.of("bucket1", "bucket2");
    List<String> keys = new ArrayList<>();
    for (int i = 0; i < 2500; i++) {
        keys.add(String.format("key-%d", i));
    }
    List<S3ResourceId> paths = new ArrayList<>();
    for (String bucket : buckets) {
        for (String key : keys) {
            paths.add(S3ResourceId.fromComponents("s3", bucket, key));
        }
    }
    s3FileSystem.delete(paths);
    // Should require 6 calls to delete 2500 objects in each of 2 buckets.
    verify(s3FileSystem.getS3Client(), times(6)).deleteObjects(any(DeleteObjectsRequest.class));
}
Also used : ArrayList(java.util.ArrayList) S3TestUtils.buildMockedS3FileSystem(org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem) DeleteObjectsRequest(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest) Test(org.junit.Test)

Example 3 with S3Options

use of org.apache.beam.sdk.io.aws2.options.S3Options in project beam by apache.

the class S3TestUtils method s3OptionsWithSSEAlgorithm.

static S3Options s3OptionsWithSSEAlgorithm() {
    S3Options options = s3Options();
    options.setSSEAlgorithm(ServerSideEncryption.AES256.name());
    return options;
}
Also used : S3Options(org.apache.beam.sdk.io.aws2.options.S3Options)

Example 4 with S3Options

use of org.apache.beam.sdk.io.aws2.options.S3Options in project beam by apache.

the class S3TestUtils method s3OptionsWithPathStyleAccessEnabled.

static S3Options s3OptionsWithPathStyleAccessEnabled() {
    S3Options options = PipelineOptionsFactory.as(S3Options.class);
    options.setAwsRegion(Region.US_WEST_1);
    options.setS3UploadBufferSizeBytes(MINIMUM_UPLOAD_BUFFER_SIZE_BYTES);
    options.setS3ClientFactoryClass(PathStyleAccessS3ClientBuilderFactory.class);
    return options;
}
Also used : S3Options(org.apache.beam.sdk.io.aws2.options.S3Options)

Example 5 with S3Options

use of org.apache.beam.sdk.io.aws2.options.S3Options in project beam by apache.

the class S3TestUtils method s3OptionsWithSSECustomerKey.

static S3Options s3OptionsWithSSECustomerKey() {
    S3Options options = s3Options();
    options.setSSECustomerKey(SSECustomerKey.builder().key("86glyTlCNZgccSxW8JxMa6ZdjdK3N141glAysPUZ3AA=").algorithm("AES256").build());
    return options;
}
Also used : S3Options(org.apache.beam.sdk.io.aws2.options.S3Options)

Aggregations

Test (org.junit.Test)9 S3Options (org.apache.beam.sdk.io.aws2.options.S3Options)8 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem)7 MatchResult (org.apache.beam.sdk.io.fs.MatchResult)4 HeadObjectResponse (software.amazon.awssdk.services.s3.model.HeadObjectResponse)3 SdkServiceException (software.amazon.awssdk.core.exception.SdkServiceException)2 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 SSECustomerKey (org.apache.beam.sdk.io.aws2.s3.SSECustomerKey)1 DeleteObjectsRequest (software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)1