Search in sources :

Example 1 with HeadObjectRequest

use of software.amazon.awssdk.services.s3.model.HeadObjectRequest in project beam by apache.

the class S3FileSystemTest method matchVariousInvokeThreadPool.

@Test
public void matchVariousInvokeThreadPool() throws IOException {
    S3FileSystem s3FileSystem = buildMockedS3FileSystem(s3Config("mys3"));
    SdkServiceException notFoundException = S3Exception.builder().message("mock exception").statusCode(404).build();
    S3ResourceId pathNotExist = S3ResourceId.fromUri("mys3://testbucket/testdirectory/nonexistentfile");
    HeadObjectRequest headObjectRequestNotExist = HeadObjectRequest.builder().bucket(pathNotExist.getBucket()).key(pathNotExist.getKey()).build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(headObjectRequestNotExist)))).thenThrow(notFoundException);
    SdkServiceException forbiddenException = SdkServiceException.builder().message("mock exception").statusCode(403).build();
    S3ResourceId pathForbidden = S3ResourceId.fromUri("mys3://testbucket/testdirectory/forbiddenfile");
    HeadObjectRequest headObjectRequestForbidden = HeadObjectRequest.builder().bucket(pathForbidden.getBucket()).key(pathForbidden.getKey()).build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(headObjectRequestForbidden)))).thenThrow(forbiddenException);
    S3ResourceId pathExist = S3ResourceId.fromUri("mys3://testbucket/testdirectory/filethatexists");
    HeadObjectRequest headObjectRequestExist = HeadObjectRequest.builder().bucket(pathExist.getBucket()).key(pathExist.getKey()).build();
    HeadObjectResponse s3ObjectMetadata = HeadObjectResponse.builder().contentLength(100L).contentEncoding("not-gzip").lastModified(Instant.ofEpochMilli(1540000000000L)).build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(headObjectRequestExist)))).thenReturn(s3ObjectMetadata);
    S3ResourceId pathGlob = S3ResourceId.fromUri("mys3://testbucket/path/part*");
    S3Object foundListObject = S3Object.builder().key("path/part-0").size(200L).lastModified(Instant.ofEpochMilli(1541000000000L)).build();
    ListObjectsV2Response listObjectsResponse = ListObjectsV2Response.builder().continuationToken(null).contents(foundListObject).build();
    when(s3FileSystem.getS3Client().listObjectsV2((ListObjectsV2Request) notNull())).thenReturn(listObjectsResponse);
    HeadObjectResponse headObjectResponse = HeadObjectResponse.builder().contentEncoding("").build();
    when(s3FileSystem.getS3Client().headObject(argThat(new GetHeadObjectRequestMatcher(HeadObjectRequest.builder().bucket(pathGlob.getBucket()).key("path/part-0").build())))).thenReturn(headObjectResponse);
    assertThat(s3FileSystem.match(ImmutableList.of(pathNotExist.toString(), pathForbidden.toString(), pathExist.toString(), pathGlob.toString())), contains(MatchResultMatcher.create(MatchResult.Status.NOT_FOUND, new FileNotFoundException()), MatchResultMatcher.create(MatchResult.Status.ERROR, new IOException(forbiddenException)), MatchResultMatcher.create(100, 1540000000000L, pathExist, true), MatchResultMatcher.create(200, 1541000000000L, S3ResourceId.fromComponents("mys3", pathGlob.getBucket(), foundListObject.key()), true)));
}
Also used : SdkServiceException(software.amazon.awssdk.core.exception.SdkServiceException) HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) FileNotFoundException(java.io.FileNotFoundException) HeadObjectRequest(software.amazon.awssdk.services.s3.model.HeadObjectRequest) S3Object(software.amazon.awssdk.services.s3.model.S3Object) IOException(java.io.IOException) ListObjectsV2Response(software.amazon.awssdk.services.s3.model.ListObjectsV2Response) S3TestUtils.buildMockedS3FileSystem(org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem) Test(org.junit.Test)

Aggregations

FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem)1 Test (org.junit.Test)1 SdkServiceException (software.amazon.awssdk.core.exception.SdkServiceException)1 HeadObjectRequest (software.amazon.awssdk.services.s3.model.HeadObjectRequest)1 HeadObjectResponse (software.amazon.awssdk.services.s3.model.HeadObjectResponse)1 ListObjectsV2Response (software.amazon.awssdk.services.s3.model.ListObjectsV2Response)1 S3Object (software.amazon.awssdk.services.s3.model.S3Object)1