Search in sources :

Example 61 with S3Client

use of org.jclouds.s3.S3Client in project flyway by flyway.

the class AwsS3Scanner method scanForResources.

/**
 * Scans S3 for the resources. In AWS SDK v2, only the region that the client is configured with can be used.
 * The format of the path is expected to be {@code s3:{bucketName}/{optional prefix}}.
 *
 * @param location The location in S3 to start searching. Subdirectories are also searched.
 * @return The resources that were found.
 */
@Override
public Collection<LoadableResource> scanForResources(final Location location) {
    String bucketName = getBucketName(location);
    String prefix = getPrefix(bucketName, location.getPath());
    S3Client s3Client = S3ClientFactory.getClient();
    try {
        ListObjectsV2Request.Builder builder = ListObjectsV2Request.builder().bucket(bucketName).prefix(prefix);
        ListObjectsV2Request request = builder.build();
        ListObjectsV2Response listObjectResult = s3Client.listObjectsV2(request);
        return getLoadableResources(bucketName, listObjectResult);
    } catch (SdkClientException e) {
        if (throwOnMissingLocations) {
            throw new FlywayException("Could not access s3 location:" + bucketName + prefix + " due to error: " + e.getMessage());
        }
        LOG.error("Skipping s3 location:" + bucketName + prefix + " due to error: " + e.getMessage());
        return Collections.emptyList();
    }
}
Also used : FlywayException(org.flywaydb.core.api.FlywayException) ListObjectsV2Request(software.amazon.awssdk.services.s3.model.ListObjectsV2Request) SdkClientException(software.amazon.awssdk.core.exception.SdkClientException) S3Client(software.amazon.awssdk.services.s3.S3Client) ListObjectsV2Response(software.amazon.awssdk.services.s3.model.ListObjectsV2Response)

Aggregations

S3Client (software.amazon.awssdk.services.s3.S3Client)59 Region (software.amazon.awssdk.regions.Region)43 S3Exception (software.amazon.awssdk.services.s3.model.S3Exception)8 S3Object (software.amazon.awssdk.services.s3.model.S3Object)7 List (java.util.List)5 GetObjectRequest (software.amazon.awssdk.services.s3.model.GetObjectRequest)5 GetObjectResponse (software.amazon.awssdk.services.s3.model.GetObjectResponse)5 SupplierEx (com.hazelcast.function.SupplierEx)4 ArrayList (java.util.ArrayList)4 InputStream (java.io.InputStream)3 UTF_8 (java.nio.charset.StandardCharsets.UTF_8)3 Pipeline (com.hazelcast.jet.pipeline.Pipeline)2 TestSources (com.hazelcast.jet.pipeline.test.TestSources)2 ILogger (com.hazelcast.logging.ILogger)2 Logger (com.hazelcast.logging.Logger)2 HazelcastSerialClassRunner (com.hazelcast.test.HazelcastSerialClassRunner)2 BufferedReader (java.io.BufferedReader)2 InputStreamReader (java.io.InputStreamReader)2 Collectors (java.util.stream.Collectors)2 AfterClass (org.junit.AfterClass)2