Examples with S3Client uk.co.automatictester.lightning.lambda.s3.S3Client used on opensource projects

Example 61 with S3Client

use of uk.co.automatictester.lightning.lambda.s3.S3Client in project hazelcast by hazelcast.

the class S3Sources method s3.

/**
 * Creates an AWS S3 {@link BatchSource} which lists all the objects in the
 * bucket-list using given {@code prefix}, reads them line by line,
 * transforms each line to the desired output object using given {@code
 * mapFn} and emits them to downstream.
 * <p>
 * The source does not save any state to snapshot. If the job is restarted,
 * it will re-emit all entries.
 * <p>
 * The default local parallelism for this processor is 2.
 * <p>
 * Here is an example which reads the objects from a single bucket with
 * applying the given prefix.
 *
 * <pre>{@code
 * Pipeline p = Pipeline.create();
 * BatchStage<String> srcStage = p.readFrom(S3Sources.s3(
 *      Arrays.asList("bucket1", "bucket2"),
 *      "prefix",
 *      StandardCharsets.UTF_8,
 *      () -> S3Client.create(),
 *      (filename, line) -> line
 * ));
 * }</pre>
 *
 * @param bucketNames    list of bucket-names
 * @param prefix         the prefix to filter the objects. Optional, passing
 *                       {@code null} will list all objects.
 * @param clientSupplier function which returns the s3 client to use
 *                       one client per processor instance is used
 * @param mapFn          the function which creates output object from each
 *                       line. Gets the object name and line as parameters
 * @param <T>            the type of the items the source emits
 */
@Nonnull
public static <T> BatchSource<T> s3(@Nonnull List<String> bucketNames, @Nullable String prefix, @Nonnull Charset charset, @Nonnull SupplierEx<? extends S3Client> clientSupplier, @Nonnull BiFunctionEx<String, String, ? extends T> mapFn) {
    String charsetName = charset.name();
    FunctionEx<InputStream, Stream<String>> readFileFn = responseInputStream -> {
        BufferedReader reader = new BufferedReader(new InputStreamReader(responseInputStream, Charset.forName(charsetName)));
        return reader.lines();
    };
    return s3(bucketNames, prefix, clientSupplier, readFileFn, mapFn);
}

Example 62 with S3Client

use of uk.co.automatictester.lightning.lambda.s3.S3Client in project flyway by flyway.

the class AwsS3Scanner method scanForResources.

/**
 * Scans S3 for the resources. In AWS SDK v2, only the region that the client is configured with can be used.
 * The format of the path is expected to be {@code s3:{bucketName}/{optional prefix}}.
 *
 * @param location The location in S3 to start searching. Subdirectories are also searched.
 * @return The resources that were found.
 */
@Override
public Collection<LoadableResource> scanForResources(final Location location) {
    String bucketName = getBucketName(location);
    String prefix = getPrefix(bucketName, location.getPath());
    S3Client s3Client = S3ClientFactory.getClient();
    try {
        ListObjectsV2Request.Builder builder = ListObjectsV2Request.builder().bucket(bucketName).prefix(prefix);
        ListObjectsV2Request request = builder.build();
        ListObjectsV2Response listObjectResult = s3Client.listObjectsV2(request);
        return getLoadableResources(bucketName, listObjectResult);
    } catch (SdkClientException e) {
        if (throwOnMissingLocations) {
            throw new FlywayException("Could not access s3 location:" + bucketName + prefix + " due to error: " + e.getMessage());
        }
        LOG.error("Skipping s3 location:" + bucketName + prefix + " due to error: " + e.getMessage());
        return Collections.emptyList();
    }
}

Example 63 with S3Client

use of uk.co.automatictester.lightning.lambda.s3.S3Client in project fluency by komamitsu.

the class AwsS3SenderTest method testSend.

private void testSend(AwsS3Sender.Config config, boolean gzipCompressed, int failures) throws IOException {
    S3Client s3Client = mock(S3Client.class);
    S3ClientBuilder s3ClientBuilder = mock(S3ClientBuilder.class);
    doReturn(s3Client).when(s3ClientBuilder).build();
    AtomicInteger retryCount = new AtomicInteger();
    doAnswer(invocation -> {
        PutObjectRequest request = invocation.getArgument(0);
        assertEquals("hello.world", request.bucket());
        assertEquals("2345/01/31/23/59-59-99999.data", request.key());
        RequestBody body = invocation.getArgument(1);
        try (InputStream s3In = body.contentStreamProvider().newStream();
            InputStream in = gzipCompressed ? new GZIPInputStream(s3In) : s3In) {
            byte[] content = ByteStreams.toByteArray(in);
            assertEquals("0123456789", new String(content, StandardCharsets.UTF_8));
        }
        if (retryCount.getAndIncrement() < failures) {
            throw new RuntimeException("Something happened");
        }
        return null;
    }).when(s3Client).putObject(any(PutObjectRequest.class), any(RequestBody.class));
    AwsS3Sender sender = new AwsS3Sender(s3ClientBuilder, config);
    sender.send("hello.world", "2345/01/31/23/59-59-99999.data", ByteBuffer.wrap("0123456789".getBytes(StandardCharsets.UTF_8)));
    verify(s3Client, times(failures + 1)).putObject(any(PutObjectRequest.class), any(RequestBody.class));
    sender.close();
}

Example 64 with S3Client

use of uk.co.automatictester.lightning.lambda.s3.S3Client in project fluency by komamitsu.

the class AwsS3SenderTest method buildClientWithCustomizedConfig.

@Test
void buildClientWithCustomizedConfig() {
    AwsS3Sender.Config config = new AwsS3Sender.Config();
    config.setEndpoint("https://another.s3endpoi.nt");
    config.setRegion("ap-northeast-1");
    config.setAwsAccessKeyId("foo");
    config.setAwsSecretAccessKey("bar");
    S3Client s3Client = mock(S3Client.class);
    S3ClientBuilder s3ClientBuilder = mock(S3ClientBuilder.class);
    doReturn(s3Client).when(s3ClientBuilder).build();
    doAnswer(invocation -> {
        AwsCredentialsProvider provider = invocation.getArgument(0);
        AwsCredentials awsCredentials = provider.resolveCredentials();
        assertEquals("foo", awsCredentials.accessKeyId());
        assertEquals("bar", awsCredentials.secretAccessKey());
        return null;
    }).when(s3ClientBuilder).credentialsProvider(any());
    new AwsS3Sender(s3ClientBuilder, config);
    verify(s3ClientBuilder, times(1)).build();
    verify(s3ClientBuilder, times(1)).endpointOverride(eq(URI.create("https://another.s3endpoi.nt")));
    verify(s3ClientBuilder, times(1)).region(eq(Region.AP_NORTHEAST_1));
    verify(s3ClientBuilder, times(1)).credentialsProvider(any());
}

Example 65 with S3Client

use of uk.co.automatictester.lightning.lambda.s3.S3Client in project fluency by komamitsu.

the class AwsS3SenderTest method buildClientWithDefaults.

@Test
void buildClientWithDefaults() {
    AwsS3Sender.Config config = new AwsS3Sender.Config();
    S3Client s3Client = mock(S3Client.class);
    S3ClientBuilder s3ClientBuilder = mock(S3ClientBuilder.class);
    doReturn(s3Client).when(s3ClientBuilder).build();
    new AwsS3Sender(s3ClientBuilder, config);
    verify(s3ClientBuilder, times(1)).build();
    verify(s3ClientBuilder, times(0)).endpointOverride(any());
    verify(s3ClientBuilder, times(0)).region(any());
    verify(s3ClientBuilder, times(0)).credentialsProvider(any());
}