Search in sources :

Example 1 with S3Client

use of software.amazon.awssdk.crt.s3.S3Client in project beam by apache.

the class S3WritableByteChannelTest method writeFromConfig.

private void writeFromConfig(S3FileSystemConfiguration config, boolean writeReadOnlyBuffer) throws IOException {
    S3Client mockS3Client = mock(S3Client.class, withSettings().defaultAnswer(RETURNS_SMART_NULLS));
    S3ResourceId path = S3ResourceId.fromUri("s3://bucket/dir/file");
    Supplier channel = () -> new S3WritableByteChannel(mockS3Client, path, "text/plain", config);
    write(mockS3Client, channel, path, config.getSSEAlgorithm(), toMd5(config.getSSECustomerKey()), config.getSSEKMSKeyId(), config.getS3UploadBufferSizeBytes(), config.getBucketKeyEnabled(), writeReadOnlyBuffer);
}
Also used : S3Client(software.amazon.awssdk.services.s3.S3Client)

Example 2 with S3Client

use of software.amazon.awssdk.crt.s3.S3Client in project beam by apache.

the class S3WritableByteChannelTest method writeFromOptions.

private void writeFromOptions(S3Options options, boolean writeReadOnlyBuffer) throws IOException {
    S3Client mockS3Client = mock(S3Client.class, withSettings().defaultAnswer(RETURNS_SMART_NULLS));
    S3ResourceId path = S3ResourceId.fromUri("s3://bucket/dir/file");
    Supplier channel = () -> new S3WritableByteChannel(mockS3Client, path, "text/plain", S3FileSystemConfiguration.fromS3Options(options));
    write(mockS3Client, channel, path, options.getSSEAlgorithm(), toMd5(options.getSSECustomerKey()), options.getSSEKMSKeyId(), options.getS3UploadBufferSizeBytes(), options.getBucketKeyEnabled(), writeReadOnlyBuffer);
}
Also used : S3Client(software.amazon.awssdk.services.s3.S3Client)

Example 3 with S3Client

use of software.amazon.awssdk.crt.s3.S3Client in project hazelcast by hazelcast.

the class S3TestBase method testSink.

void testSink(String bucketName, String prefix, int itemCount, String payload) {
    IMap<Integer, String> map = hz.getMap("map");
    for (int i = 0; i < itemCount; i++) {
        map.put(i, payload);
    }
    Pipeline p = Pipeline.create();
    p.readFrom(Sources.map(map, alwaysTrue(), Map.Entry::getValue)).writeTo(S3Sinks.s3(bucketName, prefix, CHARSET, clientSupplier(), Object::toString));
    hz.getJet().newJob(p).join();
    try (S3Client client = clientSupplier().get()) {
        assertTrueEventually(() -> {
            long lineCount = client.listObjectsV2(req -> req.bucket(bucketName).prefix(prefix)).contents().stream().flatMap(o -> s3ObjectToLines(o, client, bucketName)).peek(line -> assertEquals(payload, line)).count();
            assertEquals(itemCount, lineCount);
        });
    }
}
Also used : S3Object(software.amazon.awssdk.services.s3.model.S3Object) S3Exception(software.amazon.awssdk.services.s3.model.S3Exception) GetObjectResponse(software.amazon.awssdk.services.s3.model.GetObjectResponse) Collections.singletonList(java.util.Collections.singletonList) Charset(java.nio.charset.Charset) Map(java.util.Map) ResponseInputStream(software.amazon.awssdk.core.ResponseInputStream) Assert.fail(org.junit.Assert.fail) Before(org.junit.Before) HazelcastInstance(com.hazelcast.core.HazelcastInstance) Pipeline(com.hazelcast.jet.pipeline.Pipeline) JetTestSupport(com.hazelcast.jet.core.JetTestSupport) S3Client(software.amazon.awssdk.services.s3.S3Client) UTF_8(java.nio.charset.StandardCharsets.UTF_8) Sinks(com.hazelcast.jet.pipeline.Sinks) NoSuchBucketException(software.amazon.awssdk.services.s3.model.NoSuchBucketException) AggregateOperations(com.hazelcast.jet.aggregate.AggregateOperations) Assert.assertTrue(org.junit.Assert.assertTrue) IOException(java.io.IOException) Predicates.alwaysTrue(com.hazelcast.query.Predicates.alwaysTrue) InputStreamReader(java.io.InputStreamReader) Collectors(java.util.stream.Collectors) SupplierEx(com.hazelcast.function.SupplierEx) Sources(com.hazelcast.jet.pipeline.Sources) TestSources(com.hazelcast.jet.pipeline.test.TestSources) List(java.util.List) Stream(java.util.stream.Stream) ResponseTransformer.toInputStream(software.amazon.awssdk.core.sync.ResponseTransformer.toInputStream) BufferedReader(java.io.BufferedReader) Assertions(com.hazelcast.jet.pipeline.test.Assertions) Assert.assertEquals(org.junit.Assert.assertEquals) IMap(com.hazelcast.map.IMap) InputStream(java.io.InputStream) S3Client(software.amazon.awssdk.services.s3.S3Client) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Example 4 with S3Client

use of software.amazon.awssdk.crt.s3.S3Client in project flyway by flyway.

the class AwsS3Resource method read.

@Override
public Reader read() {
    S3Client s3 = S3ClientFactory.getClient();
    try {
        GetObjectRequest.Builder builder = GetObjectRequest.builder().bucket(bucketName).key(s3ObjectSummary.key());
        GetObjectRequest request = builder.build();
        ResponseInputStream o = s3.getObject(request);
        return Channels.newReader(Channels.newChannel(o), encoding.name());
    } catch (AwsServiceException e) {
        LOG.error(e.getMessage(), e);
        throw new FlywayException("Failed to get object from s3: " + e.getMessage(), e);
    }
}
Also used : FlywayException(org.flywaydb.core.api.FlywayException) ResponseInputStream(software.amazon.awssdk.core.ResponseInputStream) AwsServiceException(software.amazon.awssdk.awscore.exception.AwsServiceException) S3Client(software.amazon.awssdk.services.s3.S3Client) GetObjectRequest(software.amazon.awssdk.services.s3.model.GetObjectRequest)

Example 5 with S3Client

use of software.amazon.awssdk.crt.s3.S3Client in project aws-doc-sdk-examples by awsdocs.

the class PutObject method main.

public static void main(String[] args) {
    final String USAGE = "\n" + "Usage:\n" + "  <bucketName> <objectKey> <objectPath> \n\n" + "Where:\n" + "  bucketName - the Amazon S3 bucket to upload an object into.\n" + "  objectKey - the object to upload (for example, book.pdf).\n" + "  objectPath - the path where the file is located (for example, C:/AWS/book2.pdf). \n\n";
    if (args.length != 3) {
        System.out.println(USAGE);
        System.exit(1);
    }
    String bucketName = args[0];
    String objectKey = args[1];
    String objectPath = args[2];
    System.out.println("Putting object " + objectKey + " into bucket " + bucketName);
    System.out.println("  in bucket: " + bucketName);
    Region region = Region.US_EAST_1;
    S3Client s3 = S3Client.builder().region(region).build();
    String result = putS3Object(s3, bucketName, objectKey, objectPath);
    System.out.println("Tag information: " + result);
    s3.close();
}
Also used : Region(software.amazon.awssdk.regions.Region) S3Client(software.amazon.awssdk.services.s3.S3Client)

Aggregations

S3Client (software.amazon.awssdk.services.s3.S3Client)259 Test (org.junit.jupiter.api.Test)83 Region (software.amazon.awssdk.regions.Region)80 Test (org.junit.Test)51 URI (java.net.URI)36 S3Exception (software.amazon.awssdk.services.s3.model.S3Exception)28 GetObjectResponse (software.amazon.awssdk.services.s3.model.GetObjectResponse)27 ArrayList (java.util.ArrayList)25 Assertions.assertThat (org.assertj.core.api.Assertions.assertThat)24 List (java.util.List)23 Collectors (java.util.stream.Collectors)22 InputStream (java.io.InputStream)20 S3ClientBuilder (software.amazon.awssdk.services.s3.S3ClientBuilder)20 IOException (java.io.IOException)19 GetObjectRequest (software.amazon.awssdk.services.s3.model.GetObjectRequest)19 S3AsyncClient (software.amazon.awssdk.services.s3.S3AsyncClient)17 PutObjectRequest (software.amazon.awssdk.services.s3.model.PutObjectRequest)17 S3Object (software.amazon.awssdk.services.s3.model.S3Object)17 Map (java.util.Map)16 HashMap (java.util.HashMap)15