Search in sources :

Example 46 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project adeptj-modules by AdeptJ.

the class AwsS3Service method createFolder.

/**
 * {@inheritDoc}
 */
@Override
public S3Response createFolder(String bucketName, String folderName) {
    ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentLength(0);
    try {
        return new S3Response().withPutObjectResult(this.s3Client.putObject(new PutObjectRequest(bucketName, folderName + PATH_SEPARATOR, new ByteArrayInputStream(new byte[0]), objectMetadata)));
    } catch (RuntimeException ex) {
        LOGGER.error("Exception while creating folder!!", ex);
        throw new AwsException(ex.getMessage(), ex);
    }
}
Also used : S3Response(com.adeptj.modules.aws.s3.S3Response) ByteArrayInputStream(java.io.ByteArrayInputStream) AwsException(com.adeptj.modules.aws.core.AwsException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 47 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project uploader by smoketurner.

the class Uploader method upload.

/**
 * Upload a batch to S3
 *
 * @param batch
 *            Batch to upload
 */
public void upload(@Nonnull final Batch batch) {
    batchSize.update(batch.size());
    batchCount.update(batch.getCount());
    final ImmutableMap.Builder<String, String> builder = ImmutableMap.<String, String>builder().put("count", String.valueOf(batch.getCount()));
    batch.getCustomerId().ifPresent(id -> builder.put("customer_id", id));
    final Map<String, String> metadata = builder.build();
    final String key;
    if (configuration.getPrefix().isPresent()) {
        key = configuration.getPrefix().get() + "/" + batch.getKey();
    } else {
        key = batch.getKey();
    }
    LOGGER.debug("Customer: {}, S3 key: {}", batch.getCustomerId().orElse(null), key);
    final PutObjectRequest request = PutObjectRequest.builder().bucket(configuration.getBucketName()).key(key).metadata(metadata).contentLength(batch.size()).contentType(MediaType.TEXT_PLAIN).contentEncoding("gzip").serverSideEncryption(ServerSideEncryption.AES256).build();
    final long start = currentTimeProvider.get();
    final CompletableFuture<PutObjectResponse> future = s3.putObject(request, new BatchRequestProvider(batch));
    future.whenComplete((resp, err) -> {
        if (resp != null) {
            final long took = currentTimeProvider.get() - start;
            uploadTime.update(took, TimeUnit.NANOSECONDS);
            successCounter.inc();
            LOGGER.info("Finished uploading \"{}\" ({} events, {} bytes) in {}ms", key, batch.getCount(), batch.size(), (took / NANOS_IN_MILLIS));
        } else {
            failedCounter.inc();
            LOGGER.error(String.format("Failed to upload \"%s\"", key), err);
        }
    });
}
Also used : PutObjectResponse(software.amazon.awssdk.services.s3.model.PutObjectResponse) ImmutableMap(com.google.common.collect.ImmutableMap) PutObjectRequest(software.amazon.awssdk.services.s3.model.PutObjectRequest)

Example 48 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project tutorials by eugenp.

the class MultipartUploadTest method whenUploadingFileWithTransferManager_thenVerifyUploadRequested.

@Test
public void whenUploadingFileWithTransferManager_thenVerifyUploadRequested() {
    File file = mock(File.class);
    PutObjectResult s3Result = mock(PutObjectResult.class);
    when(amazonS3.putObject(anyString(), anyString(), (File) any())).thenReturn(s3Result);
    when(file.getName()).thenReturn(KEY_NAME);
    PutObjectRequest request = new PutObjectRequest(BUCKET_NAME, KEY_NAME, file);
    request.setGeneralProgressListener(progressListener);
    Upload upload = tm.upload(request);
    assertThat(upload).isNotNull();
    verify(amazonS3).putObject(request);
}
Also used : PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) Upload(com.amazonaws.services.s3.transfer.Upload) File(java.io.File) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) Test(org.junit.Test)

Example 49 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project spring-integration-aws by spring-projects.

the class S3MessageHandlerTests method testUploadByteArray.

@Test
public void testUploadByteArray() throws IOException {
    byte[] payload = "b".getBytes("UTF-8");
    Message<?> message = MessageBuilder.withPayload(payload).setHeader("s3Command", S3MessageHandler.Command.UPLOAD.name()).setHeader("key", "myStream").build();
    this.s3SendChannel.send(message);
    ArgumentCaptor<PutObjectRequest> putObjectRequestArgumentCaptor = ArgumentCaptor.forClass(PutObjectRequest.class);
    verify(this.amazonS3, atLeastOnce()).putObject(putObjectRequestArgumentCaptor.capture());
    PutObjectRequest putObjectRequest = putObjectRequestArgumentCaptor.getValue();
    assertThat(putObjectRequest.getBucketName()).isEqualTo(S3_BUCKET_NAME);
    assertThat(putObjectRequest.getKey()).isEqualTo("myStream");
    assertThat(putObjectRequest.getFile()).isNull();
    assertThat(putObjectRequest.getInputStream()).isNotNull();
    ObjectMetadata metadata = putObjectRequest.getMetadata();
    assertThat(metadata.getContentMD5()).isEqualTo(Md5Utils.md5AsBase64(payload));
    assertThat(metadata.getContentLength()).isEqualTo(1);
    assertThat(metadata.getContentType()).isEqualTo(MediaType.APPLICATION_JSON_VALUE);
    assertThat(metadata.getContentDisposition()).isEqualTo("test.json");
}
Also used : ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) Test(org.junit.Test)

Example 50 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project spring-integration-aws by spring-projects.

the class S3MessageHandlerTests method testUploadInputStream.

@Test
public void testUploadInputStream() throws IOException {
    Expression actualKeyExpression = TestUtils.getPropertyValue(this.s3MessageHandler, "keyExpression", Expression.class);
    this.s3MessageHandler.setKeyExpression(null);
    InputStream payload = new StringInputStream("a");
    Message<?> message = MessageBuilder.withPayload(payload).setHeader("s3Command", S3MessageHandler.Command.UPLOAD.name()).setHeader("key", "myStream").build();
    assertThatThrownBy(() -> this.s3SendChannel.send(message)).hasCauseExactlyInstanceOf(IllegalStateException.class).hasMessageContaining("Specify a 'keyExpression' for non-java.io.File payloads");
    this.s3MessageHandler.setKeyExpression(actualKeyExpression);
    this.s3SendChannel.send(message);
    ArgumentCaptor<PutObjectRequest> putObjectRequestArgumentCaptor = ArgumentCaptor.forClass(PutObjectRequest.class);
    verify(this.amazonS3, atLeastOnce()).putObject(putObjectRequestArgumentCaptor.capture());
    PutObjectRequest putObjectRequest = putObjectRequestArgumentCaptor.getValue();
    assertThat(putObjectRequest.getBucketName()).isEqualTo(S3_BUCKET_NAME);
    assertThat(putObjectRequest.getKey()).isEqualTo("myStream");
    assertThat(putObjectRequest.getFile()).isNull();
    assertThat(putObjectRequest.getInputStream()).isNotNull();
    ObjectMetadata metadata = putObjectRequest.getMetadata();
    assertThat(metadata.getContentMD5()).isEqualTo(Md5Utils.md5AsBase64(payload));
    assertThat(metadata.getContentLength()).isEqualTo(1);
    assertThat(metadata.getContentType()).isEqualTo(MediaType.APPLICATION_JSON_VALUE);
    assertThat(metadata.getContentDisposition()).isEqualTo("test.json");
}
Also used : StringInputStream(com.amazonaws.util.StringInputStream) ValueExpression(org.springframework.integration.expression.ValueExpression) Expression(org.springframework.expression.Expression) ByteArrayInputStream(java.io.ByteArrayInputStream) S3ObjectInputStream(com.amazonaws.services.s3.model.S3ObjectInputStream) StringInputStream(com.amazonaws.util.StringInputStream) FileInputStream(java.io.FileInputStream) InputStream(java.io.InputStream) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest) Test(org.junit.Test)

Aggregations

PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)140 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)80 ByteArrayInputStream (java.io.ByteArrayInputStream)63 Test (org.junit.Test)61 File (java.io.File)35 IOException (java.io.IOException)32 S3FileTransferRequestParamsDto (org.finra.herd.model.dto.S3FileTransferRequestParamsDto)29 PutObjectResult (com.amazonaws.services.s3.model.PutObjectResult)23 Upload (com.amazonaws.services.s3.transfer.Upload)23 AmazonClientException (com.amazonaws.AmazonClientException)20 InputStream (java.io.InputStream)18 BusinessObjectDataKey (org.finra.herd.model.api.xml.BusinessObjectDataKey)14 StorageUnitEntity (org.finra.herd.model.jpa.StorageUnitEntity)14 PutObjectRequest (software.amazon.awssdk.services.s3.model.PutObjectRequest)14 BusinessObjectDataEntity (org.finra.herd.model.jpa.BusinessObjectDataEntity)13 HashMap (java.util.HashMap)12 S3Exception (software.amazon.awssdk.services.s3.model.S3Exception)10 AmazonServiceException (com.amazonaws.AmazonServiceException)9 AmazonS3 (com.amazonaws.services.s3.AmazonS3)9 Exchange (org.apache.camel.Exchange)8