Search in sources :

Example 76 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project crate by crate.

the class S3BlobContainer method executeSingleUpload.

/**
 * Uploads a blob using a single upload request
 */
void executeSingleUpload(final S3BlobStore blobStore, final String blobName, final InputStream input, final long blobSize) throws IOException {
    // Extra safety checks
    if (blobSize > MAX_FILE_SIZE.getBytes()) {
        throw new IllegalArgumentException("Upload request size [" + blobSize + "] can't be larger than " + MAX_FILE_SIZE);
    }
    if (blobSize > blobStore.bufferSizeInBytes()) {
        throw new IllegalArgumentException("Upload request size [" + blobSize + "] can't be larger than buffer size");
    }
    final ObjectMetadata md = new ObjectMetadata();
    md.setContentLength(blobSize);
    if (blobStore.serverSideEncryption()) {
        md.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
    }
    final PutObjectRequest putRequest = new PutObjectRequest(blobStore.bucket(), blobName, input, md);
    putRequest.setStorageClass(blobStore.getStorageClass());
    putRequest.setCannedAcl(blobStore.getCannedACL());
    try (AmazonS3Reference clientReference = blobStore.clientReference()) {
        clientReference.client().putObject(putRequest);
    } catch (final AmazonClientException e) {
        throw new IOException("Unable to upload object [" + blobName + "] using a single upload", e);
    }
}
Also used : AmazonClientException(com.amazonaws.AmazonClientException) IOException(java.io.IOException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 77 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project airpal by airbnb.

the class S3FilePersistor method persist.

@Override
public URI persist(JobOutputBuilder outputBuilder, Job job) {
    File file = checkNotNull(outputBuilder.build(), "output builder resulting file was null");
    val objectMetaData = new ObjectMetadata();
    objectMetaData.setContentLength(file.length());
    objectMetaData.setContentType(MediaType.CSV_UTF_8.toString());
    if (compressedOutput) {
        objectMetaData.setContentEncoding("gzip");
    }
    val putRequest = new PutObjectRequest(outputBucket, getOutputKey(file.getName()), file).withMetadata(objectMetaData);
    try {
        s3Client.putObject(putRequest);
        return UriBuilder.fromPath("/api/s3/{filename}").build(file.getName());
    } catch (AmazonClientException e) {
        throw new ExecutionClient.ExecutionFailureException(job, "Could not upload CSV to S3", e);
    } finally {
        outputBuilder.delete();
    }
}
Also used : lombok.val(lombok.val) AmazonClientException(com.amazonaws.AmazonClientException) ExecutionClient(com.airbnb.airpal.core.execution.ExecutionClient) File(java.io.File) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 78 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project uPortal by Jasig.

the class AwsS3DynamicSkinService method saveContentToAwsS3Bucket.

private void saveContentToAwsS3Bucket(final String objectKey, final String content, final DynamicSkinInstanceData data) {
    final InputStream inputStream = IOUtils.toInputStream(content);
    final ObjectMetadata objectMetadata = this.createObjectMetadata(content, data);
    final PutObjectRequest putObjectRequest = this.createPutObjectRequest(objectKey, inputStream, objectMetadata);
    log.info(ATTEMPTING_TO_SAVE_FILE_TO_AWS_S3_LOG_MSG, this.awsS3BucketConfig.getBucketName(), objectKey);
    this.saveContentToAwsS3Bucket(putObjectRequest);
    log.info(FILE_SAVED_TO_AWS_S3_LOG_MSG, this.awsS3BucketConfig.getBucketName(), objectKey);
}
Also used : InputStream(java.io.InputStream) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 79 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project xm-ms-entity by xm-online.

the class AmazonS3Template method save.

/**
 * Save a file using authenticated session credentials.
 *
 * @param key is the name of the file to save in the bucket
 * @param inputStream is the file that will be saved
 */
public void save(String key, InputStream inputStream) throws IOException {
    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentType(URLConnection.guessContentTypeFromStream(inputStream));
    PutObjectRequest request = new PutObjectRequest(bucket, key, inputStream, metadata);
    request.setCannedAcl(CannedAccessControlList.PublicRead);
    request.getRequestClientOptions().setReadLimit(Integer.MAX_VALUE);
    Upload upload = getTransferManager().upload(request);
    try {
        upload.waitForUploadResult();
    } catch (AmazonClientException ex) {
        throw new IOException(ex);
    } catch (InterruptedException ex) {
        // reset interrupted status
        Thread.currentThread().interrupt();
        // continue interrupt
        throw new IllegalStateException(ex);
    }
}
Also used : AmazonClientException(com.amazonaws.AmazonClientException) Upload(com.amazonaws.services.s3.transfer.Upload) IOException(java.io.IOException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 80 with PutObjectRequest

use of software.amazon.awssdk.services.s3.model.PutObjectRequest in project alluxio by Alluxio.

the class S3AUnderFileSystem method createEmptyObject.

@Override
public boolean createEmptyObject(String key) {
    try {
        ObjectMetadata meta = new ObjectMetadata();
        meta.setContentLength(0);
        meta.setContentMD5(DIR_HASH);
        meta.setContentType(Mimetypes.MIMETYPE_OCTET_STREAM);
        mClient.putObject(new PutObjectRequest(mBucketName, key, new ByteArrayInputStream(new byte[0]), meta));
        return true;
    } catch (AmazonClientException e) {
        LOG.error("Failed to create object: {}", key, e);
        return false;
    }
}
Also used : ByteArrayInputStream(java.io.ByteArrayInputStream) AmazonClientException(com.amazonaws.AmazonClientException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Aggregations

PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)140 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)80 ByteArrayInputStream (java.io.ByteArrayInputStream)63 Test (org.junit.Test)61 File (java.io.File)35 IOException (java.io.IOException)32 S3FileTransferRequestParamsDto (org.finra.herd.model.dto.S3FileTransferRequestParamsDto)29 PutObjectResult (com.amazonaws.services.s3.model.PutObjectResult)23 Upload (com.amazonaws.services.s3.transfer.Upload)23 AmazonClientException (com.amazonaws.AmazonClientException)20 InputStream (java.io.InputStream)18 BusinessObjectDataKey (org.finra.herd.model.api.xml.BusinessObjectDataKey)14 StorageUnitEntity (org.finra.herd.model.jpa.StorageUnitEntity)14 PutObjectRequest (software.amazon.awssdk.services.s3.model.PutObjectRequest)14 BusinessObjectDataEntity (org.finra.herd.model.jpa.BusinessObjectDataEntity)13 HashMap (java.util.HashMap)12 S3Exception (software.amazon.awssdk.services.s3.model.S3Exception)10 AmazonServiceException (com.amazonaws.AmazonServiceException)9 AmazonS3 (com.amazonaws.services.s3.AmazonS3)9 Exchange (org.apache.camel.Exchange)8