Search in sources :

Example 71 with ObjectMetadata

use of com.ibm.watson.visual_recognition.v4.model.ObjectMetadata in project javabuilder by code-dot-org.

the class AWSContentManager method writeToOutputFile.

@Override
public String writeToOutputFile(String filename, byte[] inputBytes, String contentType) throws JavabuilderException {
    if (this.writes >= WRITES_PER_SESSION) {
        throw new UserInitiatedException(UserInitiatedExceptionKey.TOO_MANY_WRITES);
    }
    String filePath = this.generateKey(filename);
    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentType(contentType);
    metadata.setContentLength(inputBytes.length);
    ByteArrayInputStream inputStream = new ByteArrayInputStream(inputBytes);
    try {
        this.s3Client.putObject(this.bucketName, filePath, inputStream, metadata);
    } catch (AbortedException e) {
    // this is most likely because the end user interrupted program execution. We can safely
    // ignore this.
    } catch (SdkClientException e) {
        // We couldn't write to S3, send a message to the user and fail. The S3 SDK includes retries.
        throw new InternalServerError(InternalErrorKey.INTERNAL_RUNTIME_EXCEPTION, e);
    }
    this.writes++;
    return this.contentBucketUrl + "/" + filePath;
}
Also used : SdkClientException(com.amazonaws.SdkClientException) ByteArrayInputStream(java.io.ByteArrayInputStream) AbortedException(com.amazonaws.AbortedException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata)

Example 72 with ObjectMetadata

use of com.ibm.watson.visual_recognition.v4.model.ObjectMetadata in project Europeana-Cloud by europeana.

the class ThumbnailUploader method prepareObjectMetadata.

private ObjectMetadata prepareObjectMetadata(Thumbnail thumbnail) throws IOException {
    final ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentType(thumbnail.getMimeType());
    objectMetadata.setContentLength(thumbnail.getContentSize());
    return objectMetadata;
}
Also used : ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata)

Example 73 with ObjectMetadata

use of com.ibm.watson.visual_recognition.v4.model.ObjectMetadata in project presto by prestodb.

the class PrestoS3FileSystem method getS3ObjectMetadata.

@VisibleForTesting
ObjectMetadata getS3ObjectMetadata(Path path) throws IOException {
    String bucketName = getBucketName(uri);
    String key = keyFromPath(path);
    ObjectMetadata s3ObjectMetadata = getS3ObjectMetadata(path, bucketName, key);
    if (s3ObjectMetadata == null && !key.isEmpty()) {
        return getS3ObjectMetadata(path, bucketName, key + PATH_SEPARATOR);
    }
    return s3ObjectMetadata;
}
Also used : ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Example 74 with ObjectMetadata

use of com.ibm.watson.visual_recognition.v4.model.ObjectMetadata in project r5 by conveyal.

the class ShapeDataStore method writeTilesToS3.

/**
 * Write GeoBuf tiles to S3
 */
public void writeTilesToS3(String bucketName) throws IOException {
    // For the duration of this multiple-tile upload operation, manage a single upload thread for the S3 uploads.
    ExecutorService executor = Executors.newSingleThreadExecutor();
    // initialize an S3 client
    AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();
    try {
        writeTilesInternal((x, y) -> {
            PipedInputStream is = new PipedInputStream();
            PipedOutputStream os = new PipedOutputStream(is);
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentType("application/gzip");
            // perform the upload in its own thread so it doesn't deadlock
            executor.execute(() -> s3.putObject(bucketName, String.format("%d/%d.pbf.gz", x, y), is, metadata));
            return os;
        });
    } finally {
        // allow the JVM to exit
        executor.shutdown();
        try {
            executor.awaitTermination(1, TimeUnit.HOURS);
        } catch (InterruptedException e) {
            LOG.error("Interrupted while waiting for S3 uploads to finish");
        }
    }
}
Also used : AmazonS3(com.amazonaws.services.s3.AmazonS3) ExecutorService(java.util.concurrent.ExecutorService) PipedOutputStream(java.io.PipedOutputStream) PipedInputStream(java.io.PipedInputStream) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata)

Example 75 with ObjectMetadata

use of com.ibm.watson.visual_recognition.v4.model.ObjectMetadata in project druid by druid-io.

the class S3ServerSideEncryption method decorate.

@Override
public PutObjectRequest decorate(PutObjectRequest request) {
    final ObjectMetadata objectMetadata = request.getMetadata() == null ? new ObjectMetadata() : request.getMetadata().clone();
    objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
    return request.withMetadata(objectMetadata);
}
Also used : ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata)

Aggregations

ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)566 PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)191 ByteArrayInputStream (java.io.ByteArrayInputStream)157 Test (org.junit.Test)143 IOException (java.io.IOException)101 InputStream (java.io.InputStream)80 File (java.io.File)62 AmazonClientException (com.amazonaws.AmazonClientException)61 AmazonServiceException (com.amazonaws.AmazonServiceException)61 S3Object (com.amazonaws.services.s3.model.S3Object)59 AmazonS3 (com.amazonaws.services.s3.AmazonS3)54 Date (java.util.Date)46 S3FileTransferRequestParamsDto (org.finra.herd.model.dto.S3FileTransferRequestParamsDto)34 GetObjectMetadataRequest (com.amazonaws.services.s3.model.GetObjectMetadataRequest)33 PutObjectResult (com.amazonaws.services.s3.model.PutObjectResult)32 GetObjectRequest (com.amazonaws.services.s3.model.GetObjectRequest)30 AmazonS3Exception (com.amazonaws.services.s3.model.AmazonS3Exception)29 Upload (com.amazonaws.services.s3.transfer.Upload)26 SdkClientException (com.amazonaws.SdkClientException)24 CopyObjectRequest (com.amazonaws.services.s3.model.CopyObjectRequest)24