Search in sources :

Example 46 with ObjectMetadata

use of com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata in project components by Talend.

the class S3SparkRuntimeTestIT method testAvro_noEncryption.

/**
 * Basic Avro test.
 */
@Test
public void testAvro_noEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties();
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
    // Get some object metadata from the results.
    ObjectMetadata md = s3.getObjectMetadata(datasetProps);
    assertThat(md.getSSEAlgorithm(), nullValue());
    assertThat(md.getSSEAwsKmsKeyId(), nullValue());
}
Also used : S3DatasetProperties(org.talend.components.simplefileio.s3.S3DatasetProperties) ObjectMetadata(com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata) Test(org.junit.Test)

Example 47 with ObjectMetadata

use of com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata in project components by Talend.

the class S3RoundTripRuntimeTestIT method testAvro_noEncryption.

/**
 * Basic Avro test.
 */
@Test
public void testAvro_noEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties();
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
    // Get some object metadata from the results.
    ObjectMetadata md = s3.getObjectMetadata(datasetProps);
    assertThat(md.getSSEAlgorithm(), nullValue());
    assertThat(md.getSSEAwsKmsKeyId(), nullValue());
}
Also used : S3DatasetProperties(org.talend.components.simplefileio.s3.S3DatasetProperties) ObjectMetadata(com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata) Test(org.junit.Test)

Example 48 with ObjectMetadata

use of com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata in project components by Talend.

the class S3RoundTripRuntimeTestIT method testAvro_sseKmsEncryption.

/**
 * Basic Avro test with sseKmsEncryption.
 */
@Test
public void testAvro_sseKmsEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties(true, false);
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
    // Get some object metadata from the results.
    ObjectMetadata md = s3.getObjectMetadata(datasetProps);
    assertThat(md.getSSEAlgorithm(), is("aws:kms"));
    assertThat(md.getSSEAwsKmsKeyId(), is(datasetProps.kmsForDataAtRest.getValue()));
}
Also used : S3DatasetProperties(org.talend.components.simplefileio.s3.S3DatasetProperties) ObjectMetadata(com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata) Test(org.junit.Test)

Example 49 with ObjectMetadata

use of com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata in project jackrabbit-oak by apache.

the class S3Backend method addMetadataRecord.

@Override
public void addMetadataRecord(final InputStream input, final String name) throws DataStoreException {
    ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
    try {
        Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
        Upload upload = tmx.upload(s3ReqDecorator.decorate(new PutObjectRequest(bucket, addMetaKeyPrefix(name), input, new ObjectMetadata())));
        upload.waitForUploadResult();
    } catch (InterruptedException e) {
        LOG.error("Error in uploading", e);
        throw new DataStoreException("Error in uploading", e);
    } finally {
        if (contextClassLoader != null) {
            Thread.currentThread().setContextClassLoader(contextClassLoader);
        }
    }
}
Also used : DataStoreException(org.apache.jackrabbit.core.data.DataStoreException) Upload(com.amazonaws.services.s3.transfer.Upload) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Example 50 with ObjectMetadata

use of com.talend.shaded.com.amazonaws.services.s3.model.ObjectMetadata in project jackrabbit-oak by apache.

the class S3Backend method exists.

/**
 * Check if record identified by identifier exists in Amazon S3.
 */
@Override
public boolean exists(DataIdentifier identifier) throws DataStoreException {
    long start = System.currentTimeMillis();
    String key = getKeyName(identifier);
    ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
    try {
        Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
        ObjectMetadata objectMetaData = s3service.getObjectMetadata(bucket, key);
        if (objectMetaData != null) {
            LOG.trace("exists [{}]: [true] took [{}] ms.", identifier, (System.currentTimeMillis() - start));
            return true;
        }
        return false;
    } catch (AmazonServiceException e) {
        if (e.getStatusCode() == 404 || e.getStatusCode() == 403) {
            LOG.debug("exists [{}]: [false] took [{}] ms.", identifier, (System.currentTimeMillis() - start));
            return false;
        }
        throw new DataStoreException("Error occured to getObjectMetadata for key [" + identifier.toString() + "]", e);
    } finally {
        if (contextClassLoader != null) {
            Thread.currentThread().setContextClassLoader(contextClassLoader);
        }
    }
}
Also used : DataStoreException(org.apache.jackrabbit.core.data.DataStoreException) AmazonServiceException(com.amazonaws.AmazonServiceException) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata)

Aggregations

ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)163 PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)76 ByteArrayInputStream (java.io.ByteArrayInputStream)52 Test (org.junit.Test)47 IOException (java.io.IOException)33 File (java.io.File)27 AmazonClientException (com.amazonaws.AmazonClientException)25 AmazonServiceException (com.amazonaws.AmazonServiceException)22 S3FileTransferRequestParamsDto (org.finra.herd.model.dto.S3FileTransferRequestParamsDto)21 InputStream (java.io.InputStream)20 DataStoreException (org.apache.jackrabbit.core.data.DataStoreException)18 PutObjectResult (com.amazonaws.services.s3.model.PutObjectResult)15 Upload (com.amazonaws.services.s3.transfer.Upload)15 CopyObjectRequest (com.amazonaws.services.s3.model.CopyObjectRequest)11 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)10 Date (java.util.Date)9 BusinessObjectDataKey (org.finra.herd.model.api.xml.BusinessObjectDataKey)9 Copy (com.amazonaws.services.s3.transfer.Copy)8 S3Object (com.amazonaws.services.s3.model.S3Object)7 InterruptedIOException (java.io.InterruptedIOException)7