Search in sources :

Example 71 with S3Object

use of software.amazon.awssdk.services.s3.model.S3Object in project archaius by Netflix.

the class S3ConfigurationSource method poll.

@Override
public PollResult poll(boolean initial, Object checkPoint) throws IOException, AmazonServiceException {
    GetObjectRequest s3request = new GetObjectRequest(bucketName, key);
    InputStream is = null;
    try {
        S3Object result = client.getObject(s3request);
        is = result.getObjectContent();
        Map<String, Object> resultMap = inputStreamToMap(is);
        return PollResult.createFull(resultMap);
    } finally {
        if (is != null)
            is.close();
    }
}
Also used : InputStream(java.io.InputStream) S3Object(com.amazonaws.services.s3.model.S3Object) S3Object(com.amazonaws.services.s3.model.S3Object) GetObjectRequest(com.amazonaws.services.s3.model.GetObjectRequest)

Example 72 with S3Object

use of software.amazon.awssdk.services.s3.model.S3Object in project camel by apache.

the class S3Consumer method poll.

@Override
protected int poll() throws Exception {
    // must reset for each poll
    shutdownRunningTask = null;
    pendingExchanges = 0;
    String fileName = getConfiguration().getFileName();
    String bucketName = getConfiguration().getBucketName();
    Queue<Exchange> exchanges;
    if (fileName != null) {
        LOG.trace("Getting object in bucket [{}] with file name [{}]...", bucketName, fileName);
        S3Object s3Object = getAmazonS3Client().getObject(new GetObjectRequest(bucketName, fileName));
        exchanges = createExchanges(s3Object);
    } else {
        LOG.trace("Queueing objects in bucket [{}]...", bucketName);
        ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
        listObjectsRequest.setBucketName(bucketName);
        listObjectsRequest.setPrefix(getConfiguration().getPrefix());
        if (maxMessagesPerPoll > 0) {
            listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
        }
        // if there was a marker from previous poll then use that to continue from where we left last time
        if (marker != null) {
            LOG.trace("Resuming from marker: {}", marker);
            listObjectsRequest.setMarker(marker);
        }
        ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
        if (listObjects.isTruncated()) {
            marker = listObjects.getNextMarker();
            LOG.trace("Returned list is truncated, so setting next marker: {}", marker);
        } else {
            // no more data so clear marker
            marker = null;
        }
        if (LOG.isTraceEnabled()) {
            LOG.trace("Found {} objects in bucket [{}]...", listObjects.getObjectSummaries().size(), bucketName);
        }
        exchanges = createExchanges(listObjects.getObjectSummaries());
    }
    return processBatch(CastUtils.cast(exchanges));
}
Also used : Exchange(org.apache.camel.Exchange) ListObjectsRequest(com.amazonaws.services.s3.model.ListObjectsRequest) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3Object(com.amazonaws.services.s3.model.S3Object) GetObjectRequest(com.amazonaws.services.s3.model.GetObjectRequest)

Example 73 with S3Object

use of software.amazon.awssdk.services.s3.model.S3Object in project camel by apache.

the class S3Consumer method createExchanges.

protected Queue<Exchange> createExchanges(List<S3ObjectSummary> s3ObjectSummaries) {
    if (LOG.isTraceEnabled()) {
        LOG.trace("Received {} messages in this poll", s3ObjectSummaries.size());
    }
    Queue<Exchange> answer = new LinkedList<Exchange>();
    for (S3ObjectSummary s3ObjectSummary : s3ObjectSummaries) {
        S3Object s3Object = getAmazonS3Client().getObject(s3ObjectSummary.getBucketName(), s3ObjectSummary.getKey());
        Exchange exchange = getEndpoint().createExchange(s3Object);
        answer.add(exchange);
    }
    return answer;
}
Also used : Exchange(org.apache.camel.Exchange) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) S3Object(com.amazonaws.services.s3.model.S3Object) LinkedList(java.util.LinkedList)

Example 74 with S3Object

use of software.amazon.awssdk.services.s3.model.S3Object in project camel by apache.

the class S3BatchConsumerTest method createRegistry.

@Override
protected JndiRegistry createRegistry() throws Exception {
    JndiRegistry registry = super.createRegistry();
    AmazonS3ClientMock clientMock = new AmazonS3ClientMock();
    // add 6 messages, one more we will poll
    for (int counter = 0; counter < 6; counter++) {
        S3Object s3Object = new S3Object();
        s3Object.setBucketName("mycamelbucket");
        s3Object.setKey("counter-" + counter);
        clientMock.objects.add(s3Object);
    }
    registry.bind("amazonS3Client", clientMock);
    return registry;
}
Also used : JndiRegistry(org.apache.camel.impl.JndiRegistry) S3Object(com.amazonaws.services.s3.model.S3Object) MockEndpoint(org.apache.camel.component.mock.MockEndpoint)

Example 75 with S3Object

use of software.amazon.awssdk.services.s3.model.S3Object in project jackrabbit-oak by apache.

the class S3Backend method read.

@Override
public InputStream read(DataIdentifier identifier) throws DataStoreException {
    long start = System.currentTimeMillis();
    String key = getKeyName(identifier);
    ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
    try {
        Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
        S3Object object = s3service.getObject(bucket, key);
        InputStream in = object.getObjectContent();
        LOG.debug("[{}] read took [{}]ms", identifier, (System.currentTimeMillis() - start));
        return in;
    } catch (AmazonServiceException e) {
        throw new DataStoreException("Object not found: " + key, e);
    } finally {
        if (contextClassLoader != null) {
            Thread.currentThread().setContextClassLoader(contextClassLoader);
        }
    }
}
Also used : DataStoreException(org.apache.jackrabbit.core.data.DataStoreException) InputStream(java.io.InputStream) AmazonServiceException(com.amazonaws.AmazonServiceException) S3Object(com.amazonaws.services.s3.model.S3Object)

Aggregations

S3Object (com.amazonaws.services.s3.model.S3Object)110 S3ObjectInputStream (com.amazonaws.services.s3.model.S3ObjectInputStream)34 InputStream (java.io.InputStream)28 IOException (java.io.IOException)24 GetObjectRequest (com.amazonaws.services.s3.model.GetObjectRequest)23 ByteArrayInputStream (java.io.ByteArrayInputStream)21 AmazonServiceException (com.amazonaws.AmazonServiceException)20 AmazonS3 (com.amazonaws.services.s3.AmazonS3)20 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)18 Test (org.junit.Test)18 Test (org.testng.annotations.Test)18 S3Object (software.amazon.awssdk.services.s3.model.S3Object)17 File (java.io.File)14 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)13 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)13 FileInputStream (java.io.FileInputStream)13 Date (java.util.Date)11 SignedDomain (com.yahoo.athenz.zms.SignedDomain)10 ListObjectsV2Response (software.amazon.awssdk.services.s3.model.ListObjectsV2Response)9 AmazonClientException (com.amazonaws.AmazonClientException)8