Search in sources :

Example 26 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project druid by druid-io.

the class S3Entity method readFrom.

@Override
protected InputStream readFrom(long offset) throws IOException {
    final GetObjectRequest request = new GetObjectRequest(object.getBucket(), object.getPath());
    request.setRange(offset);
    try {
        final S3Object s3Object = s3Client.getObject(request);
        if (s3Object == null) {
            throw new ISE("Failed to get an s3 object for bucket[%s], key[%s], and start[%d]", object.getBucket(), object.getPath(), offset);
        }
        return s3Object.getObjectContent();
    } catch (AmazonS3Exception e) {
        throw new IOException(e);
    }
}
Also used : ISE(org.apache.druid.java.util.common.ISE) S3Object(com.amazonaws.services.s3.model.S3Object) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) IOException(java.io.IOException) GetObjectRequest(com.amazonaws.services.s3.model.GetObjectRequest)

Example 27 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project druid by druid-io.

the class StaticS3FirehoseFactory method openObjectStream.

@Override
protected InputStream openObjectStream(URI object) throws IOException {
    try {
        // Get data of the given object and open an input stream
        final String bucket = object.getAuthority();
        final String key = S3Utils.extractS3Key(object);
        final S3Object s3Object = s3Client.getObject(bucket, key);
        if (s3Object == null) {
            throw new ISE("Failed to get an s3 object for bucket[%s] and key[%s]", bucket, key);
        }
        return s3Object.getObjectContent();
    } catch (AmazonS3Exception e) {
        throw new IOException(e);
    }
}
Also used : ISE(org.apache.druid.java.util.common.ISE) S3Object(com.amazonaws.services.s3.model.S3Object) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) IOException(java.io.IOException)

Example 28 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project druid by druid-io.

the class ObjectSummaryIterator method fetchNextBatch.

private void fetchNextBatch() {
    try {
        result = S3Utils.retryS3Operation(() -> s3Client.listObjectsV2(request), maxRetries);
        request.setContinuationToken(result.getNextContinuationToken());
        objectSummaryIterator = result.getObjectSummaries().iterator();
    } catch (AmazonS3Exception e) {
        throw new RE(e, "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s", request.getBucketName(), request.getPrefix(), e.getMessage());
    } catch (Exception e) {
        throw new RE(e, "Failed to get object summaries from S3 bucket[%s], prefix[%s]", request.getBucketName(), request.getPrefix());
    }
}
Also used : RE(org.apache.druid.java.util.common.RE) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) NoSuchElementException(java.util.NoSuchElementException)

Example 29 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project druid by druid-io.

the class S3InputSourceTest method expectListObjectsAndThrowAccessDenied.

private static void expectListObjectsAndThrowAccessDenied(final URI prefix) {
    AmazonS3Exception boom = new AmazonS3Exception("oh dang, you can't list that bucket friend");
    boom.setStatusCode(403);
    EasyMock.expect(S3_CLIENT.listObjectsV2(matchListObjectsRequest(prefix))).andThrow(boom).once();
}
Also used : AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception)

Example 30 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project aws-doc-sdk-examples by awsdocs.

the class ImportEndpoints method uploadToS3.

private static void uploadToS3(File endpointsFile, String s3BucketName) {
    // Initializes Amazon S3 client.
    final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
    // Checks whether the specified bucket exists. If not, attempts to create one.
    if (!s3.doesBucketExistV2(s3BucketName)) {
        try {
            s3.createBucket(s3BucketName);
            System.out.format("Created S3 bucket %s.\n", s3BucketName);
        } catch (AmazonS3Exception e) {
            System.err.println(e.getErrorMessage());
            System.exit(1);
        }
    }
    // Uploads the endpoints file to the bucket.
    String endpointsFileName = endpointsFile.getName();
    System.out.format("Uploading %s to S3 bucket %s . . .\n", endpointsFileName, s3BucketName);
    try {
        s3.putObject(s3BucketName, "imports/" + endpointsFileName, endpointsFile);
        System.out.println("Finished uploading to S3.");
    } catch (AmazonServiceException e) {
        System.err.println(e.getErrorMessage());
        System.exit(1);
    }
}
Also used : AmazonS3(com.amazonaws.services.s3.AmazonS3) AmazonServiceException(com.amazonaws.AmazonServiceException) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception)

Aggregations

AmazonS3Exception (com.amazonaws.services.s3.model.AmazonS3Exception)62 IOException (java.io.IOException)23 Test (org.junit.Test)13 FileNotFoundException (java.io.FileNotFoundException)10 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)9 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)9 AmazonServiceException (com.amazonaws.AmazonServiceException)8 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)7 GetObjectMetadataRequest (com.amazonaws.services.s3.model.GetObjectMetadataRequest)6 S3Object (com.amazonaws.services.s3.model.S3Object)6 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws.s3.S3TestUtils.buildMockedS3FileSystem)6 AmazonClientException (com.amazonaws.AmazonClientException)5 InterruptedIOException (java.io.InterruptedIOException)5 ArrayList (java.util.ArrayList)5 Path (org.apache.hadoop.fs.Path)5 ClientConfiguration (com.amazonaws.ClientConfiguration)4 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)4 GetObjectRequest (com.amazonaws.services.s3.model.GetObjectRequest)4 PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)4 URI (java.net.URI)4