Search in sources :

Example 1 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project elasticsearch by elastic.

the class TestAmazonS3 method putObject.

@Override
public PutObjectResult putObject(String bucketName, String key, InputStream input, ObjectMetadata metadata) throws AmazonClientException, AmazonServiceException {
    if (shouldFail(bucketName, key, writeFailureRate)) {
        long length = metadata.getContentLength();
        long partToRead = (long) (length * randomDouble());
        byte[] buffer = new byte[1024];
        for (long cur = 0; cur < partToRead; cur += buffer.length) {
            try {
                input.read(buffer, 0, (int) (partToRead - cur > buffer.length ? buffer.length : partToRead - cur));
            } catch (IOException ex) {
                throw new ElasticsearchException("cannot read input stream", ex);
            }
        }
        logger.info("--> random write failure on putObject method: throwing an exception for [bucket={}, key={}]", bucketName, key);
        AmazonS3Exception ex = new AmazonS3Exception("Random S3 exception");
        ex.setStatusCode(400);
        ex.setErrorCode("RequestTimeout");
        throw ex;
    } else {
        return super.putObject(bucketName, key, input, metadata);
    }
}
Also used : IOException(java.io.IOException) ElasticsearchException(org.elasticsearch.ElasticsearchException) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception)

Example 2 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project elasticsearch by elastic.

the class MockAmazonS3 method putObject.

@Override
public PutObjectResult putObject(PutObjectRequest putObjectRequest) throws AmazonClientException, AmazonServiceException {
    String blobName = putObjectRequest.getKey();
    DigestInputStream stream = (DigestInputStream) putObjectRequest.getInputStream();
    if (blobs.containsKey(blobName)) {
        throw new AmazonS3Exception("[" + blobName + "] already exists.");
    }
    blobs.put(blobName, stream);
    // input and output md5 hashes need to match to avoid an exception
    String md5 = Base64.encodeAsString(stream.getMessageDigest().digest());
    PutObjectResult result = new PutObjectResult();
    result.setContentMd5(md5);
    return result;
}
Also used : DigestInputStream(java.security.DigestInputStream) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception)

Example 3 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project elasticsearch by elastic.

the class MockAmazonS3 method getObject.

@Override
public S3Object getObject(GetObjectRequest getObjectRequest) throws AmazonClientException, AmazonServiceException {
    // in ESBlobStoreContainerTestCase.java, the prefix is empty,
    // so the key and blobName are equivalent to each other
    String blobName = getObjectRequest.getKey();
    if (!blobs.containsKey(blobName)) {
        throw new AmazonS3Exception("[" + blobName + "] does not exist.");
    }
    // the HTTP request attribute is irrelevant for reading
    S3ObjectInputStream stream = new S3ObjectInputStream(blobs.get(blobName), null, false);
    S3Object s3Object = new S3Object();
    s3Object.setObjectContent(stream);
    return s3Object;
}
Also used : S3ObjectInputStream(com.amazonaws.services.s3.model.S3ObjectInputStream) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) S3Object(com.amazonaws.services.s3.model.S3Object)

Example 4 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project elasticsearch by elastic.

the class MockDefaultS3OutputStream method doUpload.

@Override
protected void doUpload(S3BlobStore blobStore, String bucketName, String blobName, InputStream is, int length, boolean serverSideEncryption) throws AmazonS3Exception {
    try {
        long copied = Streams.copy(is, out);
        if (copied != length) {
            throw new AmazonS3Exception("Not all the bytes were copied");
        }
        numberOfUploadRequests++;
    } catch (IOException e) {
        throw new AmazonS3Exception(e.getMessage());
    }
}
Also used : AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) IOException(java.io.IOException)

Example 5 with AmazonS3Exception

use of com.amazonaws.services.s3.model.AmazonS3Exception in project elasticsearch by elastic.

the class DefaultS3OutputStream method doUpload.

protected void doUpload(S3BlobStore blobStore, String bucketName, String blobName, InputStream is, int length, boolean serverSideEncryption) throws AmazonS3Exception {
    ObjectMetadata md = new ObjectMetadata();
    if (serverSideEncryption) {
        md.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
    }
    md.setContentLength(length);
    InputStream inputStream = is;
    // We try to compute a MD5 while reading it
    MessageDigest messageDigest;
    try {
        messageDigest = MessageDigest.getInstance("MD5");
        inputStream = new DigestInputStream(is, messageDigest);
    } catch (NoSuchAlgorithmException impossible) {
        // Every implementation of the Java platform is required to support MD5 (see MessageDigest)
        throw new RuntimeException(impossible);
    }
    PutObjectRequest putRequest = new PutObjectRequest(bucketName, blobName, inputStream, md).withStorageClass(blobStore.getStorageClass()).withCannedAcl(blobStore.getCannedACL());
    PutObjectResult putObjectResult = blobStore.client().putObject(putRequest);
    String localMd5 = Base64.encodeAsString(messageDigest.digest());
    String remoteMd5 = putObjectResult.getContentMd5();
    if (!localMd5.equals(remoteMd5)) {
        logger.debug("MD5 local [{}], remote [{}] are not equal...", localMd5, remoteMd5);
        throw new AmazonS3Exception("MD5 local [" + localMd5 + "], remote [" + remoteMd5 + "] are not equal...");
    }
}
Also used : DigestInputStream(java.security.DigestInputStream) PutObjectResult(com.amazonaws.services.s3.model.PutObjectResult) ByteArrayInputStream(java.io.ByteArrayInputStream) DigestInputStream(java.security.DigestInputStream) InputStream(java.io.InputStream) NoSuchAlgorithmException(java.security.NoSuchAlgorithmException) AmazonS3Exception(com.amazonaws.services.s3.model.AmazonS3Exception) MessageDigest(java.security.MessageDigest) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) PutObjectRequest(com.amazonaws.services.s3.model.PutObjectRequest)

Aggregations

AmazonS3Exception (com.amazonaws.services.s3.model.AmazonS3Exception)62 IOException (java.io.IOException)23 Test (org.junit.Test)13 FileNotFoundException (java.io.FileNotFoundException)10 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)9 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)9 AmazonServiceException (com.amazonaws.AmazonServiceException)8 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)7 GetObjectMetadataRequest (com.amazonaws.services.s3.model.GetObjectMetadataRequest)6 S3Object (com.amazonaws.services.s3.model.S3Object)6 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws.s3.S3TestUtils.buildMockedS3FileSystem)6 AmazonClientException (com.amazonaws.AmazonClientException)5 InterruptedIOException (java.io.InterruptedIOException)5 ArrayList (java.util.ArrayList)5 Path (org.apache.hadoop.fs.Path)5 ClientConfiguration (com.amazonaws.ClientConfiguration)4 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)4 GetObjectRequest (com.amazonaws.services.s3.model.GetObjectRequest)4 PutObjectRequest (com.amazonaws.services.s3.model.PutObjectRequest)4 URI (java.net.URI)4