Examples with S3Object com.amazonaws.services.s3.model.S3Object used on opensource projects

Example 6 with S3Object

use of com.amazonaws.services.s3.model.S3Object in project deeplearning4j by deeplearning4j.

the class S3Downloader method download.

public void download(String bucket, String key, File to) throws IOException {
    AmazonS3 s3 = getClient();
    S3Object obj = s3.getObject(bucket, key);
    InputStream is = obj.getObjectContent();
    BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(to));
    IOUtils.copy(is, bos);
    bos.close();
    is.close();
    obj.close();
}

Example 7 with S3Object

use of com.amazonaws.services.s3.model.S3Object in project crate by crate.

the class FileReadingCollectorTest method createBatchIterator.

private BatchIterator createBatchIterator(Collection<String> fileUris, String compression, final S3ObjectInputStream s3InputStream) {
    Reference raw = createReference("_raw", DataTypes.STRING);
    InputFactory.Context<LineCollectorExpression<?>> ctx = inputFactory.ctxForRefs(FileLineReferenceResolver::getImplementation);
    List<Input<?>> inputs = Collections.singletonList(ctx.add(raw));
    return FileReadingIterator.newInstance(fileUris, inputs, ctx.expressions(), compression, ImmutableMap.of(LocalFsFileInputFactory.NAME, new LocalFsFileInputFactory(), S3FileInputFactory.NAME, () -> new S3FileInput(new S3ClientHelper() {

        @Override
        protected AmazonS3 initClient(String accessKey, String secretKey) throws IOException {
            AmazonS3 client = mock(AmazonS3Client.class);
            ObjectListing objectListing = mock(ObjectListing.class);
            S3ObjectSummary summary = mock(S3ObjectSummary.class);
            S3Object s3Object = mock(S3Object.class);
            when(client.listObjects(anyString(), anyString())).thenReturn(objectListing);
            when(objectListing.getObjectSummaries()).thenReturn(Arrays.asList(summary));
            when(summary.getKey()).thenReturn("foo");
            when(client.getObject("fakebucket", "foo")).thenReturn(s3Object);
            when(s3Object.getObjectContent()).thenReturn(s3InputStream);
            when(client.listNextBatchOfObjects(any(ObjectListing.class))).thenReturn(objectListing);
            when(objectListing.isTruncated()).thenReturn(false);
            return client;
        }
    })), false, 1, 0);
}

Example 8 with S3Object

use of com.amazonaws.services.s3.model.S3Object in project elasticsearch by elastic.

the class MockAmazonS3 method getObject.

@Override
public S3Object getObject(GetObjectRequest getObjectRequest) throws AmazonClientException, AmazonServiceException {
    // in ESBlobStoreContainerTestCase.java, the prefix is empty,
    // so the key and blobName are equivalent to each other
    String blobName = getObjectRequest.getKey();
    if (!blobs.containsKey(blobName)) {
        throw new AmazonS3Exception("[" + blobName + "] does not exist.");
    }
    // the HTTP request attribute is irrelevant for reading
    S3ObjectInputStream stream = new S3ObjectInputStream(blobs.get(blobName), null, false);
    S3Object s3Object = new S3Object();
    s3Object.setObjectContent(stream);
    return s3Object;
}

Example 9 with S3Object

use of com.amazonaws.services.s3.model.S3Object in project YCSB by brianfrankcooper.

the class S3Client method readFromStorage.

/**
  * Download an object from S3.
  *
  * @param bucket
  *            The name of the bucket
  * @param key
  *            The file key of the object to upload/update.
  * @param result
  *            The Hash map where data from the object are written
  *
  */
protected Status readFromStorage(String bucket, String key, HashMap<String, ByteIterator> result, SSECustomerKey ssecLocal) {
    try {
        Map.Entry<S3Object, ObjectMetadata> objectAndMetadata = getS3ObjectAndMetadata(bucket, key, ssecLocal);
        //consuming the stream
        InputStream objectData = objectAndMetadata.getKey().getObjectContent();
        // writing the stream to bytes and to results
        int sizeOfFile = (int) objectAndMetadata.getValue().getContentLength();
        byte[] inputStreamToByte = new byte[sizeOfFile];
        objectData.read(inputStreamToByte, 0, sizeOfFile);
        result.put(key, new ByteArrayByteIterator(inputStreamToByte));
        objectData.close();
        objectAndMetadata.getKey().close();
    } catch (Exception e) {
        System.err.println("Not possible to get the object " + key);
        e.printStackTrace();
        return Status.ERROR;
    }
    return Status.OK;
}

Example 10 with S3Object

use of com.amazonaws.services.s3.model.S3Object in project gradle by gradle.

the class S3ResourceConnector method openResource.

public ExternalResourceReadResponse openResource(URI location, boolean revalidate) {
    LOGGER.debug("Attempting to get resource: {}", location);
    S3Object s3Object = s3Client.getResource(location);
    if (s3Object == null) {
        return null;
    }
    return new S3Resource(s3Object, location);
}