Examples with S3AFileSystem com.talend.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem used on opensource projects

Example 1 with S3AFileSystem

use of com.talend.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem in project components by Talend.

the class S3Connection method createFileSystem.

public static S3AFileSystem createFileSystem(S3DatasetProperties properties) throws IOException {
    Configuration config = new Configuration(true);
    ExtraHadoopConfiguration extraConfig = new ExtraHadoopConfiguration();
    S3Connection.setS3Configuration(extraConfig, properties);
    extraConfig.addTo(config);
    try {
        return (S3AFileSystem) FileSystem.get(new URI(Constants.FS_S3A + "://" + properties.bucket.getValue()), config);
    } catch (URISyntaxException e) {
        // The URI is constant, so this exception should never occur.
        throw new RuntimeException(e);
    }
}

Example 2 with S3AFileSystem

use of com.talend.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem in project components by Talend.

the class S3TestResource method getObjectMetadata.

/**
 * Get the ObjectMetadata from S3 for the first file found on the path specified by the S3DatasetProperties.
 */
public ObjectMetadata getObjectMetadata(S3DatasetProperties datasetProps) throws IOException {
    S3AFileSystem fs = S3Connection.createFileSystem(datasetProps);
    // The current path is a directory, so get a file to check the encryption.
    Path path = new Path(S3Connection.getUriPath(datasetProps));
    FileStatus[] files = fs.listStatus(path);
    assertThat(files, arrayWithSize(greaterThan(0)));
    return fs.getObjectMetadata(files[0].getPath());
}

Example 3 with S3AFileSystem

use of com.talend.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem in project components by Talend.

the class S3TestResource method after.

/**
 * Remove any resources created on the S3 bucket.
 */
@Override
protected void after() {
    try {
        S3DatasetProperties datasetProps = createS3DatasetProperties();
        S3AFileSystem fs = S3Connection.createFileSystem(datasetProps);
        Path path = new Path(S3Connection.getUriPath(datasetProps));
        if (fs.exists(path)) {
            fs.delete(path, true);
        }
    } catch (IOException ioe) {
        throw new RuntimeException(ioe);
    }
}