Examples with S3DatasetProperties org.talend.components.simplefileio.s3.S3DatasetProperties used on opensource projects

Example 1 with S3DatasetProperties

use of org.talend.components.simplefileio.s3.S3DatasetProperties in project components by Talend.

the class S3OutputRuntimeTestIT method testParquet_merge.

@Test
public void testParquet_merge() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties();
    datasetProps.format.setValue(SimpleFileIOFormat.PARQUET);
    S3OutputProperties outputProperties = new S3OutputProperties("out");
    outputProperties.init();
    outputProperties.setDatasetProperties(datasetProps);
    outputProperties.mergeOutput.setValue(true);
    // Create the runtime.
    S3OutputRuntime runtime = new S3OutputRuntime();
    runtime.initialize(null, outputProperties);
    // Use the runtime in a Spark pipeline to test.
    final Pipeline p = spark.createPipeline();
    PCollection<IndexedRecord> input = // 
    p.apply(// 
    Create.of(// 
    ConvertToIndexedRecord.convertToAvro(new String[] { "1", "one" }), // 
    ConvertToIndexedRecord.convertToAvro(new String[] { "2", "two" })));
    input.apply(runtime);
    // And run the test.
    p.run().waitUntilFinish();
    FileSystem s3FileSystem = S3Connection.createFileSystem(datasetProps);
    MiniDfsResource.assertReadParquetFile(s3FileSystem, s3.getS3APath(datasetProps), new HashSet<IndexedRecord>(// 
    Arrays.asList(// 
    ConvertToIndexedRecord.convertToAvro(new String[] { "1", "one" }), ConvertToIndexedRecord.convertToAvro(new String[] { "2", "two" }))), false);
    MiniDfsResource.assertFileNumber(s3FileSystem, s3.getS3APath(datasetProps), 1);
}

Example 2 with S3DatasetProperties

use of org.talend.components.simplefileio.s3.S3DatasetProperties in project components by Talend.

the class S3SparkRuntimeTestIT method testAvro_sseAndCseKmsEncryption.

/**
 * Basic Avro test with sseKmsEncryption.
 */
@Ignore("cse not yet supported.")
@Test
public void testAvro_sseAndCseKmsEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties(true, true);
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
}

Example 3 with S3DatasetProperties

use of org.talend.components.simplefileio.s3.S3DatasetProperties in project components by Talend.

the class S3SparkRuntimeTestIT method testAvro_noEncryption.

/**
 * Basic Avro test.
 */
@Test
public void testAvro_noEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties();
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
    // Get some object metadata from the results.
    ObjectMetadata md = s3.getObjectMetadata(datasetProps);
    assertThat(md.getSSEAlgorithm(), nullValue());
    assertThat(md.getSSEAwsKmsKeyId(), nullValue());
}

Example 4 with S3DatasetProperties

use of org.talend.components.simplefileio.s3.S3DatasetProperties in project components by Talend.

the class S3SparkRuntimeTestIT method testParquet_noEncryption.

/**
 * Basic Parquet test.
 */
@Test
public void testParquet_noEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties();
    datasetProps.format.setValue(SimpleFileIOFormat.PARQUET);
    test_noEncryption(datasetProps);
}

Example 5 with S3DatasetProperties

use of org.talend.components.simplefileio.s3.S3DatasetProperties in project components by Talend.

the class S3RoundTripRuntimeTestIT method testAvro_sseAndCseKmsEncryption.

/**
 * Basic Avro test with sseKmsEncryption.
 */
@Ignore("cse not yet supported.")
@Test
public void testAvro_sseAndCseKmsEncryption() throws IOException {
    S3DatasetProperties datasetProps = s3.createS3DatasetProperties(true, true);
    datasetProps.format.setValue(SimpleFileIOFormat.AVRO);
    test_noEncryption(datasetProps);
}