Search in sources :

Example 11 with S3OutputProperties

use of org.talend.components.simplefileio.s3.output.S3OutputProperties in project components by Talend.

the class S3RoundTripRuntimeTestIT method test_noEncryption.

public void test_noEncryption(S3DatasetProperties datasetProps) throws IOException {
    // The file that we will be creating.
    RecordSet rs = getSimpleTestData(0);
    // Configure the components.
    S3OutputProperties outputProps = new S3OutputProperties("out");
    outputProps.setDatasetProperties(datasetProps);
    S3InputProperties inputProps = new S3InputProperties("in");
    inputProps.setDatasetProperties(datasetProps);
    List<IndexedRecord> actual = runRoundTripPipelines(rs.getAllData(), outputProps, inputProps);
    List<IndexedRecord> expected = rs.getAllData();
    assertThat(actual, containsInAnyOrder(expected.toArray()));
    List<IndexedRecord> samples = getSample(datasetProps);
    assertThat(samples, containsInAnyOrder(expected.toArray()));
    Schema schema = getSchema(datasetProps);
    assertEquals(expected.get(0).getSchema(), schema);
}
Also used : IndexedRecord(org.apache.avro.generic.IndexedRecord) S3OutputProperties(org.talend.components.simplefileio.s3.output.S3OutputProperties) Schema(org.apache.avro.Schema) RecordSet(org.talend.components.test.RecordSet) S3InputProperties(org.talend.components.simplefileio.s3.input.S3InputProperties)

Aggregations

S3OutputProperties (org.talend.components.simplefileio.s3.output.S3OutputProperties)9 IndexedRecord (org.apache.avro.generic.IndexedRecord)6 Test (org.junit.Test)6 S3DatasetProperties (org.talend.components.simplefileio.s3.S3DatasetProperties)6 S3DatastoreProperties (org.talend.components.simplefileio.s3.S3DatastoreProperties)4 Schema (org.apache.avro.Schema)3 Pipeline (org.apache.beam.sdk.Pipeline)3 FileSystem (org.apache.hadoop.fs.FileSystem)3 ConvertToIndexedRecord (org.talend.components.adapter.beam.transform.ConvertToIndexedRecord)3 AmazonS3 (com.amazonaws.services.s3.AmazonS3)2 S3InputProperties (org.talend.components.simplefileio.s3.input.S3InputProperties)2 RecordSet (org.talend.components.test.RecordSet)2 ValidationResult (org.talend.daikon.properties.ValidationResult)2 AWSCredentialsProvider (com.amazonaws.auth.AWSCredentialsProvider)1 StaticCredentialsProvider (com.amazonaws.internal.StaticCredentialsProvider)1 Region (com.amazonaws.regions.Region)1 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)1 AmazonS3EncryptionClient (com.amazonaws.services.s3.AmazonS3EncryptionClient)1 CryptoConfiguration (com.amazonaws.services.s3.model.CryptoConfiguration)1 KMSEncryptionMaterialsProvider (com.amazonaws.services.s3.model.KMSEncryptionMaterialsProvider)1