Search in sources :

Example 11 with RecordSet

use of org.talend.components.test.RecordSet in project components by Talend.

the class SimpleFileIODatasetRuntimeTest method testGetSampleAvro.

@Test
public void testGetSampleAvro() throws Exception {
    RecordSet rs = getSimpleTestData(0);
    writeRandomAvroFile(mini.getFs(), "/user/test/input.avro", rs);
    String fileSpec = mini.getFs().getUri().resolve("/user/test/input.avro").toString();
    // Configure the component.
    SimpleFileIODatasetProperties props = createDatasetProperties();
    props.format.setValue(SimpleFileIOFormat.AVRO);
    props.path.setValue(fileSpec);
    // Create the runtime.
    SimpleFileIODatasetRuntime runtime = new SimpleFileIODatasetRuntime();
    runtime.initialize(null, props);
    // Attempt to get a sample using the runtime methods.
    final List<IndexedRecord> actual = new ArrayList<>();
    runtime.getSample(100, new Consumer<IndexedRecord>() {

        @Override
        public void accept(IndexedRecord ir) {
            actual.add(ir);
        }
    });
    // Check the expected values.
    assertThat(actual, (Matcher) equalTo(rs.getAllData()));
}
Also used : IndexedRecord(org.apache.avro.generic.IndexedRecord) ArrayList(java.util.ArrayList) RecordSet(org.talend.components.test.RecordSet) SimpleFileIODatasetProperties(org.talend.components.simplefileio.SimpleFileIODatasetProperties) Test(org.junit.Test)

Aggregations

IndexedRecord (org.apache.avro.generic.IndexedRecord)11 RecordSet (org.talend.components.test.RecordSet)11 Test (org.junit.Test)9 ConvertToIndexedRecord (org.talend.components.adapter.beam.transform.ConvertToIndexedRecord)6 SimpleFileIOInputProperties (org.talend.components.simplefileio.input.SimpleFileIOInputProperties)6 SimpleFileIOOutputProperties (org.talend.components.simplefileio.output.SimpleFileIOOutputProperties)4 ArrayList (java.util.ArrayList)3 SimpleFileIODatasetProperties (org.talend.components.simplefileio.SimpleFileIODatasetProperties)3 Schema (org.apache.avro.Schema)2 Pipeline (org.apache.beam.sdk.Pipeline)2 S3InputProperties (org.talend.components.simplefileio.s3.input.S3InputProperties)2 S3OutputProperties (org.talend.components.simplefileio.s3.output.S3OutputProperties)2 List (java.util.List)1 Ignore (org.junit.Ignore)1 Consumer (org.talend.daikon.java8.Consumer)1