Search in sources :

Example 1 with SimpleFileIOOutputRuntime

use of org.talend.components.simplefileio.runtime.SimpleFileIOOutputRuntime in project components by Talend.

the class GSRoundTripRuntimeTestIT method testCsv.

@Test
public void testCsv() {
    List<IndexedRecord> expected = new ArrayList<>();
    expected.add(ConvertToIndexedRecord.convertToAvro(new String[] { "1", "one" }));
    expected.add(ConvertToIndexedRecord.convertToAvro(new String[] { "2", "two" }));
    SimpleFileIOOutputProperties outputProps = createOutputProps();
    outputProps.getDatasetProperties().path.setValue(gsPath);
    SimpleFileIOOutputRuntime outputRuntime = new SimpleFileIOOutputRuntime();
    outputRuntime.initialize(null, outputProps);
    PCollection<IndexedRecord> input = writeP.apply(Create.of(expected));
    input.apply(outputRuntime);
    writeP.run(pipelineOptions).waitUntilFinish();
    SimpleFileIOInputProperties inputProps = createInputProps();
    inputProps.getDatasetProperties().path.setValue(gsPath + "*");
    SimpleFileIOInputRuntime inputRuntime = new SimpleFileIOInputRuntime();
    inputRuntime.initialize(null, inputProps);
    PCollection<IndexedRecord> readRecords = readP.apply(inputRuntime);
    PAssert.that(readRecords).containsInAnyOrder(expected);
    readP.run(pipelineOptions).waitUntilFinish();
}
Also used : SimpleFileIOOutputProperties(org.talend.components.simplefileio.output.SimpleFileIOOutputProperties) SimpleFileIOInputRuntime(org.talend.components.simplefileio.runtime.SimpleFileIOInputRuntime) ConvertToIndexedRecord(org.talend.components.adapter.beam.transform.ConvertToIndexedRecord) IndexedRecord(org.apache.avro.generic.IndexedRecord) ArrayList(java.util.ArrayList) SimpleFileIOInputProperties(org.talend.components.simplefileio.input.SimpleFileIOInputProperties) SimpleFileIOOutputRuntime(org.talend.components.simplefileio.runtime.SimpleFileIOOutputRuntime) Test(org.junit.Test) SimpleFileIODatasetRuntimeTest(org.talend.components.simplefileio.runtime.SimpleFileIODatasetRuntimeTest)

Aggregations

ArrayList (java.util.ArrayList)1 IndexedRecord (org.apache.avro.generic.IndexedRecord)1 Test (org.junit.Test)1 ConvertToIndexedRecord (org.talend.components.adapter.beam.transform.ConvertToIndexedRecord)1 SimpleFileIOInputProperties (org.talend.components.simplefileio.input.SimpleFileIOInputProperties)1 SimpleFileIOOutputProperties (org.talend.components.simplefileio.output.SimpleFileIOOutputProperties)1 SimpleFileIODatasetRuntimeTest (org.talend.components.simplefileio.runtime.SimpleFileIODatasetRuntimeTest)1 SimpleFileIOInputRuntime (org.talend.components.simplefileio.runtime.SimpleFileIOInputRuntime)1 SimpleFileIOOutputRuntime (org.talend.components.simplefileio.runtime.SimpleFileIOOutputRuntime)1