Search in sources :

Example 61 with MockEmitter

use of io.cdap.cdap.etl.mock.common.MockEmitter in project hydrator-plugins by cdapio.

the class XMLParserTest method testIgnoreError.

@Test
public void testIgnoreError() throws Exception {
    XMLParser.Config config = new XMLParser.Config("body", "ISO-8859-1", "title:/book/title,author:/book/author," + "year:/book/year", "title:string,author:string,year:int", "Ignore error and continue");
    Transform<StructuredRecord, StructuredRecord> transform = new XMLParser(config);
    transform.initialize(new MockTransformContext());
    List<StructuredRecord> input = ImmutableList.of(StructuredRecord.builder(INPUT).set("offset", 1).set("body", "<book category=\"COOKING\"><title lang=\"en\">Everyday Italian</title>" + "<author>Giada De Laurentiis</author><year>null</year><price>30.00</price></book>").build(), StructuredRecord.builder(INPUT).set("offset", 2).set("body", "<book category=\"children\"><title lang=\"en\">Harry Potter</title>" + "<author>J K. Rowling</author><year>2005</year><price>49.99</price></book>").build());
    MockEmitter<StructuredRecord> emitter = new MockEmitter<>();
    for (StructuredRecord record : input) {
        transform.transform(record, emitter);
    }
    Assert.assertEquals(1, emitter.getEmitted().size());
    Assert.assertEquals(0, emitter.getErrors().size());
    StructuredRecord record = emitter.getEmitted().get(0);
    Assert.assertEquals("Harry Potter", record.get("title"));
    Assert.assertEquals("J K. Rowling", record.get("author"));
    Assert.assertEquals(2005, record.<Integer>get("year").intValue());
}
Also used : MockTransformContext(io.cdap.cdap.etl.mock.transform.MockTransformContext) MockEmitter(io.cdap.cdap.etl.mock.common.MockEmitter) StructuredRecord(io.cdap.cdap.api.data.format.StructuredRecord) Test(org.junit.Test)

Example 62 with MockEmitter

use of io.cdap.cdap.etl.mock.common.MockEmitter in project hydrator-plugins by cdapio.

the class XMLParserTest method testEmitErrors.

@Test
public void testEmitErrors() throws Exception {
    XMLParser.Config config = new XMLParser.Config("body", "UTF-8", "title:/book/title,author:/book/author," + "year:/book/year", "title:string,author:int,year:int", "Write to error dataset");
    Transform<StructuredRecord, StructuredRecord> transform = new XMLParser(config);
    transform.initialize(new MockTransformContext());
    StructuredRecord inputRecord = StructuredRecord.builder(INPUT).set("offset", 1).set("body", "<book category=\"COOKING\"><title lang=\"en\">Everyday Italian</title>" + "<author>Giada De Laurentiis</author><year>2005</year><price>30.00</price></book>").build();
    MockEmitter<StructuredRecord> emitter = new MockEmitter<>();
    transform.transform(inputRecord, emitter);
    Assert.assertEquals(0, emitter.getEmitted().size());
    Assert.assertEquals(1, emitter.getErrors().size());
    InvalidEntry<StructuredRecord> invalidEntry = emitter.getErrors().get(0);
    Assert.assertEquals(31, invalidEntry.getErrorCode());
    Assert.assertEquals("offset", 1, invalidEntry.getInvalidRecord().<Integer>get("offset").intValue());
    Assert.assertEquals("body", "<book category=\"COOKING\"><title lang=\"en\">Everyday Italian</title><author>Giada " + "De Laurentiis</author><year>2005</year><price>30.00</price>" + "</book>", invalidEntry.getInvalidRecord().get("body"));
}
Also used : MockTransformContext(io.cdap.cdap.etl.mock.transform.MockTransformContext) MockEmitter(io.cdap.cdap.etl.mock.common.MockEmitter) StructuredRecord(io.cdap.cdap.api.data.format.StructuredRecord) Test(org.junit.Test)

Aggregations

StructuredRecord (io.cdap.cdap.api.data.format.StructuredRecord)62 MockEmitter (io.cdap.cdap.etl.mock.common.MockEmitter)62 Test (org.junit.Test)61 MockTransformContext (io.cdap.cdap.etl.mock.transform.MockTransformContext)52 TransformContext (io.cdap.cdap.etl.api.TransformContext)30 Schema (io.cdap.cdap.api.data.schema.Schema)25 LookupConfig (io.cdap.cdap.etl.api.LookupConfig)7 LookupTableConfig (io.cdap.cdap.etl.api.LookupTableConfig)7 MockPipelineConfigurer (io.cdap.cdap.etl.mock.common.MockPipelineConfigurer)4 Base32 (org.apache.commons.codec.binary.Base32)4 Base64 (org.apache.commons.codec.binary.Base64)4 KeyValue (io.cdap.cdap.api.dataset.lib.KeyValue)3 Hex (org.apache.commons.codec.binary.Hex)2 ValidationException (io.cdap.cdap.etl.api.validation.ValidationException)1 Cause (io.cdap.cdap.etl.api.validation.ValidationFailure.Cause)1 MockLookupProvider (io.cdap.cdap.etl.mock.common.MockLookupProvider)1 HashMap (java.util.HashMap)1