Examples with ETLConfig io.cdap.cdap.etl.proto.v2.ETLConfig used on opensource projects

Example 76 with ETLConfig

use of io.cdap.cdap.etl.proto.v2.ETLConfig in project cdap by caskdata.

the class PipelineSpecGeneratorTest method testConnectionOutOfSink.

@Test(expected = IllegalArgumentException.class)
public void testConnectionOutOfSink() throws ValidationException {
    ETLBatchConfig etlConfig = ETLBatchConfig.builder().addStage(new ETLStage("source", MOCK_SOURCE)).addStage(new ETLStage("sink", MOCK_SINK)).addStage(new ETLStage("transform", MOCK_TRANSFORM_A)).addConnection("source", "sink").addConnection("sink", "transform").build();
    specGenerator.generateSpec(etlConfig);
}

Example 77 with ETLConfig

use of io.cdap.cdap.etl.proto.v2.ETLConfig in project cdap by caskdata.

the class PipelineSpecGeneratorTest method testUnreachableStage.

@Test(expected = IllegalArgumentException.class)
public void testUnreachableStage() throws ValidationException {
    ETLBatchConfig etlConfig = ETLBatchConfig.builder().setTimeSchedule("* * * * *").addStage(new ETLStage("source", MOCK_SOURCE)).addStage(new ETLStage("sink", MOCK_SINK)).addStage(new ETLStage("transform", MOCK_TRANSFORM_A)).addConnection("source", "sink").addConnection("transform", "sink").build();
    specGenerator.generateSpec(etlConfig);
}

Example 78 with ETLConfig

use of io.cdap.cdap.etl.proto.v2.ETLConfig in project cdap by caskdata.

the class PipelineSpecGeneratorTest method testTwoConditionsGoingToSameStage.

@Test(expected = IllegalArgumentException.class)
public void testTwoConditionsGoingToSameStage() throws ValidationException {
    // source--condition1-----t1-----condition2------t11------sink1
    // |                      |                     |
    // |                      |-----------t12--------
    // t2---------sink2                   |
    // |
    // |
    // anotherSource--------->condition3----------------------sink3
    ETLBatchConfig etlConfig = ETLBatchConfig.builder().setTimeSchedule("* * * * *").addStage(new ETLStage("source", MOCK_SOURCE)).addStage(new ETLStage("anotherSource", MOCK_SOURCE)).addStage(new ETLStage("condition1", MOCK_CONDITION)).addStage(new ETLStage("condition2", MOCK_CONDITION)).addStage(new ETLStage("condition3", MOCK_CONDITION)).addStage(new ETLStage("t1", MOCK_TRANSFORM_A)).addStage(new ETLStage("t11", MOCK_TRANSFORM_A)).addStage(new ETLStage("t12", MOCK_TRANSFORM_A)).addStage(new ETLStage("t2", MOCK_TRANSFORM_B)).addStage(new ETLStage("sink1", MOCK_SINK)).addStage(new ETLStage("sink2", MOCK_SINK)).addStage(new ETLStage("sink3", MOCK_SINK)).addConnection("source", "condition1").addConnection("condition1", "t1", true).addConnection("t1", "condition2").addConnection("condition2", "t11", false).addConnection("condition2", "t12", true).addConnection("condition1", "t2", false).addConnection("t11", "sink1").addConnection("t12", "sink1").addConnection("t2", "sink2").addConnection("anotherSource", "condition3").addConnection("condition3", "sink3", false).addConnection("condition3", "t12", true).build();
    specGenerator.generateSpec(etlConfig);
}

Example 79 with ETLConfig

use of io.cdap.cdap.etl.proto.v2.ETLConfig in project cdap by caskdata.

the class PipelineSpecGeneratorTest method testSourceConditionWithMultipleInputStepWithinBranch.

@Test
public void testSourceConditionWithMultipleInputStepWithinBranch() {
    // 
    // condition1-----source-----condition2------t1-------sink
    // |                    |
    // |----------t2---------
    // 
    ETLBatchConfig etlConfig = ETLBatchConfig.builder().setTimeSchedule("* * * * *").addStage(new ETLStage("condition1", MOCK_CONDITION)).addStage(new ETLStage("source", MOCK_SOURCE)).addStage(new ETLStage("condition2", MOCK_CONDITION)).addStage(new ETLStage("t1", MOCK_TRANSFORM_A)).addStage(new ETLStage("t2", MOCK_TRANSFORM_A)).addStage(new ETLStage("sink", MOCK_SINK)).addConnection("condition1", "source", true).addConnection("source", "condition2").addConnection("condition2", "t1", true).addConnection("condition2", "t2", false).addConnection("t1", "sink").addConnection("t2", "sink").build();
    specGenerator.generateSpec(etlConfig);
}

Example 80 with ETLConfig

use of io.cdap.cdap.etl.proto.v2.ETLConfig in project cdap by caskdata.

the class PipelineSpecGeneratorTest method testNestedConditionConnectionWithNoBranchInfo.

@Test(expected = IllegalArgumentException.class)
public void testNestedConditionConnectionWithNoBranchInfo() throws ValidationException {
    ETLBatchConfig etlConfig = ETLBatchConfig.builder().setTimeSchedule("* * * * *").addStage(new ETLStage("source", MOCK_SOURCE)).addStage(new ETLStage("condition1", MOCK_CONDITION)).addStage(new ETLStage("t1", MOCK_TRANSFORM_A)).addStage(new ETLStage("condition2", MOCK_CONDITION)).addStage(new ETLStage("sink", MOCK_SINK)).addConnection("source", "condition1").addConnection("condition1", "t1", true).addConnection("t1", "condition2").addConnection("condition2", "sink").build();
    specGenerator.generateSpec(etlConfig);
}