Examples with StageConfigurer io.cdap.cdap.etl.api.StageConfigurer used on opensource projects

Example 6 with StageConfigurer

use of io.cdap.cdap.etl.api.StageConfigurer in project cdap by caskdata.

the class StringCaseTransform method configurePipeline.

// configurePipeline is called only once, when the pipeline is deployed. Static validation should be done here.
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
    StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
    // the output schema is always the same as the input schema
    Schema inputSchema = stageConfigurer.getInputSchema();
    // if schema is null, that means it is either not known until runtime, or it is variable
    if (inputSchema != null) {
        // if the input schema is constant and known at configure time, check that all configured fields are strings
        for (String fieldName : config.getUpperFields()) {
            validateFieldIsString(inputSchema, fieldName);
        }
        for (String fieldName : config.getLowerFields()) {
            validateFieldIsString(inputSchema, fieldName);
        }
    }
    stageConfigurer.setOutputSchema(inputSchema);
}

Example 7 with StageConfigurer

use of io.cdap.cdap.etl.api.StageConfigurer in project cdap by caskdata.

the class DistinctAggregator method configurePipeline.

@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
    StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
    Schema inputSchema = stageConfigurer.getInputSchema();
    this.outputSchema = getOutputSchema(stageConfigurer.getFailureCollector(), inputSchema, conf.getFields());
    stageConfigurer.setOutputSchema(outputSchema);
}

Example 8 with StageConfigurer

use of io.cdap.cdap.etl.api.StageConfigurer in project cdap by caskdata.

the class GroupFilterAggregator method configurePipeline.

@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
    StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
    Schema inputSchema = stageConfigurer.getInputSchema();
    if (inputSchema == null) {
        return;
    }
    Schema.Field groupField = inputSchema.getField(config.field);
    if (groupField == null) {
        throw new IllegalArgumentException(config.field + " is not in the input schema");
    }
    Schema groupSchema = groupField.getSchema();
    Schema.Type groupType = groupSchema.isNullable() ? groupSchema.getNonNullable().getType() : groupSchema.getType();
    if (groupType != Schema.Type.STRING) {
        throw new IllegalArgumentException(config.field + " is not of type string");
    }
    stageConfigurer.setOutputSchema(inputSchema);
}

Example 9 with StageConfigurer

use of io.cdap.cdap.etl.api.StageConfigurer in project cdap by caskdata.

the class IdentityAggregator method configurePipeline.

@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
    StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
    stageConfigurer.setOutputSchema(stageConfigurer.getInputSchema());
}

Example 10 with StageConfigurer

use of io.cdap.cdap.etl.api.StageConfigurer in project cdap by caskdata.

the class AllErrorTransform method configurePipeline.

@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
    StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
    stageConfigurer.setOutputSchema(stageConfigurer.getInputSchema());
}