Examples with BasicInputFormatProvider io.cdap.cdap.etl.batch.BasicInputFormatProvider used on opensource projects

Example 1 with BasicInputFormatProvider

use of io.cdap.cdap.etl.batch.BasicInputFormatProvider in project cdap by caskdata.

the class SparkBatchSourceContext method setInput.

@Override
public void setInput(Input input) {
    Input trackableInput = input;
    // Wrap the input provider with tracking counter for metrics collection via MR counter.
    if (trackableInput instanceof Input.InputFormatProviderInput) {
        InputFormatProvider provider = ((Input.InputFormatProviderInput) trackableInput).getInputFormatProvider();
        Map<String, String> conf = new HashMap<>(provider.getInputFormatConfiguration());
        conf.put(TrackingInputFormat.DELEGATE_CLASS_NAME, provider.getInputFormatClassName());
        provider = new BasicInputFormatProvider(TrackingInputFormat.class.getName(), conf);
        trackableInput = Input.of(trackableInput.getName(), provider).alias(trackableInput.getAlias());
    }
    // Limit preview input by wrapping the input
    if (isPreviewEnabled && trackableInput instanceof Input.InputFormatProviderInput) {
        InputFormatProvider inputFormatProvider = ((Input.InputFormatProviderInput) trackableInput).getInputFormatProvider();
        LimitingInputFormatProvider wrapper = new LimitingInputFormatProvider(inputFormatProvider, getMaxPreviewRecords());
        trackableInput = Input.of(trackableInput.getName(), wrapper).alias(trackableInput.getAlias());
    }
    trackableInput = ExternalDatasets.makeTrackable(admin, suffixInput(trackableInput));
    sourceFactory.addInput(getStageName(), trackableInput);
}

Example 2 with BasicInputFormatProvider

use of io.cdap.cdap.etl.batch.BasicInputFormatProvider in project cdap by caskdata.

the class InputFormatProviderTypeAdapter method deserialize.

@Override
public InputFormatProvider deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {
    JsonObject obj = json.getAsJsonObject();
    // if inputFormat is not present, return empty InputFormatProvider
    if (obj.get("inputFormatClass") == null) {
        return new BasicInputFormatProvider();
    }
    String className = obj.get("inputFormatClass").getAsString();
    Map<String, String> conf = context.deserialize(obj.get("inputFormatConfig"), mapType);
    return new BasicInputFormatProvider(className, conf);
}