Search in sources :

Example 1 with FlinkExecutableStagePruningFunction

use of org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction in project beam by apache.

the class FlinkBatchPortablePipelineTranslator method pruneOutput.

private static void pruneOutput(DataSet<RawUnionValue> taggedDataset, BatchTranslationContext context, int unionTag, Coder<WindowedValue<?>> outputCoder, String collectionId) {
    TypeInformation<WindowedValue<?>> outputType = new CoderTypeInformation<>(outputCoder, context.getPipelineOptions());
    FlinkExecutableStagePruningFunction pruningFunction = new FlinkExecutableStagePruningFunction(unionTag, context.getPipelineOptions());
    FlatMapOperator<RawUnionValue, WindowedValue<?>> pruningOperator = new FlatMapOperator<>(taggedDataset, outputType, pruningFunction, String.format("ExtractOutput[%s]", unionTag));
    context.addDataSet(collectionId, pruningOperator);
}
Also used : CoderTypeInformation(org.apache.beam.runners.flink.translation.types.CoderTypeInformation) FlatMapOperator(org.apache.flink.api.java.operators.FlatMapOperator) RawUnionValue(org.apache.beam.sdk.transforms.join.RawUnionValue) WindowedValue(org.apache.beam.sdk.util.WindowedValue) FlinkExecutableStagePruningFunction(org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction)

Aggregations

FlinkExecutableStagePruningFunction (org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction)1 CoderTypeInformation (org.apache.beam.runners.flink.translation.types.CoderTypeInformation)1 RawUnionValue (org.apache.beam.sdk.transforms.join.RawUnionValue)1 WindowedValue (org.apache.beam.sdk.util.WindowedValue)1 FlatMapOperator (org.apache.flink.api.java.operators.FlatMapOperator)1