Search in sources :

Example 1 with PCollection

use of com.google.cloud.dataflow.sdk.values.PCollection in project spark-dataflow by cloudera.

the class TransformTranslator method multiDo.

private static <I, O> TransformEvaluator<ParDo.BoundMulti<I, O>> multiDo() {
    return new TransformEvaluator<ParDo.BoundMulti<I, O>>() {

        @Override
        public void evaluate(ParDo.BoundMulti<I, O> transform, EvaluationContext context) {
            TupleTag<O> mainOutputTag = MULTIDO_FG.get("mainOutputTag", transform);
            MultiDoFnFunction<I, O> multifn = new MultiDoFnFunction<>(transform.getFn(), context.getRuntimeContext(), mainOutputTag, getSideInputs(transform.getSideInputs(), context));
            @SuppressWarnings("unchecked") JavaRDDLike<WindowedValue<I>, ?> inRDD = (JavaRDDLike<WindowedValue<I>, ?>) context.getInputRDD(transform);
            JavaPairRDD<TupleTag<?>, WindowedValue<?>> all = inRDD.mapPartitionsToPair(multifn).cache();
            PCollectionTuple pct = context.getOutput(transform);
            for (Map.Entry<TupleTag<?>, PCollection<?>> e : pct.getAll().entrySet()) {
                @SuppressWarnings("unchecked") JavaPairRDD<TupleTag<?>, WindowedValue<?>> filtered = all.filter(new TupleTagFilter(e.getKey()));
                @SuppressWarnings("unchecked") JavaRDD<WindowedValue<Object>> // Object is the best we can do since different outputs can have different tags
                values = (JavaRDD<WindowedValue<Object>>) (JavaRDD<?>) filtered.values();
                context.setRDD(e.getValue(), values);
            }
        }
    };
}
Also used : TupleTag(com.google.cloud.dataflow.sdk.values.TupleTag) TextIO(com.google.cloud.dataflow.sdk.io.TextIO) AvroIO(com.google.cloud.dataflow.sdk.io.AvroIO) HadoopIO(com.cloudera.dataflow.hadoop.HadoopIO) JavaRDD(org.apache.spark.api.java.JavaRDD) JavaRDDLike(org.apache.spark.api.java.JavaRDDLike) PCollection(com.google.cloud.dataflow.sdk.values.PCollection) WindowedValue(com.google.cloud.dataflow.sdk.util.WindowedValue) ParDo(com.google.cloud.dataflow.sdk.transforms.ParDo) PCollectionTuple(com.google.cloud.dataflow.sdk.values.PCollectionTuple) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap)

Aggregations

HadoopIO (com.cloudera.dataflow.hadoop.HadoopIO)1 AvroIO (com.google.cloud.dataflow.sdk.io.AvroIO)1 TextIO (com.google.cloud.dataflow.sdk.io.TextIO)1 ParDo (com.google.cloud.dataflow.sdk.transforms.ParDo)1 WindowedValue (com.google.cloud.dataflow.sdk.util.WindowedValue)1 PCollection (com.google.cloud.dataflow.sdk.values.PCollection)1 PCollectionTuple (com.google.cloud.dataflow.sdk.values.PCollectionTuple)1 TupleTag (com.google.cloud.dataflow.sdk.values.TupleTag)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 Map (java.util.Map)1 JavaRDD (org.apache.spark.api.java.JavaRDD)1 JavaRDDLike (org.apache.spark.api.java.JavaRDDLike)1