Search in sources :

Example 6 with Sessions

use of org.apache.beam.sdk.transforms.windowing.Sessions in project beam by apache.

the class FlinkReduceFunction method reduce.

@Override
public void reduce(Iterable<WindowedValue<KV<K, AccumT>>> elements, Collector<WindowedValue<KV<K, OutputT>>> out) throws Exception {
    PipelineOptions options = serializedOptions.get();
    FlinkSideInputReader sideInputReader = new FlinkSideInputReader(sideInputs, getRuntimeContext());
    AbstractFlinkCombineRunner<K, AccumT, AccumT, OutputT, W> reduceRunner;
    if (groupedByWindow) {
        reduceRunner = new SingleWindowFlinkCombineRunner<>();
    } else {
        if (windowingStrategy.needsMerge() && windowingStrategy.getWindowFn() instanceof Sessions) {
            reduceRunner = new SortingFlinkCombineRunner<>();
        } else {
            reduceRunner = new HashingFlinkCombineRunner<>();
        }
    }
    reduceRunner.combine(new AbstractFlinkCombineRunner.FinalFlinkCombiner<>(combineFn), windowingStrategy, sideInputReader, options, elements, out);
}
Also used : Sessions(org.apache.beam.sdk.transforms.windowing.Sessions) SerializablePipelineOptions(org.apache.beam.runners.core.construction.SerializablePipelineOptions) PipelineOptions(org.apache.beam.sdk.options.PipelineOptions)

Aggregations

Sessions (org.apache.beam.sdk.transforms.windowing.Sessions)6 SerializablePipelineOptions (org.apache.beam.runners.core.construction.SerializablePipelineOptions)3 PipelineOptions (org.apache.beam.sdk.options.PipelineOptions)3 FixedWindows (org.apache.beam.sdk.transforms.windowing.FixedWindows)3 HashMap (java.util.HashMap)2 Map (java.util.Map)2 Pipeline (org.apache.beam.sdk.Pipeline)2 PipelineResult (org.apache.beam.sdk.PipelineResult)2 GcpOptions (org.apache.beam.sdk.extensions.gcp.options.GcpOptions)2 PubsubIO (org.apache.beam.sdk.io.gcp.pubsub.PubsubIO)2 Counter (org.apache.beam.sdk.metrics.Counter)2 Metrics (org.apache.beam.sdk.metrics.Metrics)2 Default (org.apache.beam.sdk.options.Default)2 Description (org.apache.beam.sdk.options.Description)2 PipelineOptionsFactory (org.apache.beam.sdk.options.PipelineOptionsFactory)2 Combine (org.apache.beam.sdk.transforms.Combine)2 DoFn (org.apache.beam.sdk.transforms.DoFn)2 MapElements (org.apache.beam.sdk.transforms.MapElements)2 Mean (org.apache.beam.sdk.transforms.Mean)2 PTransform (org.apache.beam.sdk.transforms.PTransform)2