Examples with NamedAggregators com.cloudera.dataflow.spark.aggregators.NamedAggregators used on opensource projects

Example 1 with NamedAggregators

use of com.cloudera.dataflow.spark.aggregators.NamedAggregators in project spark-dataflow by cloudera.

the class SparkRuntimeContext method createAggregator.

/**
   * Creates and aggregator and associates it with the specified name.
   *
   * @param named     Name of aggregator.
   * @param combineFn Combine function used in aggregation.
   * @param <IN>      Type of inputs to aggregator.
   * @param <INTER>   Intermediate data type
   * @param <OUT>     Type of aggregator outputs.
   * @return Specified aggregator
   */
public synchronized <IN, INTER, OUT> Aggregator<IN, OUT> createAggregator(String named, Combine.CombineFn<? super IN, INTER, OUT> combineFn) {
    @SuppressWarnings("unchecked") Aggregator<IN, OUT> aggregator = (Aggregator<IN, OUT>) aggregators.get(named);
    if (aggregator == null) {
        @SuppressWarnings("unchecked") NamedAggregators.CombineFunctionState<IN, INTER, OUT> state = new NamedAggregators.CombineFunctionState<>((Combine.CombineFn<IN, INTER, OUT>) combineFn, (Coder<IN>) getCoder(combineFn), this);
        accum.add(new NamedAggregators(named, state));
        aggregator = new SparkAggregator<>(named, state);
        aggregators.put(named, aggregator);
    }
    return aggregator;
}