Examples with MaxPerPartitionAccumulator org.deeplearning4j.spark.text.accumulators.MaxPerPartitionAccumulator used on opensource projects

Example 1 with MaxPerPartitionAccumulator

use of org.deeplearning4j.spark.text.accumulators.MaxPerPartitionAccumulator in project deeplearning4j by deeplearning4j.

the class CountCumSum method cumSumWithinPartition.

// Do cum sum within the partition
public void cumSumWithinPartition() {
    // Accumulator to get the max of the cumulative sum in each partition
    final Accumulator<Counter<Integer>> maxPerPartitionAcc = sc.accumulator(new Counter<Integer>(), new MaxPerPartitionAccumulator());
    // Partition mapping to fold within partition
    foldWithinPartitionRDD = sentenceCountRDD.mapPartitionsWithIndex(new FoldWithinPartitionFunction(maxPerPartitionAcc), true).cache();
    actionForMapPartition(foldWithinPartitionRDD);
    // Broadcast the counter (partition index : sum of count) to all workers
    broadcastedMaxPerPartitionCounter = sc.broadcast(maxPerPartitionAcc.value());
}