Examples with Counter org.deeplearning4j.berkeley.Counter used on opensource projects

Example 11 with Counter

use of org.deeplearning4j.berkeley.Counter in project deeplearning4j by deeplearning4j.

the class CountCumSum method cumSumWithinPartition.

// Do cum sum within the partition
public void cumSumWithinPartition() {
    // Accumulator to get the max of the cumulative sum in each partition
    final Accumulator<Counter<Integer>> maxPerPartitionAcc = sc.accumulator(new Counter<Integer>(), new MaxPerPartitionAccumulator());
    // Partition mapping to fold within partition
    foldWithinPartitionRDD = sentenceCountRDD.mapPartitionsWithIndex(new FoldWithinPartitionFunction(maxPerPartitionAcc), true).cache();
    actionForMapPartition(foldWithinPartitionRDD);
    // Broadcast the counter (partition index : sum of count) to all workers
    broadcastedMaxPerPartitionCounter = sc.broadcast(maxPerPartitionAcc.value());
}

Example 12 with Counter

use of org.deeplearning4j.berkeley.Counter in project deeplearning4j by deeplearning4j.

the class UpdateWordFreqAccumulatorFunction method call.

// Function to add to word freq counter and total count of words
@Override
public Pair<List<String>, AtomicLong> call(List<String> lstOfWords) throws Exception {
    List<String> stops = stopWords.getValue();
    Counter<String> counter = new Counter<>();
    for (String w : lstOfWords) {
        if (w.isEmpty())
            continue;
        if (!stops.isEmpty()) {
            if (stops.contains(w)) {
                counter.incrementCount("STOP", 1.0);
            } else {
                counter.incrementCount(w, 1.0);
            }
        } else {
            counter.incrementCount(w, 1.0);
        }
    }
    wordFreqAcc.add(counter);
    AtomicLong lstOfWordsSize = new AtomicLong(lstOfWords.size());
    return new Pair<>(lstOfWords, lstOfWordsSize);
}