Examples with WordFreqAccumulator org.deeplearning4j.spark.text.accumulators.WordFreqAccumulator used on opensource projects

Example 1 with WordFreqAccumulator

use of org.deeplearning4j.spark.text.accumulators.WordFreqAccumulator in project deeplearning4j by deeplearning4j.

the class TextPipeline method setup.

private void setup() {
    // Set up accumulators and broadcast stopwords
    this.sc = new JavaSparkContext(corpusRDD.context());
    this.wordFreqAcc = sc.accumulator(new Counter<String>(), new WordFreqAccumulator());
    this.stopWordBroadCast = sc.broadcast(stopWords);
}