Search in sources :

Example 1 with PairDStreamCollection

use of co.cask.cdap.etl.spark.streaming.PairDStreamCollection in project cdap by caskdata.

the class SparkStreamingPipelineRunner method addJoinKey.

@Override
protected SparkPairCollection<Object, Object> addJoinKey(StageInfo stageInfo, String inputStageName, SparkCollection<Object> inputCollection) throws Exception {
    DynamicDriverContext dynamicDriverContext = new DynamicDriverContext(stageInfo, sec);
    JavaDStream<Object> dStream = inputCollection.getUnderlying();
    JavaPairDStream<Object, Object> result = dStream.transformToPair(new DynamicJoinOn<>(dynamicDriverContext, inputStageName));
    return new PairDStreamCollection<>(sec, result);
}
Also used : PairDStreamCollection(co.cask.cdap.etl.spark.streaming.PairDStreamCollection) DynamicDriverContext(co.cask.cdap.etl.spark.streaming.DynamicDriverContext)

Aggregations

DynamicDriverContext (co.cask.cdap.etl.spark.streaming.DynamicDriverContext)1 PairDStreamCollection (co.cask.cdap.etl.spark.streaming.PairDStreamCollection)1