Examples with DynamicDriverContext io.cdap.cdap.etl.spark.streaming.DynamicDriverContext used on opensource projects

Example 1 with DynamicDriverContext

use of io.cdap.cdap.etl.spark.streaming.DynamicDriverContext in project cdap by caskdata.

the class SparkStreamingPipelineRunner method mergeJoinResults.

@Override
protected SparkCollection<Object> mergeJoinResults(StageSpec stageSpec, FunctionCache.Factory functionCacheFactory, SparkPairCollection<Object, List<JoinElement<Object>>> joinedInputs, StageStatisticsCollector collector) throws Exception {
    DynamicDriverContext dynamicDriverContext = new DynamicDriverContext(stageSpec, sec, collector);
    JavaPairDStream<Object, List<JoinElement<Object>>> pairDStream = joinedInputs.getUnderlying();
    JavaDStream<Object> result = pairDStream.transform(new DynamicJoinMerge<>(dynamicDriverContext, functionCacheFactory.newCache()));
    return new DStreamCollection<>(sec, functionCacheFactory, result);
}

Example 2 with DynamicDriverContext

use of io.cdap.cdap.etl.spark.streaming.DynamicDriverContext in project cdap by caskdata.

the class SparkStreamingPipelineRunner method addJoinKey.

@Override
protected SparkPairCollection<Object, Object> addJoinKey(StageSpec stageSpec, FunctionCache.Factory functionCacheFactory, String inputStageName, SparkCollection<Object> inputCollection, StageStatisticsCollector collector) throws Exception {
    DynamicDriverContext dynamicDriverContext = new DynamicDriverContext(stageSpec, sec, collector);
    JavaDStream<Object> dStream = inputCollection.getUnderlying();
    JavaPairDStream<Object, Object> result = dStream.transformToPair(new DynamicJoinOn<>(dynamicDriverContext, functionCacheFactory.newCache(), inputStageName));
    return new PairDStreamCollection<>(sec, functionCacheFactory, result);
}