Example 16 with JavaStreamingContext
use of org.apache.spark.streaming.api.java.JavaStreamingContext in project spark-dataflow by cloudera.
the class SparkPipelineRunner method createStreamingEvaluationContext.
private EvaluationContext createStreamingEvaluationContext(JavaSparkContext jsc, Pipeline pipeline, Duration batchDuration) {
SparkStreamingPipelineOptions streamingOptions = (SparkStreamingPipelineOptions) mOptions;
JavaStreamingContext jssc = new JavaStreamingContext(jsc, batchDuration);
return new StreamingEvaluationContext(jsc, pipeline, jssc, streamingOptions.getTimeout());
}
Also used :
JavaStreamingContext(org.apache.spark.streaming.api.java.JavaStreamingContext)
SparkStreamingPipelineOptions(com.cloudera.dataflow.spark.streaming.SparkStreamingPipelineOptions)
StreamingEvaluationContext(com.cloudera.dataflow.spark.streaming.StreamingEvaluationContext)
Aggregations
JavaStreamingContext (org.apache.spark.streaming.api.java.JavaStreamingContext)16
JavaSparkContext (org.apache.spark.api.java.JavaSparkContext)8
Duration (org.apache.spark.streaming.Duration)5
EvaluationContext (org.apache.beam.runners.spark.translation.EvaluationContext)4
SparkConf (org.apache.spark.SparkConf)4
SparkPipelineTranslator (org.apache.beam.runners.spark.translation.SparkPipelineTranslator)3
TransformTranslator (org.apache.beam.runners.spark.translation.TransformTranslator)3
JavaRDD (org.apache.spark.api.java.JavaRDD)3
TransformEvaluator (org.apache.beam.runners.spark.translation.TransformEvaluator)2
Pipeline (org.apache.beam.sdk.Pipeline)2
Test (org.junit.Test)2
TxRunnable (co.cask.cdap.api.TxRunnable)1
DatasetContext (co.cask.cdap.api.data.DatasetContext)1
StructuredRecord (co.cask.cdap.api.data.format.StructuredRecord)1
Schema (co.cask.cdap.api.data.schema.Schema)1
FileSet (co.cask.cdap.api.dataset.lib.FileSet)1
PipelinePhase (co.cask.cdap.etl.common.PipelinePhase)1
PipelinePluginContext (co.cask.cdap.etl.common.plugin.PipelinePluginContext)1
StageSpec (co.cask.cdap.etl.spec.StageSpec)1
KafkaIO (com.cloudera.dataflow.io.KafkaIO)1
