Search in sources :

Example 6 with SparkExecutionPluginContext

use of co.cask.cdap.etl.api.batch.SparkExecutionPluginContext in project cdap by caskdata.

the class DynamicSparkCompute method lazyInit.

// when checkpointing is enabled, and Spark is loading DStream operations from an existing checkpoint,
// delegate will be null and the initialize() method won't have been called. So we need to instantiate
// the delegate and initialize it.
private void lazyInit(final JavaSparkContext jsc) throws Exception {
    if (delegate == null) {
        PluginFunctionContext pluginFunctionContext = dynamicDriverContext.getPluginFunctionContext();
        delegate = pluginFunctionContext.createPlugin();
        final StageInfo stageInfo = pluginFunctionContext.getStageInfo();
        final JavaSparkExecutionContext sec = dynamicDriverContext.getSparkExecutionContext();
        Transactionals.execute(sec, new TxRunnable() {

            @Override
            public void run(DatasetContext datasetContext) throws Exception {
                SparkExecutionPluginContext sparkPluginContext = new BasicSparkExecutionPluginContext(sec, jsc, datasetContext, stageInfo);
                delegate.initialize(sparkPluginContext);
            }
        }, Exception.class);
    }
}
Also used : BasicSparkExecutionPluginContext(co.cask.cdap.etl.spark.batch.BasicSparkExecutionPluginContext) PluginFunctionContext(co.cask.cdap.etl.spark.function.PluginFunctionContext) SparkExecutionPluginContext(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext) BasicSparkExecutionPluginContext(co.cask.cdap.etl.spark.batch.BasicSparkExecutionPluginContext) StageInfo(co.cask.cdap.etl.planner.StageInfo) TxRunnable(co.cask.cdap.api.TxRunnable) JavaSparkExecutionContext(co.cask.cdap.api.spark.JavaSparkExecutionContext) DatasetContext(co.cask.cdap.api.data.DatasetContext) TransactionFailureException(org.apache.tephra.TransactionFailureException)

Aggregations

SparkExecutionPluginContext (co.cask.cdap.etl.api.batch.SparkExecutionPluginContext)6 CountingFunction (co.cask.cdap.etl.spark.function.CountingFunction)4 TxRunnable (co.cask.cdap.api.TxRunnable)3 DatasetContext (co.cask.cdap.api.data.DatasetContext)3 BasicSparkExecutionPluginContext (co.cask.cdap.etl.spark.batch.BasicSparkExecutionPluginContext)2 SparkStreamingExecutionContext (co.cask.cdap.etl.spark.streaming.SparkStreamingExecutionContext)2 TransactionFailureException (org.apache.tephra.TransactionFailureException)2 MacroEvaluator (co.cask.cdap.api.macro.MacroEvaluator)1 PluginContext (co.cask.cdap.api.plugin.PluginContext)1 JavaSparkExecutionContext (co.cask.cdap.api.spark.JavaSparkExecutionContext)1 SparkPluginContext (co.cask.cdap.etl.api.batch.SparkPluginContext)1 DefaultMacroEvaluator (co.cask.cdap.etl.common.DefaultMacroEvaluator)1 StageInfo (co.cask.cdap.etl.planner.StageInfo)1 BasicSparkPluginContext (co.cask.cdap.etl.spark.batch.BasicSparkPluginContext)1 PluginFunctionContext (co.cask.cdap.etl.spark.function.PluginFunctionContext)1 SparkPipelinePluginContext (co.cask.cdap.etl.spark.plugin.SparkPipelinePluginContext)1 ComputeTransformFunction (co.cask.cdap.etl.spark.streaming.function.ComputeTransformFunction)1 DynamicSparkCompute (co.cask.cdap.etl.spark.streaming.function.DynamicSparkCompute)1