Examples with SparkContextOptions org.apache.beam.runners.spark.SparkContextOptions used on opensource projects

Example 6 with SparkContextOptions

use of org.apache.beam.runners.spark.SparkContextOptions in project components by Talend.

the class PubSubOutputRuntimeTestIT method createSparkRunnerPipeline.

// TODO extract this to utils
private Pipeline createSparkRunnerPipeline() {
    JavaSparkContext jsc = new JavaSparkContext("local[2]", this.getClass().getName());
    PipelineOptions o = PipelineOptionsFactory.create();
    SparkContextOptions options = o.as(SparkContextOptions.class);
    options.setProvidedSparkContext(jsc);
    options.setUsesProvidedSparkContext(true);
    options.setRunner(SparkRunner.class);
    runtimeContainer = new BeamJobRuntimeContainer(options);
    return Pipeline.create(options);
}