Examples with BeamJobRuntimeContainer org.talend.components.adapter.beam.BeamJobRuntimeContainer used on opensource projects

Example 6 with BeamJobRuntimeContainer

use of org.talend.components.adapter.beam.BeamJobRuntimeContainer in project components by Talend.

the class PubSubOutputRuntimeTestIT method init.

@Before
public void init() {
    datastoreProperties = createDatastore();
    datasetProperties = createDataset(datastoreProperties, topicName);
    runtimeContainer = new BeamJobRuntimeContainer(pipeline.getOptions());
}

Example 7 with BeamJobRuntimeContainer

use of org.talend.components.adapter.beam.BeamJobRuntimeContainer in project components by Talend.

the class PubSubOutputRuntimeTestIT method createSparkRunnerPipeline.

// TODO extract this to utils
private Pipeline createSparkRunnerPipeline() {
    JavaSparkContext jsc = new JavaSparkContext("local[2]", this.getClass().getName());
    PipelineOptions o = PipelineOptionsFactory.create();
    SparkContextOptions options = o.as(SparkContextOptions.class);
    options.setProvidedSparkContext(jsc);
    options.setUsesProvidedSparkContext(true);
    options.setRunner(SparkRunner.class);
    runtimeContainer = new BeamJobRuntimeContainer(options);
    return Pipeline.create(options);
}