Examples with SparkStructuredStreamingPipelineOptions org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions used on opensource projects

Example 1 with SparkStructuredStreamingPipelineOptions

use of org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions in project beam by apache.

the class SparkMetricsSinkTest method beforeClass.

@BeforeClass
public static void beforeClass() {
    SparkStructuredStreamingPipelineOptions options = PipelineOptionsFactory.create().as(SparkStructuredStreamingPipelineOptions.class);
    options.setRunner(SparkStructuredStreamingRunner.class);
    options.setTestMode(true);
    pipeline = Pipeline.create(options);
}

Example 2 with SparkStructuredStreamingPipelineOptions

use of org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions in project beam by apache.

the class ParDoTest method beforeClass.

@BeforeClass
public static void beforeClass() {
    SparkStructuredStreamingPipelineOptions options = PipelineOptionsFactory.create().as(SparkStructuredStreamingPipelineOptions.class);
    options.setRunner(SparkStructuredStreamingRunner.class);
    options.setTestMode(true);
    pipeline = Pipeline.create(options);
}

Example 3 with SparkStructuredStreamingPipelineOptions

use of org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions in project beam by apache.

the class SimpleSourceTest method beforeClass.

@BeforeClass
public static void beforeClass() {
    SparkStructuredStreamingPipelineOptions options = PipelineOptionsFactory.create().as(SparkStructuredStreamingPipelineOptions.class);
    options.setRunner(SparkStructuredStreamingRunner.class);
    options.setTestMode(true);
    pipeline = Pipeline.create(options);
}

Example 4 with SparkStructuredStreamingPipelineOptions

use of org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions in project beam by apache.

the class AbstractTranslationContext method startPipeline.

// --------------------------------------------------------------------------------------------
// Pipeline methods
// --------------------------------------------------------------------------------------------
/**
 * Starts the pipeline.
 */
public void startPipeline() {
    SparkStructuredStreamingPipelineOptions options = serializablePipelineOptions.get().as(SparkStructuredStreamingPipelineOptions.class);
    int datasetIndex = 0;
    for (Dataset<?> dataset : leaves) {
        if (options.isStreaming()) {
            // TODO: deal with Beam Discarding, Accumulating and Accumulating & Retracting	outputmodes
            // with DatastreamWriter.outputMode
            DataStreamWriter<?> dataStreamWriter = dataset.writeStream();
            // spark sets a default checkpoint dir if not set.
            if (options.getCheckpointDir() != null) {
                dataStreamWriter = dataStreamWriter.option("checkpointLocation", options.getCheckpointDir());
            }
            launchStreaming(dataStreamWriter.foreach(new NoOpForeachWriter<>()));
        } else {
            if (options.getTestMode()) {
                LOG.debug("**** dataset {} catalyst execution plans ****", ++datasetIndex);
                dataset.explain(true);
            }
            // apply a dummy fn just to apply foreach action that will trigger the pipeline run in
            // spark
            dataset.foreach((ForeachFunction) t -> {
            });
        }
    }
}

Example 5 with SparkStructuredStreamingPipelineOptions

use of org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions in project beam by apache.

the class ComplexSourceTest method beforeClass.

@BeforeClass
public static void beforeClass() throws IOException {
    SparkStructuredStreamingPipelineOptions options = PipelineOptionsFactory.create().as(SparkStructuredStreamingPipelineOptions.class);
    options.setRunner(SparkStructuredStreamingRunner.class);
    options.setTestMode(true);
    pipeline = Pipeline.create(options);
    file = createFile(lines);
}