Search in sources :

Example 1 with DataflowPipelineTranslator

use of org.apache.beam.runners.dataflow.DataflowPipelineTranslator in project beam by apache.

the class WorkerCustomSourcesTest method translateIOToCloudSource.

static com.google.api.services.dataflow.model.Source translateIOToCloudSource(BoundedSource<?> io, DataflowPipelineOptions options) throws Exception {
    DataflowPipelineTranslator translator = DataflowPipelineTranslator.fromOptions(options);
    Pipeline p = Pipeline.create(options);
    p.begin().apply(Read.from(io));
    // Note that we specifically perform this replacement since this is what the DataflowRunner
    // does and the DataflowRunner class does not expose a way to perform these replacements
    // without running the pipeline.
    p.replaceAll(Collections.singletonList(SplittableParDo.PRIMITIVE_BOUNDED_READ_OVERRIDE));
    DataflowRunner runner = DataflowRunner.fromOptions(options);
    SdkComponents sdkComponents = SdkComponents.create();
    RunnerApi.Environment defaultEnvironmentForDataflow = Environments.createDockerEnvironment("dummy-image-url");
    sdkComponents.registerEnvironment(defaultEnvironmentForDataflow);
    RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p, sdkComponents, true);
    Job workflow = translator.translate(p, pipelineProto, sdkComponents, runner, new ArrayList<DataflowPackage>()).getJob();
    Step step = workflow.getSteps().get(0);
    return stepToCloudSource(step);
}
Also used : RunnerApi(org.apache.beam.model.pipeline.v1.RunnerApi) ArrayList(java.util.ArrayList) DataflowRunner(org.apache.beam.runners.dataflow.DataflowRunner) Step(com.google.api.services.dataflow.model.Step) DataflowPipelineTranslator(org.apache.beam.runners.dataflow.DataflowPipelineTranslator) SdkComponents(org.apache.beam.runners.core.construction.SdkComponents) Job(com.google.api.services.dataflow.model.Job) Pipeline(org.apache.beam.sdk.Pipeline)

Aggregations

Job (com.google.api.services.dataflow.model.Job)1 Step (com.google.api.services.dataflow.model.Step)1 ArrayList (java.util.ArrayList)1 RunnerApi (org.apache.beam.model.pipeline.v1.RunnerApi)1 SdkComponents (org.apache.beam.runners.core.construction.SdkComponents)1 DataflowPipelineTranslator (org.apache.beam.runners.dataflow.DataflowPipelineTranslator)1 DataflowRunner (org.apache.beam.runners.dataflow.DataflowRunner)1 Pipeline (org.apache.beam.sdk.Pipeline)1