Search in sources :

Example 6 with DirectStepContext

use of org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext in project beam by apache.

the class ParDoEvaluator method create.

public static <InputT, OutputT> ParDoEvaluator<InputT> create(EvaluationContext evaluationContext, PipelineOptions options, DirectStepContext stepContext, AppliedPTransform<?, ?, ?> application, Coder<InputT> inputCoder, WindowingStrategy<?, ? extends BoundedWindow> windowingStrategy, DoFn<InputT, OutputT> fn, StructuralKey<?> key, List<PCollectionView<?>> sideInputs, TupleTag<OutputT> mainOutputTag, List<TupleTag<?>> additionalOutputTags, Map<TupleTag<?>, PCollection<?>> outputs, DoFnSchemaInformation doFnSchemaInformation, Map<String, PCollectionView<?>> sideInputMapping, DoFnRunnerFactory<InputT, OutputT> runnerFactory) {
    BundleOutputManager outputManager = createOutputManager(evaluationContext, key, outputs);
    ReadyCheckingSideInputReader sideInputReader = evaluationContext.createSideInputReader(sideInputs);
    Map<TupleTag<?>, Coder<?>> outputCoders = outputs.entrySet().stream().collect(Collectors.toMap(e -> e.getKey(), e -> e.getValue().getCoder()));
    PushbackSideInputDoFnRunner<InputT, OutputT> runner = runnerFactory.createRunner(options, fn, sideInputs, sideInputReader, outputManager, mainOutputTag, additionalOutputTags, stepContext, inputCoder, outputCoders, windowingStrategy, doFnSchemaInformation, sideInputMapping);
    return create(runner, stepContext, application, outputManager);
}
Also used : StatefulDoFnRunner(org.apache.beam.runners.core.StatefulDoFnRunner) UserCodeException(org.apache.beam.sdk.util.UserCodeException) PushbackSideInputDoFnRunner(org.apache.beam.runners.core.PushbackSideInputDoFnRunner) WindowedValue(org.apache.beam.sdk.util.WindowedValue) KeyedWorkItemCoder(org.apache.beam.runners.core.KeyedWorkItemCoder) DoFnRunner(org.apache.beam.runners.core.DoFnRunner) Coder(org.apache.beam.sdk.coders.Coder) HashMap(java.util.HashMap) DoFnRunners(org.apache.beam.runners.core.DoFnRunners) DoFnSchemaInformation(org.apache.beam.sdk.transforms.DoFnSchemaInformation) DoFnSignatures(org.apache.beam.sdk.transforms.reflect.DoFnSignatures) TupleTag(org.apache.beam.sdk.values.TupleTag) Map(java.util.Map) Preconditions.checkArgument(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument) TimerData(org.apache.beam.runners.core.TimerInternals.TimerData) PipelineOptions(org.apache.beam.sdk.options.PipelineOptions) AppliedPTransform(org.apache.beam.sdk.runners.AppliedPTransform) StructuralKey(org.apache.beam.runners.local.StructuralKey) DoFn(org.apache.beam.sdk.transforms.DoFn) ReadyCheckingSideInputReader(org.apache.beam.runners.core.ReadyCheckingSideInputReader) PCollection(org.apache.beam.sdk.values.PCollection) Collectors(java.util.stream.Collectors) List(java.util.List) DirectStepContext(org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext) SimplePushbackSideInputDoFnRunner(org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner) PCollectionView(org.apache.beam.sdk.values.PCollectionView) BoundedWindow(org.apache.beam.sdk.transforms.windowing.BoundedWindow) OutputManager(org.apache.beam.runners.core.DoFnRunners.OutputManager) ImmutableList(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList) WindowingStrategy(org.apache.beam.sdk.values.WindowingStrategy) KeyedWorkItemCoder(org.apache.beam.runners.core.KeyedWorkItemCoder) Coder(org.apache.beam.sdk.coders.Coder) TupleTag(org.apache.beam.sdk.values.TupleTag) ReadyCheckingSideInputReader(org.apache.beam.runners.core.ReadyCheckingSideInputReader)

Aggregations

DirectStepContext (org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext)6 StructuralKey (org.apache.beam.runners.local.StructuralKey)2 AppliedPTransform (org.apache.beam.sdk.runners.AppliedPTransform)2 PCollection (org.apache.beam.sdk.values.PCollection)2 HashMap (java.util.HashMap)1 List (java.util.List)1 Map (java.util.Map)1 Collectors (java.util.stream.Collectors)1 DoFnRunner (org.apache.beam.runners.core.DoFnRunner)1 DoFnRunners (org.apache.beam.runners.core.DoFnRunners)1 OutputManager (org.apache.beam.runners.core.DoFnRunners.OutputManager)1 KeyedWorkItemCoder (org.apache.beam.runners.core.KeyedWorkItemCoder)1 PushbackSideInputDoFnRunner (org.apache.beam.runners.core.PushbackSideInputDoFnRunner)1 ReadyCheckingSideInputReader (org.apache.beam.runners.core.ReadyCheckingSideInputReader)1 SimplePushbackSideInputDoFnRunner (org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner)1 StatefulDoFnRunner (org.apache.beam.runners.core.StatefulDoFnRunner)1 TimerData (org.apache.beam.runners.core.TimerInternals.TimerData)1 Coder (org.apache.beam.sdk.coders.Coder)1 PipelineOptions (org.apache.beam.sdk.options.PipelineOptions)1 BagState (org.apache.beam.sdk.state.BagState)1