Search in sources :

Example 6 with WireCoderSetting

use of org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting in project beam by apache.

the class ProcessBundleDescriptors method addStageInput.

private static RemoteInputDestination<WindowedValue<?>> addStageInput(ApiServiceDescriptor dataEndpoint, PCollectionNode inputPCollection, Components.Builder components, WireCoderSetting wireCoderSetting) throws IOException {
    String inputWireCoderId = WireCoders.addSdkWireCoder(inputPCollection, components, wireCoderSetting);
    @SuppressWarnings("unchecked") Coder<WindowedValue<?>> wireCoder = (Coder) WireCoders.instantiateRunnerWireCoder(inputPCollection, components.build(), wireCoderSetting);
    RemoteGrpcPort inputPort = RemoteGrpcPort.newBuilder().setApiServiceDescriptor(dataEndpoint).setCoderId(inputWireCoderId).build();
    String inputId = uniqueId(String.format("fn/read/%s", inputPCollection.getId()), components::containsTransforms);
    PTransform inputTransform = RemoteGrpcPortRead.readFromPort(inputPort, inputPCollection.getId()).toPTransform();
    components.putTransforms(inputId, inputTransform);
    return RemoteInputDestination.of(wireCoder, inputId);
}
Also used : Coder(org.apache.beam.sdk.coders.Coder) ByteStringCoder(org.apache.beam.runners.fnexecution.wire.ByteStringCoder) FullWindowedValueCoder(org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder) RemoteGrpcPort(org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort) WindowedValue(org.apache.beam.sdk.util.WindowedValue) PTransform(org.apache.beam.model.pipeline.v1.RunnerApi.PTransform)

Aggregations

ByteStringCoder (org.apache.beam.runners.fnexecution.wire.ByteStringCoder)5 Coder (org.apache.beam.sdk.coders.Coder)5 FullWindowedValueCoder (org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder)5 WireCoderSetting (org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting)4 PTransform (org.apache.beam.model.pipeline.v1.RunnerApi.PTransform)4 LinkedHashMap (java.util.LinkedHashMap)2 RemoteGrpcPort (org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort)2 RunnerApi (org.apache.beam.model.pipeline.v1.RunnerApi)2 Components (org.apache.beam.model.pipeline.v1.RunnerApi.Components)2 PCollectionNode (org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode)2 WindowedValue (org.apache.beam.sdk.util.WindowedValue)2 Collection (java.util.Collection)1 Collections (java.util.Collections)1 List (java.util.List)1 Map (java.util.Map)1 Collectors (java.util.stream.Collectors)1 ProcessBundleDescriptor (org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor)1 Environment (org.apache.beam.model.pipeline.v1.RunnerApi.Environment)1 ExecutableStagePayload (org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload)1 SideInputId (org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId)1