Search in sources :

Example 6 with Parameter

use of org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter in project beam by apache.

the class DoFnSignatures method analyzeOnTimerFamilyMethod.

@VisibleForTesting
static DoFnSignature.OnTimerFamilyMethod analyzeOnTimerFamilyMethod(ErrorReporter errors, TypeDescriptor<? extends DoFn<?, ?>> fnClass, Method m, String timerFamilyId, TypeDescriptor<?> inputT, TypeDescriptor<?> outputT, FnAnalysisContext fnContext) {
    errors.checkArgument(void.class.equals(m.getReturnType()), "Must return void");
    Type[] params = m.getGenericParameterTypes();
    MethodAnalysisContext methodContext = MethodAnalysisContext.create();
    boolean requiresStableInput = m.isAnnotationPresent(DoFn.RequiresStableInput.class);
    @Nullable TypeDescriptor<? extends BoundedWindow> windowT = getWindowType(fnClass, m);
    List<DoFnSignature.Parameter> extraParameters = new ArrayList<>();
    ErrorReporter onTimerErrors = errors.forMethod(DoFn.OnTimerFamily.class, m);
    for (int i = 0; i < params.length; ++i) {
        Parameter parameter = analyzeExtraParameter(onTimerErrors, fnContext, methodContext, ParameterDescription.of(m, i, fnClass.resolveType(params[i]), Arrays.asList(m.getParameterAnnotations()[i])), inputT, outputT);
        checkParameterOneOf(errors, parameter, ALLOWED_ON_TIMER_FAMILY_PARAMETERS);
        extraParameters.add(parameter);
    }
    return DoFnSignature.OnTimerFamilyMethod.create(m, timerFamilyId, requiresStableInput, windowT, extraParameters);
}
Also used : Type(java.lang.reflect.Type) ParameterizedType(java.lang.reflect.ParameterizedType) DoFn(org.apache.beam.sdk.transforms.DoFn) ArrayList(java.util.ArrayList) Parameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter) WatermarkEstimatorParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorParameter) BundleFinalizerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.BundleFinalizerParameter) PipelineOptionsParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter) WindowParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter) TimerFamilyParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerFamilyParameter) TypeParameter(org.apache.beam.sdk.values.TypeParameter) RestrictionTrackerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter) TimerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter) WatermarkEstimatorStateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorStateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) StateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter) Nullable(org.checkerframework.checker.nullness.qual.Nullable) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)

Example 7 with Parameter

use of org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter in project beam by apache.

the class DoFnSignatures method analyzeGetInitialRestrictionMethod.

@VisibleForTesting
static DoFnSignature.GetInitialRestrictionMethod analyzeGetInitialRestrictionMethod(ErrorReporter errors, TypeDescriptor<? extends DoFn<?, ?>> fnT, Method m, TypeDescriptor<?> inputT, TypeDescriptor<?> outputT, FnAnalysisContext fnContext) {
    // Method is of the form:
    // @GetInitialRestriction
    // RestrictionT getInitialRestriction(... parameters ...);
    Type[] params = m.getGenericParameterTypes();
    MethodAnalysisContext methodContext = MethodAnalysisContext.create();
    TypeDescriptor<? extends BoundedWindow> windowT = getWindowType(fnT, m);
    for (int i = 0; i < params.length; ++i) {
        Parameter extraParam = analyzeExtraParameter(errors, fnContext, methodContext, ParameterDescription.of(m, i, fnT.resolveType(params[i]), Arrays.asList(m.getParameterAnnotations()[i])), inputT, outputT);
        if (extraParam instanceof SchemaElementParameter) {
            errors.throwIllegalArgument("Schema @%s are not supported for @%s method. Found %s, did you mean to use %s?", format(DoFn.Element.class), format(DoFn.GetInitialRestriction.class), format(((SchemaElementParameter) extraParam).elementT()), format(inputT));
        }
        methodContext.addParameter(extraParam);
    }
    for (Parameter parameter : methodContext.getExtraParameters()) {
        checkParameterOneOf(errors, parameter, ALLOWED_GET_INITIAL_RESTRICTION_PARAMETERS);
    }
    return DoFnSignature.GetInitialRestrictionMethod.create(m, fnT.resolveType(m.getGenericReturnType()), windowT, methodContext.extraParameters);
}
Also used : Type(java.lang.reflect.Type) ParameterizedType(java.lang.reflect.ParameterizedType) Parameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter) WatermarkEstimatorParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorParameter) BundleFinalizerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.BundleFinalizerParameter) PipelineOptionsParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter) WindowParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter) TimerFamilyParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerFamilyParameter) TypeParameter(org.apache.beam.sdk.values.TypeParameter) RestrictionTrackerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter) TimerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter) WatermarkEstimatorStateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorStateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) StateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)

Example 8 with Parameter

use of org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter in project beam by apache.

the class DoFnSignatures method analyzeTruncateRestrictionMethod.

@VisibleForTesting
static DoFnSignature.TruncateRestrictionMethod analyzeTruncateRestrictionMethod(ErrorReporter errors, TypeDescriptor<? extends DoFn<?, ?>> fnT, Method m, TypeDescriptor<?> inputT, TypeDescriptor<?> restrictionT, FnAnalysisContext fnContext) {
    // Method is of the form:
    // @TruncateRestriction
    // TruncateResult<RestrictionT> truncateRestriction(... parameters ...);
    errors.checkArgument(TruncateResult.class.equals(m.getReturnType()), "Must return TruncateResult<Restriction>");
    Type[] params = m.getGenericParameterTypes();
    MethodAnalysisContext methodContext = MethodAnalysisContext.create();
    TypeDescriptor<? extends BoundedWindow> windowT = getWindowType(fnT, m);
    for (int i = 0; i < params.length; ++i) {
        Parameter extraParam = analyzeExtraParameter(errors, fnContext, methodContext, ParameterDescription.of(m, i, fnT.resolveType(params[i]), Arrays.asList(m.getParameterAnnotations()[i])), inputT, restrictionT);
        if (extraParam instanceof SchemaElementParameter) {
            errors.throwIllegalArgument("Schema @%s are not supported for @%s method. Found %s, did you mean to use %s?", format(DoFn.Element.class), format(TruncateRestriction.class), format(((SchemaElementParameter) extraParam).elementT()), format(inputT));
        } else if (extraParam instanceof RestrictionParameter) {
            errors.checkArgument(restrictionT.equals(((RestrictionParameter) extraParam).restrictionT()), "Uses restriction type %s, but @%s method uses restriction type %s", format(((RestrictionParameter) extraParam).restrictionT()), format(DoFn.GetInitialRestriction.class), format(restrictionT));
        }
        methodContext.addParameter(extraParam);
    }
    for (Parameter parameter : methodContext.getExtraParameters()) {
        checkParameterOneOf(errors, parameter, ALLOWED_TRUNCATE_RESTRICTION_PARAMETERS);
    }
    return DoFnSignature.TruncateRestrictionMethod.create(m, windowT, methodContext.getExtraParameters());
}
Also used : RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) Type(java.lang.reflect.Type) ParameterizedType(java.lang.reflect.ParameterizedType) TruncateRestriction(org.apache.beam.sdk.transforms.DoFn.TruncateRestriction) DoFn(org.apache.beam.sdk.transforms.DoFn) Parameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter) WatermarkEstimatorParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorParameter) BundleFinalizerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.BundleFinalizerParameter) PipelineOptionsParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter) WindowParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter) TimerFamilyParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerFamilyParameter) TypeParameter(org.apache.beam.sdk.values.TypeParameter) RestrictionTrackerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter) TimerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter) WatermarkEstimatorStateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorStateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) StateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter) TruncateResult(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.TruncateResult) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)

Example 9 with Parameter

use of org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter in project beam by apache.

the class DoFnSignatures method analyzeNewTrackerMethod.

@VisibleForTesting
static DoFnSignature.NewTrackerMethod analyzeNewTrackerMethod(ErrorReporter errors, TypeDescriptor<? extends DoFn<?, ?>> fnT, Method m, TypeDescriptor<?> inputT, TypeDescriptor<?> outputT, TypeDescriptor<?> restrictionT, FnAnalysisContext fnContext) {
    // Method is of the form:
    // @NewTracker
    // TrackerT newTracker(... parameters ...);
    Type[] params = m.getGenericParameterTypes();
    TypeDescriptor<?> trackerT = fnT.resolveType(m.getGenericReturnType());
    TypeDescriptor<?> expectedTrackerT = restrictionTrackerTypeOf(restrictionT);
    errors.checkArgument(trackerT.isSubtypeOf(expectedTrackerT), "Returns %s, but must return a subtype of %s", format(trackerT), format(expectedTrackerT));
    MethodAnalysisContext methodContext = MethodAnalysisContext.create();
    TypeDescriptor<? extends BoundedWindow> windowT = getWindowType(fnT, m);
    for (int i = 0; i < params.length; ++i) {
        Parameter extraParam = analyzeExtraParameter(errors, fnContext, methodContext, ParameterDescription.of(m, i, fnT.resolveType(params[i]), Arrays.asList(m.getParameterAnnotations()[i])), inputT, outputT);
        if (extraParam instanceof SchemaElementParameter) {
            errors.throwIllegalArgument("Schema @%s are not supported for @%s method. Found %s, did you mean to use %s?", format(DoFn.Element.class), format(DoFn.NewTracker.class), format(((SchemaElementParameter) extraParam).elementT()), format(inputT));
        } else if (extraParam instanceof RestrictionParameter) {
            errors.checkArgument(restrictionT.equals(((RestrictionParameter) extraParam).restrictionT()), "Uses restriction type %s, but @%s method uses restriction type %s", format(((RestrictionParameter) extraParam).restrictionT()), format(DoFn.GetInitialRestriction.class), format(restrictionT));
        }
        methodContext.addParameter(extraParam);
    }
    for (Parameter parameter : methodContext.getExtraParameters()) {
        checkParameterOneOf(errors, parameter, ALLOWED_NEW_TRACKER_PARAMETERS);
    }
    return DoFnSignature.NewTrackerMethod.create(m, fnT.resolveType(m.getGenericReturnType()), windowT, methodContext.getExtraParameters());
}
Also used : RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) Type(java.lang.reflect.Type) ParameterizedType(java.lang.reflect.ParameterizedType) DoFn(org.apache.beam.sdk.transforms.DoFn) Parameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter) WatermarkEstimatorParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorParameter) BundleFinalizerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.BundleFinalizerParameter) PipelineOptionsParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter) WindowParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter) TimerFamilyParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerFamilyParameter) TypeParameter(org.apache.beam.sdk.values.TypeParameter) RestrictionTrackerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter) TimerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter) WatermarkEstimatorStateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorStateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) RestrictionParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter) StateParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter) SchemaElementParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter) VisibleForTesting(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)

Example 10 with Parameter

use of org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter in project beam by apache.

the class ParDoTranslation method toProto.

public static ParDoPayload toProto(ParDo.MultiOutput<?, ?> parDo, SdkComponents components) throws IOException {
    DoFn<?, ?> doFn = parDo.getFn();
    DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
    Map<String, StateDeclaration> states = signature.stateDeclarations();
    Map<String, TimerDeclaration> timers = signature.timerDeclarations();
    List<Parameter> parameters = signature.processElement().extraParameters();
    ParDoPayload.Builder builder = ParDoPayload.newBuilder();
    builder.setDoFn(toProto(parDo.getFn(), parDo.getMainOutputTag()));
    for (PCollectionView<?> sideInput : parDo.getSideInputs()) {
        builder.putSideInputs(sideInput.getTagInternal().getId(), toProto(sideInput));
    }
    for (Parameter parameter : parameters) {
        Optional<RunnerApi.Parameter> protoParameter = toProto(parameter);
        if (protoParameter.isPresent()) {
            builder.addParameters(protoParameter.get());
        }
    }
    for (Map.Entry<String, StateDeclaration> state : states.entrySet()) {
        RunnerApi.StateSpec spec = toProto(getStateSpecOrCrash(state.getValue(), doFn), components);
        builder.putStateSpecs(state.getKey(), spec);
    }
    for (Map.Entry<String, TimerDeclaration> timer : timers.entrySet()) {
        RunnerApi.TimerSpec spec = toProto(getTimerSpecOrCrash(timer.getValue(), doFn));
        builder.putTimerSpecs(timer.getKey(), spec);
    }
    return builder.build();
}
Also used : ParDoPayload(org.apache.beam.sdk.common.runner.v1.RunnerApi.ParDoPayload) ByteString(com.google.protobuf.ByteString) TimerDeclaration(org.apache.beam.sdk.transforms.reflect.DoFnSignature.TimerDeclaration) RunnerApi(org.apache.beam.sdk.common.runner.v1.RunnerApi) Parameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter) WindowParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter) RestrictionTrackerParameter(org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter) Map(java.util.Map) DoFnSignature(org.apache.beam.sdk.transforms.reflect.DoFnSignature) StateDeclaration(org.apache.beam.sdk.transforms.reflect.DoFnSignature.StateDeclaration)

Aggregations

Parameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter)16 WindowParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter)16 BundleFinalizerParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.BundleFinalizerParameter)15 PipelineOptionsParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter)15 RestrictionTrackerParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter)15 SchemaElementParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter)15 StateParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter)15 TimerParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter)15 ParameterizedType (java.lang.reflect.ParameterizedType)14 Type (java.lang.reflect.Type)14 RestrictionParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionParameter)14 TimerFamilyParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerFamilyParameter)14 WatermarkEstimatorParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorParameter)14 WatermarkEstimatorStateParameter (org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WatermarkEstimatorStateParameter)14 TypeParameter (org.apache.beam.sdk.values.TypeParameter)14 VisibleForTesting (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting)14 DoFn (org.apache.beam.sdk.transforms.DoFn)10 ArrayList (java.util.ArrayList)4 Nullable (org.checkerframework.checker.nullness.qual.Nullable)4 Map (java.util.Map)2