Search in sources :

Example 1 with UnionTransformation

use of org.apache.flink.streaming.api.transformations.UnionTransformation in project flink by apache.

the class PythonOperatorChainingOptimizer method replaceInput.

private static void replaceInput(Transformation<?> transformation, Transformation<?> oldInput, Transformation<?> newInput) {
    try {
        if (transformation instanceof OneInputTransformation || transformation instanceof FeedbackTransformation || transformation instanceof SideOutputTransformation || transformation instanceof ReduceTransformation || transformation instanceof SinkTransformation || transformation instanceof LegacySinkTransformation || transformation instanceof TimestampsAndWatermarksTransformation || transformation instanceof PartitionTransformation) {
            final Field inputField = transformation.getClass().getDeclaredField("input");
            inputField.setAccessible(true);
            inputField.set(transformation, newInput);
        } else if (transformation instanceof TwoInputTransformation) {
            final Field inputField;
            if (((TwoInputTransformation<?, ?, ?>) transformation).getInput1() == oldInput) {
                inputField = transformation.getClass().getDeclaredField("input1");
            } else {
                inputField = transformation.getClass().getDeclaredField("input2");
            }
            inputField.setAccessible(true);
            inputField.set(transformation, newInput);
        } else if (transformation instanceof UnionTransformation || transformation instanceof AbstractMultipleInputTransformation) {
            final Field inputsField = transformation.getClass().getDeclaredField("inputs");
            inputsField.setAccessible(true);
            List<Transformation<?>> newInputs = Lists.newArrayList();
            newInputs.addAll(transformation.getInputs());
            newInputs.remove(oldInput);
            newInputs.add(newInput);
            inputsField.set(transformation, newInputs);
        } else if (transformation instanceof AbstractBroadcastStateTransformation) {
            final Field inputField;
            if (((AbstractBroadcastStateTransformation<?, ?, ?>) transformation).getRegularInput() == oldInput) {
                inputField = transformation.getClass().getDeclaredField("regularInput");
            } else {
                inputField = transformation.getClass().getDeclaredField("broadcastInput");
            }
            inputField.setAccessible(true);
            inputField.set(transformation, newInput);
        } else {
            throw new RuntimeException("Unsupported transformation: " + transformation);
        }
    } catch (NoSuchFieldException | IllegalAccessException e) {
        // This should never happen
        throw new RuntimeException(e);
    }
}
Also used : FeedbackTransformation(org.apache.flink.streaming.api.transformations.FeedbackTransformation) ReduceTransformation(org.apache.flink.streaming.api.transformations.ReduceTransformation) TimestampsAndWatermarksTransformation(org.apache.flink.streaming.api.transformations.TimestampsAndWatermarksTransformation) AbstractMultipleInputTransformation(org.apache.flink.streaming.api.transformations.AbstractMultipleInputTransformation) PhysicalTransformation(org.apache.flink.streaming.api.transformations.PhysicalTransformation) SinkTransformation(org.apache.flink.streaming.api.transformations.SinkTransformation) PartitionTransformation(org.apache.flink.streaming.api.transformations.PartitionTransformation) TwoInputTransformation(org.apache.flink.streaming.api.transformations.TwoInputTransformation) UnionTransformation(org.apache.flink.streaming.api.transformations.UnionTransformation) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) SideOutputTransformation(org.apache.flink.streaming.api.transformations.SideOutputTransformation) LegacySinkTransformation(org.apache.flink.streaming.api.transformations.LegacySinkTransformation) Transformation(org.apache.flink.api.dag.Transformation) AbstractBroadcastStateTransformation(org.apache.flink.streaming.api.transformations.AbstractBroadcastStateTransformation) AbstractMultipleInputTransformation(org.apache.flink.streaming.api.transformations.AbstractMultipleInputTransformation) PartitionTransformation(org.apache.flink.streaming.api.transformations.PartitionTransformation) SideOutputTransformation(org.apache.flink.streaming.api.transformations.SideOutputTransformation) Field(java.lang.reflect.Field) TimestampsAndWatermarksTransformation(org.apache.flink.streaming.api.transformations.TimestampsAndWatermarksTransformation) AbstractBroadcastStateTransformation(org.apache.flink.streaming.api.transformations.AbstractBroadcastStateTransformation) LegacySinkTransformation(org.apache.flink.streaming.api.transformations.LegacySinkTransformation) TwoInputTransformation(org.apache.flink.streaming.api.transformations.TwoInputTransformation) UnionTransformation(org.apache.flink.streaming.api.transformations.UnionTransformation) SinkTransformation(org.apache.flink.streaming.api.transformations.SinkTransformation) LegacySinkTransformation(org.apache.flink.streaming.api.transformations.LegacySinkTransformation) ReduceTransformation(org.apache.flink.streaming.api.transformations.ReduceTransformation) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) FeedbackTransformation(org.apache.flink.streaming.api.transformations.FeedbackTransformation)

Example 2 with UnionTransformation

use of org.apache.flink.streaming.api.transformations.UnionTransformation in project flink by apache.

the class StreamExecIntervalJoin method createNegativeWindowSizeJoin.

private Transformation<RowData> createNegativeWindowSizeJoin(JoinSpec joinSpec, Transformation<RowData> leftInputTransform, Transformation<RowData> rightInputTransform, int leftArity, int rightArity, InternalTypeInfo<RowData> returnTypeInfo, ReadableConfig config) {
    // We filter all records instead of adding an empty source to preserve the watermarks.
    FilterAllFlatMapFunction allFilter = new FilterAllFlatMapFunction(returnTypeInfo);
    OuterJoinPaddingUtil paddingUtil = new OuterJoinPaddingUtil(leftArity, rightArity);
    PaddingLeftMapFunction leftPadder = new PaddingLeftMapFunction(paddingUtil, returnTypeInfo);
    PaddingRightMapFunction rightPadder = new PaddingRightMapFunction(paddingUtil, returnTypeInfo);
    int leftParallelism = leftInputTransform.getParallelism();
    int rightParallelism = rightInputTransform.getParallelism();
    OneInputTransformation<RowData, RowData> filterAllLeftStream = new OneInputTransformation<>(leftInputTransform, "FilterLeft", new StreamFlatMap<>(allFilter), returnTypeInfo, leftParallelism);
    filterAllLeftStream.setUid(createTransformationUid(FILTER_LEFT_TRANSFORMATION));
    filterAllLeftStream.setDescription(createFormattedTransformationDescription("filter all left input transformation", config));
    filterAllLeftStream.setName(createFormattedTransformationName(filterAllLeftStream.getDescription(), "FilterLeft", config));
    OneInputTransformation<RowData, RowData> filterAllRightStream = new OneInputTransformation<>(rightInputTransform, "FilterRight", new StreamFlatMap<>(allFilter), returnTypeInfo, rightParallelism);
    filterAllRightStream.setUid(createTransformationUid(FILTER_RIGHT_TRANSFORMATION));
    filterAllRightStream.setDescription(createFormattedTransformationDescription("filter all right input transformation", config));
    filterAllRightStream.setName(createFormattedTransformationName(filterAllRightStream.getDescription(), "FilterRight", config));
    OneInputTransformation<RowData, RowData> padLeftStream = new OneInputTransformation<>(leftInputTransform, "PadLeft", new StreamMap<>(leftPadder), returnTypeInfo, leftParallelism);
    padLeftStream.setUid(createTransformationUid(PAD_LEFT_TRANSFORMATION));
    padLeftStream.setDescription(createFormattedTransformationDescription("pad left input transformation", config));
    padLeftStream.setName(createFormattedTransformationName(padLeftStream.getDescription(), "PadLeft", config));
    OneInputTransformation<RowData, RowData> padRightStream = new OneInputTransformation<>(rightInputTransform, "PadRight", new StreamMap<>(rightPadder), returnTypeInfo, rightParallelism);
    padRightStream.setUid(createTransformationUid(PAD_RIGHT_TRANSFORMATION));
    padRightStream.setDescription(createFormattedTransformationDescription("pad right input transformation", config));
    padRightStream.setName(createFormattedTransformationName(padRightStream.getDescription(), "PadRight", config));
    switch(joinSpec.getJoinType()) {
        case INNER:
            return new UnionTransformation<>(Lists.newArrayList(filterAllLeftStream, filterAllRightStream));
        case LEFT:
            return new UnionTransformation<>(Lists.newArrayList(padLeftStream, filterAllRightStream));
        case RIGHT:
            return new UnionTransformation<>(Lists.newArrayList(filterAllLeftStream, padRightStream));
        case FULL:
            return new UnionTransformation<>(Lists.newArrayList(padLeftStream, padRightStream));
        default:
            throw new TableException("should no reach here");
    }
}
Also used : FilterAllFlatMapFunction(org.apache.flink.table.runtime.operators.join.interval.FilterAllFlatMapFunction) RowData(org.apache.flink.table.data.RowData) TableException(org.apache.flink.table.api.TableException) PaddingLeftMapFunction(org.apache.flink.table.runtime.operators.join.interval.PaddingLeftMapFunction) UnionTransformation(org.apache.flink.streaming.api.transformations.UnionTransformation) PaddingRightMapFunction(org.apache.flink.table.runtime.operators.join.interval.PaddingRightMapFunction) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) OuterJoinPaddingUtil(org.apache.flink.table.runtime.operators.join.OuterJoinPaddingUtil)

Example 3 with UnionTransformation

use of org.apache.flink.streaming.api.transformations.UnionTransformation in project flink by apache.

the class MultipleInputITCase method testNonKeyed.

public void testNonKeyed(boolean withUnion) throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(1);
    TestListResultSink<Long> resultSink = new TestListResultSink<>();
    DataStream<Long> source1 = env.fromElements(1L, 10L);
    DataStream<Long> source2 = env.fromElements(2L, 11L);
    DataStream<String> source3 = env.fromElements("42", "44");
    MultipleInputTransformation<Long> multipleInput = new MultipleInputTransformation<>("My Operator", new SumAllInputOperatorFactory(withUnion ? 2 : 3), BasicTypeInfo.LONG_TYPE_INFO, 1);
    MultipleInputTransformation<Long> multipleInputTransformation;
    if (withUnion) {
        UnionTransformation<Long> union = new UnionTransformation<>(Arrays.asList(source1.getTransformation(), source2.getTransformation()));
        multipleInputTransformation = multipleInput.addInput(union);
    } else {
        multipleInputTransformation = multipleInput.addInput(source1.getTransformation()).addInput(source2.getTransformation());
    }
    env.addOperator(multipleInputTransformation.addInput(source3.getTransformation()));
    new MultipleConnectedStreams(env).transform(multipleInput).addSink(resultSink);
    env.execute();
    List<Long> result = resultSink.getResult();
    Collections.sort(result);
    long actualSum = result.get(result.size() - 1);
    assertEquals(1 + 10 + 2 + 11 + 42 + 44, actualSum);
}
Also used : MultipleConnectedStreams(org.apache.flink.streaming.api.datastream.MultipleConnectedStreams) TestListResultSink(org.apache.flink.test.streaming.runtime.util.TestListResultSink) UnionTransformation(org.apache.flink.streaming.api.transformations.UnionTransformation) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) MultipleInputTransformation(org.apache.flink.streaming.api.transformations.MultipleInputTransformation) KeyedMultipleInputTransformation(org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation)

Aggregations

UnionTransformation (org.apache.flink.streaming.api.transformations.UnionTransformation)3 OneInputTransformation (org.apache.flink.streaming.api.transformations.OneInputTransformation)2 Field (java.lang.reflect.Field)1 Transformation (org.apache.flink.api.dag.Transformation)1 MultipleConnectedStreams (org.apache.flink.streaming.api.datastream.MultipleConnectedStreams)1 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)1 AbstractBroadcastStateTransformation (org.apache.flink.streaming.api.transformations.AbstractBroadcastStateTransformation)1 AbstractMultipleInputTransformation (org.apache.flink.streaming.api.transformations.AbstractMultipleInputTransformation)1 FeedbackTransformation (org.apache.flink.streaming.api.transformations.FeedbackTransformation)1 KeyedMultipleInputTransformation (org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation)1 LegacySinkTransformation (org.apache.flink.streaming.api.transformations.LegacySinkTransformation)1 MultipleInputTransformation (org.apache.flink.streaming.api.transformations.MultipleInputTransformation)1 PartitionTransformation (org.apache.flink.streaming.api.transformations.PartitionTransformation)1 PhysicalTransformation (org.apache.flink.streaming.api.transformations.PhysicalTransformation)1 ReduceTransformation (org.apache.flink.streaming.api.transformations.ReduceTransformation)1 SideOutputTransformation (org.apache.flink.streaming.api.transformations.SideOutputTransformation)1 SinkTransformation (org.apache.flink.streaming.api.transformations.SinkTransformation)1 TimestampsAndWatermarksTransformation (org.apache.flink.streaming.api.transformations.TimestampsAndWatermarksTransformation)1 TwoInputTransformation (org.apache.flink.streaming.api.transformations.TwoInputTransformation)1 TableException (org.apache.flink.table.api.TableException)1