Search in sources :

Example 31 with OneInputTransformation

use of org.apache.flink.streaming.api.transformations.OneInputTransformation in project flink by apache.

the class CommonExecPythonCorrelate method translateToPlanInternal.

@SuppressWarnings("unchecked")
@Override
protected Transformation<RowData> translateToPlanInternal(PlannerBase planner, ExecNodeConfig config) {
    final ExecEdge inputEdge = getInputEdges().get(0);
    final Transformation<RowData> inputTransform = (Transformation<RowData>) inputEdge.translateToPlan(planner);
    final Configuration pythonConfig = CommonPythonUtil.getMergedConfig(planner.getExecEnv(), config.getTableConfig());
    OneInputTransformation<RowData, RowData> transform = createPythonOneInputTransformation(inputTransform, config, pythonConfig);
    if (CommonPythonUtil.isPythonWorkerUsingManagedMemory(pythonConfig)) {
        transform.declareManagedMemoryUseCaseAtSlotScope(ManagedMemoryUseCase.PYTHON);
    }
    return transform;
}
Also used : RowData(org.apache.flink.table.data.RowData) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) Transformation(org.apache.flink.api.dag.Transformation) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) Configuration(org.apache.flink.configuration.Configuration)

Example 32 with OneInputTransformation

use of org.apache.flink.streaming.api.transformations.OneInputTransformation in project flink by apache.

the class BatchExecPythonGroupWindowAggregate method createPythonOneInputTransformation.

private OneInputTransformation<RowData, RowData> createPythonOneInputTransformation(Transformation<RowData> inputTransform, RowType inputRowType, RowType outputRowType, int maxLimitSize, long windowSize, long slideSize, Configuration pythonConfig, ExecNodeConfig config) {
    int[] namePropertyTypeArray = Arrays.stream(namedWindowProperties).mapToInt(p -> {
        WindowProperty property = p.getProperty();
        if (property instanceof WindowStart) {
            return 0;
        }
        if (property instanceof WindowEnd) {
            return 1;
        }
        if (property instanceof RowtimeAttribute) {
            return 2;
        }
        throw new TableException("Unexpected property " + property);
    }).toArray();
    Tuple2<int[], PythonFunctionInfo[]> aggInfos = CommonPythonUtil.extractPythonAggregateFunctionInfosFromAggregateCall(aggCalls);
    int[] pythonUdafInputOffsets = aggInfos.f0;
    PythonFunctionInfo[] pythonFunctionInfos = aggInfos.f1;
    OneInputStreamOperator<RowData, RowData> pythonOperator = getPythonGroupWindowAggregateFunctionOperator(config, pythonConfig, inputRowType, outputRowType, maxLimitSize, windowSize, slideSize, namePropertyTypeArray, pythonUdafInputOffsets, pythonFunctionInfos);
    return ExecNodeUtil.createOneInputTransformation(inputTransform, createTransformationName(config), createTransformationDescription(config), pythonOperator, InternalTypeInfo.of(outputRowType), inputTransform.getParallelism());
}
Also used : Arrays(java.util.Arrays) InputProperty(org.apache.flink.table.planner.plan.nodes.exec.InputProperty) Tuple2(org.apache.flink.api.java.tuple.Tuple2) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) RowType(org.apache.flink.table.types.logical.RowType) Constructor(java.lang.reflect.Constructor) ExecNode(org.apache.flink.table.planner.plan.nodes.exec.ExecNode) ExecNodeUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.ExecNodeUtil) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd) ManagedMemoryUseCase(org.apache.flink.core.memory.ManagedMemoryUseCase) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) Projection(org.apache.flink.table.connector.Projection) ProjectionCodeGenerator(org.apache.flink.table.planner.codegen.ProjectionCodeGenerator) WindowCodeGenerator(org.apache.flink.table.planner.codegen.agg.batch.WindowCodeGenerator) ExecNodeContext(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeContext) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) RowData(org.apache.flink.table.data.RowData) PlannerBase(org.apache.flink.table.planner.delegation.PlannerBase) CommonPythonUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.CommonPythonUtil) SingleTransformationTranslator(org.apache.flink.table.planner.plan.nodes.exec.SingleTransformationTranslator) ExecNodeConfig(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeConfig) Configuration(org.apache.flink.configuration.Configuration) TableException(org.apache.flink.table.api.TableException) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) InvocationTargetException(java.lang.reflect.InvocationTargetException) InternalTypeInfo(org.apache.flink.table.runtime.typeutils.InternalTypeInfo) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) LogicalWindow(org.apache.flink.table.planner.plan.logical.LogicalWindow) AggregateCall(org.apache.calcite.rel.core.AggregateCall) ExecNodeBase(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase) GeneratedProjection(org.apache.flink.table.runtime.generated.GeneratedProjection) Transformation(org.apache.flink.api.dag.Transformation) OneInputStreamOperator(org.apache.flink.streaming.api.operators.OneInputStreamOperator) ExecutionConfigOptions(org.apache.flink.table.api.config.ExecutionConfigOptions) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) Collections(java.util.Collections) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) TableException(org.apache.flink.table.api.TableException) RowData(org.apache.flink.table.data.RowData) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd)

Example 33 with OneInputTransformation

use of org.apache.flink.streaming.api.transformations.OneInputTransformation in project flink by apache.

the class BatchExecPythonGroupWindowAggregate method translateToPlanInternal.

@SuppressWarnings("unchecked")
@Override
protected Transformation<RowData> translateToPlanInternal(PlannerBase planner, ExecNodeConfig config) {
    final ExecEdge inputEdge = getInputEdges().get(0);
    final Transformation<RowData> inputTransform = (Transformation<RowData>) inputEdge.translateToPlan(planner);
    final RowType inputRowType = (RowType) inputEdge.getOutputType();
    final RowType outputRowType = InternalTypeInfo.of(getOutputType()).toRowType();
    final Tuple2<Long, Long> windowSizeAndSlideSize = WindowCodeGenerator.getWindowDef(window);
    final Configuration pythonConfig = CommonPythonUtil.getMergedConfig(planner.getExecEnv(), config.getTableConfig());
    int groupBufferLimitSize = pythonConfig.getInteger(ExecutionConfigOptions.TABLE_EXEC_WINDOW_AGG_BUFFER_SIZE_LIMIT);
    OneInputTransformation<RowData, RowData> transform = createPythonOneInputTransformation(inputTransform, inputRowType, outputRowType, groupBufferLimitSize, windowSizeAndSlideSize.f0, windowSizeAndSlideSize.f1, pythonConfig, config);
    if (CommonPythonUtil.isPythonWorkerUsingManagedMemory(pythonConfig)) {
        transform.declareManagedMemoryUseCaseAtSlotScope(ManagedMemoryUseCase.PYTHON);
    }
    return transform;
}
Also used : RowData(org.apache.flink.table.data.RowData) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) Transformation(org.apache.flink.api.dag.Transformation) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) Configuration(org.apache.flink.configuration.Configuration) RowType(org.apache.flink.table.types.logical.RowType)

Example 34 with OneInputTransformation

use of org.apache.flink.streaming.api.transformations.OneInputTransformation in project flink by apache.

the class AllWindowTranslationTest method testApplyEventTime.

// ------------------------------------------------------------------------
// apply() translation tests
// ------------------------------------------------------------------------
@Test
@SuppressWarnings("rawtypes")
public void testApplyEventTime() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStream<Tuple2<String, Integer>> source = env.fromElements(Tuple2.of("hello", 1), Tuple2.of("hello", 2));
    DataStream<Tuple2<String, Integer>> window1 = source.windowAll(TumblingEventTimeWindows.of(Time.of(1, TimeUnit.SECONDS))).apply(new AllWindowFunction<Tuple2<String, Integer>, Tuple2<String, Integer>, TimeWindow>() {

        private static final long serialVersionUID = 1L;

        @Override
        public void apply(TimeWindow window, Iterable<Tuple2<String, Integer>> values, Collector<Tuple2<String, Integer>> out) throws Exception {
            for (Tuple2<String, Integer> in : values) {
                out.collect(in);
            }
        }
    });
    OneInputTransformation<Tuple2<String, Integer>, Tuple2<String, Integer>> transform = (OneInputTransformation<Tuple2<String, Integer>, Tuple2<String, Integer>>) window1.getTransformation();
    OneInputStreamOperator<Tuple2<String, Integer>, Tuple2<String, Integer>> operator = transform.getOperator();
    Assert.assertTrue(operator instanceof WindowOperator);
    WindowOperator<String, Tuple2<String, Integer>, ?, ?, ?> winOperator = (WindowOperator<String, Tuple2<String, Integer>, ?, ?, ?>) operator;
    Assert.assertTrue(winOperator.getTrigger() instanceof EventTimeTrigger);
    Assert.assertTrue(winOperator.getWindowAssigner() instanceof TumblingEventTimeWindows);
    Assert.assertTrue(winOperator.getStateDescriptor() instanceof ListStateDescriptor);
    processElementAndEnsureOutput(winOperator, winOperator.getKeySelector(), BasicTypeInfo.STRING_TYPE_INFO, new Tuple2<>("hello", 1));
}
Also used : TumblingEventTimeWindows(org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows) ListStateDescriptor(org.apache.flink.api.common.state.ListStateDescriptor) TimeWindow(org.apache.flink.streaming.api.windowing.windows.TimeWindow) ExpectedException(org.junit.rules.ExpectedException) Tuple2(org.apache.flink.api.java.tuple.Tuple2) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) EventTimeTrigger(org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger) Test(org.junit.Test)

Example 35 with OneInputTransformation

use of org.apache.flink.streaming.api.transformations.OneInputTransformation in project flink by apache.

the class AllWindowTranslationTest method testAggregateProcessingTime.

@Test
public void testAggregateProcessingTime() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStream<Tuple2<String, Integer>> source = env.fromElements(Tuple2.of("hello", 1), Tuple2.of("hello", 2));
    DataStream<Tuple2<String, Integer>> window1 = source.windowAll(SlidingProcessingTimeWindows.of(Time.of(1, TimeUnit.SECONDS), Time.of(100, TimeUnit.MILLISECONDS))).aggregate(new DummyAggregationFunction());
    OneInputTransformation<Tuple2<String, Integer>, Tuple2<String, Integer>> transform = (OneInputTransformation<Tuple2<String, Integer>, Tuple2<String, Integer>>) window1.getTransformation();
    OneInputStreamOperator<Tuple2<String, Integer>, Tuple2<String, Integer>> operator = transform.getOperator();
    Assert.assertTrue(operator instanceof WindowOperator);
    WindowOperator<String, Tuple2<String, Integer>, ?, ?, ?> winOperator = (WindowOperator<String, Tuple2<String, Integer>, ?, ?, ?>) operator;
    Assert.assertTrue(winOperator.getTrigger() instanceof ProcessingTimeTrigger);
    Assert.assertTrue(winOperator.getWindowAssigner() instanceof SlidingProcessingTimeWindows);
    Assert.assertTrue(winOperator.getStateDescriptor() instanceof AggregatingStateDescriptor);
    processElementAndEnsureOutput(winOperator, winOperator.getKeySelector(), BasicTypeInfo.STRING_TYPE_INFO, new Tuple2<>("hello", 1));
}
Also used : AggregatingStateDescriptor(org.apache.flink.api.common.state.AggregatingStateDescriptor) ProcessingTimeTrigger(org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger) Tuple2(org.apache.flink.api.java.tuple.Tuple2) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) SlidingProcessingTimeWindows(org.apache.flink.streaming.api.windowing.assigners.SlidingProcessingTimeWindows) Test(org.junit.Test)

Aggregations

OneInputTransformation (org.apache.flink.streaming.api.transformations.OneInputTransformation)125 Test (org.junit.Test)95 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)88 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)76 TimeWindow (org.apache.flink.streaming.api.windowing.windows.TimeWindow)47 EventTimeTrigger (org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger)44 Tuple3 (org.apache.flink.api.java.tuple.Tuple3)43 ListStateDescriptor (org.apache.flink.api.common.state.ListStateDescriptor)34 Transformation (org.apache.flink.api.dag.Transformation)34 TumblingEventTimeWindows (org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows)30 RowData (org.apache.flink.table.data.RowData)26 SlidingEventTimeWindows (org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows)24 ExecEdge (org.apache.flink.table.planner.plan.nodes.exec.ExecEdge)24 ProcessingTimeTrigger (org.apache.flink.streaming.api.windowing.triggers.ProcessingTimeTrigger)21 RowType (org.apache.flink.table.types.logical.RowType)21 RowDataKeySelector (org.apache.flink.table.runtime.keyselector.RowDataKeySelector)19 FoldingStateDescriptor (org.apache.flink.api.common.state.FoldingStateDescriptor)17 ReducingStateDescriptor (org.apache.flink.api.common.state.ReducingStateDescriptor)17 TumblingProcessingTimeWindows (org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows)15 ExpectedException (org.junit.rules.ExpectedException)15