Search in sources :

Example 1 with WindowStart

use of org.apache.flink.table.runtime.groupwindow.WindowStart in project flink by apache.

the class BatchExecPythonGroupWindowAggregate method createPythonOneInputTransformation.

private OneInputTransformation<RowData, RowData> createPythonOneInputTransformation(Transformation<RowData> inputTransform, RowType inputRowType, RowType outputRowType, int maxLimitSize, long windowSize, long slideSize, Configuration pythonConfig, ExecNodeConfig config) {
    int[] namePropertyTypeArray = Arrays.stream(namedWindowProperties).mapToInt(p -> {
        WindowProperty property = p.getProperty();
        if (property instanceof WindowStart) {
            return 0;
        }
        if (property instanceof WindowEnd) {
            return 1;
        }
        if (property instanceof RowtimeAttribute) {
            return 2;
        }
        throw new TableException("Unexpected property " + property);
    }).toArray();
    Tuple2<int[], PythonFunctionInfo[]> aggInfos = CommonPythonUtil.extractPythonAggregateFunctionInfosFromAggregateCall(aggCalls);
    int[] pythonUdafInputOffsets = aggInfos.f0;
    PythonFunctionInfo[] pythonFunctionInfos = aggInfos.f1;
    OneInputStreamOperator<RowData, RowData> pythonOperator = getPythonGroupWindowAggregateFunctionOperator(config, pythonConfig, inputRowType, outputRowType, maxLimitSize, windowSize, slideSize, namePropertyTypeArray, pythonUdafInputOffsets, pythonFunctionInfos);
    return ExecNodeUtil.createOneInputTransformation(inputTransform, createTransformationName(config), createTransformationDescription(config), pythonOperator, InternalTypeInfo.of(outputRowType), inputTransform.getParallelism());
}
Also used : Arrays(java.util.Arrays) InputProperty(org.apache.flink.table.planner.plan.nodes.exec.InputProperty) Tuple2(org.apache.flink.api.java.tuple.Tuple2) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) RowType(org.apache.flink.table.types.logical.RowType) Constructor(java.lang.reflect.Constructor) ExecNode(org.apache.flink.table.planner.plan.nodes.exec.ExecNode) ExecNodeUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.ExecNodeUtil) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd) ManagedMemoryUseCase(org.apache.flink.core.memory.ManagedMemoryUseCase) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) Projection(org.apache.flink.table.connector.Projection) ProjectionCodeGenerator(org.apache.flink.table.planner.codegen.ProjectionCodeGenerator) WindowCodeGenerator(org.apache.flink.table.planner.codegen.agg.batch.WindowCodeGenerator) ExecNodeContext(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeContext) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) RowData(org.apache.flink.table.data.RowData) PlannerBase(org.apache.flink.table.planner.delegation.PlannerBase) CommonPythonUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.CommonPythonUtil) SingleTransformationTranslator(org.apache.flink.table.planner.plan.nodes.exec.SingleTransformationTranslator) ExecNodeConfig(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeConfig) Configuration(org.apache.flink.configuration.Configuration) TableException(org.apache.flink.table.api.TableException) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) InvocationTargetException(java.lang.reflect.InvocationTargetException) InternalTypeInfo(org.apache.flink.table.runtime.typeutils.InternalTypeInfo) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) LogicalWindow(org.apache.flink.table.planner.plan.logical.LogicalWindow) AggregateCall(org.apache.calcite.rel.core.AggregateCall) ExecNodeBase(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase) GeneratedProjection(org.apache.flink.table.runtime.generated.GeneratedProjection) Transformation(org.apache.flink.api.dag.Transformation) OneInputStreamOperator(org.apache.flink.streaming.api.operators.OneInputStreamOperator) ExecutionConfigOptions(org.apache.flink.table.api.config.ExecutionConfigOptions) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) Collections(java.util.Collections) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) TableException(org.apache.flink.table.api.TableException) RowData(org.apache.flink.table.data.RowData) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd)

Example 2 with WindowStart

use of org.apache.flink.table.runtime.groupwindow.WindowStart in project flink by apache.

the class StreamArrowPythonGroupWindowAggregateFunctionOperatorTest method getTestOperator.

@Override
public AbstractArrowPythonAggregateFunctionOperator getTestOperator(Configuration config, PythonFunctionInfo[] pandasAggregateFunctions, RowType inputType, RowType outputType, int[] groupingSet, int[] udafInputOffsets) {
    long size = 10000L;
    long slide = 5000L;
    SlidingWindowAssigner windowAssigner = SlidingWindowAssigner.of(Duration.ofMillis(size), Duration.ofMillis(slide)).withEventTime();
    EventTimeTriggers.AfterEndOfWindow<Window> trigger = EventTimeTriggers.afterEndOfWindow();
    RowType udfInputType = (RowType) Projection.of(udafInputOffsets).project(inputType);
    RowType udfOutputType = (RowType) Projection.range(groupingSet.length, outputType.getFieldCount() - 2).project(outputType);
    return new PassThroughStreamArrowPythonGroupWindowAggregateFunctionOperator(config, pandasAggregateFunctions, inputType, udfInputType, udfOutputType, 3, windowAssigner, trigger, 0, new NamedWindowProperty[] { new NamedWindowProperty("start", new WindowStart(null)), new NamedWindowProperty("end", new WindowEnd(null)) }, UTC_ZONE_ID, ProjectionCodeGenerator.generateProjection(CodeGeneratorContext.apply(new TableConfig()), "UdafInputProjection", inputType, udfInputType, udafInputOffsets));
}
Also used : Window(org.apache.flink.table.runtime.operators.window.Window) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) SlidingWindowAssigner(org.apache.flink.table.runtime.operators.window.assigners.SlidingWindowAssigner) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) RowType(org.apache.flink.table.types.logical.RowType) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd) TableConfig(org.apache.flink.table.api.TableConfig) EventTimeTriggers(org.apache.flink.table.runtime.operators.window.triggers.EventTimeTriggers)

Example 3 with WindowStart

use of org.apache.flink.table.runtime.groupwindow.WindowStart in project flink by apache.

the class PythonStreamGroupWindowAggregateOperatorTest method getTestOperator.

@Override
OneInputStreamOperator getTestOperator(Configuration config) {
    long size = 10000L;
    long slide = 5000L;
    SlidingWindowAssigner windowAssigner = SlidingWindowAssigner.of(Duration.ofMillis(size), Duration.ofMillis(slide)).withEventTime();
    WindowReference windowRef = new WindowReference("w$", new TimestampType(3));
    return new PassThroughPythonStreamGroupWindowAggregateOperator(config, getInputType(), getOutputType(), new PythonAggregateFunctionInfo[] { new PythonAggregateFunctionInfo(PythonScalarFunctionOperatorTestBase.DummyPythonFunction.INSTANCE, new Integer[] { 2 }, -1, false) }, getGrouping(), -1, false, false, 3, windowAssigner, FlinkFnApi.GroupWindow.WindowType.SLIDING_GROUP_WINDOW, true, true, size, slide, 0L, 0L, new NamedWindowProperty[] { new NamedWindowProperty("start", new WindowStart(null)), new NamedWindowProperty("end", new WindowEnd(null)) }, UTC_ZONE_ID);
}
Also used : NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) SlidingWindowAssigner(org.apache.flink.table.runtime.operators.window.assigners.SlidingWindowAssigner) PythonAggregateFunctionInfo(org.apache.flink.table.functions.python.PythonAggregateFunctionInfo) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) TimestampType(org.apache.flink.table.types.logical.TimestampType) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd) WindowReference(org.apache.flink.table.runtime.groupwindow.WindowReference)

Aggregations

NamedWindowProperty (org.apache.flink.table.runtime.groupwindow.NamedWindowProperty)3 WindowEnd (org.apache.flink.table.runtime.groupwindow.WindowEnd)3 WindowStart (org.apache.flink.table.runtime.groupwindow.WindowStart)3 SlidingWindowAssigner (org.apache.flink.table.runtime.operators.window.assigners.SlidingWindowAssigner)2 RowType (org.apache.flink.table.types.logical.RowType)2 Constructor (java.lang.reflect.Constructor)1 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 Arrays (java.util.Arrays)1 Collections (java.util.Collections)1 AggregateCall (org.apache.calcite.rel.core.AggregateCall)1 Transformation (org.apache.flink.api.dag.Transformation)1 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)1 Configuration (org.apache.flink.configuration.Configuration)1 ManagedMemoryUseCase (org.apache.flink.core.memory.ManagedMemoryUseCase)1 OneInputStreamOperator (org.apache.flink.streaming.api.operators.OneInputStreamOperator)1 OneInputTransformation (org.apache.flink.streaming.api.transformations.OneInputTransformation)1 TableConfig (org.apache.flink.table.api.TableConfig)1 TableException (org.apache.flink.table.api.TableException)1 ExecutionConfigOptions (org.apache.flink.table.api.config.ExecutionConfigOptions)1 Projection (org.apache.flink.table.connector.Projection)1