Search in sources :

Example 1 with WindowProperty

use of org.apache.flink.table.runtime.groupwindow.WindowProperty in project flink by apache.

the class StreamExecGroupWindowAggregate method createAggsHandler.

private GeneratedClass<?> createAggsHandler(AggregateInfoList aggInfoList, ExecNodeConfig config, RelBuilder relBuilder, List<LogicalType> fieldTypes, ZoneId shiftTimeZone) {
    final boolean needMerge;
    final Class<?> windowClass;
    if (window instanceof SlidingGroupWindow) {
        ValueLiteralExpression size = ((SlidingGroupWindow) window).size();
        needMerge = hasTimeIntervalType(size);
        windowClass = hasRowIntervalType(size) ? CountWindow.class : TimeWindow.class;
    } else if (window instanceof TumblingGroupWindow) {
        needMerge = false;
        ValueLiteralExpression size = ((TumblingGroupWindow) window).size();
        windowClass = hasRowIntervalType(size) ? CountWindow.class : TimeWindow.class;
    } else if (window instanceof SessionGroupWindow) {
        needMerge = true;
        windowClass = TimeWindow.class;
    } else {
        throw new TableException("Unsupported window: " + window.toString());
    }
    final AggsHandlerCodeGenerator generator = new AggsHandlerCodeGenerator(new CodeGeneratorContext(config.getTableConfig()), relBuilder, JavaScalaConversionUtil.toScala(fieldTypes), // copyInputField
    false).needAccumulate();
    if (needMerge) {
        generator.needMerge(0, false, null);
    }
    if (needRetraction) {
        generator.needRetract();
    }
    final List<WindowProperty> windowProperties = Arrays.asList(Arrays.stream(namedWindowProperties).map(NamedWindowProperty::getProperty).toArray(WindowProperty[]::new));
    final boolean isTableAggregate = isTableAggregate(Arrays.asList(aggInfoList.getActualAggregateCalls()));
    if (isTableAggregate) {
        return generator.generateNamespaceTableAggsHandler("GroupingWindowTableAggsHandler", aggInfoList, JavaScalaConversionUtil.toScala(windowProperties), windowClass, shiftTimeZone);
    } else {
        return generator.generateNamespaceAggsHandler("GroupingWindowAggsHandler", aggInfoList, JavaScalaConversionUtil.toScala(windowProperties), windowClass, shiftTimeZone);
    }
}
Also used : NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) CountWindow(org.apache.flink.table.runtime.operators.window.CountWindow) TableException(org.apache.flink.table.api.TableException) ValueLiteralExpression(org.apache.flink.table.expressions.ValueLiteralExpression) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) AggsHandlerCodeGenerator(org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator) TimeWindow(org.apache.flink.table.runtime.operators.window.TimeWindow) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) TumblingGroupWindow(org.apache.flink.table.planner.plan.logical.TumblingGroupWindow) SlidingGroupWindow(org.apache.flink.table.planner.plan.logical.SlidingGroupWindow) SessionGroupWindow(org.apache.flink.table.planner.plan.logical.SessionGroupWindow)

Example 2 with WindowProperty

use of org.apache.flink.table.runtime.groupwindow.WindowProperty in project flink by apache.

the class BatchExecPythonGroupWindowAggregate method createPythonOneInputTransformation.

private OneInputTransformation<RowData, RowData> createPythonOneInputTransformation(Transformation<RowData> inputTransform, RowType inputRowType, RowType outputRowType, int maxLimitSize, long windowSize, long slideSize, Configuration pythonConfig, ExecNodeConfig config) {
    int[] namePropertyTypeArray = Arrays.stream(namedWindowProperties).mapToInt(p -> {
        WindowProperty property = p.getProperty();
        if (property instanceof WindowStart) {
            return 0;
        }
        if (property instanceof WindowEnd) {
            return 1;
        }
        if (property instanceof RowtimeAttribute) {
            return 2;
        }
        throw new TableException("Unexpected property " + property);
    }).toArray();
    Tuple2<int[], PythonFunctionInfo[]> aggInfos = CommonPythonUtil.extractPythonAggregateFunctionInfosFromAggregateCall(aggCalls);
    int[] pythonUdafInputOffsets = aggInfos.f0;
    PythonFunctionInfo[] pythonFunctionInfos = aggInfos.f1;
    OneInputStreamOperator<RowData, RowData> pythonOperator = getPythonGroupWindowAggregateFunctionOperator(config, pythonConfig, inputRowType, outputRowType, maxLimitSize, windowSize, slideSize, namePropertyTypeArray, pythonUdafInputOffsets, pythonFunctionInfos);
    return ExecNodeUtil.createOneInputTransformation(inputTransform, createTransformationName(config), createTransformationDescription(config), pythonOperator, InternalTypeInfo.of(outputRowType), inputTransform.getParallelism());
}
Also used : Arrays(java.util.Arrays) InputProperty(org.apache.flink.table.planner.plan.nodes.exec.InputProperty) Tuple2(org.apache.flink.api.java.tuple.Tuple2) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) RowType(org.apache.flink.table.types.logical.RowType) Constructor(java.lang.reflect.Constructor) ExecNode(org.apache.flink.table.planner.plan.nodes.exec.ExecNode) ExecNodeUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.ExecNodeUtil) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd) ManagedMemoryUseCase(org.apache.flink.core.memory.ManagedMemoryUseCase) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) Projection(org.apache.flink.table.connector.Projection) ProjectionCodeGenerator(org.apache.flink.table.planner.codegen.ProjectionCodeGenerator) WindowCodeGenerator(org.apache.flink.table.planner.codegen.agg.batch.WindowCodeGenerator) ExecNodeContext(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeContext) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) RowData(org.apache.flink.table.data.RowData) PlannerBase(org.apache.flink.table.planner.delegation.PlannerBase) CommonPythonUtil(org.apache.flink.table.planner.plan.nodes.exec.utils.CommonPythonUtil) SingleTransformationTranslator(org.apache.flink.table.planner.plan.nodes.exec.SingleTransformationTranslator) ExecNodeConfig(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeConfig) Configuration(org.apache.flink.configuration.Configuration) TableException(org.apache.flink.table.api.TableException) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) InvocationTargetException(java.lang.reflect.InvocationTargetException) InternalTypeInfo(org.apache.flink.table.runtime.typeutils.InternalTypeInfo) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) LogicalWindow(org.apache.flink.table.planner.plan.logical.LogicalWindow) AggregateCall(org.apache.calcite.rel.core.AggregateCall) ExecNodeBase(org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase) GeneratedProjection(org.apache.flink.table.runtime.generated.GeneratedProjection) Transformation(org.apache.flink.api.dag.Transformation) OneInputStreamOperator(org.apache.flink.streaming.api.operators.OneInputStreamOperator) ExecutionConfigOptions(org.apache.flink.table.api.config.ExecutionConfigOptions) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) Collections(java.util.Collections) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) RowtimeAttribute(org.apache.flink.table.runtime.groupwindow.RowtimeAttribute) PythonFunctionInfo(org.apache.flink.table.functions.python.PythonFunctionInfo) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) TableException(org.apache.flink.table.api.TableException) RowData(org.apache.flink.table.data.RowData) WindowStart(org.apache.flink.table.runtime.groupwindow.WindowStart) WindowEnd(org.apache.flink.table.runtime.groupwindow.WindowEnd)

Example 3 with WindowProperty

use of org.apache.flink.table.runtime.groupwindow.WindowProperty in project flink by apache.

the class StreamExecGlobalWindowAggregate method createAggsHandler.

private GeneratedNamespaceAggsHandleFunction<Long> createAggsHandler(String name, SliceAssigner sliceAssigner, AggregateInfoList aggInfoList, int mergedAccOffset, boolean mergedAccIsOnHeap, DataType[] mergedAccExternalTypes, ExecNodeConfig config, RelBuilder relBuilder, ZoneId shifTimeZone) {
    final AggsHandlerCodeGenerator generator = new AggsHandlerCodeGenerator(new CodeGeneratorContext(config.getTableConfig()), relBuilder, JavaScalaConversionUtil.toScala(localAggInputRowType.getChildren()), // copyInputField
    true).needAccumulate().needMerge(mergedAccOffset, mergedAccIsOnHeap, mergedAccExternalTypes);
    final List<WindowProperty> windowProperties = Arrays.asList(Arrays.stream(namedWindowProperties).map(NamedWindowProperty::getProperty).toArray(WindowProperty[]::new));
    return generator.generateNamespaceAggsHandler(name, aggInfoList, JavaScalaConversionUtil.toScala(windowProperties), sliceAssigner, shifTimeZone);
}
Also used : NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) AggsHandlerCodeGenerator(org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator)

Example 4 with WindowProperty

use of org.apache.flink.table.runtime.groupwindow.WindowProperty in project flink by apache.

the class StreamExecWindowAggregate method createAggsHandler.

private GeneratedNamespaceAggsHandleFunction<Long> createAggsHandler(SliceAssigner sliceAssigner, AggregateInfoList aggInfoList, ExecNodeConfig config, RelBuilder relBuilder, List<LogicalType> fieldTypes, ZoneId shiftTimeZone) {
    final AggsHandlerCodeGenerator generator = new AggsHandlerCodeGenerator(new CodeGeneratorContext(config.getTableConfig()), relBuilder, JavaScalaConversionUtil.toScala(fieldTypes), // copyInputField
    false).needAccumulate();
    if (sliceAssigner instanceof SliceSharedAssigner) {
        generator.needMerge(0, false, null);
    }
    final List<WindowProperty> windowProperties = Arrays.asList(Arrays.stream(namedWindowProperties).map(NamedWindowProperty::getProperty).toArray(WindowProperty[]::new));
    return generator.generateNamespaceAggsHandler("WindowAggsHandler", aggInfoList, JavaScalaConversionUtil.toScala(windowProperties), sliceAssigner, shiftTimeZone);
}
Also used : NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) WindowProperty(org.apache.flink.table.runtime.groupwindow.WindowProperty) NamedWindowProperty(org.apache.flink.table.runtime.groupwindow.NamedWindowProperty) CodeGeneratorContext(org.apache.flink.table.planner.codegen.CodeGeneratorContext) AggsHandlerCodeGenerator(org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator) SliceSharedAssigner(org.apache.flink.table.runtime.operators.window.slicing.SliceSharedAssigner)

Aggregations

CodeGeneratorContext (org.apache.flink.table.planner.codegen.CodeGeneratorContext)4 NamedWindowProperty (org.apache.flink.table.runtime.groupwindow.NamedWindowProperty)4 WindowProperty (org.apache.flink.table.runtime.groupwindow.WindowProperty)4 AggsHandlerCodeGenerator (org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator)3 TableException (org.apache.flink.table.api.TableException)2 Constructor (java.lang.reflect.Constructor)1 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 Arrays (java.util.Arrays)1 Collections (java.util.Collections)1 AggregateCall (org.apache.calcite.rel.core.AggregateCall)1 Transformation (org.apache.flink.api.dag.Transformation)1 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)1 Configuration (org.apache.flink.configuration.Configuration)1 ManagedMemoryUseCase (org.apache.flink.core.memory.ManagedMemoryUseCase)1 OneInputStreamOperator (org.apache.flink.streaming.api.operators.OneInputStreamOperator)1 OneInputTransformation (org.apache.flink.streaming.api.transformations.OneInputTransformation)1 ExecutionConfigOptions (org.apache.flink.table.api.config.ExecutionConfigOptions)1 Projection (org.apache.flink.table.connector.Projection)1 RowData (org.apache.flink.table.data.RowData)1 ValueLiteralExpression (org.apache.flink.table.expressions.ValueLiteralExpression)1