Search in sources :

Example 1 with ProcTimeDeduplicateKeepLastRowFunction

use of org.apache.flink.table.runtime.operators.deduplicate.ProcTimeDeduplicateKeepLastRowFunction in project flink by apache.

the class StreamExecChangelogNormalize method translateToPlanInternal.

@SuppressWarnings("unchecked")
@Override
protected Transformation<RowData> translateToPlanInternal(PlannerBase planner, ExecNodeConfig config) {
    final ExecEdge inputEdge = getInputEdges().get(0);
    final Transformation<RowData> inputTransform = (Transformation<RowData>) inputEdge.translateToPlan(planner);
    final InternalTypeInfo<RowData> rowTypeInfo = (InternalTypeInfo<RowData>) inputTransform.getOutputType();
    final OneInputStreamOperator<RowData, RowData> operator;
    final long stateIdleTime = config.getStateRetentionTime();
    final boolean isMiniBatchEnabled = config.get(ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED);
    GeneratedRecordEqualiser generatedEqualiser = new EqualiserCodeGenerator(rowTypeInfo.toRowType()).generateRecordEqualiser("DeduplicateRowEqualiser");
    if (isMiniBatchEnabled) {
        TypeSerializer<RowData> rowSerializer = rowTypeInfo.createSerializer(planner.getExecEnv().getConfig());
        ProcTimeMiniBatchDeduplicateKeepLastRowFunction processFunction = new ProcTimeMiniBatchDeduplicateKeepLastRowFunction(rowTypeInfo, rowSerializer, stateIdleTime, generateUpdateBefore, // generateInsert
        true, // inputInsertOnly
        false, generatedEqualiser);
        CountBundleTrigger<RowData> trigger = AggregateUtil.createMiniBatchTrigger(config);
        operator = new KeyedMapBundleOperator<>(processFunction, trigger);
    } else {
        ProcTimeDeduplicateKeepLastRowFunction processFunction = new ProcTimeDeduplicateKeepLastRowFunction(rowTypeInfo, stateIdleTime, generateUpdateBefore, // generateInsert
        true, // inputInsertOnly
        false, generatedEqualiser);
        operator = new KeyedProcessOperator<>(processFunction);
    }
    final OneInputTransformation<RowData, RowData> transform = ExecNodeUtil.createOneInputTransformation(inputTransform, createTransformationMeta(CHANGELOG_NORMALIZE_TRANSFORMATION, config), operator, rowTypeInfo, inputTransform.getParallelism());
    final RowDataKeySelector selector = KeySelectorUtil.getRowDataSelector(uniqueKeys, rowTypeInfo);
    transform.setStateKeySelector(selector);
    transform.setStateKeyType(selector.getProducedType());
    return transform;
}
Also used : OneInputTransformation(org.apache.flink.streaming.api.transformations.OneInputTransformation) Transformation(org.apache.flink.api.dag.Transformation) ExecEdge(org.apache.flink.table.planner.plan.nodes.exec.ExecEdge) InternalTypeInfo(org.apache.flink.table.runtime.typeutils.InternalTypeInfo) EqualiserCodeGenerator(org.apache.flink.table.planner.codegen.EqualiserCodeGenerator) GeneratedRecordEqualiser(org.apache.flink.table.runtime.generated.GeneratedRecordEqualiser) RowData(org.apache.flink.table.data.RowData) ProcTimeMiniBatchDeduplicateKeepLastRowFunction(org.apache.flink.table.runtime.operators.deduplicate.ProcTimeMiniBatchDeduplicateKeepLastRowFunction) RowDataKeySelector(org.apache.flink.table.runtime.keyselector.RowDataKeySelector) ProcTimeDeduplicateKeepLastRowFunction(org.apache.flink.table.runtime.operators.deduplicate.ProcTimeDeduplicateKeepLastRowFunction)

Aggregations

Transformation (org.apache.flink.api.dag.Transformation)1 OneInputTransformation (org.apache.flink.streaming.api.transformations.OneInputTransformation)1 RowData (org.apache.flink.table.data.RowData)1 EqualiserCodeGenerator (org.apache.flink.table.planner.codegen.EqualiserCodeGenerator)1 ExecEdge (org.apache.flink.table.planner.plan.nodes.exec.ExecEdge)1 GeneratedRecordEqualiser (org.apache.flink.table.runtime.generated.GeneratedRecordEqualiser)1 RowDataKeySelector (org.apache.flink.table.runtime.keyselector.RowDataKeySelector)1 ProcTimeDeduplicateKeepLastRowFunction (org.apache.flink.table.runtime.operators.deduplicate.ProcTimeDeduplicateKeepLastRowFunction)1 ProcTimeMiniBatchDeduplicateKeepLastRowFunction (org.apache.flink.table.runtime.operators.deduplicate.ProcTimeMiniBatchDeduplicateKeepLastRowFunction)1 InternalTypeInfo (org.apache.flink.table.runtime.typeutils.InternalTypeInfo)1