Search in sources :

Example 16 with ResolvedExpression

use of org.apache.flink.table.expressions.ResolvedExpression in project flink by apache.

the class ValuesOperationFactory method create.

/**
 * Creates a valid {@link ValuesQueryOperation} operation.
 *
 * <p>It derives a row type based on {@link LogicalTypeMerging}. It flattens any row
 * constructors. It does not flatten ROWs which are a result of e.g. a function call.
 *
 * <p>The resulting schema can be provided manually. If it is not, the schema will be
 * automatically derived from the types of the expressions.
 */
QueryOperation create(@Nullable ResolvedSchema expectedSchema, List<ResolvedExpression> resolvedExpressions, ExpressionResolver.PostResolverFactory postResolverFactory) {
    List<List<ResolvedExpression>> resolvedRows = unwrapFromRowConstructor(resolvedExpressions);
    if (expectedSchema != null) {
        verifyAllSameSize(resolvedRows, expectedSchema.getColumnCount());
    }
    ResolvedSchema schema = Optional.ofNullable(expectedSchema).orElseGet(() -> extractSchema(resolvedRows));
    List<List<ResolvedExpression>> castedExpressions = resolvedRows.stream().map(row -> convertTopLevelExpressionToExpectedRowType(postResolverFactory, schema.getColumnDataTypes(), row)).collect(Collectors.toList());
    return new ValuesQueryOperation(castedExpressions, schema);
}
Also used : IntStream(java.util.stream.IntStream) DataType(org.apache.flink.table.types.DataType) QueryOperation(org.apache.flink.table.operations.QueryOperation) ApiExpressionUtils.valueLiteral(org.apache.flink.table.expressions.ApiExpressionUtils.valueLiteral) KeyValueDataType(org.apache.flink.table.types.KeyValueDataType) CallExpression(org.apache.flink.table.expressions.CallExpression) ResolvedSchema(org.apache.flink.table.catalog.ResolvedSchema) LogicalTypeMerging(org.apache.flink.table.types.logical.utils.LogicalTypeMerging) Expression(org.apache.flink.table.expressions.Expression) ExpressionDefaultVisitor(org.apache.flink.table.expressions.ExpressionDefaultVisitor) ArrayList(java.util.ArrayList) NULL(org.apache.flink.table.types.logical.LogicalTypeRoot.NULL) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) FieldsDataType(org.apache.flink.table.types.FieldsDataType) ExpressionResolver(org.apache.flink.table.expressions.resolver.ExpressionResolver) MAP(org.apache.flink.table.types.logical.LogicalTypeRoot.MAP) LogicalTypeCasts.supportsExplicitCast(org.apache.flink.table.types.logical.utils.LogicalTypeCasts.supportsExplicitCast) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) Nullable(javax.annotation.Nullable) LinkedHashSet(java.util.LinkedHashSet) ROW(org.apache.flink.table.types.logical.LogicalTypeRoot.ROW) FunctionDefinition(org.apache.flink.table.functions.FunctionDefinition) BuiltInFunctionDefinitions(org.apache.flink.table.functions.BuiltInFunctionDefinitions) TableException(org.apache.flink.table.api.TableException) Set(java.util.Set) ValueLiteralExpression(org.apache.flink.table.expressions.ValueLiteralExpression) Collectors(java.util.stream.Collectors) List(java.util.List) CollectionDataType(org.apache.flink.table.types.CollectionDataType) LogicalType(org.apache.flink.table.types.logical.LogicalType) ARRAY(org.apache.flink.table.types.logical.LogicalTypeRoot.ARRAY) ValidationException(org.apache.flink.table.api.ValidationException) Optional(java.util.Optional) Internal(org.apache.flink.annotation.Internal) TypeConversions(org.apache.flink.table.types.utils.TypeConversions) Collections(java.util.Collections) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) ArrayList(java.util.ArrayList) List(java.util.List) ResolvedSchema(org.apache.flink.table.catalog.ResolvedSchema)

Example 17 with ResolvedExpression

use of org.apache.flink.table.expressions.ResolvedExpression in project flink by apache.

the class OperationTreeBuilder method joinLateral.

public QueryOperation joinLateral(QueryOperation left, Expression tableFunction, JoinType joinType, Optional<Expression> condition) {
    ExpressionResolver resolver = getResolver(left);
    ResolvedExpression resolvedFunction = resolveSingleExpression(tableFunction, resolver);
    QueryOperation temporalTable = calculatedTableFactory.create(resolvedFunction, left.getResolvedSchema().getColumnNames());
    return join(left, temporalTable, joinType, condition, true);
}
Also used : ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) ExpressionResolver(org.apache.flink.table.expressions.resolver.ExpressionResolver) SqlExpressionResolver(org.apache.flink.table.expressions.resolver.SqlExpressionResolver) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) QueryOperation(org.apache.flink.table.operations.QueryOperation) DistinctQueryOperation(org.apache.flink.table.operations.DistinctQueryOperation) FilterQueryOperation(org.apache.flink.table.operations.FilterQueryOperation)

Example 18 with ResolvedExpression

use of org.apache.flink.table.expressions.ResolvedExpression in project flink by apache.

the class OperationTreeBuilder method flatMap.

public QueryOperation flatMap(Expression tableFunctionCall, QueryOperation child) {
    final ExpressionResolver resolver = getResolverBuilder(child).build();
    final ResolvedExpression resolvedCall = resolveSingleExpression(tableFunctionCall, resolver);
    if (!isFunctionOfKind(resolvedCall, FunctionKind.TABLE)) {
        throw new ValidationException("Only a table function can be used in the flatMap operator.");
    }
    final List<String> originFieldNames = DataTypeUtils.flattenToNames(resolvedCall.getOutputDataType());
    List<String> childFields = child.getResolvedSchema().getColumnNames();
    Set<String> usedFieldNames = new HashSet<>(childFields);
    List<Expression> args = new ArrayList<>();
    for (String originFieldName : originFieldNames) {
        String resultName = getUniqueName(originFieldName, usedFieldNames);
        usedFieldNames.add(resultName);
        args.add(valueLiteral(resultName));
    }
    args.add(0, tableFunctionCall);
    Expression renamedTableFunction = unresolvedCall(BuiltInFunctionDefinitions.AS, args.toArray(new Expression[0]));
    QueryOperation joinNode = joinLateral(child, renamedTableFunction, JoinType.INNER, Optional.empty());
    QueryOperation rightNode = dropColumns(childFields.stream().map(ApiExpressionUtils::unresolvedRef).collect(Collectors.toList()), joinNode);
    return alias(originFieldNames.stream().map(ApiExpressionUtils::unresolvedRef).collect(Collectors.toList()), rightNode);
}
Also used : ValidationException(org.apache.flink.table.api.ValidationException) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) Expression(org.apache.flink.table.expressions.Expression) UnresolvedCallExpression(org.apache.flink.table.expressions.UnresolvedCallExpression) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) ArrayList(java.util.ArrayList) ApiExpressionUtils(org.apache.flink.table.expressions.ApiExpressionUtils) ExpressionResolver(org.apache.flink.table.expressions.resolver.ExpressionResolver) SqlExpressionResolver(org.apache.flink.table.expressions.resolver.SqlExpressionResolver) HashSet(java.util.HashSet) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) QueryOperation(org.apache.flink.table.operations.QueryOperation) DistinctQueryOperation(org.apache.flink.table.operations.DistinctQueryOperation) FilterQueryOperation(org.apache.flink.table.operations.FilterQueryOperation)

Example 19 with ResolvedExpression

use of org.apache.flink.table.expressions.ResolvedExpression in project flink by apache.

the class OperationTreeBuilder method windowAggregate.

public QueryOperation windowAggregate(List<Expression> groupingExpressions, GroupWindow window, List<Expression> windowProperties, Expression aggregateFunction, QueryOperation child) {
    ExpressionResolver resolver = getAggResolver(child, groupingExpressions);
    Expression resolvedAggregate = aggregateFunction.accept(lookupResolver);
    AggregateWithAlias aggregateWithAlias = resolvedAggregate.accept(new ExtractAliasAndAggregate(true, resolver));
    List<Expression> groupsAndAggregate = new ArrayList<>(groupingExpressions);
    groupsAndAggregate.add(aggregateWithAlias.aggregate);
    List<Expression> namedGroupsAndAggregate = addAliasToTheCallInAggregate(child.getResolvedSchema().getColumnNames(), groupsAndAggregate);
    // Step1: add a default name to the call in the grouping expressions, e.g., groupBy(a % 5)
    // to
    // groupBy(a % 5 as TMP_0). We need a name for every column so that to perform alias for the
    // table aggregate function in Step6.
    List<Expression> newGroupingExpressions = namedGroupsAndAggregate.subList(0, groupingExpressions.size());
    // Step2: turn agg to a named agg, because it will be verified later.
    Expression aggregateRenamed = namedGroupsAndAggregate.get(groupingExpressions.size());
    // Step3: resolve expressions, including grouping, aggregates and window properties.
    ResolvedGroupWindow resolvedWindow = aggregateOperationFactory.createResolvedWindow(window, resolver);
    ExpressionResolver resolverWithWindowReferences = getResolverBuilder(child).withLocalReferences(localRef(resolvedWindow.getAlias(), resolvedWindow.getTimeAttribute().getOutputDataType())).build();
    List<ResolvedExpression> convertedGroupings = resolverWithWindowReferences.resolve(newGroupingExpressions);
    List<ResolvedExpression> convertedAggregates = resolverWithWindowReferences.resolve(Collections.singletonList(aggregateRenamed));
    List<ResolvedExpression> convertedProperties = resolverWithWindowReferences.resolve(windowProperties);
    // Step4: create window agg operation
    QueryOperation aggregateOperation = aggregateOperationFactory.createWindowAggregate(convertedGroupings, Collections.singletonList(convertedAggregates.get(0)), convertedProperties, resolvedWindow, child);
    // Step5: flatten the aggregate function
    List<String> aggNames = aggregateOperation.getResolvedSchema().getColumnNames();
    List<Expression> flattenedExpressions = aggNames.stream().map(ApiExpressionUtils::unresolvedRef).collect(Collectors.toCollection(ArrayList::new));
    flattenedExpressions.set(groupingExpressions.size(), unresolvedCall(BuiltInFunctionDefinitions.FLATTEN, unresolvedRef(aggNames.get(groupingExpressions.size()))));
    QueryOperation flattenedProjection = this.project(flattenedExpressions, aggregateOperation);
    // window attribute.
    return aliasBackwardFields(flattenedProjection, aggregateWithAlias.aliases, groupingExpressions.size());
}
Also used : ResolvedGroupWindow(org.apache.flink.table.operations.WindowAggregateQueryOperation.ResolvedGroupWindow) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) Expression(org.apache.flink.table.expressions.Expression) UnresolvedCallExpression(org.apache.flink.table.expressions.UnresolvedCallExpression) ArrayList(java.util.ArrayList) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) ExpressionResolver(org.apache.flink.table.expressions.resolver.ExpressionResolver) SqlExpressionResolver(org.apache.flink.table.expressions.resolver.SqlExpressionResolver) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) QueryOperation(org.apache.flink.table.operations.QueryOperation) DistinctQueryOperation(org.apache.flink.table.operations.DistinctQueryOperation) FilterQueryOperation(org.apache.flink.table.operations.FilterQueryOperation)

Example 20 with ResolvedExpression

use of org.apache.flink.table.expressions.ResolvedExpression in project flink by apache.

the class OperationTreeBuilder method tableAggregate.

public QueryOperation tableAggregate(List<Expression> groupingExpressions, Expression tableAggFunction, QueryOperation child) {
    // Step1: add a default name to the call in the grouping expressions, e.g., groupBy(a % 5)
    // to
    // groupBy(a % 5 as TMP_0). We need a name for every column so that to perform alias for the
    // table aggregate function in Step4.
    List<Expression> newGroupingExpressions = addAliasToTheCallInAggregate(child.getResolvedSchema().getColumnNames(), groupingExpressions);
    // Step2: resolve expressions
    ExpressionResolver resolver = getAggResolver(child, groupingExpressions);
    List<ResolvedExpression> resolvedGroupings = resolver.resolve(newGroupingExpressions);
    Tuple2<ResolvedExpression, List<String>> resolvedFunctionAndAlias = aggregateOperationFactory.extractTableAggFunctionAndAliases(resolveSingleExpression(tableAggFunction, resolver));
    // Step3: create table agg operation
    QueryOperation tableAggOperation = aggregateOperationFactory.createAggregate(resolvedGroupings, Collections.singletonList(resolvedFunctionAndAlias.f0), child);
    // Step4: add a top project to alias the output fields of the table aggregate.
    return aliasBackwardFields(tableAggOperation, resolvedFunctionAndAlias.f1, groupingExpressions.size());
}
Also used : ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) Expression(org.apache.flink.table.expressions.Expression) UnresolvedCallExpression(org.apache.flink.table.expressions.UnresolvedCallExpression) ResolvedExpression(org.apache.flink.table.expressions.ResolvedExpression) List(java.util.List) ArrayList(java.util.ArrayList) ExpressionResolver(org.apache.flink.table.expressions.resolver.ExpressionResolver) SqlExpressionResolver(org.apache.flink.table.expressions.resolver.SqlExpressionResolver) ValuesQueryOperation(org.apache.flink.table.operations.ValuesQueryOperation) QueryOperation(org.apache.flink.table.operations.QueryOperation) DistinctQueryOperation(org.apache.flink.table.operations.DistinctQueryOperation) FilterQueryOperation(org.apache.flink.table.operations.FilterQueryOperation)

Aggregations

ResolvedExpression (org.apache.flink.table.expressions.ResolvedExpression)29 ExpressionResolver (org.apache.flink.table.expressions.resolver.ExpressionResolver)15 Expression (org.apache.flink.table.expressions.Expression)13 ArrayList (java.util.ArrayList)11 SqlExpressionResolver (org.apache.flink.table.expressions.resolver.SqlExpressionResolver)11 List (java.util.List)8 QueryOperation (org.apache.flink.table.operations.QueryOperation)8 DataType (org.apache.flink.table.types.DataType)8 TableException (org.apache.flink.table.api.TableException)7 ValidationException (org.apache.flink.table.api.ValidationException)7 CallExpression (org.apache.flink.table.expressions.CallExpression)7 ValueLiteralExpression (org.apache.flink.table.expressions.ValueLiteralExpression)7 LogicalType (org.apache.flink.table.types.logical.LogicalType)7 Optional (java.util.Optional)6 Collectors (java.util.stream.Collectors)6 FilterQueryOperation (org.apache.flink.table.operations.FilterQueryOperation)6 ValuesQueryOperation (org.apache.flink.table.operations.ValuesQueryOperation)6 FieldReferenceExpression (org.apache.flink.table.expressions.FieldReferenceExpression)5 UnresolvedCallExpression (org.apache.flink.table.expressions.UnresolvedCallExpression)5 FunctionDefinition (org.apache.flink.table.functions.FunctionDefinition)5