Search in sources :

Example 1 with EquiJoinClause

use of com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause in project presto by prestodb.

the class TestPredicatePushdown method testPredicatePushDownCreatesValidJoin.

@Test
public void testPredicatePushDownCreatesValidJoin() {
    RuleTester tester = new RuleTester();
    tester.assertThat(new PredicatePushDown(tester.getMetadata(), tester.getSqlParser())).on(p -> p.join(INNER, p.filter(p.comparison(OperatorType.EQUAL, p.variable("a1"), constant(1L, INTEGER)), p.values(p.variable("a1"))), p.values(p.variable("b1")), ImmutableList.of(new EquiJoinClause(p.variable("a1"), p.variable("b1"))), ImmutableList.of(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.of(PARTITIONED), ImmutableMap.of())).matches(project(join(INNER, ImmutableList.of(), Optional.empty(), Optional.of(REPLICATED), project(filter("a1=1", values("a1"))), project(filter("1=b1", values("b1"))))));
}
Also used : EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) PlanOptimizer(com.facebook.presto.sql.planner.optimizations.PlanOptimizer) PlanMatchPattern.filter(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.filter) BasePlanTest(com.facebook.presto.sql.planner.assertions.BasePlanTest) PredicatePushDown(com.facebook.presto.sql.planner.optimizations.PredicatePushDown) PlanMatchPattern.anyTree(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.anyTree) PlanMatchPattern.output(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.output) PlanMatchPattern.assignUniqueId(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.assignUniqueId) Test(org.testng.annotations.Test) Expressions.constant(com.facebook.presto.sql.relational.Expressions.constant) PlanMatchPattern(com.facebook.presto.sql.planner.assertions.PlanMatchPattern) PlanMatchPattern.join(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.join) ImmutableList(com.google.common.collect.ImmutableList) PlanMatchPattern.equiJoinClause(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.equiJoinClause) Map(java.util.Map) PlanMatchPattern.expression(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.expression) PlanMatchPattern.project(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.project) WindowNode(com.facebook.presto.sql.planner.plan.WindowNode) ImmutableMap(com.google.common.collect.ImmutableMap) PlanMatchPattern.semiJoin(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.semiJoin) PlanMatchPattern.tableScan(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.tableScan) PlanMatchPattern.exchange(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.exchange) OperatorType(com.facebook.presto.common.function.OperatorType) List(java.util.List) REPLICATED(com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.REPLICATED) LEFT(com.facebook.presto.sql.planner.plan.JoinNode.Type.LEFT) INTEGER(com.facebook.presto.common.type.IntegerType.INTEGER) Optional(java.util.Optional) PlanMatchPattern.values(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.values) PARTITIONED(com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.PARTITIONED) RuleTester(com.facebook.presto.sql.planner.iterative.rule.test.RuleTester) ExchangeNode(com.facebook.presto.sql.planner.plan.ExchangeNode) PlanMatchPattern.node(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.node) INNER(com.facebook.presto.sql.planner.plan.JoinNode.Type.INNER) RuleTester(com.facebook.presto.sql.planner.iterative.rule.test.RuleTester) EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) PredicatePushDown(com.facebook.presto.sql.planner.optimizations.PredicatePushDown) BasePlanTest(com.facebook.presto.sql.planner.assertions.BasePlanTest) Test(org.testng.annotations.Test)

Example 2 with EquiJoinClause

use of com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause in project presto by prestodb.

the class JoinStatsRule method filterByEquiJoinClauses.

private PlanNodeStatsEstimate filterByEquiJoinClauses(PlanNodeStatsEstimate stats, EquiJoinClause drivingClause, Collection<EquiJoinClause> remainingClauses, Session session, TypeProvider types) {
    ComparisonExpression drivingPredicate = new ComparisonExpression(EQUAL, new SymbolReference(getNodeLocation(drivingClause.getLeft().getSourceLocation()), drivingClause.getLeft().getName()), new SymbolReference(getNodeLocation(drivingClause.getRight().getSourceLocation()), drivingClause.getRight().getName()));
    PlanNodeStatsEstimate filteredStats = filterStatsCalculator.filterStats(stats, drivingPredicate, session, types);
    for (EquiJoinClause clause : remainingClauses) {
        filteredStats = filterByAuxiliaryClause(filteredStats, clause);
    }
    return filteredStats;
}
Also used : ComparisonExpression(com.facebook.presto.sql.tree.ComparisonExpression) SymbolReference(com.facebook.presto.sql.tree.SymbolReference) EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause)

Example 3 with EquiJoinClause

use of com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause in project presto by prestodb.

the class TestJoinStatsRule method testStatsForInnerJoinWithTwoEquiClausesAndNonEqualityFunction.

@Test
public void testStatsForInnerJoinWithTwoEquiClausesAndNonEqualityFunction() {
    double innerJoinRowCount = // driver join clause
    LEFT_ROWS_COUNT * RIGHT_ROWS_COUNT / LEFT_JOIN_COLUMN_2_NDV * LEFT_JOIN_COLUMN_2_NON_NULLS * RIGHT_JOIN_COLUMN_2_NON_NULLS * // auxiliary join clause
    UNKNOWN_FILTER_COEFFICIENT * // LEFT_JOIN_COLUMN < 10 non equality filter
    0.3333333333;
    PlanNodeStatsEstimate innerJoinStats = planNodeStats(innerJoinRowCount, variableStatistics(LEFT_JOIN_COLUMN, 5.0, 10.0, 0.0, RIGHT_JOIN_COLUMN_NDV * 0.3333333333), variableStatistics(RIGHT_JOIN_COLUMN, 5.0, 20.0, 0.0, RIGHT_JOIN_COLUMN_NDV), variableStatistics(LEFT_JOIN_COLUMN_2, 100.0, 200.0, 0.0, RIGHT_JOIN_COLUMN_2_NDV), variableStatistics(RIGHT_JOIN_COLUMN_2, 100.0, 200.0, 0.0, RIGHT_JOIN_COLUMN_2_NDV));
    tester().assertStatsFor(pb -> {
        VariableReferenceExpression leftJoinColumn = pb.variable(LEFT_JOIN_COLUMN);
        VariableReferenceExpression rightJoinColumn = pb.variable(RIGHT_JOIN_COLUMN);
        VariableReferenceExpression leftJoinColumn2 = pb.variable(LEFT_JOIN_COLUMN_2);
        VariableReferenceExpression rightJoinColumn2 = pb.variable(RIGHT_JOIN_COLUMN_2);
        ComparisonExpression leftJoinColumnLessThanTen = new ComparisonExpression(ComparisonExpression.Operator.LESS_THAN, new SymbolReference(leftJoinColumn.getName()), new LongLiteral("10"));
        return pb.join(INNER, pb.values(leftJoinColumn, leftJoinColumn2), pb.values(rightJoinColumn, rightJoinColumn2), ImmutableList.of(new EquiJoinClause(leftJoinColumn2, rightJoinColumn2), new EquiJoinClause(leftJoinColumn, rightJoinColumn)), ImmutableList.of(leftJoinColumn, leftJoinColumn2, rightJoinColumn, rightJoinColumn2), Optional.of(castToRowExpression(leftJoinColumnLessThanTen)));
    }).withSourceStats(0, planNodeStats(LEFT_ROWS_COUNT, LEFT_JOIN_COLUMN_STATS, LEFT_JOIN_COLUMN_2_STATS)).withSourceStats(1, planNodeStats(RIGHT_ROWS_COUNT, RIGHT_JOIN_COLUMN_STATS, RIGHT_JOIN_COLUMN_2_STATS)).check(stats -> stats.equalTo(innerJoinStats));
}
Also used : EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) PlanNodeStatsAssertion.assertThat(com.facebook.presto.cost.PlanNodeStatsAssertion.assertThat) JoinNode(com.facebook.presto.sql.planner.plan.JoinNode) BIGINT(com.facebook.presto.common.type.BigintType.BIGINT) SymbolReference(com.facebook.presto.sql.tree.SymbolReference) MetadataManager(com.facebook.presto.metadata.MetadataManager) DOUBLE(com.facebook.presto.common.type.DoubleType.DOUBLE) UNKNOWN_FILTER_COEFFICIENT(com.facebook.presto.cost.FilterStatsCalculator.UNKNOWN_FILTER_COEFFICIENT) Assert.assertEquals(org.testng.Assert.assertEquals) VariableReferenceExpression(com.facebook.presto.spi.relation.VariableReferenceExpression) Test(org.testng.annotations.Test) ComparisonExpression(com.facebook.presto.sql.tree.ComparisonExpression) FULL(com.facebook.presto.sql.planner.plan.JoinNode.Type.FULL) RIGHT(com.facebook.presto.sql.planner.plan.JoinNode.Type.RIGHT) ImmutableList(com.google.common.collect.ImmutableList) LEFT(com.facebook.presto.sql.planner.plan.JoinNode.Type.LEFT) NaN(java.lang.Double.NaN) MetadataManager.createTestMetadataManager(com.facebook.presto.metadata.MetadataManager.createTestMetadataManager) LongLiteral(com.facebook.presto.sql.tree.LongLiteral) Optional(java.util.Optional) INNER(com.facebook.presto.sql.planner.plan.JoinNode.Type.INNER) OriginalExpressionUtils.castToRowExpression(com.facebook.presto.sql.relational.OriginalExpressionUtils.castToRowExpression) ComparisonExpression(com.facebook.presto.sql.tree.ComparisonExpression) LongLiteral(com.facebook.presto.sql.tree.LongLiteral) VariableReferenceExpression(com.facebook.presto.spi.relation.VariableReferenceExpression) SymbolReference(com.facebook.presto.sql.tree.SymbolReference) EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) Test(org.testng.annotations.Test)

Example 4 with EquiJoinClause

use of com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause in project presto by prestodb.

the class JoinStatsRule method filterByEquiJoinClauses.

private PlanNodeStatsEstimate filterByEquiJoinClauses(PlanNodeStatsEstimate stats, Collection<EquiJoinClause> clauses, Session session, TypeProvider types) {
    checkArgument(!clauses.isEmpty(), "clauses is empty");
    PlanNodeStatsEstimate result = PlanNodeStatsEstimate.unknown();
    // Join equality clauses are usually correlated. Therefore we shouldn't treat each join equality
    // clause separately because stats estimates would be way off. Instead we choose so called
    // "driving clause" which mostly reduces join output rows cardinality and apply UNKNOWN_FILTER_COEFFICIENT
    // for other (auxiliary) clauses.
    Queue<EquiJoinClause> remainingClauses = new LinkedList<>(clauses);
    EquiJoinClause drivingClause = remainingClauses.poll();
    for (int i = 0; i < clauses.size(); i++) {
        PlanNodeStatsEstimate estimate = filterByEquiJoinClauses(stats, drivingClause, remainingClauses, session, types);
        if (result.isOutputRowCountUnknown() || (!estimate.isOutputRowCountUnknown() && estimate.getOutputRowCount() < result.getOutputRowCount())) {
            result = estimate;
        }
        remainingClauses.add(drivingClause);
        drivingClause = remainingClauses.poll();
    }
    return result;
}
Also used : EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) LinkedList(java.util.LinkedList)

Example 5 with EquiJoinClause

use of com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause in project presto by prestodb.

the class TestReorderJoins method testReplicatedScalarJoinEvenWhereSessionRequiresRepartitioned.

@Test
public void testReplicatedScalarJoinEvenWhereSessionRequiresRepartitioned() {
    PlanMatchPattern expectedPlan = join(INNER, ImmutableList.of(equiJoinClause("A1", "B1")), Optional.empty(), Optional.of(REPLICATED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0)));
    PlanNodeStatsEstimate valuesA = PlanNodeStatsEstimate.builder().setOutputRowCount(10000).addVariableStatistics(ImmutableMap.of(variable("A1", BIGINT), new VariableStatsEstimate(0, 100, 0, 640000, 100))).build();
    PlanNodeStatsEstimate valuesB = PlanNodeStatsEstimate.builder().setOutputRowCount(10000).addVariableStatistics(ImmutableMap.of(variable("B1", BIGINT), new VariableStatsEstimate(0, 100, 0, 640000, 100))).build();
    assertReorderJoins().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.PARTITIONED.name()).on(p -> p.join(INNER, // matches isAtMostScalar
    p.values(new PlanNodeId("valuesA"), p.variable("A1")), p.values(new PlanNodeId("valuesB"), ImmutableList.of(p.variable("B1")), TWO_ROWS), ImmutableList.of(new EquiJoinClause(p.variable("A1"), p.variable("B1"))), ImmutableList.of(p.variable("A1"), p.variable("B1")), Optional.empty())).overrideStats("valuesA", valuesA).overrideStats("valuesB", valuesB).matches(expectedPlan);
    assertReorderJoins().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.PARTITIONED.name()).on(p -> p.join(INNER, p.values(new PlanNodeId("valuesB"), ImmutableList.of(p.variable("B1")), TWO_ROWS), // matches isAtMostScalar
    p.values(new PlanNodeId("valuesA"), p.variable("A1")), ImmutableList.of(new EquiJoinClause(p.variable("A1"), p.variable("B1"))), ImmutableList.of(p.variable("A1"), p.variable("B1")), Optional.empty())).overrideStats("valuesA", valuesA).overrideStats("valuesB", valuesB).matches(expectedPlan);
}
Also used : JoinReorderingStrategy(com.facebook.presto.sql.analyzer.FeaturesConfig.JoinReorderingStrategy) EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) PlanNodeId(com.facebook.presto.spi.plan.PlanNodeId) AUTOMATIC(com.facebook.presto.sql.analyzer.FeaturesConfig.JoinDistributionType.AUTOMATIC) QualifiedName(com.facebook.presto.sql.tree.QualifiedName) VARCHAR(com.facebook.presto.common.type.VarcharType.VARCHAR) VariableReferenceExpression(com.facebook.presto.spi.relation.VariableReferenceExpression) Test(org.testng.annotations.Test) Expressions.call(com.facebook.presto.sql.relational.Expressions.call) PlanMatchPattern(com.facebook.presto.sql.planner.assertions.PlanMatchPattern) PlanMatchPattern.join(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.join) JoinDistributionType(com.facebook.presto.sql.analyzer.FeaturesConfig.JoinDistributionType) ImmutableList(com.google.common.collect.ImmutableList) JOIN_MAX_BROADCAST_TABLE_SIZE(com.facebook.presto.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) PlanMatchPattern.equiJoinClause(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.equiJoinClause) EQUAL(com.facebook.presto.common.function.OperatorType.EQUAL) LESS_THAN(com.facebook.presto.common.function.OperatorType.LESS_THAN) BOOLEAN(com.facebook.presto.common.type.BooleanType.BOOLEAN) FunctionResolution(com.facebook.presto.sql.relational.FunctionResolution) Expressions.variable(com.facebook.presto.sql.relational.Expressions.variable) RuleAssert(com.facebook.presto.sql.planner.iterative.rule.test.RuleAssert) FunctionAndTypeManager.qualifyObjectName(com.facebook.presto.metadata.FunctionAndTypeManager.qualifyObjectName) RowExpression(com.facebook.presto.spi.relation.RowExpression) AfterClass(org.testng.annotations.AfterClass) BIGINT(com.facebook.presto.common.type.BigintType.BIGINT) PlanNodeStatsEstimate(com.facebook.presto.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) JOIN_DISTRIBUTION_TYPE(com.facebook.presto.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) OperatorType(com.facebook.presto.common.function.OperatorType) List(java.util.List) REPLICATED(com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.REPLICATED) JOIN_REORDERING_STRATEGY(com.facebook.presto.SystemSessionProperties.JOIN_REORDERING_STRATEGY) CostComparator(com.facebook.presto.cost.CostComparator) Optional(java.util.Optional) PlanMatchPattern.values(com.facebook.presto.sql.planner.assertions.PlanMatchPattern.values) PARTITIONED(com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.PARTITIONED) VariableStatsEstimate(com.facebook.presto.cost.VariableStatsEstimate) Closeables.closeAllRuntimeException(com.facebook.airlift.testing.Closeables.closeAllRuntimeException) RuleTester(com.facebook.presto.sql.planner.iterative.rule.test.RuleTester) BROADCAST(com.facebook.presto.sql.analyzer.FeaturesConfig.JoinDistributionType.BROADCAST) INNER(com.facebook.presto.sql.planner.plan.JoinNode.Type.INNER) PlanNodeId(com.facebook.presto.spi.plan.PlanNodeId) PlanNodeStatsEstimate(com.facebook.presto.cost.PlanNodeStatsEstimate) EquiJoinClause(com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause) PlanMatchPattern(com.facebook.presto.sql.planner.assertions.PlanMatchPattern) VariableStatsEstimate(com.facebook.presto.cost.VariableStatsEstimate) Test(org.testng.annotations.Test)

Aggregations

EquiJoinClause (com.facebook.presto.sql.planner.plan.JoinNode.EquiJoinClause)7 INNER (com.facebook.presto.sql.planner.plan.JoinNode.Type.INNER)5 ImmutableList (com.google.common.collect.ImmutableList)5 Optional (java.util.Optional)5 Test (org.testng.annotations.Test)5 OperatorType (com.facebook.presto.common.function.OperatorType)4 BIGINT (com.facebook.presto.common.type.BigintType.BIGINT)4 VariableReferenceExpression (com.facebook.presto.spi.relation.VariableReferenceExpression)4 PlanMatchPattern (com.facebook.presto.sql.planner.assertions.PlanMatchPattern)4 PlanMatchPattern.equiJoinClause (com.facebook.presto.sql.planner.assertions.PlanMatchPattern.equiJoinClause)4 PlanMatchPattern.join (com.facebook.presto.sql.planner.assertions.PlanMatchPattern.join)4 PlanMatchPattern.values (com.facebook.presto.sql.planner.assertions.PlanMatchPattern.values)4 RuleTester (com.facebook.presto.sql.planner.iterative.rule.test.RuleTester)4 PARTITIONED (com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.PARTITIONED)4 REPLICATED (com.facebook.presto.sql.planner.plan.JoinNode.DistributionType.REPLICATED)4 ImmutableMap (com.google.common.collect.ImmutableMap)4 List (java.util.List)4 Closeables.closeAllRuntimeException (com.facebook.airlift.testing.Closeables.closeAllRuntimeException)3 JOIN_DISTRIBUTION_TYPE (com.facebook.presto.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE)3 JOIN_MAX_BROADCAST_TABLE_SIZE (com.facebook.presto.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE)3