Search in sources :

Example 1 with REPLICATED

use of io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED in project hetu-core by openlookeng.

the class TestDetermineSemiJoinDistributionType method testReplicatesWhenNotRestricted.

@Test
public void testReplicatesWhenNotRestricted() {
    // variable width so that average row size is respected
    Type symbolType = createUnboundedVarcharType();
    int aRows = 10_000;
    int bRows = 10;
    PlanNodeStatsEstimate probeSideStatsEstimate = PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), new SymbolStatsEstimate(0, 100, 0, 640000, 10))).build();
    PlanNodeStatsEstimate buildSideStatsEstimate = PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), new SymbolStatsEstimate(0, 100, 0, 640000, 10))).build();
    // B table is small enough to be replicated in AUTOMATIC_RESTRICTED mode
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).setSystemProperty(JOIN_MAX_BROADCAST_TABLE_SIZE, "100MB").overrideStats("valuesA", probeSideStatsEstimate).overrideStats("valuesB", buildSideStatsEstimate).on(p -> {
        Symbol a1 = p.symbol("A1", symbolType);
        Symbol b1 = p.symbol("B1", symbolType);
        return p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, a1), p.values(new PlanNodeId("valuesB"), bRows, b1), a1, b1, p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty());
    }).matches(semiJoin("A1", "B1", "output", Optional.of(REPLICATED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
    probeSideStatsEstimate = PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), new SymbolStatsEstimate(0, 100, 0, 640000d * 10000, 10))).build();
    buildSideStatsEstimate = PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), new SymbolStatsEstimate(0, 100, 0, 640000d * 10000, 10))).build();
    // B table exceeds AUTOMATIC_RESTRICTED limit therefore it is partitioned
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).setSystemProperty(JOIN_MAX_BROADCAST_TABLE_SIZE, "100MB").overrideStats("valuesA", probeSideStatsEstimate).overrideStats("valuesB", buildSideStatsEstimate).on(p -> {
        Symbol a1 = p.symbol("A1", symbolType);
        Symbol b1 = p.symbol("B1", symbolType);
        return p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, a1), p.values(new PlanNodeId("valuesB"), bRows, b1), a1, b1, p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty());
    }).matches(semiJoin("A1", "B1", "output", Optional.of(PARTITIONED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED) PlanMatchPattern.semiJoin(io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Test(org.testng.annotations.Test) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) Type(io.prestosql.spi.type.Type) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) PARTITIONED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Type(io.prestosql.spi.type.Type) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) Symbol(io.prestosql.spi.plan.Symbol) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) Test(org.testng.annotations.Test)

Example 2 with REPLICATED

use of io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED in project hetu-core by openlookeng.

the class TestDetermineSemiJoinDistributionType method testReplicatesWhenFilterMuchSmaller.

@Test
public void testReplicatesWhenFilterMuchSmaller() {
    int aRows = 10_000;
    int bRows = 100;
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), SymbolStatsEstimate.unknown())).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), SymbolStatsEstimate.unknown())).build()).on(p -> p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, p.symbol("A1", BIGINT)), p.values(new PlanNodeId("valuesB"), bRows, p.symbol("B1", BIGINT)), p.symbol("A1"), p.symbol("B1"), p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())).matches(semiJoin("A1", "B1", "output", Optional.of(REPLICATED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED) PlanMatchPattern.semiJoin(io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Test(org.testng.annotations.Test) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) Type(io.prestosql.spi.type.Type) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) PARTITIONED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Symbol(io.prestosql.spi.plan.Symbol) Test(org.testng.annotations.Test)

Example 3 with REPLICATED

use of io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED in project hetu-core by openlookeng.

the class TestDetermineSemiJoinDistributionType method testReplicatesWhenRequiredBySession.

@Test
public void testReplicatesWhenRequiredBySession() {
    int aRows = 10_000;
    int bRows = 10_000;
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.BROADCAST.name()).setSystemProperty(JOIN_MAX_BROADCAST_TABLE_SIZE, "1B").overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), SymbolStatsEstimate.unknown())).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), SymbolStatsEstimate.unknown())).build()).on(p -> p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, p.symbol("A1", BIGINT)), p.values(new PlanNodeId("valuesB"), bRows, p.symbol("B1", BIGINT)), p.symbol("A1"), p.symbol("B1"), p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())).matches(semiJoin("A1", "B1", "output", Optional.of(REPLICATED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED) PlanMatchPattern.semiJoin(io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Test(org.testng.annotations.Test) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) Type(io.prestosql.spi.type.Type) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) PARTITIONED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Symbol(io.prestosql.spi.plan.Symbol) Test(org.testng.annotations.Test)

Aggregations

ImmutableList (com.google.common.collect.ImmutableList)3 ImmutableMap (com.google.common.collect.ImmutableMap)3 JOIN_DISTRIBUTION_TYPE (io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE)3 JOIN_MAX_BROADCAST_TABLE_SIZE (io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE)3 CostComparator (io.prestosql.cost.CostComparator)3 PlanNodeStatsEstimate (io.prestosql.cost.PlanNodeStatsEstimate)3 SymbolStatsEstimate (io.prestosql.cost.SymbolStatsEstimate)3 TaskCountEstimator (io.prestosql.cost.TaskCountEstimator)3 PlanNodeId (io.prestosql.spi.plan.PlanNodeId)3 Symbol (io.prestosql.spi.plan.Symbol)3 BIGINT (io.prestosql.spi.type.BigintType.BIGINT)3 Type (io.prestosql.spi.type.Type)3 VarcharType.createUnboundedVarcharType (io.prestosql.spi.type.VarcharType.createUnboundedVarcharType)3 JoinDistributionType (io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType)3 PlanMatchPattern.semiJoin (io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin)3 PlanMatchPattern.values (io.prestosql.sql.planner.assertions.PlanMatchPattern.values)3 PlanBuilder.constantExpressions (io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions)3 RuleAssert (io.prestosql.sql.planner.iterative.rule.test.RuleAssert)3 RuleTester (io.prestosql.sql.planner.iterative.rule.test.RuleTester)3 PARTITIONED (io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED)3