Search in sources :

Example 6 with JOIN_DISTRIBUTION_TYPE

use of io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE in project hetu-core by openlookeng.

the class TestDetermineSemiJoinDistributionType method testPartitionsWhenBothTablesEqual.

@Test
public void testPartitionsWhenBothTablesEqual() {
    int aRows = 10_000;
    int bRows = 10_000;
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), SymbolStatsEstimate.unknown())).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), SymbolStatsEstimate.unknown())).build()).on(p -> p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, p.symbol("A1", BIGINT)), p.values(new PlanNodeId("valuesB"), bRows, p.symbol("B1", BIGINT)), p.symbol("A1"), p.symbol("B1"), p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())).matches(semiJoin("A1", "B1", "output", Optional.of(PARTITIONED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED) PlanMatchPattern.semiJoin(io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Test(org.testng.annotations.Test) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) Type(io.prestosql.spi.type.Type) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) PARTITIONED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Symbol(io.prestosql.spi.plan.Symbol) Test(org.testng.annotations.Test)

Example 7 with JOIN_DISTRIBUTION_TYPE

use of io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE in project hetu-core by openlookeng.

the class TestDetermineSemiJoinDistributionType method testReplicatesWhenFilterMuchSmaller.

@Test
public void testReplicatesWhenFilterMuchSmaller() {
    int aRows = 10_000;
    int bRows = 100;
    assertDetermineSemiJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), SymbolStatsEstimate.unknown())).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), SymbolStatsEstimate.unknown())).build()).on(p -> p.semiJoin(p.values(new PlanNodeId("valuesA"), aRows, p.symbol("A1", BIGINT)), p.values(new PlanNodeId("valuesB"), bRows, p.symbol("B1", BIGINT)), p.symbol("A1"), p.symbol("B1"), p.symbol("output"), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())).matches(semiJoin("A1", "B1", "output", Optional.of(REPLICATED), values(ImmutableMap.of("A1", 0)), values(ImmutableMap.of("B1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.REPLICATED) PlanMatchPattern.semiJoin(io.prestosql.sql.planner.assertions.PlanMatchPattern.semiJoin) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) Test(org.testng.annotations.Test) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) Type(io.prestosql.spi.type.Type) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) PARTITIONED(io.prestosql.sql.planner.plan.SemiJoinNode.DistributionType.PARTITIONED) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Symbol(io.prestosql.spi.plan.Symbol) Test(org.testng.annotations.Test)

Example 8 with JOIN_DISTRIBUTION_TYPE

use of io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE in project hetu-core by openlookeng.

the class TestMemorySmoke method testSemiJoinDynamicFilteringNone.

@Test
public void testSemiJoinDynamicFilteringNone() {
    final long buildSideRowsCount = 15_000L;
    Session session = Session.builder(getSession()).setSystemProperty(ENABLE_DYNAMIC_FILTERING, "true").setSystemProperty(JOIN_DISTRIBUTION_TYPE, FeaturesConfig.JoinDistributionType.BROADCAST.name()).build();
    DistributedQueryRunner runner = (DistributedQueryRunner) getQueryRunner();
    ResultWithQueryId<MaterializedResult> result = runner.executeWithQueryId(session, "SELECT * FROM lineitem WHERE orderkey IN (SELECT orderkey from orders WHERE totalprice < 0)");
    assertEquals(result.getResult().getRowCount(), 0);
    QueryStats stats = runner.getCoordinator().getQueryManager().getFullQueryInfo(result.getQueryId()).getQueryStats();
    Set rowsRead = stats.getOperatorSummaries().stream().filter(summary -> summary.getOperatorType().equals("ScanFilterAndProjectOperator")).map(summary -> summary.getInputPositions()).collect(toImmutableSet());
    assertEquals(rowsRead, ImmutableSet.of(0L, buildSideRowsCount));
}
Also used : ImmutableSet(com.google.common.collect.ImmutableSet) ResultWithQueryId(io.prestosql.tests.ResultWithQueryId) Language(org.intellij.lang.annotations.Language) Set(java.util.Set) Test(org.testng.annotations.Test) QueryStats(io.prestosql.execution.QueryStats) MaterializedRow(io.prestosql.testing.MaterializedRow) MaterializedResult(io.prestosql.testing.MaterializedResult) String.format(java.lang.String.format) QualifiedObjectName(io.prestosql.spi.connector.QualifiedObjectName) AbstractTestQueryFramework(io.prestosql.tests.AbstractTestQueryFramework) Assert.assertEquals(io.prestosql.testing.assertions.Assert.assertEquals) List(java.util.List) ENABLE_DYNAMIC_FILTERING(io.prestosql.SystemSessionProperties.ENABLE_DYNAMIC_FILTERING) DistributedQueryRunner(io.prestosql.tests.DistributedQueryRunner) Session(io.prestosql.Session) FeaturesConfig(io.prestosql.sql.analyzer.FeaturesConfig) ImmutableSet.toImmutableSet(com.google.common.collect.ImmutableSet.toImmutableSet) Assert.assertTrue(org.testng.Assert.assertTrue) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) DistributedQueryRunner(io.prestosql.tests.DistributedQueryRunner) ImmutableSet(com.google.common.collect.ImmutableSet) Set(java.util.Set) ImmutableSet.toImmutableSet(com.google.common.collect.ImmutableSet.toImmutableSet) QueryStats(io.prestosql.execution.QueryStats) MaterializedResult(io.prestosql.testing.MaterializedResult) Session(io.prestosql.Session) Test(org.testng.annotations.Test)

Example 9 with JOIN_DISTRIBUTION_TYPE

use of io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE in project hetu-core by openlookeng.

the class TestDetermineJoinDistributionType method testFlipAndReplicateWhenOneTableMuchSmaller.

@Test
public void testFlipAndReplicateWhenOneTableMuchSmaller() {
    // variable width so that average row size is respected
    VarcharType symbolType = createUnboundedVarcharType();
    int aRows = 100;
    int bRows = 10_000;
    assertDetermineJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), new SymbolStatsEstimate(0, 100, 0, 6400, 100))).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), new SymbolStatsEstimate(0, 100, 0, 640000, 100))).build()).on(p -> {
        Symbol a1 = p.symbol("A1", symbolType);
        Symbol b1 = p.symbol("B1", symbolType);
        return p.join(INNER, p.values(new PlanNodeId("valuesA"), aRows, a1), p.values(new PlanNodeId("valuesB"), bRows, b1), ImmutableList.of(new JoinNode.EquiJoinClause(a1, b1)), ImmutableList.of(a1, b1), Optional.empty());
    }).matches(join(INNER, ImmutableList.of(equiJoinClause("B1", "A1")), Optional.empty(), Optional.of(REPLICATED), values(ImmutableMap.of("B1", 0)), values(ImmutableMap.of("A1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.spi.plan.JoinNode.DistributionType.REPLICATED) PlanMatchPattern.equiJoinClause(io.prestosql.sql.planner.assertions.PlanMatchPattern.equiJoinClause) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) PlanMatchPattern.enforceSingleRow(io.prestosql.sql.planner.assertions.PlanMatchPattern.enforceSingleRow) Test(org.testng.annotations.Test) PARTITIONED(io.prestosql.spi.plan.JoinNode.DistributionType.PARTITIONED) PlanBuilder.castToRowExpression(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.castToRowExpression) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) Type(io.prestosql.spi.plan.JoinNode.Type) JoinNode(io.prestosql.spi.plan.JoinNode) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanMatchPattern.join(io.prestosql.sql.planner.assertions.PlanMatchPattern.join) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) FULL(io.prestosql.spi.plan.JoinNode.Type.FULL) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) RIGHT(io.prestosql.spi.plan.JoinNode.Type.RIGHT) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) DistributionType(io.prestosql.spi.plan.JoinNode.DistributionType) INNER(io.prestosql.spi.plan.JoinNode.Type.INNER) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) VarcharType(io.prestosql.spi.type.VarcharType) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) LEFT(io.prestosql.spi.plan.JoinNode.Type.LEFT) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) VarcharType(io.prestosql.spi.type.VarcharType) Symbol(io.prestosql.spi.plan.Symbol) JoinNode(io.prestosql.spi.plan.JoinNode) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) Test(org.testng.annotations.Test)

Example 10 with JOIN_DISTRIBUTION_TYPE

use of io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE in project hetu-core by openlookeng.

the class TestDetermineJoinDistributionType method testFlipAndReplicateWhenOneTableMuchSmallerAndJoinCardinalityUnknown.

@Test
public void testFlipAndReplicateWhenOneTableMuchSmallerAndJoinCardinalityUnknown() {
    int aRows = 100;
    int bRows = 10_000;
    assertDetermineJoinDistributionType().setSystemProperty(JOIN_DISTRIBUTION_TYPE, JoinDistributionType.AUTOMATIC.name()).overrideStats("valuesA", PlanNodeStatsEstimate.builder().setOutputRowCount(aRows).addSymbolStatistics(ImmutableMap.of(new Symbol("A1"), SymbolStatsEstimate.unknown())).build()).overrideStats("valuesB", PlanNodeStatsEstimate.builder().setOutputRowCount(bRows).addSymbolStatistics(ImmutableMap.of(new Symbol("B1"), SymbolStatsEstimate.unknown())).build()).on(p -> p.join(INNER, p.values(new PlanNodeId("valuesA"), aRows, p.symbol("A1", BIGINT)), p.values(new PlanNodeId("valuesB"), bRows, p.symbol("B1", BIGINT)), ImmutableList.of(new JoinNode.EquiJoinClause(p.symbol("A1", BIGINT), p.symbol("B1", BIGINT))), ImmutableList.of(p.symbol("A1", BIGINT), p.symbol("B1", BIGINT)), Optional.empty())).matches(join(INNER, ImmutableList.of(equiJoinClause("B1", "A1")), Optional.empty(), Optional.of(REPLICATED), values(ImmutableMap.of("B1", 0)), values(ImmutableMap.of("A1", 0))));
}
Also used : PlanBuilder.constantExpressions(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.constantExpressions) SymbolStatsEstimate(io.prestosql.cost.SymbolStatsEstimate) REPLICATED(io.prestosql.spi.plan.JoinNode.DistributionType.REPLICATED) PlanMatchPattern.equiJoinClause(io.prestosql.sql.planner.assertions.PlanMatchPattern.equiJoinClause) JoinDistributionType(io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType) PlanMatchPattern.enforceSingleRow(io.prestosql.sql.planner.assertions.PlanMatchPattern.enforceSingleRow) Test(org.testng.annotations.Test) PARTITIONED(io.prestosql.spi.plan.JoinNode.DistributionType.PARTITIONED) PlanBuilder.castToRowExpression(io.prestosql.sql.planner.iterative.rule.test.PlanBuilder.castToRowExpression) PlanMatchPattern.values(io.prestosql.sql.planner.assertions.PlanMatchPattern.values) ImmutableList(com.google.common.collect.ImmutableList) RuleAssert(io.prestosql.sql.planner.iterative.rule.test.RuleAssert) BIGINT(io.prestosql.spi.type.BigintType.BIGINT) Type(io.prestosql.spi.plan.JoinNode.Type) JoinNode(io.prestosql.spi.plan.JoinNode) TaskCountEstimator(io.prestosql.cost.TaskCountEstimator) Symbol(io.prestosql.spi.plan.Symbol) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) AfterClass(org.testng.annotations.AfterClass) PlanMatchPattern.join(io.prestosql.sql.planner.assertions.PlanMatchPattern.join) PlanNodeStatsEstimate(io.prestosql.cost.PlanNodeStatsEstimate) ImmutableMap(com.google.common.collect.ImmutableMap) FULL(io.prestosql.spi.plan.JoinNode.Type.FULL) BeforeClass(org.testng.annotations.BeforeClass) CostComparator(io.prestosql.cost.CostComparator) RIGHT(io.prestosql.spi.plan.JoinNode.Type.RIGHT) VarcharType.createUnboundedVarcharType(io.prestosql.spi.type.VarcharType.createUnboundedVarcharType) DistributionType(io.prestosql.spi.plan.JoinNode.DistributionType) INNER(io.prestosql.spi.plan.JoinNode.Type.INNER) Optional(java.util.Optional) JOIN_MAX_BROADCAST_TABLE_SIZE(io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE) RuleTester(io.prestosql.sql.planner.iterative.rule.test.RuleTester) VarcharType(io.prestosql.spi.type.VarcharType) JOIN_DISTRIBUTION_TYPE(io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE) LEFT(io.prestosql.spi.plan.JoinNode.Type.LEFT) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Symbol(io.prestosql.spi.plan.Symbol) Test(org.testng.annotations.Test)

Aggregations

JOIN_DISTRIBUTION_TYPE (io.prestosql.SystemSessionProperties.JOIN_DISTRIBUTION_TYPE)21 Test (org.testng.annotations.Test)21 ImmutableList (com.google.common.collect.ImmutableList)17 ImmutableMap (com.google.common.collect.ImmutableMap)17 Symbol (io.prestosql.spi.plan.Symbol)17 JoinDistributionType (io.prestosql.sql.analyzer.FeaturesConfig.JoinDistributionType)17 PlanMatchPattern.values (io.prestosql.sql.planner.assertions.PlanMatchPattern.values)17 Optional (java.util.Optional)17 JOIN_MAX_BROADCAST_TABLE_SIZE (io.prestosql.SystemSessionProperties.JOIN_MAX_BROADCAST_TABLE_SIZE)16 CostComparator (io.prestosql.cost.CostComparator)16 PlanNodeStatsEstimate (io.prestosql.cost.PlanNodeStatsEstimate)16 SymbolStatsEstimate (io.prestosql.cost.SymbolStatsEstimate)16 PlanNodeId (io.prestosql.spi.plan.PlanNodeId)16 BIGINT (io.prestosql.spi.type.BigintType.BIGINT)16 VarcharType.createUnboundedVarcharType (io.prestosql.spi.type.VarcharType.createUnboundedVarcharType)16 RuleAssert (io.prestosql.sql.planner.iterative.rule.test.RuleAssert)16 RuleTester (io.prestosql.sql.planner.iterative.rule.test.RuleTester)16 AfterClass (org.testng.annotations.AfterClass)16 BeforeClass (org.testng.annotations.BeforeClass)16 TaskCountEstimator (io.prestosql.cost.TaskCountEstimator)14