Search in sources :

Example 1 with StatisticAggregationsDescriptor

use of io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor in project hetu-core by openlookeng.

the class TestExternalFunctionPushDownChecker method testTableWriterNodeWithExternalCall.

@Test(expectedExceptions = ExternalFunctionPushDownChecker.IllegalExternalFunctionUsageException.class, expectedExceptionsMessageRegExp = "The external function jdbc.v1.foo does not support to push down to data source for this query.")
public void testTableWriterNodeWithExternalCall() {
    TableWriterNode.DeleteTarget deleteTarget = new TableWriterNode.DeleteTarget(tableHandle, new SchemaTableName("sch", "tab"));
    PlanNode node = new TableWriterNode(idAllocator.getNextId(), builder.values(), deleteTarget, columnA, columnB, ImmutableList.of(columnA, columnB), ImmutableList.of("a", "b"), Optional.empty(), Optional.of(new StatisticAggregations(ImmutableMap.of(columnA, new AggregationNode.Aggregation(externalFooCall1, externalFooCall1.getArguments(), false, Optional.empty(), Optional.empty(), Optional.empty())), ImmutableList.of(columnA))), Optional.of(new StatisticAggregationsDescriptor<>(ImmutableMap.of(), ImmutableMap.of(), ImmutableMap.of())));
    validatePlan(node);
}
Also used : PlanNode(io.prestosql.spi.plan.PlanNode) StatisticAggregationsDescriptor(io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor) TableWriterNode(io.prestosql.sql.planner.plan.TableWriterNode) SchemaTableName(io.prestosql.spi.connector.SchemaTableName) StatisticAggregations(io.prestosql.sql.planner.plan.StatisticAggregations) Test(org.testng.annotations.Test) BasePlanTest(io.prestosql.sql.planner.assertions.BasePlanTest)

Example 2 with StatisticAggregationsDescriptor

use of io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor in project hetu-core by openlookeng.

the class TestExternalFunctionPushDownChecker method testTableWriterNode.

@Test
public void testTableWriterNode() {
    TableWriterNode.DeleteTarget deleteTarget = new TableWriterNode.DeleteTarget(tableHandle, new SchemaTableName("sch", "tab"));
    PlanNode node = new TableWriterNode(idAllocator.getNextId(), builder.values(), deleteTarget, columnA, columnB, ImmutableList.of(columnA, columnB), ImmutableList.of("a", "b"), Optional.empty(), Optional.of(new StatisticAggregations(ImmutableMap.of(columnA, new AggregationNode.Aggregation(sumCall, sumCall.getArguments(), false, Optional.empty(), Optional.empty(), Optional.empty())), ImmutableList.of(columnA))), Optional.of(new StatisticAggregationsDescriptor<>(ImmutableMap.of(), ImmutableMap.of(), ImmutableMap.of())));
    validatePlan(node);
}
Also used : PlanNode(io.prestosql.spi.plan.PlanNode) StatisticAggregationsDescriptor(io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor) TableWriterNode(io.prestosql.sql.planner.plan.TableWriterNode) SchemaTableName(io.prestosql.spi.connector.SchemaTableName) StatisticAggregations(io.prestosql.sql.planner.plan.StatisticAggregations) Test(org.testng.annotations.Test) BasePlanTest(io.prestosql.sql.planner.assertions.BasePlanTest)

Example 3 with StatisticAggregationsDescriptor

use of io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor in project hetu-core by openlookeng.

the class TestExternalFunctionPushDownChecker method testVacuumTableNode.

@Test
public void testVacuumTableNode() {
    TableWriterNode.DeleteTarget deleteTarget = new TableWriterNode.DeleteTarget(tableHandle, new SchemaTableName("sch", "tab"));
    PlanNode node = new VacuumTableNode(idAllocator.getNextId(), tableHandle, deleteTarget, columnA, columnB, "p1", false, ImmutableList.of(), Optional.of(new StatisticAggregations(ImmutableMap.of(columnA, new AggregationNode.Aggregation(sumCall, sumCall.getArguments(), false, Optional.empty(), Optional.empty(), Optional.empty())), ImmutableList.of(columnA))), Optional.of(new StatisticAggregationsDescriptor<>(ImmutableMap.of(), ImmutableMap.of(), ImmutableMap.of())));
    validatePlan(node);
}
Also used : PlanNode(io.prestosql.spi.plan.PlanNode) VacuumTableNode(io.prestosql.sql.planner.plan.VacuumTableNode) StatisticAggregationsDescriptor(io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor) TableWriterNode(io.prestosql.sql.planner.plan.TableWriterNode) SchemaTableName(io.prestosql.spi.connector.SchemaTableName) StatisticAggregations(io.prestosql.sql.planner.plan.StatisticAggregations) Test(org.testng.annotations.Test) BasePlanTest(io.prestosql.sql.planner.assertions.BasePlanTest)

Example 4 with StatisticAggregationsDescriptor

use of io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor in project hetu-core by openlookeng.

the class TestTableFinishOperator method testStatisticsAggregationSnapshot.

@Test
public void testStatisticsAggregationSnapshot() throws Exception {
    TestTableFinisher tableFinisher = new TestTableFinisher();
    ColumnStatisticMetadata statisticMetadata = new ColumnStatisticMetadata("column", MAX_VALUE);
    StatisticAggregationsDescriptor<Integer> descriptor = new StatisticAggregationsDescriptor<>(ImmutableMap.of(), ImmutableMap.of(), ImmutableMap.of(statisticMetadata, 0));
    Session session = testSessionBuilder().setSystemProperty("statistics_cpu_timer_enabled", "true").build();
    TableFinishOperatorFactory operatorFactory = new TableFinishOperatorFactory(0, new PlanNodeId("node"), tableFinisher, new AggregationOperator.AggregationOperatorFactory(1, new PlanNodeId("test"), AggregationNode.Step.SINGLE, ImmutableList.of(LONG_MAX.bind(ImmutableList.of(2), Optional.empty())), true), descriptor, session);
    DriverContext driverContext = createTaskContext(scheduledExecutor, scheduledExecutor, session).addPipelineContext(0, true, true, false).addDriverContext();
    TableFinishOperator operator = (TableFinishOperator) operatorFactory.createOperator(driverContext);
    List<Type> inputTypes = ImmutableList.of(BIGINT, VARBINARY, BIGINT);
    operator.addInput(rowPagesBuilder(inputTypes).row(4, null, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(5, null, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, new byte[] { 1 }, null).build().get(0));
    Object snapshot = operator.capture(operator.getOperatorContext().getDriverContext().getSerde());
    Map<String, Object> snapshotMapping = (Map<String, Object>) SnapshotTestUtil.toFullSnapshotMapping(snapshot);
    assertEquals(snapshotMapping, createExpectedMapping());
    operator.addInput(rowPagesBuilder(inputTypes).row(null, new byte[] { 2 }, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 6).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 7).build().get(0));
    operator.restore(snapshot, operator.getOperatorContext().getDriverContext().getSerde());
    operator.addInput(rowPagesBuilder(inputTypes).row(null, new byte[] { 2 }, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 6).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 7).build().get(0));
    assertThat(driverContext.getSystemMemoryUsage()).isGreaterThan(0);
    assertEquals(driverContext.getMemoryUsage(), 0);
    assertTrue(operator.isBlocked().isDone());
    assertTrue(operator.needsInput());
    operator.finish();
    assertFalse(operator.isFinished());
    assertNull(operator.getOutput());
    List<Type> outputTypes = ImmutableList.of(BIGINT);
    assertPageEquals(outputTypes, operator.getOutput(), rowPagesBuilder(outputTypes).row(9).build().get(0));
    assertTrue(operator.isBlocked().isDone());
    assertFalse(operator.needsInput());
    assertTrue(operator.isFinished());
    operator.close();
    assertEquals(tableFinisher.getFragments(), ImmutableList.of(Slices.wrappedBuffer(new byte[] { 1 }), Slices.wrappedBuffer(new byte[] { 2 })));
    assertEquals(tableFinisher.getComputedStatistics().size(), 1);
    assertEquals(getOnlyElement(tableFinisher.getComputedStatistics()).getColumnStatistics().size(), 1);
    Block expectedStatisticsBlock = new LongArrayBlockBuilder(null, 1).writeLong(7).closeEntry().build();
    assertBlockEquals(BIGINT, getOnlyElement(tableFinisher.getComputedStatistics()).getColumnStatistics().get(statisticMetadata), expectedStatisticsBlock);
    TableFinishInfo tableFinishInfo = operator.getInfo();
    assertThat(tableFinishInfo.getStatisticsWallTime().getValue(NANOSECONDS)).isGreaterThan(0);
    assertThat(tableFinishInfo.getStatisticsCpuTime().getValue(NANOSECONDS)).isGreaterThan(0);
    assertEquals(driverContext.getSystemMemoryUsage(), 0);
    assertEquals(driverContext.getMemoryUsage(), 0);
}
Also used : TableFinishOperatorFactory(io.prestosql.operator.TableFinishOperator.TableFinishOperatorFactory) ColumnStatisticMetadata(io.prestosql.spi.statistics.ColumnStatisticMetadata) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Type(io.prestosql.spi.type.Type) Block(io.prestosql.spi.block.Block) LongArrayBlockBuilder(io.prestosql.spi.block.LongArrayBlockBuilder) StatisticAggregationsDescriptor(io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap) Session(io.prestosql.Session) Test(org.testng.annotations.Test)

Example 5 with StatisticAggregationsDescriptor

use of io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor in project hetu-core by openlookeng.

the class TestTableFinishOperator method testStatisticsAggregation.

@Test
public void testStatisticsAggregation() throws Exception {
    TestTableFinisher tableFinisher = new TestTableFinisher();
    ColumnStatisticMetadata statisticMetadata = new ColumnStatisticMetadata("column", MAX_VALUE);
    StatisticAggregationsDescriptor<Integer> descriptor = new StatisticAggregationsDescriptor<>(ImmutableMap.of(), ImmutableMap.of(), ImmutableMap.of(statisticMetadata, 0));
    Session session = testSessionBuilder().setSystemProperty("statistics_cpu_timer_enabled", "true").build();
    TableFinishOperatorFactory operatorFactory = new TableFinishOperatorFactory(0, new PlanNodeId("node"), tableFinisher, new AggregationOperator.AggregationOperatorFactory(1, new PlanNodeId("test"), AggregationNode.Step.SINGLE, ImmutableList.of(LONG_MAX.bind(ImmutableList.of(2), Optional.empty())), true), descriptor, session);
    DriverContext driverContext = createTaskContext(scheduledExecutor, scheduledExecutor, session).addPipelineContext(0, true, true, false).addDriverContext();
    TableFinishOperator operator = (TableFinishOperator) operatorFactory.createOperator(driverContext);
    List<Type> inputTypes = ImmutableList.of(BIGINT, VARBINARY, BIGINT);
    operator.addInput(rowPagesBuilder(inputTypes).row(4, null, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(5, null, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, new byte[] { 1 }, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, new byte[] { 2 }, null).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 6).build().get(0));
    operator.addInput(rowPagesBuilder(inputTypes).row(null, null, 7).build().get(0));
    assertThat(driverContext.getSystemMemoryUsage()).isGreaterThan(0);
    assertEquals(driverContext.getMemoryUsage(), 0);
    assertTrue(operator.isBlocked().isDone());
    assertTrue(operator.needsInput());
    operator.finish();
    assertFalse(operator.isFinished());
    assertNull(operator.getOutput());
    List<Type> outputTypes = ImmutableList.of(BIGINT);
    assertPageEquals(outputTypes, operator.getOutput(), rowPagesBuilder(outputTypes).row(9).build().get(0));
    assertTrue(operator.isBlocked().isDone());
    assertFalse(operator.needsInput());
    assertTrue(operator.isFinished());
    operator.close();
    assertEquals(tableFinisher.getFragments(), ImmutableList.of(Slices.wrappedBuffer(new byte[] { 1 }), Slices.wrappedBuffer(new byte[] { 2 })));
    assertEquals(tableFinisher.getComputedStatistics().size(), 1);
    assertEquals(getOnlyElement(tableFinisher.getComputedStatistics()).getColumnStatistics().size(), 1);
    Block expectedStatisticsBlock = new LongArrayBlockBuilder(null, 1).writeLong(7).closeEntry().build();
    assertBlockEquals(BIGINT, getOnlyElement(tableFinisher.getComputedStatistics()).getColumnStatistics().get(statisticMetadata), expectedStatisticsBlock);
    TableFinishInfo tableFinishInfo = operator.getInfo();
    assertThat(tableFinishInfo.getStatisticsWallTime().getValue(NANOSECONDS)).isGreaterThan(0);
    assertThat(tableFinishInfo.getStatisticsCpuTime().getValue(NANOSECONDS)).isGreaterThan(0);
    assertEquals(driverContext.getSystemMemoryUsage(), 0);
    assertEquals(driverContext.getMemoryUsage(), 0);
}
Also used : TableFinishOperatorFactory(io.prestosql.operator.TableFinishOperator.TableFinishOperatorFactory) ColumnStatisticMetadata(io.prestosql.spi.statistics.ColumnStatisticMetadata) PlanNodeId(io.prestosql.spi.plan.PlanNodeId) Type(io.prestosql.spi.type.Type) Block(io.prestosql.spi.block.Block) LongArrayBlockBuilder(io.prestosql.spi.block.LongArrayBlockBuilder) StatisticAggregationsDescriptor(io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor) Session(io.prestosql.Session) Test(org.testng.annotations.Test)

Aggregations

StatisticAggregationsDescriptor (io.prestosql.sql.planner.plan.StatisticAggregationsDescriptor)9 Test (org.testng.annotations.Test)8 StatisticAggregations (io.prestosql.sql.planner.plan.StatisticAggregations)7 SchemaTableName (io.prestosql.spi.connector.SchemaTableName)6 PlanNode (io.prestosql.spi.plan.PlanNode)6 BasePlanTest (io.prestosql.sql.planner.assertions.BasePlanTest)6 TableWriterNode (io.prestosql.sql.planner.plan.TableWriterNode)6 TableFinishNode (io.prestosql.sql.planner.plan.TableFinishNode)3 VacuumTableNode (io.prestosql.sql.planner.plan.VacuumTableNode)3 Session (io.prestosql.Session)2 TableFinishOperatorFactory (io.prestosql.operator.TableFinishOperator.TableFinishOperatorFactory)2 Block (io.prestosql.spi.block.Block)2 LongArrayBlockBuilder (io.prestosql.spi.block.LongArrayBlockBuilder)2 PlanNodeId (io.prestosql.spi.plan.PlanNodeId)2 ColumnStatisticMetadata (io.prestosql.spi.statistics.ColumnStatisticMetadata)2 Type (io.prestosql.spi.type.Type)2 ImmutableMap (com.google.common.collect.ImmutableMap)1 Symbol (io.prestosql.spi.plan.Symbol)1 TableStatisticAggregation (io.prestosql.sql.planner.StatisticsAggregationPlanner.TableStatisticAggregation)1 HashMap (java.util.HashMap)1