Search in sources :

Example 1 with ExpressionCompiler

use of io.trino.sql.gen.ExpressionCompiler in project trino by trinodb.

the class TestFilterAndProjectOperator method testMergeOutput.

@Test
public void testMergeOutput() {
    List<Page> input = rowPagesBuilder(VARCHAR, BIGINT).addSequencePage(100, 0, 0).addSequencePage(100, 0, 0).addSequencePage(100, 0, 0).addSequencePage(100, 0, 0).build();
    TestingFunctionResolution functionResolution = new TestingFunctionResolution();
    RowExpression filter = call(functionResolution.resolveOperator(EQUAL, ImmutableList.of(BIGINT, BIGINT)), field(1, BIGINT), constant(10L, BIGINT));
    ExpressionCompiler compiler = functionResolution.getExpressionCompiler();
    Supplier<PageProcessor> processor = compiler.compilePageProcessor(Optional.of(filter), ImmutableList.of(field(1, BIGINT)));
    OperatorFactory operatorFactory = FilterAndProjectOperator.createOperatorFactory(0, new PlanNodeId("test"), processor, ImmutableList.of(BIGINT), DataSize.of(64, KILOBYTE), 2);
    List<Page> expected = rowPagesBuilder(BIGINT).row(10L).row(10L).row(10L).row(10L).build();
    assertOperatorEquals(operatorFactory, ImmutableList.of(BIGINT), driverContext, input, expected);
}
Also used : TestingFunctionResolution(io.trino.metadata.TestingFunctionResolution) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) PageProcessor(io.trino.operator.project.PageProcessor) RowExpression(io.trino.sql.relational.RowExpression) Page(io.trino.spi.Page) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) Test(org.testng.annotations.Test)

Example 2 with ExpressionCompiler

use of io.trino.sql.gen.ExpressionCompiler in project trino by trinodb.

the class TaskTestUtils method createTestingPlanner.

public static LocalExecutionPlanner createTestingPlanner() {
    PageSourceManager pageSourceManager = new PageSourceManager();
    pageSourceManager.addConnectorPageSourceProvider(CONNECTOR_ID, new TestingPageSourceProvider());
    // we don't start the finalizer so nothing will be collected, which is ok for a test
    FinalizerService finalizerService = new FinalizerService();
    BlockTypeOperators blockTypeOperators = new BlockTypeOperators(PLANNER_CONTEXT.getTypeOperators());
    NodeScheduler nodeScheduler = new NodeScheduler(new UniformNodeSelectorFactory(new InMemoryNodeManager(), new NodeSchedulerConfig().setIncludeCoordinator(true), new NodeTaskMap(finalizerService)));
    NodePartitioningManager nodePartitioningManager = new NodePartitioningManager(nodeScheduler, blockTypeOperators);
    PageFunctionCompiler pageFunctionCompiler = new PageFunctionCompiler(PLANNER_CONTEXT.getFunctionManager(), 0);
    return new LocalExecutionPlanner(PLANNER_CONTEXT, createTestingTypeAnalyzer(PLANNER_CONTEXT), Optional.empty(), pageSourceManager, new IndexManager(), nodePartitioningManager, new PageSinkManager(), new MockDirectExchangeClientSupplier(), new ExpressionCompiler(PLANNER_CONTEXT.getFunctionManager(), pageFunctionCompiler), pageFunctionCompiler, new JoinFilterFunctionCompiler(PLANNER_CONTEXT.getFunctionManager()), new IndexJoinLookupStats(), new TaskManagerConfig(), new GenericSpillerFactory((types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }), (types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, (types, partitionFunction, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, new PagesIndex.TestingFactory(false), new JoinCompiler(PLANNER_CONTEXT.getTypeOperators()), new TrinoOperatorFactories(), new OrderingCompiler(PLANNER_CONTEXT.getTypeOperators()), new DynamicFilterConfig(), blockTypeOperators, new TableExecuteContextManager(), new ExchangeManagerRegistry(new ExchangeHandleResolver()));
}
Also used : PlanFragment(io.trino.sql.planner.PlanFragment) JoinFilterFunctionCompiler(io.trino.sql.gen.JoinFilterFunctionCompiler) ObjectMapperProvider(io.airlift.json.ObjectMapperProvider) CatalogName(io.trino.connector.CatalogName) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) FinalizerService(io.trino.util.FinalizerService) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) TEST_SESSION(io.trino.SessionTestUtils.TEST_SESSION) SINGLE_DISTRIBUTION(io.trino.sql.planner.SystemPartitioningHandle.SINGLE_DISTRIBUTION) TableScanNode(io.trino.sql.planner.plan.TableScanNode) ImmutableMap(com.google.common.collect.ImmutableMap) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) PageSinkManager(io.trino.split.PageSinkManager) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) TrinoOperatorFactories(io.trino.operator.TrinoOperatorFactories) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) IndexJoinLookupStats(io.trino.operator.index.IndexJoinLookupStats) SplitMonitor(io.trino.event.SplitMonitor) MockDirectExchangeClientSupplier(io.trino.execution.TestSqlTaskManager.MockDirectExchangeClientSupplier) List(java.util.List) BIGINT(io.trino.spi.type.BigintType.BIGINT) Split(io.trino.metadata.Split) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) Optional(java.util.Optional) PlanFragmentId(io.trino.sql.planner.plan.PlanFragmentId) TestingColumnHandle(io.trino.testing.TestingMetadata.TestingColumnHandle) TypeAnalyzer.createTestingTypeAnalyzer(io.trino.sql.planner.TypeAnalyzer.createTestingTypeAnalyzer) OrderingCompiler(io.trino.sql.gen.OrderingCompiler) NodePartitioningManager(io.trino.sql.planner.NodePartitioningManager) JoinCompiler(io.trino.sql.gen.JoinCompiler) ExchangeHandleResolver(io.trino.metadata.ExchangeHandleResolver) Partitioning(io.trino.sql.planner.Partitioning) PartitioningScheme(io.trino.sql.planner.PartitioningScheme) StatsAndCosts(io.trino.cost.StatsAndCosts) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) VARCHAR(io.trino.spi.type.VarcharType.VARCHAR) ImmutableList(com.google.common.collect.ImmutableList) TestingSplit(io.trino.testing.TestingSplit) StageExecutionDescriptor.ungroupedExecution(io.trino.operator.StageExecutionDescriptor.ungroupedExecution) EventListenerConfig(io.trino.eventlistener.EventListenerConfig) IndexManager(io.trino.index.IndexManager) PagesIndex(io.trino.operator.PagesIndex) Symbol(io.trino.sql.planner.Symbol) BlockTypeOperators(io.trino.type.BlockTypeOperators) GenericSpillerFactory(io.trino.spiller.GenericSpillerFactory) PageSourceManager(io.trino.split.PageSourceManager) LocalExecutionPlanner(io.trino.sql.planner.LocalExecutionPlanner) PLANNER_CONTEXT(io.trino.sql.planner.TestingPlannerContext.PLANNER_CONTEXT) EventListenerManager(io.trino.eventlistener.EventListenerManager) TEST_TABLE_HANDLE(io.trino.testing.TestingHandles.TEST_TABLE_HANDLE) OutputBuffers(io.trino.execution.buffer.OutputBuffers) SOURCE_DISTRIBUTION(io.trino.sql.planner.SystemPartitioningHandle.SOURCE_DISTRIBUTION) ExchangeManagerRegistry(io.trino.exchange.ExchangeManagerRegistry) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) MockDirectExchangeClientSupplier(io.trino.execution.TestSqlTaskManager.MockDirectExchangeClientSupplier) TrinoOperatorFactories(io.trino.operator.TrinoOperatorFactories) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) PagesIndex(io.trino.operator.PagesIndex) PageSourceManager(io.trino.split.PageSourceManager) NodePartitioningManager(io.trino.sql.planner.NodePartitioningManager) ExchangeHandleResolver(io.trino.metadata.ExchangeHandleResolver) BlockTypeOperators(io.trino.type.BlockTypeOperators) OrderingCompiler(io.trino.sql.gen.OrderingCompiler) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) GenericSpillerFactory(io.trino.spiller.GenericSpillerFactory) PageSinkManager(io.trino.split.PageSinkManager) JoinCompiler(io.trino.sql.gen.JoinCompiler) LocalExecutionPlanner(io.trino.sql.planner.LocalExecutionPlanner) IndexJoinLookupStats(io.trino.operator.index.IndexJoinLookupStats) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) JoinFilterFunctionCompiler(io.trino.sql.gen.JoinFilterFunctionCompiler) ExchangeManagerRegistry(io.trino.exchange.ExchangeManagerRegistry) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) IndexManager(io.trino.index.IndexManager) FinalizerService(io.trino.util.FinalizerService) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler)

Example 3 with ExpressionCompiler

use of io.trino.sql.gen.ExpressionCompiler in project trino by trinodb.

the class TestScanFilterAndProjectOperator method testPageYield.

@Test
public void testPageYield() {
    int totalRows = 1000;
    Page input = SequencePageBuilder.createSequencePage(ImmutableList.of(BIGINT), totalRows, 1);
    DriverContext driverContext = newDriverContext();
    // 20 columns; each column is associated with a function that will force yield per projection
    int totalColumns = 20;
    ImmutableList.Builder<SqlScalarFunction> functions = ImmutableList.builder();
    for (int i = 0; i < totalColumns; i++) {
        functions.add(new GenericLongFunction("page_col" + i, value -> {
            driverContext.getYieldSignal().forceYieldForTesting();
            return value;
        }));
    }
    functionAssertions.addFunctions(new InternalFunctionBundle(functions.build()));
    // match each column with a projection
    ExpressionCompiler expressionCompiler = new ExpressionCompiler(functionAssertions.getFunctionManager(), new PageFunctionCompiler(functionAssertions.getFunctionManager(), 0));
    ImmutableList.Builder<RowExpression> projections = ImmutableList.builder();
    for (int i = 0; i < totalColumns; i++) {
        projections.add(call(functionAssertions.getMetadata().resolveFunction(session, QualifiedName.of("generic_long_page_col" + i), fromTypes(BIGINT)), field(0, BIGINT)));
    }
    Supplier<CursorProcessor> cursorProcessor = expressionCompiler.compileCursorProcessor(Optional.empty(), projections.build(), "key");
    Supplier<PageProcessor> pageProcessor = expressionCompiler.compilePageProcessor(Optional.empty(), projections.build(), MAX_BATCH_SIZE);
    ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory factory = new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(0, new PlanNodeId("test"), new PlanNodeId("0"), (session, split, table, columns, dynamicFilter) -> new FixedPageSource(ImmutableList.of(input)), cursorProcessor, pageProcessor, TEST_TABLE_HANDLE, ImmutableList.of(), DynamicFilter.EMPTY, ImmutableList.of(BIGINT), DataSize.ofBytes(0), 0);
    SourceOperator operator = factory.createOperator(driverContext);
    operator.addSplit(new Split(new CatalogName("test"), TestingSplit.createLocalSplit(), Lifespan.taskWide()));
    operator.noMoreSplits();
    // exactly 20 blocks (one for each column) and the PageProcessor will be able to create a Page out of it.
    for (int i = 1; i <= totalRows * totalColumns; i++) {
        driverContext.getYieldSignal().setWithDelay(SECONDS.toNanos(1000), driverContext.getYieldExecutor());
        Page page = operator.getOutput();
        if (i == totalColumns) {
            assertNotNull(page);
            assertEquals(page.getPositionCount(), totalRows);
            assertEquals(page.getChannelCount(), totalColumns);
            for (int j = 0; j < totalColumns; j++) {
                assertEquals(toValues(BIGINT, page.getBlock(j)), toValues(BIGINT, input.getBlock(0)));
            }
        } else {
            assertNull(page);
        }
        driverContext.getYieldSignal().reset();
    }
}
Also used : MaterializedResult(io.trino.testing.MaterializedResult) TypeSignatureProvider.fromTypes(io.trino.sql.analyzer.TypeSignatureProvider.fromTypes) BlockAssertions(io.trino.block.BlockAssertions) Test(org.testng.annotations.Test) Expressions.field(io.trino.sql.relational.Expressions.field) LazyPagePageProjection(io.trino.operator.project.TestPageProcessor.LazyPagePageProjection) SequencePageBuilder(io.trino.SequencePageBuilder) LazyBlock(io.trino.spi.block.LazyBlock) CatalogName(io.trino.connector.CatalogName) Block(io.trino.spi.block.Block) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) Executors.newScheduledThreadPool(java.util.concurrent.Executors.newScheduledThreadPool) CursorProcessor(io.trino.operator.project.CursorProcessor) TEST_SESSION(io.trino.SessionTestUtils.TEST_SESSION) PageRecordSet(io.trino.operator.index.PageRecordSet) RowPagesBuilder.rowPagesBuilder(io.trino.RowPagesBuilder.rowPagesBuilder) KILOBYTE(io.airlift.units.DataSize.Unit.KILOBYTE) EQUAL(io.trino.spi.function.OperatorType.EQUAL) ConnectorPageSource(io.trino.spi.connector.ConnectorPageSource) Assert.assertEquals(io.trino.testing.assertions.Assert.assertEquals) Expressions.constant(io.trino.sql.relational.Expressions.constant) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) SelectAllFilter(io.trino.operator.project.TestPageProcessor.SelectAllFilter) Assert.assertNotNull(org.testng.Assert.assertNotNull) AbstractTestFunctions(io.trino.operator.scalar.AbstractTestFunctions) DataSize(io.airlift.units.DataSize) List(java.util.List) FixedPageSource(io.trino.spi.connector.FixedPageSource) BIGINT(io.trino.spi.type.BigintType.BIGINT) PageAssertions.assertPageEquals(io.trino.operator.PageAssertions.assertPageEquals) Split(io.trino.metadata.Split) DynamicFilter(io.trino.spi.connector.DynamicFilter) Optional(java.util.Optional) BlockAssertions.toValues(io.trino.block.BlockAssertions.toValues) Assert.assertNull(org.testng.Assert.assertNull) Page(io.trino.spi.Page) Supplier(java.util.function.Supplier) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) MAX_BATCH_SIZE(io.trino.operator.project.PageProcessor.MAX_BATCH_SIZE) VARCHAR(io.trino.spi.type.VarcharType.VARCHAR) ImmutableList(com.google.common.collect.ImmutableList) Threads.daemonThreadsNamed(io.airlift.concurrent.Threads.daemonThreadsNamed) RecordPageSource(io.trino.spi.connector.RecordPageSource) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) PageProcessor(io.trino.operator.project.PageProcessor) TestingSplit(io.trino.testing.TestingSplit) Lifespan(io.trino.execution.Lifespan) ExecutorService(java.util.concurrent.ExecutorService) TestingTaskContext.createTaskContext(io.trino.testing.TestingTaskContext.createTaskContext) AfterClass(org.testng.annotations.AfterClass) SqlScalarFunction(io.trino.metadata.SqlScalarFunction) FunctionManager(io.trino.metadata.FunctionManager) QualifiedName(io.trino.sql.tree.QualifiedName) OperatorAssertion.toMaterializedResult(io.trino.operator.OperatorAssertion.toMaterializedResult) Expressions.call(io.trino.sql.relational.Expressions.call) RowExpression(io.trino.sql.relational.RowExpression) InternalFunctionBundle(io.trino.metadata.InternalFunctionBundle) TEST_TABLE_HANDLE(io.trino.testing.TestingHandles.TEST_TABLE_HANDLE) Executors.newCachedThreadPool(java.util.concurrent.Executors.newCachedThreadPool) Assert.assertTrue(org.testng.Assert.assertTrue) SECONDS(java.util.concurrent.TimeUnit.SECONDS) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) CursorProcessor(io.trino.operator.project.CursorProcessor) ImmutableList(com.google.common.collect.ImmutableList) Page(io.trino.spi.Page) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) PageProcessor(io.trino.operator.project.PageProcessor) InternalFunctionBundle(io.trino.metadata.InternalFunctionBundle) RowExpression(io.trino.sql.relational.RowExpression) FixedPageSource(io.trino.spi.connector.FixedPageSource) SqlScalarFunction(io.trino.metadata.SqlScalarFunction) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) CatalogName(io.trino.connector.CatalogName) Split(io.trino.metadata.Split) TestingSplit(io.trino.testing.TestingSplit) Test(org.testng.annotations.Test)

Example 4 with ExpressionCompiler

use of io.trino.sql.gen.ExpressionCompiler in project trino by trinodb.

the class TestScanFilterAndProjectOperator method testRecordCursorYield.

@Test
public void testRecordCursorYield() {
    // create a generic long function that yields for projection on every row
    // verify we will yield #row times totally
    // create a table with 15 rows
    int length = 15;
    Page input = SequencePageBuilder.createSequencePage(ImmutableList.of(BIGINT), length, 0);
    DriverContext driverContext = newDriverContext();
    // set up generic long function with a callback to force yield
    functionAssertions.addFunctions(new InternalFunctionBundle(new GenericLongFunction("record_cursor", value -> {
        driverContext.getYieldSignal().forceYieldForTesting();
        return value;
    })));
    FunctionManager functionManager = functionAssertions.getFunctionManager();
    ExpressionCompiler expressionCompiler = new ExpressionCompiler(functionManager, new PageFunctionCompiler(functionManager, 0));
    List<RowExpression> projections = ImmutableList.of(call(functionAssertions.getMetadata().resolveFunction(session, QualifiedName.of("generic_long_record_cursor"), fromTypes(BIGINT)), field(0, BIGINT)));
    Supplier<CursorProcessor> cursorProcessor = expressionCompiler.compileCursorProcessor(Optional.empty(), projections, "key");
    Supplier<PageProcessor> pageProcessor = expressionCompiler.compilePageProcessor(Optional.empty(), projections);
    ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory factory = new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(0, new PlanNodeId("test"), new PlanNodeId("0"), (session, split, table, columns, dynamicFilter) -> new RecordPageSource(new PageRecordSet(ImmutableList.of(BIGINT), input)), cursorProcessor, pageProcessor, TEST_TABLE_HANDLE, ImmutableList.of(), DynamicFilter.EMPTY, ImmutableList.of(BIGINT), DataSize.ofBytes(0), 0);
    SourceOperator operator = factory.createOperator(driverContext);
    operator.addSplit(new Split(new CatalogName("test"), TestingSplit.createLocalSplit(), Lifespan.taskWide()));
    operator.noMoreSplits();
    // start driver; get null value due to yield for the first 15 times
    for (int i = 0; i < length; i++) {
        driverContext.getYieldSignal().setWithDelay(SECONDS.toNanos(1000), driverContext.getYieldExecutor());
        assertNull(operator.getOutput());
        driverContext.getYieldSignal().reset();
    }
    // the 16th yield is not going to prevent the operator from producing a page
    driverContext.getYieldSignal().setWithDelay(SECONDS.toNanos(1000), driverContext.getYieldExecutor());
    Page output = operator.getOutput();
    driverContext.getYieldSignal().reset();
    assertNotNull(output);
    assertEquals(toValues(BIGINT, output.getBlock(0)), toValues(BIGINT, input.getBlock(0)));
}
Also used : PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) CursorProcessor(io.trino.operator.project.CursorProcessor) RowExpression(io.trino.sql.relational.RowExpression) Page(io.trino.spi.Page) PageRecordSet(io.trino.operator.index.PageRecordSet) RecordPageSource(io.trino.spi.connector.RecordPageSource) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) PageProcessor(io.trino.operator.project.PageProcessor) InternalFunctionBundle(io.trino.metadata.InternalFunctionBundle) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) CatalogName(io.trino.connector.CatalogName) Split(io.trino.metadata.Split) TestingSplit(io.trino.testing.TestingSplit) FunctionManager(io.trino.metadata.FunctionManager) Test(org.testng.annotations.Test)

Example 5 with ExpressionCompiler

use of io.trino.sql.gen.ExpressionCompiler in project trino by trinodb.

the class BenchmarkEqualsConjunctsOperator method setup.

@Setup
public void setup() {
    ExpressionCompiler expressionCompiler = functionResolution.getExpressionCompiler();
    RowExpression projection = generateComplexComparisonProjection(FIELDS_COUNT, COMPARISONS_COUNT);
    compiledProcessor = expressionCompiler.compilePageProcessor(Optional.empty(), ImmutableList.of(projection)).get();
}
Also used : RowExpression(io.trino.sql.relational.RowExpression) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) Setup(org.openjdk.jmh.annotations.Setup)

Aggregations

ExpressionCompiler (io.trino.sql.gen.ExpressionCompiler)7 PlanNodeId (io.trino.sql.planner.plan.PlanNodeId)6 PageProcessor (io.trino.operator.project.PageProcessor)5 RowExpression (io.trino.sql.relational.RowExpression)5 Page (io.trino.spi.Page)4 PageFunctionCompiler (io.trino.sql.gen.PageFunctionCompiler)4 CatalogName (io.trino.connector.CatalogName)3 Split (io.trino.metadata.Split)3 Test (org.testng.annotations.Test)3 ImmutableList (com.google.common.collect.ImmutableList)2 TEST_SESSION (io.trino.SessionTestUtils.TEST_SESSION)2 FunctionManager (io.trino.metadata.FunctionManager)2 InternalFunctionBundle (io.trino.metadata.InternalFunctionBundle)2 TestingFunctionResolution (io.trino.metadata.TestingFunctionResolution)2 PageRecordSet (io.trino.operator.index.PageRecordSet)2 CursorProcessor (io.trino.operator.project.CursorProcessor)2 RecordPageSource (io.trino.spi.connector.RecordPageSource)2 MaterializedResult (io.trino.testing.MaterializedResult)2 TestingSplit (io.trino.testing.TestingSplit)2 ImmutableMap (com.google.common.collect.ImmutableMap)1