Search in sources :

Example 1 with FinalizerService

use of io.trino.util.FinalizerService in project trino by trinodb.

the class TaskTestUtils method createTestingPlanner.

public static LocalExecutionPlanner createTestingPlanner() {
    PageSourceManager pageSourceManager = new PageSourceManager();
    pageSourceManager.addConnectorPageSourceProvider(CONNECTOR_ID, new TestingPageSourceProvider());
    // we don't start the finalizer so nothing will be collected, which is ok for a test
    FinalizerService finalizerService = new FinalizerService();
    BlockTypeOperators blockTypeOperators = new BlockTypeOperators(PLANNER_CONTEXT.getTypeOperators());
    NodeScheduler nodeScheduler = new NodeScheduler(new UniformNodeSelectorFactory(new InMemoryNodeManager(), new NodeSchedulerConfig().setIncludeCoordinator(true), new NodeTaskMap(finalizerService)));
    NodePartitioningManager nodePartitioningManager = new NodePartitioningManager(nodeScheduler, blockTypeOperators);
    PageFunctionCompiler pageFunctionCompiler = new PageFunctionCompiler(PLANNER_CONTEXT.getFunctionManager(), 0);
    return new LocalExecutionPlanner(PLANNER_CONTEXT, createTestingTypeAnalyzer(PLANNER_CONTEXT), Optional.empty(), pageSourceManager, new IndexManager(), nodePartitioningManager, new PageSinkManager(), new MockDirectExchangeClientSupplier(), new ExpressionCompiler(PLANNER_CONTEXT.getFunctionManager(), pageFunctionCompiler), pageFunctionCompiler, new JoinFilterFunctionCompiler(PLANNER_CONTEXT.getFunctionManager()), new IndexJoinLookupStats(), new TaskManagerConfig(), new GenericSpillerFactory((types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }), (types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, (types, partitionFunction, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, new PagesIndex.TestingFactory(false), new JoinCompiler(PLANNER_CONTEXT.getTypeOperators()), new TrinoOperatorFactories(), new OrderingCompiler(PLANNER_CONTEXT.getTypeOperators()), new DynamicFilterConfig(), blockTypeOperators, new TableExecuteContextManager(), new ExchangeManagerRegistry(new ExchangeHandleResolver()));
}
Also used : PlanFragment(io.trino.sql.planner.PlanFragment) JoinFilterFunctionCompiler(io.trino.sql.gen.JoinFilterFunctionCompiler) ObjectMapperProvider(io.airlift.json.ObjectMapperProvider) CatalogName(io.trino.connector.CatalogName) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) FinalizerService(io.trino.util.FinalizerService) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) TEST_SESSION(io.trino.SessionTestUtils.TEST_SESSION) SINGLE_DISTRIBUTION(io.trino.sql.planner.SystemPartitioningHandle.SINGLE_DISTRIBUTION) TableScanNode(io.trino.sql.planner.plan.TableScanNode) ImmutableMap(com.google.common.collect.ImmutableMap) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) PageSinkManager(io.trino.split.PageSinkManager) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) TrinoOperatorFactories(io.trino.operator.TrinoOperatorFactories) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) IndexJoinLookupStats(io.trino.operator.index.IndexJoinLookupStats) SplitMonitor(io.trino.event.SplitMonitor) MockDirectExchangeClientSupplier(io.trino.execution.TestSqlTaskManager.MockDirectExchangeClientSupplier) List(java.util.List) BIGINT(io.trino.spi.type.BigintType.BIGINT) Split(io.trino.metadata.Split) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) Optional(java.util.Optional) PlanFragmentId(io.trino.sql.planner.plan.PlanFragmentId) TestingColumnHandle(io.trino.testing.TestingMetadata.TestingColumnHandle) TypeAnalyzer.createTestingTypeAnalyzer(io.trino.sql.planner.TypeAnalyzer.createTestingTypeAnalyzer) OrderingCompiler(io.trino.sql.gen.OrderingCompiler) NodePartitioningManager(io.trino.sql.planner.NodePartitioningManager) JoinCompiler(io.trino.sql.gen.JoinCompiler) ExchangeHandleResolver(io.trino.metadata.ExchangeHandleResolver) Partitioning(io.trino.sql.planner.Partitioning) PartitioningScheme(io.trino.sql.planner.PartitioningScheme) StatsAndCosts(io.trino.cost.StatsAndCosts) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler) VARCHAR(io.trino.spi.type.VarcharType.VARCHAR) ImmutableList(com.google.common.collect.ImmutableList) TestingSplit(io.trino.testing.TestingSplit) StageExecutionDescriptor.ungroupedExecution(io.trino.operator.StageExecutionDescriptor.ungroupedExecution) EventListenerConfig(io.trino.eventlistener.EventListenerConfig) IndexManager(io.trino.index.IndexManager) PagesIndex(io.trino.operator.PagesIndex) Symbol(io.trino.sql.planner.Symbol) BlockTypeOperators(io.trino.type.BlockTypeOperators) GenericSpillerFactory(io.trino.spiller.GenericSpillerFactory) PageSourceManager(io.trino.split.PageSourceManager) LocalExecutionPlanner(io.trino.sql.planner.LocalExecutionPlanner) PLANNER_CONTEXT(io.trino.sql.planner.TestingPlannerContext.PLANNER_CONTEXT) EventListenerManager(io.trino.eventlistener.EventListenerManager) TEST_TABLE_HANDLE(io.trino.testing.TestingHandles.TEST_TABLE_HANDLE) OutputBuffers(io.trino.execution.buffer.OutputBuffers) SOURCE_DISTRIBUTION(io.trino.sql.planner.SystemPartitioningHandle.SOURCE_DISTRIBUTION) ExchangeManagerRegistry(io.trino.exchange.ExchangeManagerRegistry) PageFunctionCompiler(io.trino.sql.gen.PageFunctionCompiler) MockDirectExchangeClientSupplier(io.trino.execution.TestSqlTaskManager.MockDirectExchangeClientSupplier) TrinoOperatorFactories(io.trino.operator.TrinoOperatorFactories) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) PagesIndex(io.trino.operator.PagesIndex) PageSourceManager(io.trino.split.PageSourceManager) NodePartitioningManager(io.trino.sql.planner.NodePartitioningManager) ExchangeHandleResolver(io.trino.metadata.ExchangeHandleResolver) BlockTypeOperators(io.trino.type.BlockTypeOperators) OrderingCompiler(io.trino.sql.gen.OrderingCompiler) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) GenericSpillerFactory(io.trino.spiller.GenericSpillerFactory) PageSinkManager(io.trino.split.PageSinkManager) JoinCompiler(io.trino.sql.gen.JoinCompiler) LocalExecutionPlanner(io.trino.sql.planner.LocalExecutionPlanner) IndexJoinLookupStats(io.trino.operator.index.IndexJoinLookupStats) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) JoinFilterFunctionCompiler(io.trino.sql.gen.JoinFilterFunctionCompiler) ExchangeManagerRegistry(io.trino.exchange.ExchangeManagerRegistry) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) IndexManager(io.trino.index.IndexManager) FinalizerService(io.trino.util.FinalizerService) ExpressionCompiler(io.trino.sql.gen.ExpressionCompiler)

Example 2 with FinalizerService

use of io.trino.util.FinalizerService in project trino by trinodb.

the class TestLocalExchange method setUp.

@BeforeMethod
public void setUp() {
    NodeScheduler nodeScheduler = new NodeScheduler(new UniformNodeSelectorFactory(new InMemoryNodeManager(), new NodeSchedulerConfig().setIncludeCoordinator(true), new NodeTaskMap(new FinalizerService())));
    nodePartitioningManager = new NodePartitioningManager(nodeScheduler, new BlockTypeOperators(new TypeOperators()));
}
Also used : BlockTypeOperators(io.trino.type.BlockTypeOperators) NodeTaskMap(io.trino.execution.NodeTaskMap) FinalizerService(io.trino.util.FinalizerService) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) NodePartitioningManager(io.trino.sql.planner.NodePartitioningManager) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) TypeOperators(io.trino.spi.type.TypeOperators) BlockTypeOperators(io.trino.type.BlockTypeOperators) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 3 with FinalizerService

use of io.trino.util.FinalizerService in project trino by trinodb.

the class TestNodeScheduler method setUp.

@BeforeMethod
public void setUp() {
    session = TestingSession.testSessionBuilder().build();
    finalizerService = new FinalizerService();
    nodeTaskMap = new NodeTaskMap(finalizerService);
    nodeManager = new InMemoryNodeManager();
    nodeSchedulerConfig = new NodeSchedulerConfig().setMaxSplitsPerNode(20).setIncludeCoordinator(false).setMaxPendingSplitsPerTask(10);
    nodeScheduler = new NodeScheduler(new UniformNodeSelectorFactory(nodeManager, nodeSchedulerConfig, nodeTaskMap));
    // contents of taskMap indicate the node-task map for the current stage
    taskMap = new HashMap<>();
    nodeSelector = nodeScheduler.createNodeSelector(session, Optional.of(CONNECTOR_ID));
    remoteTaskExecutor = newCachedThreadPool(daemonThreadsNamed("remoteTaskExecutor-%s"));
    remoteTaskScheduledExecutor = newScheduledThreadPool(2, daemonThreadsNamed("remoteTaskScheduledExecutor-%s"));
    finalizerService.start();
}
Also used : FinalizerService(io.trino.util.FinalizerService) UniformNodeSelectorFactory(io.trino.execution.scheduler.UniformNodeSelectorFactory) NodeSchedulerConfig(io.trino.execution.scheduler.NodeSchedulerConfig) NodeScheduler(io.trino.execution.scheduler.NodeScheduler) InMemoryNodeManager(io.trino.metadata.InMemoryNodeManager) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 4 with FinalizerService

use of io.trino.util.FinalizerService in project trino by trinodb.

the class TestSqlStage method testFinalStageInfoInternal.

private void testFinalStageInfoInternal() throws Exception {
    NodeTaskMap nodeTaskMap = new NodeTaskMap(new FinalizerService());
    StageId stageId = new StageId(new QueryId("query"), 0);
    SqlStage stage = createSqlStage(stageId, createExchangePlanFragment(), ImmutableMap.of(), new MockRemoteTaskFactory(executor, scheduledExecutor), TEST_SESSION, true, nodeTaskMap, executor, new SplitSchedulerStats());
    // add listener that fetches stage info when the final status is available
    SettableFuture<StageInfo> finalStageInfo = SettableFuture.create();
    stage.addFinalStageInfoListener(finalStageInfo::set);
    // in a background thread add a ton of tasks
    CountDownLatch latch = new CountDownLatch(1000);
    Future<?> addTasksTask = executor.submit(() -> {
        try {
            for (int i = 0; i < 1_000_000; i++) {
                if (Thread.interrupted()) {
                    return;
                }
                InternalNode node = new InternalNode("source" + i, URI.create("http://10.0.0." + (i / 10_000) + ":" + (i % 10_000)), NodeVersion.UNKNOWN, false);
                stage.createTask(node, i, 0, Optional.empty(), createInitialEmptyOutputBuffers(ARBITRARY), ImmutableMultimap.of(), ImmutableMultimap.of(), ImmutableSet.of());
                latch.countDown();
            }
        } finally {
            while (latch.getCount() > 0) {
                latch.countDown();
            }
        }
    });
    // wait for some tasks to be created, and then abort the query
    latch.await(1, MINUTES);
    assertFalse(stage.getStageInfo().getTasks().isEmpty());
    stage.finish();
    // once the final stage info is available, verify that it is complete
    StageInfo stageInfo = finalStageInfo.get(1, MINUTES);
    assertFalse(stageInfo.getTasks().isEmpty());
    assertTrue(stageInfo.isCompleteInfo());
    assertSame(stage.getStageInfo(), stageInfo);
    // cancel the background thread adding tasks
    addTasksTask.cancel(true);
}
Also used : SqlStage.createSqlStage(io.trino.execution.SqlStage.createSqlStage) QueryId(io.trino.spi.QueryId) CountDownLatch(java.util.concurrent.CountDownLatch) SplitSchedulerStats(io.trino.execution.scheduler.SplitSchedulerStats) FinalizerService(io.trino.util.FinalizerService) InternalNode(io.trino.metadata.InternalNode)

Example 5 with FinalizerService

use of io.trino.util.FinalizerService in project trino by trinodb.

the class TestFaultTolerantStageScheduler method beforeClass.

@BeforeClass
public void beforeClass() {
    finalizerService = new FinalizerService();
    finalizerService.start();
    nodeTaskMap = new NodeTaskMap(finalizerService);
}
Also used : FinalizerService(io.trino.util.FinalizerService) NodeTaskMap(io.trino.execution.NodeTaskMap) BeforeClass(org.testng.annotations.BeforeClass)

Aggregations

FinalizerService (io.trino.util.FinalizerService)6 NodeScheduler (io.trino.execution.scheduler.NodeScheduler)4 NodeSchedulerConfig (io.trino.execution.scheduler.NodeSchedulerConfig)4 UniformNodeSelectorFactory (io.trino.execution.scheduler.UniformNodeSelectorFactory)4 InMemoryNodeManager (io.trino.metadata.InMemoryNodeManager)4 NodeTaskMap (io.trino.execution.NodeTaskMap)3 NodePartitioningManager (io.trino.sql.planner.NodePartitioningManager)3 BlockTypeOperators (io.trino.type.BlockTypeOperators)3 TypeOperators (io.trino.spi.type.TypeOperators)2 BeforeMethod (org.testng.annotations.BeforeMethod)2 ImmutableList (com.google.common.collect.ImmutableList)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 ObjectMapperProvider (io.airlift.json.ObjectMapperProvider)1 TEST_SESSION (io.trino.SessionTestUtils.TEST_SESSION)1 CatalogName (io.trino.connector.CatalogName)1 StatsAndCosts (io.trino.cost.StatsAndCosts)1 SplitMonitor (io.trino.event.SplitMonitor)1 EventListenerConfig (io.trino.eventlistener.EventListenerConfig)1 EventListenerManager (io.trino.eventlistener.EventListenerManager)1 ExchangeManagerRegistry (io.trino.exchange.ExchangeManagerRegistry)1