Search in sources :

Example 6 with SimpleTtlNodeSelectorConfig

use of com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig in project presto by prestodb.

the class TaskTestUtils method createTestingPlanner.

public static LocalExecutionPlanner createTestingPlanner() {
    MetadataManager metadata = MetadataManager.createTestMetadataManager();
    PageSourceManager pageSourceManager = new PageSourceManager();
    pageSourceManager.addConnectorPageSourceProvider(CONNECTOR_ID, new TestingPageSourceProvider());
    // we don't start the finalizer so nothing will be collected, which is ok for a test
    FinalizerService finalizerService = new FinalizerService();
    NodeScheduler nodeScheduler = new NodeScheduler(new LegacyNetworkTopology(), new InMemoryNodeManager(), new NodeSelectionStats(), new NodeSchedulerConfig().setIncludeCoordinator(true), new NodeTaskMap(finalizerService), new ThrowingNodeTtlFetcherManager(), new NoOpQueryManager(), new SimpleTtlNodeSelectorConfig());
    PartitioningProviderManager partitioningProviderManager = new PartitioningProviderManager();
    NodePartitioningManager nodePartitioningManager = new NodePartitioningManager(nodeScheduler, partitioningProviderManager, new NodeSelectionStats());
    PageFunctionCompiler pageFunctionCompiler = new PageFunctionCompiler(metadata, 0);
    return new LocalExecutionPlanner(metadata, Optional.empty(), pageSourceManager, new IndexManager(), partitioningProviderManager, nodePartitioningManager, new PageSinkManager(), new ConnectorMetadataUpdaterManager(), new ExpressionCompiler(metadata, pageFunctionCompiler), pageFunctionCompiler, new JoinFilterFunctionCompiler(metadata), new IndexJoinLookupStats(), new TaskManagerConfig(), new MemoryManagerConfig(), new GenericSpillerFactory((types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }), (types, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, (types, partitionFunction, spillContext, memoryContext) -> {
        throw new UnsupportedOperationException();
    }, new BlockEncodingManager(), new PagesIndex.TestingFactory(false), new JoinCompiler(MetadataManager.createTestMetadataManager(), new FeaturesConfig()), new LookupJoinOperators(), new OrderingCompiler(), jsonCodec(TableCommitContext.class), new RowExpressionDeterminismEvaluator(metadata), new NoOpFragmentResultCacheManager(), new ObjectMapper(), (session) -> {
        throw new UnsupportedOperationException();
    });
}
Also used : WarningCollector(com.facebook.presto.spi.WarningCollector) LocalExecutionPlanner(com.facebook.presto.sql.planner.LocalExecutionPlanner) MetadataManager(com.facebook.presto.metadata.MetadataManager) VariableReferenceExpression(com.facebook.presto.spi.relation.VariableReferenceExpression) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) EventListenerManager(com.facebook.presto.eventlistener.EventListenerManager) ConnectorTransactionHandle(com.facebook.presto.spi.connector.ConnectorTransactionHandle) TableWriteInfo(com.facebook.presto.execution.scheduler.TableWriteInfo) PagesIndex(com.facebook.presto.operator.PagesIndex) StageExecutionDescriptor(com.facebook.presto.operator.StageExecutionDescriptor) PlanFragment(com.facebook.presto.sql.planner.PlanFragment) ThrowingNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager) NoOpFragmentResultCacheManager(com.facebook.presto.operator.NoOpFragmentResultCacheManager) FinalizerService(com.facebook.presto.util.FinalizerService) JsonObjectMapperProvider(com.facebook.airlift.json.JsonObjectMapperProvider) PartitioningScheme(com.facebook.presto.sql.planner.PartitioningScheme) URI(java.net.URI) PageFunctionCompiler(com.facebook.presto.sql.gen.PageFunctionCompiler) ConnectorMetadataUpdaterManager(com.facebook.presto.metadata.ConnectorMetadataUpdaterManager) ImmutableSet(com.google.common.collect.ImmutableSet) PageSinkManager(com.facebook.presto.split.PageSinkManager) ImmutableMap(com.google.common.collect.ImmutableMap) ResourceGroupId(com.facebook.presto.spi.resourceGroups.ResourceGroupId) JsonCodec.jsonCodec(com.facebook.airlift.json.JsonCodec.jsonCodec) SOURCE_DISTRIBUTION(com.facebook.presto.sql.planner.SystemPartitioningHandle.SOURCE_DISTRIBUTION) FeaturesConfig(com.facebook.presto.sql.analyzer.FeaturesConfig) List(java.util.List) Optional(java.util.Optional) ConnectorId(com.facebook.presto.spi.ConnectorId) AllowAllAccessControl(com.facebook.presto.security.AllowAllAccessControl) ExpressionCompiler(com.facebook.presto.sql.gen.ExpressionCompiler) NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) PlanNodeId(com.facebook.presto.spi.plan.PlanNodeId) OutputBuffers(com.facebook.presto.execution.buffer.OutputBuffers) SINGLE_DISTRIBUTION(com.facebook.presto.sql.planner.SystemPartitioningHandle.SINGLE_DISTRIBUTION) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) RowExpressionDeterminismEvaluator(com.facebook.presto.sql.relational.RowExpressionDeterminismEvaluator) LookupJoinOperators(com.facebook.presto.operator.LookupJoinOperators) NoOpQueryManager(com.facebook.presto.dispatcher.NoOpQueryManager) TestingSplit(com.facebook.presto.testing.TestingSplit) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) TEST_SESSION(com.facebook.presto.SessionTestUtils.TEST_SESSION) ImmutableList(com.google.common.collect.ImmutableList) PageSourceManager(com.facebook.presto.split.PageSourceManager) OrderingCompiler(com.facebook.presto.sql.gen.OrderingCompiler) BlockEncodingManager(com.facebook.presto.common.block.BlockEncodingManager) GenericSpillerFactory(com.facebook.presto.spiller.GenericSpillerFactory) IndexManager(com.facebook.presto.index.IndexManager) TableHandle(com.facebook.presto.spi.TableHandle) MemoryManagerConfig(com.facebook.presto.memory.MemoryManagerConfig) PlanFragmentId(com.facebook.presto.sql.planner.plan.PlanFragmentId) IndexJoinLookupStats(com.facebook.presto.operator.index.IndexJoinLookupStats) TransactionManager(com.facebook.presto.transaction.TransactionManager) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) Partitioning(com.facebook.presto.sql.planner.Partitioning) BIGINT(com.facebook.presto.common.type.BigintType.BIGINT) NodePartitioningManager(com.facebook.presto.sql.planner.NodePartitioningManager) TableCommitContext(com.facebook.presto.operator.TableCommitContext) SplitMonitor(com.facebook.presto.event.SplitMonitor) Executor(java.util.concurrent.Executor) Session(com.facebook.presto.Session) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) JoinFilterFunctionCompiler(com.facebook.presto.sql.gen.JoinFilterFunctionCompiler) TestingTransactionHandle(com.facebook.presto.testing.TestingTransactionHandle) TupleDomain(com.facebook.presto.common.predicate.TupleDomain) TestingColumnHandle(com.facebook.presto.testing.TestingMetadata.TestingColumnHandle) TableScanNode(com.facebook.presto.spi.plan.TableScanNode) StatsAndCosts(com.facebook.presto.cost.StatsAndCosts) Split(com.facebook.presto.metadata.Split) TestingTableHandle(com.facebook.presto.testing.TestingMetadata.TestingTableHandle) JoinCompiler(com.facebook.presto.sql.gen.JoinCompiler) PartitioningProviderManager(com.facebook.presto.sql.planner.PartitioningProviderManager) PageFunctionCompiler(com.facebook.presto.sql.gen.PageFunctionCompiler) FeaturesConfig(com.facebook.presto.sql.analyzer.FeaturesConfig) PartitioningProviderManager(com.facebook.presto.sql.planner.PartitioningProviderManager) NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) TableCommitContext(com.facebook.presto.operator.TableCommitContext) PagesIndex(com.facebook.presto.operator.PagesIndex) PageSourceManager(com.facebook.presto.split.PageSourceManager) NodePartitioningManager(com.facebook.presto.sql.planner.NodePartitioningManager) NoOpFragmentResultCacheManager(com.facebook.presto.operator.NoOpFragmentResultCacheManager) OrderingCompiler(com.facebook.presto.sql.gen.OrderingCompiler) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) GenericSpillerFactory(com.facebook.presto.spiller.GenericSpillerFactory) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) PageSinkManager(com.facebook.presto.split.PageSinkManager) LookupJoinOperators(com.facebook.presto.operator.LookupJoinOperators) JoinCompiler(com.facebook.presto.sql.gen.JoinCompiler) RowExpressionDeterminismEvaluator(com.facebook.presto.sql.relational.RowExpressionDeterminismEvaluator) LocalExecutionPlanner(com.facebook.presto.sql.planner.LocalExecutionPlanner) IndexJoinLookupStats(com.facebook.presto.operator.index.IndexJoinLookupStats) JoinFilterFunctionCompiler(com.facebook.presto.sql.gen.JoinFilterFunctionCompiler) MemoryManagerConfig(com.facebook.presto.memory.MemoryManagerConfig) ThrowingNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) IndexManager(com.facebook.presto.index.IndexManager) NoOpQueryManager(com.facebook.presto.dispatcher.NoOpQueryManager) MetadataManager(com.facebook.presto.metadata.MetadataManager) BlockEncodingManager(com.facebook.presto.common.block.BlockEncodingManager) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) FinalizerService(com.facebook.presto.util.FinalizerService) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) ConnectorMetadataUpdaterManager(com.facebook.presto.metadata.ConnectorMetadataUpdaterManager) ExpressionCompiler(com.facebook.presto.sql.gen.ExpressionCompiler)

Example 7 with SimpleTtlNodeSelectorConfig

use of com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig in project presto by prestodb.

the class TestCostCalculator method setUp.

@BeforeClass
public void setUp() {
    TaskCountEstimator taskCountEstimator = new TaskCountEstimator(() -> NUMBER_OF_NODES);
    costCalculatorUsingExchanges = new CostCalculatorUsingExchanges(taskCountEstimator);
    costCalculatorWithEstimatedExchanges = new CostCalculatorWithEstimatedExchanges(costCalculatorUsingExchanges, taskCountEstimator);
    session = testSessionBuilder().setCatalog("tpch").build();
    CatalogManager catalogManager = new CatalogManager();
    catalogManager.registerCatalog(createBogusTestingCatalog("tpch"));
    transactionManager = createTestTransactionManager(catalogManager);
    metadata = createTestMetadataManager(transactionManager, new FeaturesConfig());
    finalizerService = new FinalizerService();
    finalizerService.start();
    nodeScheduler = new NodeScheduler(new LegacyNetworkTopology(), new InMemoryNodeManager(), new NodeSelectionStats(), new NodeSchedulerConfig().setIncludeCoordinator(true), new NodeTaskMap(finalizerService), new ThrowingNodeTtlFetcherManager(), new NoOpQueryManager(), new SimpleTtlNodeSelectorConfig());
    PartitioningProviderManager partitioningProviderManager = new PartitioningProviderManager();
    nodePartitioningManager = new NodePartitioningManager(nodeScheduler, partitioningProviderManager, new NodeSelectionStats());
    planFragmenter = new PlanFragmenter(metadata, nodePartitioningManager, new QueryManagerConfig(), new SqlParser(), new FeaturesConfig());
}
Also used : NodeTaskMap(com.facebook.presto.execution.NodeTaskMap) FeaturesConfig(com.facebook.presto.sql.analyzer.FeaturesConfig) PartitioningProviderManager(com.facebook.presto.sql.planner.PartitioningProviderManager) SqlParser(com.facebook.presto.sql.parser.SqlParser) NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) PlanFragmenter(com.facebook.presto.sql.planner.PlanFragmenter) ThrowingNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager) CatalogManager(com.facebook.presto.metadata.CatalogManager) NodePartitioningManager(com.facebook.presto.sql.planner.NodePartitioningManager) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) NoOpQueryManager(com.facebook.presto.dispatcher.NoOpQueryManager) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) FinalizerService(com.facebook.presto.util.FinalizerService) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) QueryManagerConfig(com.facebook.presto.execution.QueryManagerConfig) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) BeforeClass(org.testng.annotations.BeforeClass)

Example 8 with SimpleTtlNodeSelectorConfig

use of com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig in project presto by prestodb.

the class TestNodeScheduler method testTtlAwareScheduling.

@Test
public void testTtlAwareScheduling() {
    InMemoryNodeManager nodeManager = new InMemoryNodeManager();
    InternalNode node1 = new InternalNode("other1", URI.create("http://127.0.0.1:11"), NodeVersion.UNKNOWN, false);
    InternalNode node2 = new InternalNode("other2", URI.create("http://127.0.0.1:12"), NodeVersion.UNKNOWN, false);
    InternalNode node3 = new InternalNode("other3", URI.create("http://127.0.0.1:13"), NodeVersion.UNKNOWN, false);
    List<InternalNode> nodes = ImmutableList.of(node1, node2, node3);
    nodeManager.addNode(CONNECTOR_ID, nodes);
    NodeSchedulerConfig nodeSchedulerConfig = new NodeSchedulerConfig().setMaxSplitsPerNode(20).setIncludeCoordinator(false).setMaxPendingSplitsPerTask(10);
    Instant currentInstant = Instant.now();
    NodeTtl ttl1 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(5, ChronoUnit.MINUTES).getEpochSecond(), 100)));
    NodeTtl ttl2 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(30, ChronoUnit.MINUTES).getEpochSecond(), 100)));
    NodeTtl ttl3 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(2, ChronoUnit.HOURS).getEpochSecond(), 100)));
    Map<NodeInfo, NodeTtl> nodeToTtl = ImmutableMap.of(new NodeInfo(node1.getNodeIdentifier(), node1.getHost()), ttl1, new NodeInfo(node2.getNodeIdentifier(), node2.getHost()), ttl2, new NodeInfo(node3.getNodeIdentifier(), node3.getHost()), ttl3);
    ConfidenceBasedNodeTtlFetcherManager nodeTtlFetcherManager = new ConfidenceBasedNodeTtlFetcherManager(nodeManager, new NodeSchedulerConfig(), new NodeTtlFetcherManagerConfig());
    NodeTtlFetcherFactory nodeTtlFetcherFactory = new TestingNodeTtlFetcherFactory(nodeToTtl);
    nodeTtlFetcherManager.addNodeTtlFetcherFactory(nodeTtlFetcherFactory);
    nodeTtlFetcherManager.load(nodeTtlFetcherFactory.getName(), ImmutableMap.of());
    nodeTtlFetcherManager.refreshTtlInfo();
    TestingQueryManager queryManager = new TestingQueryManager();
    NodeScheduler nodeScheduler = new NodeScheduler(new LegacyNetworkTopology(), nodeManager, new NodeSelectionStats(), nodeSchedulerConfig, nodeTaskMap, nodeTtlFetcherManager, queryManager, new SimpleTtlNodeSelectorConfig());
    // Query is estimated to take 20 mins and has been executing for 3 mins, i.e, 17 mins left
    // So only node2 and node3 have enough TTL to run additional work
    Session session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(20, TimeUnit.MINUTES));
    NodeSelector nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(3, TimeUnit.MINUTES));
    assertEquals(ImmutableSet.copyOf(nodeSelector.selectRandomNodes(3)), ImmutableSet.of(node2, node3));
    // Query is estimated to take 1 hour and has been executing for 45 mins, i.e, 15 mins left
    // So only node2 and node3 have enough TTL to work on new splits
    session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(1, TimeUnit.HOURS));
    nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(45, TimeUnit.MINUTES));
    Set<Split> splits = new HashSet<>();
    for (int i = 0; i < 2; i++) {
        splits.add(new Split(CONNECTOR_ID, TestingTransactionHandle.create(), new TestSplitRemote()));
    }
    Multimap<InternalNode, Split> assignments = nodeSelector.computeAssignments(splits, ImmutableList.copyOf(taskMap.values())).getAssignments();
    assertEquals(assignments.size(), 2);
    assertTrue(assignments.keySet().contains(node2));
    assertTrue(assignments.keySet().contains(node3));
    // Query is estimated to take 1 hour and has been executing for 20 mins, i.e, 40 mins left
    // So only node3 has enough TTL to work on new splits
    MockRemoteTaskFactory remoteTaskFactory = new MockRemoteTaskFactory(remoteTaskExecutor, remoteTaskScheduledExecutor);
    TaskId taskId = new TaskId("test", 1, 0, 1);
    RemoteTask newRemoteTask = remoteTaskFactory.createTableScanTask(taskId, node2, ImmutableList.of(), nodeTaskMap.createTaskStatsTracker(node2, taskId));
    taskMap.put(node2, newRemoteTask);
    nodeTaskMap.addTask(node2, newRemoteTask);
    session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(1, TimeUnit.HOURS));
    nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(20, TimeUnit.MINUTES));
    splits.clear();
    for (int i = 0; i < 2; i++) {
        splits.add(new Split(CONNECTOR_ID, TestingTransactionHandle.create(), new TestSplitRemote()));
    }
    assignments = nodeSelector.computeAssignments(splits, ImmutableList.copyOf(taskMap.values())).getAssignments();
    assertEquals(assignments.size(), 2);
    assertEquals(assignments.keySet().size(), 1);
    assertTrue(assignments.keySet().contains(node3));
}
Also used : NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) HashSet(java.util.HashSet) NodeTtlFetcherFactory(com.facebook.presto.spi.ttl.NodeTtlFetcherFactory) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) ConfidenceBasedNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager) ConfidenceBasedTtlInfo(com.facebook.presto.spi.ttl.ConfidenceBasedTtlInfo) Instant(java.time.Instant) NodeTtl(com.facebook.presto.spi.ttl.NodeTtl) Duration(io.airlift.units.Duration) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) NodeTtlFetcherManagerConfig(com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManagerConfig) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) NodeInfo(com.facebook.presto.spi.ttl.NodeInfo) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) InternalNode(com.facebook.presto.metadata.InternalNode) NodeSelector(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) Split(com.facebook.presto.metadata.Split) TestingSession(com.facebook.presto.testing.TestingSession) Session(com.facebook.presto.Session) Test(org.testng.annotations.Test)

Example 9 with SimpleTtlNodeSelectorConfig

use of com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig in project presto by prestodb.

the class TestNodeScheduler method testAffinityAssignmentNotSupported.

@Test
public void testAffinityAssignmentNotSupported() {
    NodeTaskMap nodeTaskMap = new NodeTaskMap(finalizerService);
    TestingTransactionHandle transactionHandle = TestingTransactionHandle.create();
    NodeSchedulerConfig nodeSchedulerConfig = new NodeSchedulerConfig().setMaxSplitsPerNode(20).setIncludeCoordinator(false).setMaxPendingSplitsPerTask(10);
    LegacyNetworkTopology legacyNetworkTopology = new LegacyNetworkTopology();
    NodeScheduler nodeScheduler = new NodeScheduler(new NetworkLocationCache(legacyNetworkTopology), legacyNetworkTopology, nodeManager, new NodeSelectionStats(), nodeSchedulerConfig, nodeTaskMap, new Duration(0, SECONDS), new ThrowingNodeTtlFetcherManager(), new NoOpQueryManager(), new SimpleTtlNodeSelectorConfig());
    NodeSelector nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID, 2);
    Set<Split> splits = new HashSet<>();
    splits.add(new Split(CONNECTOR_ID, transactionHandle, new TestSplitRemote(HostAddress.fromString("127.0.0.1:10"))));
    splits.add(new Split(CONNECTOR_ID, transactionHandle, new TestSplitRemote(HostAddress.fromString("127.0.0.1:10"))));
    SplitPlacementResult splitPlacementResult = nodeSelector.computeAssignments(splits, ImmutableList.of());
    Set<InternalNode> internalNodes = splitPlacementResult.getAssignments().keySet();
    // Split doesn't support affinity schedule, fall back to random schedule
    assertEquals(internalNodes.size(), 2);
}
Also used : NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) NetworkLocationCache(com.facebook.presto.execution.scheduler.NetworkLocationCache) Duration(io.airlift.units.Duration) ThrowingNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager) NoOpQueryManager(com.facebook.presto.dispatcher.NoOpQueryManager) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) TestingTransactionHandle(com.facebook.presto.testing.TestingTransactionHandle) NodeSelector(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector) InternalNode(com.facebook.presto.metadata.InternalNode) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) Split(com.facebook.presto.metadata.Split) SplitPlacementResult(com.facebook.presto.execution.scheduler.SplitPlacementResult) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) HashSet(java.util.HashSet) Test(org.testng.annotations.Test)

Example 10 with SimpleTtlNodeSelectorConfig

use of com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig in project presto by prestodb.

the class TestNodeScheduler method testMaxTasksPerStageWittLimit.

@Test
public void testMaxTasksPerStageWittLimit() {
    NodeTaskMap nodeTaskMap = new NodeTaskMap(finalizerService);
    TestingTransactionHandle transactionHandle = TestingTransactionHandle.create();
    NodeSchedulerConfig nodeSchedulerConfig = new NodeSchedulerConfig().setMaxSplitsPerNode(20).setIncludeCoordinator(false).setMaxPendingSplitsPerTask(10);
    NodeScheduler nodeScheduler = new NodeScheduler(new LegacyNetworkTopology(), nodeManager, new NodeSelectionStats(), nodeSchedulerConfig, nodeTaskMap, new ThrowingNodeTtlFetcherManager(), new NoOpQueryManager(), new SimpleTtlNodeSelectorConfig());
    NodeSelector nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID, 2);
    Set<Split> splits = new HashSet<>();
    splits.add(new Split(CONNECTOR_ID, transactionHandle, new TestSplitRemote()));
    SplitPlacementResult splitPlacementResult = nodeSelector.computeAssignments(splits, ImmutableList.of());
    Set<InternalNode> internalNodes = splitPlacementResult.getAssignments().keySet();
    assertEquals(internalNodes.size(), 1);
    // adding one more split. Total 2
    splits.add(new Split(CONNECTOR_ID, transactionHandle, new TestSplitRemote()));
    splitPlacementResult = nodeSelector.computeAssignments(splits, getRemoteTableScanTask(splitPlacementResult));
    Set<InternalNode> internalNodesSecondCall = splitPlacementResult.getAssignments().keySet();
    assertEquals(internalNodesSecondCall.size(), 2);
    assertTrue(internalNodesSecondCall.containsAll(internalNodes));
    // adding one more split. Total 3
    splits.add(new Split(CONNECTOR_ID, transactionHandle, new TestSplitRemote()));
    splitPlacementResult = nodeSelector.computeAssignments(splits, getRemoteTableScanTask(splitPlacementResult));
    assertEquals(splitPlacementResult.getAssignments().keySet().size(), 2);
    assertEquals(splitPlacementResult.getAssignments().keySet(), internalNodesSecondCall);
}
Also used : NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) ThrowingNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager) NoOpQueryManager(com.facebook.presto.dispatcher.NoOpQueryManager) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) TestingTransactionHandle(com.facebook.presto.testing.TestingTransactionHandle) NodeSelector(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector) InternalNode(com.facebook.presto.metadata.InternalNode) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) Split(com.facebook.presto.metadata.Split) SplitPlacementResult(com.facebook.presto.execution.scheduler.SplitPlacementResult) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) HashSet(java.util.HashSet) Test(org.testng.annotations.Test)

Aggregations

SimpleTtlNodeSelectorConfig (com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig)15 NodeSelectionStats (com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats)14 NoOpQueryManager (com.facebook.presto.dispatcher.NoOpQueryManager)13 ThrowingNodeTtlFetcherManager (com.facebook.presto.ttl.nodettlfetchermanagers.ThrowingNodeTtlFetcherManager)13 NodeScheduler (com.facebook.presto.execution.scheduler.NodeScheduler)12 NodeSchedulerConfig (com.facebook.presto.execution.scheduler.NodeSchedulerConfig)12 LegacyNetworkTopology (com.facebook.presto.execution.scheduler.LegacyNetworkTopology)11 Test (org.testng.annotations.Test)11 Split (com.facebook.presto.metadata.Split)10 NodeSelector (com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector)9 InternalNode (com.facebook.presto.metadata.InternalNode)9 ConnectorSplit (com.facebook.presto.spi.ConnectorSplit)9 TestingTransactionHandle (com.facebook.presto.testing.TestingTransactionHandle)9 HashSet (java.util.HashSet)9 SplitPlacementResult (com.facebook.presto.execution.scheduler.SplitPlacementResult)7 InMemoryNodeManager (com.facebook.presto.metadata.InMemoryNodeManager)7 NetworkLocationCache (com.facebook.presto.execution.scheduler.NetworkLocationCache)3 ImmutableList (com.google.common.collect.ImmutableList)3 Duration (io.airlift.units.Duration)3 Session (com.facebook.presto.Session)2