Search in sources :

Example 1 with ConfidenceBasedNodeTtlFetcherManager

use of com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager in project presto by prestodb.

the class TestConfidenceBasedClusterTtlProviderManager method setup.

@BeforeClass
public void setup() {
    InMemoryNodeManager nodeManager = new InMemoryNodeManager();
    nodeManager.addNode(new ConnectorId("prism"), ImmutableSet.of(node1, node2, node3));
    ConfidenceBasedNodeTtlFetcherManager nodeTtlFetcherManager = new ConfidenceBasedNodeTtlFetcherManager(nodeManager, new NodeSchedulerConfig(), new NodeTtlFetcherManagerConfig());
    NodeTtlFetcherFactory nodeTtlFetcherFactory = new TestingNodeTtlFetcherFactory(nodeToTtl);
    nodeTtlFetcherManager.addNodeTtlFetcherFactory(nodeTtlFetcherFactory);
    nodeTtlFetcherManager.load(nodeTtlFetcherFactory.getName(), ImmutableMap.of());
    nodeTtlFetcherManager.refreshTtlInfo();
    clusterTtlProviderManager = new ConfidenceBasedClusterTtlProviderManager(nodeTtlFetcherManager);
    ClusterTtlProviderFactory clusterTtlProviderFactory = new TestingClusterTtlProviderFactory();
    clusterTtlProviderManager.addClusterTtlProviderFactory(clusterTtlProviderFactory);
    clusterTtlProviderManager.load(clusterTtlProviderFactory.getName(), ImmutableMap.of());
}
Also used : ConfidenceBasedClusterTtlProviderManager(com.facebook.presto.ttl.clusterttlprovidermanagers.ConfidenceBasedClusterTtlProviderManager) NodeTtlFetcherFactory(com.facebook.presto.spi.ttl.NodeTtlFetcherFactory) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) ConfidenceBasedNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager) NodeTtlFetcherManagerConfig(com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManagerConfig) TestingClusterTtlProviderFactory(com.facebook.presto.spi.ttl.TestingClusterTtlProviderFactory) TestingClusterTtlProviderFactory(com.facebook.presto.spi.ttl.TestingClusterTtlProviderFactory) ClusterTtlProviderFactory(com.facebook.presto.spi.ttl.ClusterTtlProviderFactory) NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) ConnectorId(com.facebook.presto.spi.ConnectorId) BeforeClass(org.testng.annotations.BeforeClass)

Example 2 with ConfidenceBasedNodeTtlFetcherManager

use of com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager in project presto by prestodb.

the class TestConfidenceBasedNodeTtlFetcherManager method setup.

@BeforeClass
public void setup() {
    nodeManager = new InMemoryNodeManager();
    nodeManager.addNode(new ConnectorId("prism"), ImmutableSet.of(node1, node2, node3));
    nodeTtlFetcherManager = new ConfidenceBasedNodeTtlFetcherManager(nodeManager, new NodeSchedulerConfig(), new NodeTtlFetcherManagerConfig());
    NodeTtlFetcherFactory factory = new TestingNodeTtlFetcherFactory(nodeToTtl);
    nodeTtlFetcherManager.addNodeTtlFetcherFactory(factory);
    nodeTtlFetcherManager.load(factory.getName(), ImmutableMap.of());
    nodeTtlFetcherManager.refreshTtlInfo();
}
Also used : NodeTtlFetcherFactory(com.facebook.presto.spi.ttl.NodeTtlFetcherFactory) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) ConfidenceBasedNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager) NodeTtlFetcherManagerConfig(com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManagerConfig) NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) ConnectorId(com.facebook.presto.spi.ConnectorId) BeforeClass(org.testng.annotations.BeforeClass)

Example 3 with ConfidenceBasedNodeTtlFetcherManager

use of com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager in project presto by prestodb.

the class TestNodeScheduler method testTtlAwareScheduling.

@Test
public void testTtlAwareScheduling() {
    InMemoryNodeManager nodeManager = new InMemoryNodeManager();
    InternalNode node1 = new InternalNode("other1", URI.create("http://127.0.0.1:11"), NodeVersion.UNKNOWN, false);
    InternalNode node2 = new InternalNode("other2", URI.create("http://127.0.0.1:12"), NodeVersion.UNKNOWN, false);
    InternalNode node3 = new InternalNode("other3", URI.create("http://127.0.0.1:13"), NodeVersion.UNKNOWN, false);
    List<InternalNode> nodes = ImmutableList.of(node1, node2, node3);
    nodeManager.addNode(CONNECTOR_ID, nodes);
    NodeSchedulerConfig nodeSchedulerConfig = new NodeSchedulerConfig().setMaxSplitsPerNode(20).setIncludeCoordinator(false).setMaxPendingSplitsPerTask(10);
    Instant currentInstant = Instant.now();
    NodeTtl ttl1 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(5, ChronoUnit.MINUTES).getEpochSecond(), 100)));
    NodeTtl ttl2 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(30, ChronoUnit.MINUTES).getEpochSecond(), 100)));
    NodeTtl ttl3 = new NodeTtl(ImmutableSet.of(new ConfidenceBasedTtlInfo(currentInstant.plus(2, ChronoUnit.HOURS).getEpochSecond(), 100)));
    Map<NodeInfo, NodeTtl> nodeToTtl = ImmutableMap.of(new NodeInfo(node1.getNodeIdentifier(), node1.getHost()), ttl1, new NodeInfo(node2.getNodeIdentifier(), node2.getHost()), ttl2, new NodeInfo(node3.getNodeIdentifier(), node3.getHost()), ttl3);
    ConfidenceBasedNodeTtlFetcherManager nodeTtlFetcherManager = new ConfidenceBasedNodeTtlFetcherManager(nodeManager, new NodeSchedulerConfig(), new NodeTtlFetcherManagerConfig());
    NodeTtlFetcherFactory nodeTtlFetcherFactory = new TestingNodeTtlFetcherFactory(nodeToTtl);
    nodeTtlFetcherManager.addNodeTtlFetcherFactory(nodeTtlFetcherFactory);
    nodeTtlFetcherManager.load(nodeTtlFetcherFactory.getName(), ImmutableMap.of());
    nodeTtlFetcherManager.refreshTtlInfo();
    TestingQueryManager queryManager = new TestingQueryManager();
    NodeScheduler nodeScheduler = new NodeScheduler(new LegacyNetworkTopology(), nodeManager, new NodeSelectionStats(), nodeSchedulerConfig, nodeTaskMap, nodeTtlFetcherManager, queryManager, new SimpleTtlNodeSelectorConfig());
    // Query is estimated to take 20 mins and has been executing for 3 mins, i.e, 17 mins left
    // So only node2 and node3 have enough TTL to run additional work
    Session session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(20, TimeUnit.MINUTES));
    NodeSelector nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(3, TimeUnit.MINUTES));
    assertEquals(ImmutableSet.copyOf(nodeSelector.selectRandomNodes(3)), ImmutableSet.of(node2, node3));
    // Query is estimated to take 1 hour and has been executing for 45 mins, i.e, 15 mins left
    // So only node2 and node3 have enough TTL to work on new splits
    session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(1, TimeUnit.HOURS));
    nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(45, TimeUnit.MINUTES));
    Set<Split> splits = new HashSet<>();
    for (int i = 0; i < 2; i++) {
        splits.add(new Split(CONNECTOR_ID, TestingTransactionHandle.create(), new TestSplitRemote()));
    }
    Multimap<InternalNode, Split> assignments = nodeSelector.computeAssignments(splits, ImmutableList.copyOf(taskMap.values())).getAssignments();
    assertEquals(assignments.size(), 2);
    assertTrue(assignments.keySet().contains(node2));
    assertTrue(assignments.keySet().contains(node3));
    // Query is estimated to take 1 hour and has been executing for 20 mins, i.e, 40 mins left
    // So only node3 has enough TTL to work on new splits
    MockRemoteTaskFactory remoteTaskFactory = new MockRemoteTaskFactory(remoteTaskExecutor, remoteTaskScheduledExecutor);
    TaskId taskId = new TaskId("test", 1, 0, 1);
    RemoteTask newRemoteTask = remoteTaskFactory.createTableScanTask(taskId, node2, ImmutableList.of(), nodeTaskMap.createTaskStatsTracker(node2, taskId));
    taskMap.put(node2, newRemoteTask);
    nodeTaskMap.addTask(node2, newRemoteTask);
    session = sessionWithTtlAwareSchedulingStrategyAndEstimatedExecutionTime(new Duration(1, TimeUnit.HOURS));
    nodeSelector = nodeScheduler.createNodeSelector(session, CONNECTOR_ID);
    queryManager.setExecutionTime(new Duration(20, TimeUnit.MINUTES));
    splits.clear();
    for (int i = 0; i < 2; i++) {
        splits.add(new Split(CONNECTOR_ID, TestingTransactionHandle.create(), new TestSplitRemote()));
    }
    assignments = nodeSelector.computeAssignments(splits, ImmutableList.copyOf(taskMap.values())).getAssignments();
    assertEquals(assignments.size(), 2);
    assertEquals(assignments.keySet().size(), 1);
    assertTrue(assignments.keySet().contains(node3));
}
Also used : NodeSchedulerConfig(com.facebook.presto.execution.scheduler.NodeSchedulerConfig) NodeScheduler(com.facebook.presto.execution.scheduler.NodeScheduler) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) SimpleTtlNodeSelectorConfig(com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig) HashSet(java.util.HashSet) NodeTtlFetcherFactory(com.facebook.presto.spi.ttl.NodeTtlFetcherFactory) TestingNodeTtlFetcherFactory(com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory) ConfidenceBasedNodeTtlFetcherManager(com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager) ConfidenceBasedTtlInfo(com.facebook.presto.spi.ttl.ConfidenceBasedTtlInfo) Instant(java.time.Instant) NodeTtl(com.facebook.presto.spi.ttl.NodeTtl) Duration(io.airlift.units.Duration) InMemoryNodeManager(com.facebook.presto.metadata.InMemoryNodeManager) NodeTtlFetcherManagerConfig(com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManagerConfig) NodeSelectionStats(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats) NodeInfo(com.facebook.presto.spi.ttl.NodeInfo) LegacyNetworkTopology(com.facebook.presto.execution.scheduler.LegacyNetworkTopology) InternalNode(com.facebook.presto.metadata.InternalNode) NodeSelector(com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) Split(com.facebook.presto.metadata.Split) TestingSession(com.facebook.presto.testing.TestingSession) Session(com.facebook.presto.Session) Test(org.testng.annotations.Test)

Aggregations

NodeSchedulerConfig (com.facebook.presto.execution.scheduler.NodeSchedulerConfig)3 InMemoryNodeManager (com.facebook.presto.metadata.InMemoryNodeManager)3 NodeTtlFetcherFactory (com.facebook.presto.spi.ttl.NodeTtlFetcherFactory)3 TestingNodeTtlFetcherFactory (com.facebook.presto.spi.ttl.TestingNodeTtlFetcherFactory)3 ConfidenceBasedNodeTtlFetcherManager (com.facebook.presto.ttl.nodettlfetchermanagers.ConfidenceBasedNodeTtlFetcherManager)3 NodeTtlFetcherManagerConfig (com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManagerConfig)3 ConnectorId (com.facebook.presto.spi.ConnectorId)2 BeforeClass (org.testng.annotations.BeforeClass)2 Session (com.facebook.presto.Session)1 LegacyNetworkTopology (com.facebook.presto.execution.scheduler.LegacyNetworkTopology)1 NodeScheduler (com.facebook.presto.execution.scheduler.NodeScheduler)1 NodeSelectionStats (com.facebook.presto.execution.scheduler.nodeSelection.NodeSelectionStats)1 NodeSelector (com.facebook.presto.execution.scheduler.nodeSelection.NodeSelector)1 SimpleTtlNodeSelectorConfig (com.facebook.presto.execution.scheduler.nodeSelection.SimpleTtlNodeSelectorConfig)1 InternalNode (com.facebook.presto.metadata.InternalNode)1 Split (com.facebook.presto.metadata.Split)1 ConnectorSplit (com.facebook.presto.spi.ConnectorSplit)1 ClusterTtlProviderFactory (com.facebook.presto.spi.ttl.ClusterTtlProviderFactory)1 ConfidenceBasedTtlInfo (com.facebook.presto.spi.ttl.ConfidenceBasedTtlInfo)1 NodeInfo (com.facebook.presto.spi.ttl.NodeInfo)1