Search in sources :

Example 1 with NodeSupplier

use of com.facebook.presto.raptor.NodeSupplier in project presto by prestodb.

the class TestRaptorSplitManager method setup.

@BeforeMethod
public void setup() throws Exception {
    FunctionAndTypeManager functionAndTypeManager = createTestFunctionAndTypeManager();
    DBI dbi = new DBI("jdbc:h2:mem:test" + System.nanoTime() + "_" + ThreadLocalRandom.current().nextInt());
    dbi.registerMapper(new TableColumn.Mapper(functionAndTypeManager));
    dummyHandle = dbi.open();
    createTablesWithRetry(dbi);
    temporary = createTempDir();
    AssignmentLimiter assignmentLimiter = new AssignmentLimiter(ImmutableSet::of, systemTicker(), new MetadataConfig());
    shardManager = new DatabaseShardManager(dbi, new DaoSupplier<>(dbi, ShardDao.class), ImmutableSet::of, assignmentLimiter, systemTicker(), new Duration(0, MINUTES));
    TestingNodeManager nodeManager = new TestingNodeManager();
    NodeSupplier nodeSupplier = nodeManager::getWorkerNodes;
    String nodeName = UUID.randomUUID().toString();
    nodeManager.addNode(new InternalNode(nodeName, new URI("http://127.0.0.1/"), NodeVersion.UNKNOWN, false));
    RaptorConnectorId connectorId = new RaptorConnectorId("raptor");
    metadata = new RaptorMetadata(connectorId.toString(), dbi, shardManager, createTestFunctionAndTypeManager());
    metadata.createTable(SESSION, TEST_TABLE, false);
    tableHandle = metadata.getTableHandle(SESSION, TEST_TABLE.getTable());
    List<ShardInfo> shards = ImmutableList.<ShardInfo>builder().add(shardInfo(UUID.randomUUID(), nodeName)).add(shardInfo(UUID.randomUUID(), nodeName)).add(shardInfo(UUID.randomUUID(), nodeName)).add(shardInfo(UUID.randomUUID(), nodeName)).build();
    tableId = ((RaptorTableHandle) tableHandle).getTableId();
    List<ColumnInfo> columns = metadata.getColumnHandles(SESSION, tableHandle).values().stream().map(RaptorColumnHandle.class::cast).map(ColumnInfo::fromHandle).collect(toList());
    long transactionId = shardManager.beginTransaction();
    shardManager.commitShards(transactionId, tableId, columns, shards, Optional.empty(), 0);
    raptorSplitManager = new RaptorSplitManager(connectorId, nodeSupplier, shardManager, false);
}
Also used : RaptorColumnHandle(com.facebook.presto.raptor.RaptorColumnHandle) RaptorMetadata(com.facebook.presto.raptor.RaptorMetadata) DBI(org.skife.jdbi.v2.DBI) Duration(io.airlift.units.Duration) DaoSupplier(com.facebook.presto.raptor.util.DaoSupplier) URI(java.net.URI) FunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager) FunctionAndTypeManager.createTestFunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager.createTestFunctionAndTypeManager) ImmutableSet(com.google.common.collect.ImmutableSet) TestingNodeManager(com.facebook.presto.testing.TestingNodeManager) InternalNode(com.facebook.presto.metadata.InternalNode) RaptorConnectorId(com.facebook.presto.raptor.RaptorConnectorId) NodeSupplier(com.facebook.presto.raptor.NodeSupplier) RaptorSplitManager(com.facebook.presto.raptor.RaptorSplitManager) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 2 with NodeSupplier

use of com.facebook.presto.raptor.NodeSupplier in project presto by prestodb.

the class TestBucketBalancer method setup.

@BeforeMethod
public void setup() {
    FunctionAndTypeManager functionAndTypeManager = createTestFunctionAndTypeManager();
    dbi = new DBI("jdbc:h2:mem:test" + System.nanoTime() + "_" + ThreadLocalRandom.current().nextInt());
    dbi.registerMapper(new Distribution.Mapper(functionAndTypeManager));
    dummyHandle = dbi.open();
    createTablesWithRetry(dbi);
    metadataDao = dbi.onDemand(MetadataDao.class);
    nodeManager = new TestingNodeManager(AVAILABLE_WORKERS.stream().map(TestBucketBalancer::createTestingNode).collect(Collectors.toList()));
    NodeSupplier nodeSupplier = nodeManager::getWorkerNodes;
    shardManager = createShardManager(dbi, nodeSupplier);
    balancer = new BucketBalancer(nodeSupplier, shardManager, true, new Duration(1, DAYS), 0, true, true, "test");
}
Also used : FunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager) FunctionAndTypeManager.createTestFunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager.createTestFunctionAndTypeManager) TestingNodeManager(com.facebook.presto.testing.TestingNodeManager) Distribution(com.facebook.presto.raptor.metadata.Distribution) DBI(org.skife.jdbi.v2.DBI) Duration(io.airlift.units.Duration) NodeSupplier(com.facebook.presto.raptor.NodeSupplier) MetadataDao(com.facebook.presto.raptor.metadata.MetadataDao) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 3 with NodeSupplier

use of com.facebook.presto.raptor.NodeSupplier in project presto by prestodb.

the class TestRaptorMetadata method setupDatabase.

@BeforeMethod
public void setupDatabase() {
    FunctionAndTypeManager functionAndTypeManager = createTestFunctionAndTypeManager();
    dbi = new DBI("jdbc:h2:mem:test" + System.nanoTime() + "_" + ThreadLocalRandom.current().nextInt());
    dbi.registerMapper(new TableColumn.Mapper(functionAndTypeManager));
    dbi.registerMapper(new Distribution.Mapper(functionAndTypeManager));
    dummyHandle = dbi.open();
    createTablesWithRetry(dbi);
    RaptorConnectorId connectorId = new RaptorConnectorId("raptor");
    NodeManager nodeManager = new TestingNodeManager();
    NodeSupplier nodeSupplier = nodeManager::getWorkerNodes;
    shardManager = createShardManager(dbi, nodeSupplier, systemTicker());
    metadata = new RaptorMetadata(connectorId.toString(), dbi, shardManager, createTestFunctionAndTypeManager());
}
Also used : NodeManager(com.facebook.presto.spi.NodeManager) TestingNodeManager(com.facebook.presto.testing.TestingNodeManager) FunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager) FunctionAndTypeManager.createTestFunctionAndTypeManager(com.facebook.presto.metadata.FunctionAndTypeManager.createTestFunctionAndTypeManager) TestingNodeManager(com.facebook.presto.testing.TestingNodeManager) RaptorMetadata(com.facebook.presto.raptor.RaptorMetadata) DBI(org.skife.jdbi.v2.DBI) RaptorConnectorId(com.facebook.presto.raptor.RaptorConnectorId) NodeSupplier(com.facebook.presto.raptor.NodeSupplier) BeforeMethod(org.testng.annotations.BeforeMethod)

Example 4 with NodeSupplier

use of com.facebook.presto.raptor.NodeSupplier in project presto by prestodb.

the class TestRaptorSplitManager method testAssignRandomNodeWhenBackupAvailable.

@Test
public void testAssignRandomNodeWhenBackupAvailable() throws URISyntaxException {
    TestingNodeManager nodeManager = new TestingNodeManager();
    RaptorConnectorId connectorId = new RaptorConnectorId("raptor");
    NodeSupplier nodeSupplier = nodeManager::getWorkerNodes;
    InternalNode node = new InternalNode(UUID.randomUUID().toString(), new URI("http://127.0.0.1/"), NodeVersion.UNKNOWN, false);
    nodeManager.addNode(node);
    RaptorSplitManager raptorSplitManagerWithBackup = new RaptorSplitManager(connectorId, nodeSupplier, shardManager, true);
    deleteShardNodes();
    ConnectorTableLayoutResult layout = getOnlyElement(metadata.getTableLayouts(SESSION, tableHandle, Constraint.alwaysTrue(), Optional.empty()));
    ConnectorSplitSource partitionSplit = getSplits(raptorSplitManagerWithBackup, layout);
    List<ConnectorSplit> batch = getSplits(partitionSplit, 1);
    assertEquals(getOnlyElement(((RaptorSplit) getOnlyElement(batch)).getAddresses()), node.getHostAndPort());
}
Also used : ConnectorTableLayoutResult(com.facebook.presto.spi.ConnectorTableLayoutResult) RaptorSplit(com.facebook.presto.raptor.RaptorSplit) TestingNodeManager(com.facebook.presto.testing.TestingNodeManager) InternalNode(com.facebook.presto.metadata.InternalNode) ConnectorSplitSource(com.facebook.presto.spi.ConnectorSplitSource) RaptorConnectorId(com.facebook.presto.raptor.RaptorConnectorId) URI(java.net.URI) ConnectorSplit(com.facebook.presto.spi.ConnectorSplit) NodeSupplier(com.facebook.presto.raptor.NodeSupplier) RaptorSplitManager(com.facebook.presto.raptor.RaptorSplitManager) Test(org.testng.annotations.Test)

Aggregations

NodeSupplier (com.facebook.presto.raptor.NodeSupplier)4 TestingNodeManager (com.facebook.presto.testing.TestingNodeManager)4 FunctionAndTypeManager (com.facebook.presto.metadata.FunctionAndTypeManager)3 FunctionAndTypeManager.createTestFunctionAndTypeManager (com.facebook.presto.metadata.FunctionAndTypeManager.createTestFunctionAndTypeManager)3 RaptorConnectorId (com.facebook.presto.raptor.RaptorConnectorId)3 DBI (org.skife.jdbi.v2.DBI)3 BeforeMethod (org.testng.annotations.BeforeMethod)3 InternalNode (com.facebook.presto.metadata.InternalNode)2 RaptorMetadata (com.facebook.presto.raptor.RaptorMetadata)2 RaptorSplitManager (com.facebook.presto.raptor.RaptorSplitManager)2 Duration (io.airlift.units.Duration)2 URI (java.net.URI)2 RaptorColumnHandle (com.facebook.presto.raptor.RaptorColumnHandle)1 RaptorSplit (com.facebook.presto.raptor.RaptorSplit)1 Distribution (com.facebook.presto.raptor.metadata.Distribution)1 MetadataDao (com.facebook.presto.raptor.metadata.MetadataDao)1 DaoSupplier (com.facebook.presto.raptor.util.DaoSupplier)1 ConnectorSplit (com.facebook.presto.spi.ConnectorSplit)1 ConnectorSplitSource (com.facebook.presto.spi.ConnectorSplitSource)1 ConnectorTableLayoutResult (com.facebook.presto.spi.ConnectorTableLayoutResult)1