Search in sources :

Example 11 with Node

use of io.trino.spi.Node in project trino by trinodb.

the class TpchSplitManager method getSplits.

@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableHandle table, SplitSchedulingStrategy splitSchedulingStrategy, DynamicFilter dynamicFilter) {
    Set<Node> nodes = nodeManager.getRequiredWorkerNodes();
    int totalParts = nodes.size() * splitsPerNode;
    int partNumber = 0;
    // Split the data using split and skew by the number of nodes available.
    ImmutableList.Builder<ConnectorSplit> splits = ImmutableList.builder();
    for (Node node : nodes) {
        for (int i = 0; i < splitsPerNode; i++) {
            splits.add(new TpchSplit(partNumber, totalParts, ImmutableList.of(node.getHostAndPort())));
            partNumber++;
        }
    }
    return new FixedSplitSource(splits.build());
}
Also used : ImmutableList(com.google.common.collect.ImmutableList) FixedSplitSource(io.trino.spi.connector.FixedSplitSource) Node(io.trino.spi.Node) ConnectorSplit(io.trino.spi.connector.ConnectorSplit)

Aggregations

Node (io.trino.spi.Node)11 ImmutableList (com.google.common.collect.ImmutableList)5 InternalNode (io.trino.metadata.InternalNode)4 ConnectorSplit (io.trino.spi.connector.ConnectorSplit)4 FixedSplitSource (io.trino.spi.connector.FixedSplitSource)3 ImmutableList.toImmutableList (com.google.common.collect.ImmutableList.toImmutableList)2 ImmutableSet (com.google.common.collect.ImmutableSet)2 Duration (io.airlift.units.Duration)2 CatalogName (io.trino.plugin.base.CatalogName)2 HashMap (java.util.HashMap)2 List (java.util.List)2 Test (org.testng.annotations.Test)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Joiner (com.google.common.base.Joiner)1 VerifyException (com.google.common.base.VerifyException)1 HashMultimap (com.google.common.collect.HashMultimap)1 HashMultiset (com.google.common.collect.HashMultiset)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 ImmutableMultimap (com.google.common.collect.ImmutableMultimap)1 Multimap (com.google.common.collect.Multimap)1