Examples with GlobalPartitionInformation org.apache.storm.kafka.trident.GlobalPartitionInformation used on opensource projects

Example 21 with GlobalPartitionInformation

use of org.apache.storm.kafka.trident.GlobalPartitionInformation in project storm by apache.

the class TestUtils method getBrokerHosts.

private static BrokerHosts getBrokerHosts(KafkaTestBroker broker) {
    GlobalPartitionInformation globalPartitionInformation = new GlobalPartitionInformation(TOPIC);
    globalPartitionInformation.addPartition(0, Broker.fromString(broker.getBrokerConnectionString()));
    return new StaticHosts(globalPartitionInformation);
}

Example 22 with GlobalPartitionInformation

use of org.apache.storm.kafka.trident.GlobalPartitionInformation in project heron by twitter.

the class KafkaUtils method calculatePartitionsForTask.

public static List<Partition> calculatePartitionsForTask(List<GlobalPartitionInformation> partitons, int totalTasks, int taskIndex) {
    Preconditions.checkArgument(taskIndex < totalTasks, "task index must be less that total tasks");
    List<Partition> taskPartitions = new ArrayList<Partition>();
    List<Partition> partitions = new ArrayList<Partition>();
    for (GlobalPartitionInformation partitionInformation : partitons) {
        partitions.addAll(partitionInformation.getOrderedPartitions());
    }
    int numPartitions = partitions.size();
    if (numPartitions < totalTasks) {
        LOG.warn("there are more tasks than partitions (tasks: " + totalTasks + "; partitions: " + numPartitions + "), some tasks will be idle");
    }
    for (int i = taskIndex; i < numPartitions; i += totalTasks) {
        Partition taskPartition = partitions.get(i);
        taskPartitions.add(taskPartition);
    }
    logPartitionMapping(totalTasks, taskIndex, taskPartitions);
    return taskPartitions;
}

Example 23 with GlobalPartitionInformation

use of org.apache.storm.kafka.trident.GlobalPartitionInformation in project heron by twitter.

the class DynamicBrokersReaderTest method testMultiplePartitionsOnDifferentHosts.

@Test
public void testMultiplePartitionsOnDifferentHosts() throws Exception {
    String host = "localhost";
    int port = 9092;
    int secondPort = 9093;
    int partition = 0;
    int secondPartition = partition + 1;
    addPartition(partition, 0, host, port, topic);
    addPartition(secondPartition, 1, host, secondPort, topic);
    List<GlobalPartitionInformation> partitions = dynamicBrokersReader.getBrokerInfo();
    GlobalPartitionInformation brokerInfo = getByTopic(partitions, topic);
    assertNotNull(brokerInfo);
    assertEquals(2, brokerInfo.getOrderedPartitions().size());
    assertEquals(port, brokerInfo.getBrokerFor(partition).port);
    assertEquals(host, brokerInfo.getBrokerFor(partition).host);
    assertEquals(secondPort, brokerInfo.getBrokerFor(secondPartition).port);
    assertEquals(host, brokerInfo.getBrokerFor(secondPartition).host);
}

Example 24 with GlobalPartitionInformation

use of org.apache.storm.kafka.trident.GlobalPartitionInformation in project heron by twitter.

the class KafkaUtilsTest method moreTasksThanPartitions.

@Test
public void moreTasksThanPartitions() {
    GlobalPartitionInformation globalPartitionInformation = TestUtils.buildPartitionInfo(1);
    List<GlobalPartitionInformation> partitions = new ArrayList<GlobalPartitionInformation>();
    partitions.add(globalPartitionInformation);
    int numTasks = 2;
    assertEquals(1, KafkaUtils.calculatePartitionsForTask(partitions, numTasks, 0).size());
    assertEquals(0, KafkaUtils.calculatePartitionsForTask(partitions, numTasks, 1).size());
}

Example 25 with GlobalPartitionInformation

use of org.apache.storm.kafka.trident.GlobalPartitionInformation in project heron by twitter.

the class TestUtils method getBrokerHosts.

private static BrokerHosts getBrokerHosts(KafkaTestBroker broker) {
    GlobalPartitionInformation globalPartitionInformation = new GlobalPartitionInformation(TOPIC);
    globalPartitionInformation.addPartition(0, Broker.fromString(broker.getBrokerConnectionString()));
    return new StaticHosts(globalPartitionInformation);
}