Examples with KafkaStream kafka.consumer.KafkaStream used on opensource projects

Example 11 with KafkaStream

use of kafka.consumer.KafkaStream in project voltdb by VoltDB.

the class KafkaLoader method getConsumerExecutor.

private ExecutorService getConsumerExecutor(KafkaConsumerConnector consumer, CSVDataLoader loader) throws Exception {
    Map<String, Integer> topicCountMap = new HashMap<>();
    // generate as many threads as there are partitions defined in kafka config
    ExecutorService executor = Executors.newFixedThreadPool(m_config.kpartitions);
    topicCountMap.put(m_config.topic, m_config.kpartitions);
    Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.m_consumer.createMessageStreams(topicCountMap);
    List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(m_config.topic);
    // now launch all the threads for partitions.
    for (final KafkaStream stream : streams) {
        KafkaConsumer bconsumer = new KafkaConsumer(stream, loader, m_config);
        executor.submit(bconsumer);
    }
    return executor;
}

Example 12 with KafkaStream

use of kafka.consumer.KafkaStream in project incubator-gobblin by apache.

the class MockKafkaStream method createMockStream.

@SuppressWarnings("unchecked")
private static KafkaStream<byte[], byte[]> createMockStream(BlockingQueue<FetchedDataChunk> queue) {
    KafkaStream<byte[], byte[]> stream = (KafkaStream<byte[], byte[]>) Mockito.mock(KafkaStream.class);
    ConsumerIterator<byte[], byte[]> it = new ConsumerIterator<>(queue, -1, new DefaultDecoder(new VerifiableProperties()), new DefaultDecoder(new VerifiableProperties()), "clientId");
    Mockito.when(stream.iterator()).thenReturn(it);
    return stream;
}