Examples with SimpleConsumer kafka.javaapi.consumer.SimpleConsumer used on opensource projects

Example 1 with SimpleConsumer

use of kafka.javaapi.consumer.SimpleConsumer in project kafka by apache.

the class SimpleConsumerDemo method main.

public static void main(String[] args) throws Exception {
    generateData();
    SimpleConsumer simpleConsumer = new SimpleConsumer(KafkaProperties.KAFKA_SERVER_URL, KafkaProperties.KAFKA_SERVER_PORT, KafkaProperties.CONNECTION_TIMEOUT, KafkaProperties.KAFKA_PRODUCER_BUFFER_SIZE, KafkaProperties.CLIENT_ID);
    System.out.println("Testing single fetch");
    FetchRequest req = new FetchRequestBuilder().clientId(KafkaProperties.CLIENT_ID).addFetch(KafkaProperties.TOPIC2, 0, 0L, 100).build();
    FetchResponse fetchResponse = simpleConsumer.fetch(req);
    printMessages(fetchResponse.messageSet(KafkaProperties.TOPIC2, 0));
    System.out.println("Testing single multi-fetch");
    Map<String, List<Integer>> topicMap = new HashMap<>();
    topicMap.put(KafkaProperties.TOPIC2, Collections.singletonList(0));
    topicMap.put(KafkaProperties.TOPIC3, Collections.singletonList(0));
    req = new FetchRequestBuilder().clientId(KafkaProperties.CLIENT_ID).addFetch(KafkaProperties.TOPIC2, 0, 0L, 100).addFetch(KafkaProperties.TOPIC3, 0, 0L, 100).build();
    fetchResponse = simpleConsumer.fetch(req);
    int fetchReq = 0;
    for (Map.Entry<String, List<Integer>> entry : topicMap.entrySet()) {
        String topic = entry.getKey();
        for (Integer offset : entry.getValue()) {
            System.out.println("Response from fetch request no: " + ++fetchReq);
            printMessages(fetchResponse.messageSet(topic, offset));
        }
    }
}

Example 2 with SimpleConsumer

use of kafka.javaapi.consumer.SimpleConsumer in project storm by apache.

the class StaticPartitionConnections method getConsumer.

public SimpleConsumer getConsumer(int partition) {
    if (!_kafka.containsKey(partition)) {
        Broker hp = hosts.getPartitionInformation().getBrokerFor(partition);
        _kafka.put(partition, new SimpleConsumer(hp.host, hp.port, _config.socketTimeoutMs, _config.bufferSizeBytes, _config.clientId));
    }
    return _kafka.get(partition);
}

Example 3 with SimpleConsumer

use of kafka.javaapi.consumer.SimpleConsumer in project storm by apache.

the class KafkaBoltTest method setupKafkaConsumer.

private void setupKafkaConsumer() {
    GlobalPartitionInformation globalPartitionInformation = new GlobalPartitionInformation(TEST_TOPIC);
    globalPartitionInformation.addPartition(0, Broker.fromString(broker.getBrokerConnectionString()));
    BrokerHosts brokerHosts = new StaticHosts(globalPartitionInformation);
    kafkaConfig = new KafkaConfig(brokerHosts, TEST_TOPIC);
    simpleConsumer = new SimpleConsumer("localhost", broker.getPort(), 60000, 1024, "testClient");
}

Example 4 with SimpleConsumer

use of kafka.javaapi.consumer.SimpleConsumer in project storm by apache.

the class KafkaBoltTest method mockSimpleConsumer.

private static SimpleConsumer mockSimpleConsumer(ByteBufferMessageSet mockMsg) {
    SimpleConsumer simpleConsumer = mock(SimpleConsumer.class);
    FetchResponse resp = mock(FetchResponse.class);
    doReturn(resp).when(simpleConsumer).fetch(any(FetchRequest.class));
    OffsetResponse mockOffsetResponse = mock(OffsetResponse.class);
    doReturn(new long[] {}).when(mockOffsetResponse).offsets(anyString(), anyInt());
    doReturn(mockOffsetResponse).when(simpleConsumer).getOffsetsBefore(any(kafka.javaapi.OffsetRequest.class));
    doReturn(mockMsg).when(resp).messageSet(anyString(), anyInt());
    return simpleConsumer;
}

Example 5 with SimpleConsumer

use of kafka.javaapi.consumer.SimpleConsumer in project druid by druid-io.

the class KafkaSimpleConsumer method ensureConsumer.

private void ensureConsumer(Broker leader) throws InterruptedException {
    if (consumer == null) {
        while (leaderBroker == null) {
            leaderBroker = findNewLeader(leader);
        }
        log.info("making SimpleConsumer[%s][%s], leader broker[%s:%s]", topic, partitionId, leaderBroker.host(), leaderBroker.port());
        consumer = new SimpleConsumer(leaderBroker.host(), leaderBroker.port(), SO_TIMEOUT, BUFFER_SIZE, clientId);
    }
}