Examples with ByteBufferMessageSet kafka.javaapi.message.ByteBufferMessageSet used on opensource projects

Example 31 with ByteBufferMessageSet

use of kafka.javaapi.message.ByteBufferMessageSet in project heron by twitter.

the class KafkaUtilsTest method generateTuplesWithValueAndStringMultiSchemeWithTopic.

@Test
public void generateTuplesWithValueAndStringMultiSchemeWithTopic() {
    config.scheme = new StringMultiSchemeWithTopic();
    String value = "value";
    createTopicAndSendMessage(value);
    ByteBufferMessageSet messageAndOffsets = getLastMessage();
    for (MessageAndOffset msg : messageAndOffsets) {
        Iterable<List<Object>> lists = KafkaUtils.generateTuples(config, msg.message(), config.topic);
        List<Object> list = lists.iterator().next();
        assertEquals(value, list.get(0));
        assertEquals(config.topic, list.get(1));
    }
}

Example 32 with ByteBufferMessageSet

use of kafka.javaapi.message.ByteBufferMessageSet in project heron by twitter.

the class KafkaUtilsTest method generateTuplesWithValueSchemeAndKeyValueMessage.

@Test
public void generateTuplesWithValueSchemeAndKeyValueMessage() {
    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    String value = "value";
    String key = "key";
    createTopicAndSendMessage(key, value);
    ByteBufferMessageSet messageAndOffsets = getLastMessage();
    for (MessageAndOffset msg : messageAndOffsets) {
        Iterable<List<Object>> lists = KafkaUtils.generateTuples(config, msg.message(), config.topic);
        assertEquals(value, lists.iterator().next().get(0));
    }
}

Example 33 with ByteBufferMessageSet

use of kafka.javaapi.message.ByteBufferMessageSet in project heron by twitter.

the class KafkaUtilsTest method fetchMessage.

@Test
public void fetchMessage() throws Exception {
    String value = "test";
    createTopicAndSendMessage(value);
    long offset = KafkaUtils.getOffset(simpleConsumer, config.topic, 0, OffsetRequest.LatestTime()) - 1;
    ByteBufferMessageSet messageAndOffsets = KafkaUtils.fetchMessages(config, simpleConsumer, new Partition(Broker.fromString(broker.getBrokerConnectionString()), TEST_TOPIC, 0), offset);
    String message = new String(Utils.toByteArray(messageAndOffsets.iterator().next().message().payload()));
    assertThat(message, is(equalTo(value)));
}

Example 34 with ByteBufferMessageSet

use of kafka.javaapi.message.ByteBufferMessageSet in project cdap by caskdata.

the class KafkaConsumer method fetchMessages.

/**
   * Fetches Kafka messages from an offset.
   * @param offset message offset to start.
   * @param callback callback to handle the messages fetched.
   * @return number of messages fetched.
   */
public int fetchMessages(long offset, Callback callback) throws OffsetOutOfRangeException {
    ByteBufferMessageSet messageSet = fetchMessageSet(offset);
    int msgCount = 0;
    for (MessageAndOffset msg : messageSet) {
        ++msgCount;
        callback.handle(msg.offset(), msg.message().payload());
    }
    return msgCount;
}