Examples with KafkaMessageId nl.minvenj.nfi.storm.kafka.util.KafkaMessageId used on opensource projects

Example 6 with KafkaMessageId

use of nl.minvenj.nfi.storm.kafka.util.KafkaMessageId in project kafka-spout by HolmesNL.

the class ReliableFailHandlerTest method testRefuseFail.

@Test(expected = IllegalStateException.class)
public void testRefuseFail() {
    final FailHandler subject = new ReliableFailHandler();
    final KafkaMessageId id = new KafkaMessageId(1, 1234);
    final byte[] message = { 1, 2, 3, 4 };
    // failing a message to the reliable handler should *always* throw an exception
    subject.fail(id, message);
}

Example 7 with KafkaMessageId

use of nl.minvenj.nfi.storm.kafka.util.KafkaMessageId in project kafka-spout by HolmesNL.

the class ReliableFailHandlerTest method testSideEffects.

@Test
public void testSideEffects() {
    final FailHandler subject = new ReliableFailHandler();
    final KafkaMessageId id = new KafkaMessageId(1, 1234);
    // convenience methods should have no effect
    subject.open(mock(Map.class), mock(TopologyContext.class), mock(SpoutOutputCollector.class));
    subject.activate();
    subject.deactivate();
    subject.close();
    // ack should be ignored
    subject.ack(id);
}

Example 8 with KafkaMessageId

use of nl.minvenj.nfi.storm.kafka.util.KafkaMessageId in project kafka-spout by HolmesNL.

the class UnreliableFailHandlerTest method testFail.

@Test
public void testFail() {
    final FailHandler subject = new UnreliableFailHandler();
    final KafkaMessageId id = new KafkaMessageId(1, 1234);
    final byte[] message = { 1, 2, 3, 4 };
    // failing a message to the unreliable handler should not throw an exception
    subject.fail(id, message);
}

Example 9 with KafkaMessageId

use of nl.minvenj.nfi.storm.kafka.util.KafkaMessageId in project kafka-spout by HolmesNL.

the class KafkaSpout method fillBuffer.

/**
 * Refills the buffer with messages from the configured kafka topic if available.
 *
 * @return Whether the buffer contains messages to be emitted after this call.
 * @throws IllegalStateException When current buffer is not empty or messages not acknowledged by topology.
 */
protected boolean fillBuffer() {
    if (!_inProgress.isEmpty() || !_queue.isEmpty()) {
        throw new IllegalStateException("cannot fill buffer when buffer or pending messages are non-empty");
    }
    if (_iterator == null) {
        // create a stream of messages from _consumer using the streams as defined on construction
        final Map<String, List<KafkaStream<byte[], byte[]>>> streams = _consumer.createMessageStreams(Collections.singletonMap(_topic, 1));
        _iterator = streams.get(_topic).get(0).iterator();
    }
    // throwing a ConsumerTimeoutException when the configured timeout is exceeded.
    try {
        int size = 0;
        while (size < _bufSize && _iterator.hasNext()) {
            final MessageAndMetadata<byte[], byte[]> message = _iterator.next();
            final KafkaMessageId id = new KafkaMessageId(message.partition(), message.offset());
            _inProgress.put(id, message.message());
            size++;
        }
    } catch (final InvalidMessageException e) {
        LOG.warn(e.getMessage(), e);
    } catch (final ConsumerTimeoutException e) {
    // ignore, storm will call nextTuple again at some point in the near future
    // timeout does *not* mean that no messages were read (state is checked below)
    }
    if (_inProgress.size() > 0) {
        // set _queue to all currently pending kafka message ids
        _queue.addAll(_inProgress.keySet());
        LOG.debug("buffer now has {} messages to be emitted", _queue.size());
        // message(s) appended to buffer
        return true;
    } else {
        // no messages appended to buffer
        return false;
    }
}

Example 10 with KafkaMessageId

use of nl.minvenj.nfi.storm.kafka.util.KafkaMessageId in project kafka-spout by HolmesNL.

the class KafkaSpout method fail.

@Override
public void fail(final Object o) {
    if (o instanceof KafkaMessageId) {
        final KafkaMessageId id = (KafkaMessageId) o;
        // delegate decision of replaying the message to failure policy
        if (_failHandler.shouldReplay(id)) {
            LOG.debug("kafka message id {} failed in topology, adding to buffer again", id);
            _queue.add(id);
        } else {
            LOG.debug("kafka message id {} failed in topology, delegating failure to policy", id);
            // remove message from pending; _failHandler will take action if needed
            _failHandler.fail(id, _inProgress.remove(id));
        }
    }
}