Search in sources :

Example 1 with OffsetOutOfRangeException

use of kafka.common.OffsetOutOfRangeException in project graylog2-server by Graylog2.

the class KafkaJournal method read.

public List<JournalReadEntry> read(long readOffset, long requestedMaximumCount) {
    // Always read at least one!
    final long maximumCount = Math.max(1, requestedMaximumCount);
    long maxOffset = readOffset + maximumCount;
    if (shuttingDown) {
        return Collections.emptyList();
    }
    final List<JournalReadEntry> messages = new ArrayList<>(Ints.saturatedCast(maximumCount));
    try (Timer.Context ignored = readTime.time()) {
        final long logStartOffset = getLogStartOffset();
        if (readOffset < logStartOffset) {
            LOG.info("Read offset {} before start of log at {}, starting to read from the beginning of the journal.", readOffset, logStartOffset);
            readOffset = logStartOffset;
            maxOffset = readOffset + maximumCount;
        }
        LOG.debug("Requesting to read a maximum of {} messages (or 5MB) from the journal, offset interval [{}, {})", maximumCount, readOffset, maxOffset);
        // TODO benchmark and make read-ahead strategy configurable for performance tuning
        final MessageSet messageSet = kafkaLog.read(readOffset, 5 * 1024 * 1024, Option.<Object>apply(maxOffset)).messageSet();
        final Iterator<MessageAndOffset> iterator = messageSet.iterator();
        long firstOffset = Long.MIN_VALUE;
        long lastOffset = Long.MIN_VALUE;
        long totalBytes = 0;
        while (iterator.hasNext()) {
            final MessageAndOffset messageAndOffset = iterator.next();
            if (firstOffset == Long.MIN_VALUE)
                firstOffset = messageAndOffset.offset();
            // always remember the last seen offset for debug purposes below
            lastOffset = messageAndOffset.offset();
            final byte[] payloadBytes = ByteBufferUtils.readBytes(messageAndOffset.message().payload());
            if (LOG.isTraceEnabled()) {
                final byte[] keyBytes = ByteBufferUtils.readBytes(messageAndOffset.message().key());
                LOG.trace("Read message {} contains {}", bytesToHex(keyBytes), bytesToHex(payloadBytes));
            }
            totalBytes += payloadBytes.length;
            messages.add(new JournalReadEntry(payloadBytes, messageAndOffset.offset()));
            // remember where to read from
            nextReadOffset = messageAndOffset.nextOffset();
        }
        if (messages.isEmpty()) {
            LOG.debug("No messages available to read for offset interval [{}, {}).", readOffset, maxOffset);
        } else {
            LOG.debug("Read {} messages, total payload size {}, from journal, offset interval [{}, {}], requested read at {}", messages.size(), totalBytes, firstOffset, lastOffset, readOffset);
        }
    } catch (OffsetOutOfRangeException e) {
        // This is fine, the reader tries to read faster than the writer committed data. Next read will get the data.
        LOG.debug("Offset out of range, no messages available starting at offset {}", readOffset);
    } catch (Exception e) {
        // sigh.
        if (shuttingDown) {
            LOG.debug("Caught exception during shutdown, ignoring it because we might have been blocked on a read.");
            return Collections.emptyList();
        }
        //noinspection ConstantConditions
        if (e instanceof ClosedByInterruptException) {
            LOG.debug("Interrupted while reading from journal, during shutdown this is harmless and ignored.", e);
        } else {
            throw e;
        }
    }
    readMessages.mark(messages.size());
    return messages;
}
Also used : ArrayList(java.util.ArrayList) MessageAndOffset(kafka.message.MessageAndOffset) ClosedByInterruptException(java.nio.channels.ClosedByInterruptException) SyncFailedException(java.io.SyncFailedException) AccessDeniedException(java.nio.file.AccessDeniedException) KafkaException(kafka.common.KafkaException) OffsetOutOfRangeException(kafka.common.OffsetOutOfRangeException) IOException(java.io.IOException) MessageSet(kafka.message.MessageSet) ByteBufferMessageSet(kafka.message.ByteBufferMessageSet) ClosedByInterruptException(java.nio.channels.ClosedByInterruptException) HdrTimer(org.graylog2.shared.metrics.HdrTimer) Timer(com.codahale.metrics.Timer) OffsetOutOfRangeException(kafka.common.OffsetOutOfRangeException)

Example 2 with OffsetOutOfRangeException

use of kafka.common.OffsetOutOfRangeException in project cdap by caskdata.

the class KafkaConsumer method fetchMessageSet.

private ByteBufferMessageSet fetchMessageSet(long fetchOffset) throws OffsetOutOfRangeException {
    Preconditions.checkArgument(fetchOffset >= 0, String.format("Illegal fetch offset %d", fetchOffset));
    int failureCount = 0;
    while (true) {
        SimpleConsumer consumer = getConsumer();
        FetchRequest req = new FetchRequestBuilder().clientId(clientName).addFetch(topic, partition, fetchOffset, BUFFER_SIZE_BYTES).maxWait(fetchTimeoutMs).build();
        FetchResponse fetchResponse = consumer.fetch(req);
        if (!fetchResponse.hasError()) {
            return fetchResponse.messageSet(topic, partition);
        }
        short errorCode = fetchResponse.errorCode(topic, partition);
        if (++failureCount >= MAX_KAFKA_FETCH_RETRIES) {
            throw new RuntimeException(String.format("Error fetching data from broker %s:%d for topic %s, partition %d. Error code: %d", consumer.host(), consumer.port(), topic, partition, errorCode));
        }
        LOG.warn("Error fetching data from broker {}:{} for topic {}, partition {}. Error code: {}", consumer.host(), consumer.port(), topic, partition, errorCode);
        if (errorCode == ErrorMapping.OffsetOutOfRangeCode()) {
            throw new OffsetOutOfRangeException(String.format("Requested offset %d is out of range for topic %s partition %d", fetchOffset, topic, partition));
        }
        closeConsumer();
    }
}
Also used : FetchRequestBuilder(kafka.api.FetchRequestBuilder) FetchRequest(kafka.api.FetchRequest) FetchResponse(kafka.javaapi.FetchResponse) OffsetOutOfRangeException(kafka.common.OffsetOutOfRangeException) SimpleConsumer(kafka.javaapi.consumer.SimpleConsumer)

Aggregations

OffsetOutOfRangeException (kafka.common.OffsetOutOfRangeException)2 Timer (com.codahale.metrics.Timer)1 IOException (java.io.IOException)1 SyncFailedException (java.io.SyncFailedException)1 ClosedByInterruptException (java.nio.channels.ClosedByInterruptException)1 AccessDeniedException (java.nio.file.AccessDeniedException)1 ArrayList (java.util.ArrayList)1 FetchRequest (kafka.api.FetchRequest)1 FetchRequestBuilder (kafka.api.FetchRequestBuilder)1 KafkaException (kafka.common.KafkaException)1 FetchResponse (kafka.javaapi.FetchResponse)1 SimpleConsumer (kafka.javaapi.consumer.SimpleConsumer)1 ByteBufferMessageSet (kafka.message.ByteBufferMessageSet)1 MessageAndOffset (kafka.message.MessageAndOffset)1 MessageSet (kafka.message.MessageSet)1 HdrTimer (org.graylog2.shared.metrics.HdrTimer)1