Examples with LeaderNotAvailableException org.apache.kafka.common.errors.LeaderNotAvailableException used on opensource projects

Example 1 with LeaderNotAvailableException

use of org.apache.kafka.common.errors.LeaderNotAvailableException in project cdap by caskdata.

the class KafkaLogProcessorPipeline method fetchMessages.

/**
 * Fetch messages from Kafka.
 *
 * @param partition the partition to fetch from
 * @param offset the Kafka offset to fetch from
 * @return An {@link Iterable} of {@link MessageAndOffset}.
 *
 * @throws LeaderNotAvailableException if there is no Kafka broker to talk to.
 * @throws OffsetOutOfRangeException if the given offset is out of range.
 * @throws NotLeaderForPartitionException if the broker that the consumer is talking to is not the leader
 *                                        for the given topic and partition.
 * @throws UnknownTopicOrPartitionException if the topic or partition is not known by the Kafka server
 * @throws UnknownServerException if the Kafka server responded with error.
 */
private Iterable<MessageAndOffset> fetchMessages(int partition, long offset) throws KafkaException {
    String topic = config.getTopic();
    KafkaSimpleConsumer consumer = getKafkaConsumer(topic, partition);
    if (consumer == null) {
        throw new LeaderNotAvailableException("No broker to fetch messages for " + topic + ":" + partition);
    }
    LOG.trace("Fetching messages from Kafka on {}:{} for pipeline {} with offset {}", topic, partition, name, offset);
    try {
        ByteBufferMessageSet result = KafkaUtil.fetchMessages(consumer, topic, partition, config.getKafkaFetchBufferSize(), offset);
        LOG.trace("Fetched {} bytes from Kafka on {}:{} for pipeline {}", result.sizeInBytes(), topic, partition, name);
        return result;
    } catch (OffsetOutOfRangeException e) {
        // If the error is not offset out of range, clear the consumer cache
        kafkaConsumers.remove(consumer.getBrokerInfo());
        throw e;
    }
}

Example 2 with LeaderNotAvailableException

use of org.apache.kafka.common.errors.LeaderNotAvailableException in project kafka by apache.

the class TopicAdmin method endOffsets.

/**
 * Fetch the most recent offset for each of the supplied {@link TopicPartition} objects.
 *
 * @param partitions the topic partitions
 * @return the map of offset for each topic partition, or an empty map if the supplied partitions
 *         are null or empty
 * @throws UnsupportedVersionException if the admin client cannot read end offsets
 * @throws TimeoutException if the offset metadata could not be fetched before the amount of time allocated
 *         by {@code request.timeout.ms} expires, and this call can be retried
 * @throws LeaderNotAvailableException if the leader was not available and this call can be retried
 * @throws RetriableException if a retriable error occurs, or the thread is interrupted while attempting
 *         to perform this operation
 * @throws ConnectException if a non retriable error occurs
 */
public Map<TopicPartition, Long> endOffsets(Set<TopicPartition> partitions) {
    if (partitions == null || partitions.isEmpty()) {
        return Collections.emptyMap();
    }
    Map<TopicPartition, OffsetSpec> offsetSpecMap = partitions.stream().collect(Collectors.toMap(Function.identity(), tp -> OffsetSpec.latest()));
    ListOffsetsResult resultFuture = admin.listOffsets(offsetSpecMap);
    // Get the individual result for each topic partition so we have better error messages
    Map<TopicPartition, Long> result = new HashMap<>();
    for (TopicPartition partition : partitions) {
        try {
            ListOffsetsResultInfo info = resultFuture.partitionResult(partition).get();
            result.put(partition, info.offset());
        } catch (ExecutionException e) {
            Throwable cause = e.getCause();
            String topic = partition.topic();
            if (cause instanceof AuthorizationException) {
                String msg = String.format("Not authorized to get the end offsets for topic '%s' on brokers at %s", topic, bootstrapServers());
                throw new ConnectException(msg, e);
            } else if (cause instanceof UnsupportedVersionException) {
                // Should theoretically never happen, because this method is the same as what the consumer uses and therefore
                // should exist in the broker since before the admin client was added
                String msg = String.format("API to get the get the end offsets for topic '%s' is unsupported on brokers at %s", topic, bootstrapServers());
                throw new UnsupportedVersionException(msg, e);
            } else if (cause instanceof TimeoutException) {
                String msg = String.format("Timed out while waiting to get end offsets for topic '%s' on brokers at %s", topic, bootstrapServers());
                throw new TimeoutException(msg, e);
            } else if (cause instanceof LeaderNotAvailableException) {
                String msg = String.format("Unable to get end offsets during leader election for topic '%s' on brokers at %s", topic, bootstrapServers());
                throw new LeaderNotAvailableException(msg, e);
            } else if (cause instanceof org.apache.kafka.common.errors.RetriableException) {
                throw (org.apache.kafka.common.errors.RetriableException) cause;
            } else {
                String msg = String.format("Error while getting end offsets for topic '%s' on brokers at %s", topic, bootstrapServers());
                throw new ConnectException(msg, e);
            }
        } catch (InterruptedException e) {
            Thread.interrupted();
            String msg = String.format("Interrupted while attempting to read end offsets for topic '%s' on brokers at %s", partition.topic(), bootstrapServers());
            throw new RetriableException(msg, e);
        }
    }
    return result;
}

Example 3 with LeaderNotAvailableException

use of org.apache.kafka.common.errors.LeaderNotAvailableException in project kafka by apache.

the class InternalTopicManagerTest method shouldThrowExceptionWhenKeepsTopicLeaderNotAvailable.

@Test
public void shouldThrowExceptionWhenKeepsTopicLeaderNotAvailable() {
    final AdminClient admin = EasyMock.createNiceMock(AdminClient.class);
    final MockTime time = new MockTime((Integer) config.get(StreamsConfig.consumerPrefix(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG)) / 15);
    final InternalTopicManager topicManager = new InternalTopicManager(time, admin, new StreamsConfig(config));
    final KafkaFutureImpl<TopicDescription> topicDescriptionFailFuture = new KafkaFutureImpl<>();
    topicDescriptionFailFuture.completeExceptionally(new LeaderNotAvailableException("Leader Not Available!"));
    // simulate describeTopics got LeaderNotAvailableException
    EasyMock.expect(admin.describeTopics(Collections.singleton(topic1))).andReturn(new MockDescribeTopicsResult(Collections.singletonMap(topic1, topicDescriptionFailFuture))).anyTimes();
    EasyMock.replay(admin);
    final InternalTopicConfig internalTopicConfig = new RepartitionTopicConfig(topic1, Collections.emptyMap());
    internalTopicConfig.setNumberOfPartitions(1);
    final TimeoutException exception = assertThrows(TimeoutException.class, () -> topicManager.makeReady(Collections.singletonMap(topic1, internalTopicConfig)));
    assertNull(exception.getCause());
    assertThat(exception.getMessage(), equalTo("Could not create topics within 50 milliseconds." + " This can happen if the Kafka cluster is temporarily not available."));
    EasyMock.verify(admin);
}

Example 4 with LeaderNotAvailableException

use of org.apache.kafka.common.errors.LeaderNotAvailableException in project kafka by apache.

the class InternalTopicManagerTest method shouldCompleteValidateWhenTopicLeaderNotAvailableAndThenDescribeSuccess.

@Test
public void shouldCompleteValidateWhenTopicLeaderNotAvailableAndThenDescribeSuccess() {
    final AdminClient admin = EasyMock.createNiceMock(AdminClient.class);
    final InternalTopicManager topicManager = new InternalTopicManager(time, admin, new StreamsConfig(config));
    final TopicPartitionInfo partitionInfo = new TopicPartitionInfo(0, broker1, Collections.singletonList(broker1), Collections.singletonList(broker1));
    final KafkaFutureImpl<TopicDescription> topicDescriptionFailFuture = new KafkaFutureImpl<>();
    topicDescriptionFailFuture.completeExceptionally(new LeaderNotAvailableException("Leader Not Available!"));
    final KafkaFutureImpl<TopicDescription> topicDescriptionSuccessFuture = new KafkaFutureImpl<>();
    topicDescriptionSuccessFuture.complete(new TopicDescription(topic1, false, Collections.singletonList(partitionInfo), Collections.emptySet()));
    EasyMock.expect(admin.describeTopics(Collections.singleton(topic1))).andReturn(new MockDescribeTopicsResult(Collections.singletonMap(topic1, topicDescriptionFailFuture))).once();
    EasyMock.expect(admin.describeTopics(Collections.singleton(topic1))).andReturn(new MockDescribeTopicsResult(Collections.singletonMap(topic1, topicDescriptionSuccessFuture))).once();
    EasyMock.replay(admin);
    final InternalTopicConfig internalTopicConfig = new RepartitionTopicConfig(topic1, Collections.emptyMap());
    internalTopicConfig.setNumberOfPartitions(1);
    topicManager.makeReady(Collections.singletonMap(topic1, internalTopicConfig));
    EasyMock.verify(admin);
}

Example 5 with LeaderNotAvailableException

use of org.apache.kafka.common.errors.LeaderNotAvailableException in project kafka by apache.

the class InternalTopicManager method cleanUpCreatedTopics.

private void cleanUpCreatedTopics(final Set<String> topicsToCleanUp) {
    log.info("Starting to clean up internal topics {}.", topicsToCleanUp);
    final long now = time.milliseconds();
    final long deadline = now + retryTimeoutMs;
    final Set<String> topicsStillToCleanup = new HashSet<>(topicsToCleanUp);
    while (!topicsStillToCleanup.isEmpty()) {
        log.info("Going to cleanup internal topics: " + topicsStillToCleanup);
        final DeleteTopicsResult deleteTopicsResult = adminClient.deleteTopics(topicsStillToCleanup);
        final Map<String, KafkaFuture<Void>> deleteResultForTopic = deleteTopicsResult.topicNameValues();
        while (!deleteResultForTopic.isEmpty()) {
            for (final String topicName : new HashSet<>(topicsStillToCleanup)) {
                if (!deleteResultForTopic.containsKey(topicName)) {
                    throw new IllegalStateException("Delete topic results do not contain internal topic " + topicName + " to clean up. " + BUG_ERROR_MESSAGE);
                }
                final KafkaFuture<Void> deleteResult = deleteResultForTopic.get(topicName);
                if (deleteResult.isDone()) {
                    try {
                        deleteResult.get();
                        topicsStillToCleanup.remove(topicName);
                    } catch (final ExecutionException executionException) {
                        final Throwable cause = executionException.getCause();
                        if (cause instanceof UnknownTopicOrPartitionException) {
                            log.info("Internal topic {} to clean up is missing", topicName);
                        } else if (cause instanceof LeaderNotAvailableException) {
                            log.info("The leader of internal topic {} to clean up is not available.", topicName);
                        } else if (cause instanceof TimeoutException) {
                            log.info("Cleaning up internal topic {} timed out.", topicName);
                        } else {
                            log.error("Unexpected error during cleanup of internal topics: ", cause);
                            throw new StreamsException(String.format("Could not clean up internal topics %s, because during the cleanup " + "of topic %s the following error occurred: ", topicsStillToCleanup, topicName), cause);
                        }
                    } catch (final InterruptedException interruptedException) {
                        throw new InterruptException(interruptedException);
                    } finally {
                        deleteResultForTopic.remove(topicName);
                    }
                }
            }
            maybeThrowTimeoutException(Collections.singletonList(topicsStillToCleanup), deadline, String.format("Could not cleanup internal topics within %d milliseconds. This can happen if the " + "Kafka cluster is temporarily not available or the broker did not complete topic creation " + "before the cleanup. The following internal topics could not be cleaned up: %s", retryTimeoutMs, topicsStillToCleanup));
            if (!deleteResultForTopic.isEmpty()) {
                Utils.sleep(100);
            }
        }
        maybeSleep(Collections.singletonList(topicsStillToCleanup), deadline, "validated");
    }
    log.info("Completed cleanup of internal topics {}.", topicsToCleanUp);
}