Search in sources :

Example 1 with KafkaTopicPartitionState

use of org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState in project flink by apache.

the class Kafka09Fetcher method commitInternalOffsetsToKafka.

@Override
public void commitInternalOffsetsToKafka(Map<KafkaTopicPartition, Long> offsets) throws Exception {
    KafkaTopicPartitionState<TopicPartition>[] partitions = subscribedPartitionStates();
    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>(partitions.length);
    for (KafkaTopicPartitionState<TopicPartition> partition : partitions) {
        Long lastProcessedOffset = offsets.get(partition.getKafkaTopicPartition());
        if (lastProcessedOffset != null) {
            // committed offsets through the KafkaConsumer need to be 1 more than the last processed offset.
            // This does not affect Flink's checkpoints/saved state.
            long offsetToCommit = lastProcessedOffset + 1;
            offsetsToCommit.put(partition.getKafkaPartitionHandle(), new OffsetAndMetadata(offsetToCommit));
            partition.setCommittedOffset(offsetToCommit);
        }
    }
    // record the work to be committed by the main consumer thread and make sure the consumer notices that
    consumerThread.setOffsetsToCommit(offsetsToCommit);
}
Also used : KafkaTopicPartitionState(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState) HashMap(java.util.HashMap) TopicPartition(org.apache.kafka.common.TopicPartition) KafkaTopicPartition(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition) OffsetAndMetadata(org.apache.kafka.clients.consumer.OffsetAndMetadata)

Aggregations

HashMap (java.util.HashMap)1 KafkaTopicPartition (org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition)1 KafkaTopicPartitionState (org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState)1 OffsetAndMetadata (org.apache.kafka.clients.consumer.OffsetAndMetadata)1 TopicPartition (org.apache.kafka.common.TopicPartition)1