Examples with KafkaTopicPartitionState org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState used on opensource projects

Example 1 with KafkaTopicPartitionState

use of org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState in project flink by apache.

the class Kafka09Fetcher method commitInternalOffsetsToKafka.

@Override
public void commitInternalOffsetsToKafka(Map<KafkaTopicPartition, Long> offsets) throws Exception {
    KafkaTopicPartitionState<TopicPartition>[] partitions = subscribedPartitionStates();
    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>(partitions.length);
    for (KafkaTopicPartitionState<TopicPartition> partition : partitions) {
        Long lastProcessedOffset = offsets.get(partition.getKafkaTopicPartition());
        if (lastProcessedOffset != null) {
            // committed offsets through the KafkaConsumer need to be 1 more than the last processed offset.
            // This does not affect Flink's checkpoints/saved state.
            long offsetToCommit = lastProcessedOffset + 1;
            offsetsToCommit.put(partition.getKafkaPartitionHandle(), new OffsetAndMetadata(offsetToCommit));
            partition.setCommittedOffset(offsetToCommit);
        }
    }
    // record the work to be committed by the main consumer thread and make sure the consumer notices that
    consumerThread.setOffsetsToCommit(offsetsToCommit);
}