Examples with KafkaPartitionSplitReader org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader used on opensource projects

Example 1 with KafkaPartitionSplitReader

use of org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader in project flink by apache.

the class KafkaSourceFetcherManager method enqueueOffsetsCommitTask.

private void enqueueOffsetsCommitTask(SplitFetcher<ConsumerRecord<byte[], byte[]>, KafkaPartitionSplit> splitFetcher, Map<TopicPartition, OffsetAndMetadata> offsetsToCommit, OffsetCommitCallback callback) {
    KafkaPartitionSplitReader kafkaReader = (KafkaPartitionSplitReader) splitFetcher.getSplitReader();
    splitFetcher.enqueueTask(new SplitFetcherTask() {

        @Override
        public boolean run() throws IOException {
            kafkaReader.notifyCheckpointComplete(offsetsToCommit, callback);
            return true;
        }

        @Override
        public void wakeUp() {
        }
    });
}

Example 2 with KafkaPartitionSplitReader

use of org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader in project flink by apache.

the class KafkaSource method createReader.

@VisibleForTesting
SourceReader<OUT, KafkaPartitionSplit> createReader(SourceReaderContext readerContext, Consumer<Collection<String>> splitFinishedHook) throws Exception {
    FutureCompletingBlockingQueue<RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>>> elementsQueue = new FutureCompletingBlockingQueue<>();
    deserializationSchema.open(new DeserializationSchema.InitializationContext() {

        @Override
        public MetricGroup getMetricGroup() {
            return readerContext.metricGroup().addGroup("deserializer");
        }

        @Override
        public UserCodeClassLoader getUserCodeClassLoader() {
            return readerContext.getUserCodeClassLoader();
        }
    });
    final KafkaSourceReaderMetrics kafkaSourceReaderMetrics = new KafkaSourceReaderMetrics(readerContext.metricGroup());
    Supplier<KafkaPartitionSplitReader> splitReaderSupplier = () -> new KafkaPartitionSplitReader(props, readerContext, kafkaSourceReaderMetrics);
    KafkaRecordEmitter<OUT> recordEmitter = new KafkaRecordEmitter<>(deserializationSchema);
    return new KafkaSourceReader<>(elementsQueue, new KafkaSourceFetcherManager(elementsQueue, splitReaderSupplier::get, splitFinishedHook), recordEmitter, toConfiguration(props), readerContext, kafkaSourceReaderMetrics);
}