Search in sources :

Example 1 with KafkaSourceReaderMetrics

use of org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics in project flink by apache.

the class KafkaSource method createReader.

@VisibleForTesting
SourceReader<OUT, KafkaPartitionSplit> createReader(SourceReaderContext readerContext, Consumer<Collection<String>> splitFinishedHook) throws Exception {
    FutureCompletingBlockingQueue<RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>>> elementsQueue = new FutureCompletingBlockingQueue<>();
    deserializationSchema.open(new DeserializationSchema.InitializationContext() {

        @Override
        public MetricGroup getMetricGroup() {
            return readerContext.metricGroup().addGroup("deserializer");
        }

        @Override
        public UserCodeClassLoader getUserCodeClassLoader() {
            return readerContext.getUserCodeClassLoader();
        }
    });
    final KafkaSourceReaderMetrics kafkaSourceReaderMetrics = new KafkaSourceReaderMetrics(readerContext.metricGroup());
    Supplier<KafkaPartitionSplitReader> splitReaderSupplier = () -> new KafkaPartitionSplitReader(props, readerContext, kafkaSourceReaderMetrics);
    KafkaRecordEmitter<OUT> recordEmitter = new KafkaRecordEmitter<>(deserializationSchema);
    return new KafkaSourceReader<>(elementsQueue, new KafkaSourceFetcherManager(elementsQueue, splitReaderSupplier::get, splitFinishedHook), recordEmitter, toConfiguration(props), readerContext, kafkaSourceReaderMetrics);
}
Also used : KafkaSourceFetcherManager(org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager) MetricGroup(org.apache.flink.metrics.MetricGroup) KafkaSourceReaderMetrics(org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics) KafkaSourceReader(org.apache.flink.connector.kafka.source.reader.KafkaSourceReader) RecordsWithSplitIds(org.apache.flink.connector.base.source.reader.RecordsWithSplitIds) KafkaRecordDeserializationSchema(org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema) DeserializationSchema(org.apache.flink.api.common.serialization.DeserializationSchema) UserCodeClassLoader(org.apache.flink.util.UserCodeClassLoader) FutureCompletingBlockingQueue(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue) KafkaRecordEmitter(org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter) KafkaPartitionSplitReader(org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader) VisibleForTesting(org.apache.flink.annotation.VisibleForTesting)

Example 2 with KafkaSourceReaderMetrics

use of org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics in project flink by apache.

the class KafkaPartitionSplitReaderTest method createReader.

private KafkaPartitionSplitReader createReader(Properties additionalProperties, SourceReaderMetricGroup sourceReaderMetricGroup) {
    Properties props = new Properties();
    props.putAll(KafkaSourceTestEnv.getConsumerProperties(ByteArrayDeserializer.class));
    props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
    if (!additionalProperties.isEmpty()) {
        props.putAll(additionalProperties);
    }
    KafkaSourceReaderMetrics kafkaSourceReaderMetrics = new KafkaSourceReaderMetrics(sourceReaderMetricGroup);
    return new KafkaPartitionSplitReader(props, new TestingReaderContext(new Configuration(), sourceReaderMetricGroup), kafkaSourceReaderMetrics);
}
Also used : Configuration(org.apache.flink.configuration.Configuration) KafkaSourceReaderMetrics(org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics) TestingReaderContext(org.apache.flink.connector.testutils.source.reader.TestingReaderContext) Properties(java.util.Properties) ByteArrayDeserializer(org.apache.kafka.common.serialization.ByteArrayDeserializer)

Aggregations

KafkaSourceReaderMetrics (org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics)2 Properties (java.util.Properties)1 VisibleForTesting (org.apache.flink.annotation.VisibleForTesting)1 DeserializationSchema (org.apache.flink.api.common.serialization.DeserializationSchema)1 Configuration (org.apache.flink.configuration.Configuration)1 RecordsWithSplitIds (org.apache.flink.connector.base.source.reader.RecordsWithSplitIds)1 FutureCompletingBlockingQueue (org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue)1 KafkaPartitionSplitReader (org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader)1 KafkaRecordEmitter (org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter)1 KafkaSourceReader (org.apache.flink.connector.kafka.source.reader.KafkaSourceReader)1 KafkaRecordDeserializationSchema (org.apache.flink.connector.kafka.source.reader.deserializer.KafkaRecordDeserializationSchema)1 KafkaSourceFetcherManager (org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager)1 TestingReaderContext (org.apache.flink.connector.testutils.source.reader.TestingReaderContext)1 MetricGroup (org.apache.flink.metrics.MetricGroup)1 UserCodeClassLoader (org.apache.flink.util.UserCodeClassLoader)1 ByteArrayDeserializer (org.apache.kafka.common.serialization.ByteArrayDeserializer)1