Search in sources :

Example 1 with PulsarOrderedPartitionSplitReader

use of org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader in project flink by apache.

the class PulsarSourceReaderFactory method create.

@SuppressWarnings("java:S2095")
public static <OUT> SourceReader<OUT, PulsarPartitionSplit> create(SourceReaderContext readerContext, PulsarDeserializationSchema<OUT> deserializationSchema, SourceConfiguration sourceConfiguration) {
    PulsarClient pulsarClient = createClient(sourceConfiguration);
    PulsarAdmin pulsarAdmin = createAdmin(sourceConfiguration);
    // Create a message queue with the predefined source option.
    int queueCapacity = sourceConfiguration.getMessageQueueCapacity();
    FutureCompletingBlockingQueue<RecordsWithSplitIds<PulsarMessage<OUT>>> elementsQueue = new FutureCompletingBlockingQueue<>(queueCapacity);
    // Create different pulsar source reader by subscription type.
    SubscriptionType subscriptionType = sourceConfiguration.getSubscriptionType();
    if (subscriptionType == SubscriptionType.Failover || subscriptionType == SubscriptionType.Exclusive) {
        // Create a ordered split reader supplier.
        Supplier<PulsarOrderedPartitionSplitReader<OUT>> splitReaderSupplier = () -> new PulsarOrderedPartitionSplitReader<>(pulsarClient, pulsarAdmin, sourceConfiguration, deserializationSchema);
        return new PulsarOrderedSourceReader<>(elementsQueue, splitReaderSupplier, readerContext, sourceConfiguration, pulsarClient, pulsarAdmin);
    } else if (subscriptionType == SubscriptionType.Shared || subscriptionType == SubscriptionType.Key_Shared) {
        TransactionCoordinatorClient coordinatorClient = ((PulsarClientImpl) pulsarClient).getTcClient();
        if (coordinatorClient == null && !sourceConfiguration.isEnableAutoAcknowledgeMessage()) {
            throw new IllegalStateException("Transaction is required but didn't enabled");
        }
        Supplier<PulsarUnorderedPartitionSplitReader<OUT>> splitReaderSupplier = () -> new PulsarUnorderedPartitionSplitReader<>(pulsarClient, pulsarAdmin, sourceConfiguration, deserializationSchema, coordinatorClient);
        return new PulsarUnorderedSourceReader<>(elementsQueue, splitReaderSupplier, readerContext, sourceConfiguration, pulsarClient, pulsarAdmin, coordinatorClient);
    } else {
        throw new UnsupportedOperationException("This subscription type is not " + subscriptionType + " supported currently.");
    }
}
Also used : PulsarAdmin(org.apache.pulsar.client.admin.PulsarAdmin) PulsarOrderedSourceReader(org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader) PulsarOrderedPartitionSplitReader(org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader) RecordsWithSplitIds(org.apache.flink.connector.base.source.reader.RecordsWithSplitIds) SubscriptionType(org.apache.pulsar.client.api.SubscriptionType) FutureCompletingBlockingQueue(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue) Supplier(java.util.function.Supplier) PulsarClient(org.apache.pulsar.client.api.PulsarClient) TransactionCoordinatorClient(org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient)

Aggregations

Supplier (java.util.function.Supplier)1 RecordsWithSplitIds (org.apache.flink.connector.base.source.reader.RecordsWithSplitIds)1 FutureCompletingBlockingQueue (org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue)1 PulsarOrderedSourceReader (org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader)1 PulsarOrderedPartitionSplitReader (org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader)1 PulsarAdmin (org.apache.pulsar.client.admin.PulsarAdmin)1 PulsarClient (org.apache.pulsar.client.api.PulsarClient)1 SubscriptionType (org.apache.pulsar.client.api.SubscriptionType)1 TransactionCoordinatorClient (org.apache.pulsar.client.api.transaction.TransactionCoordinatorClient)1