Search in sources :

Example 6 with PulsarPartitionSplit

use of org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit in project flink by apache.

the class PulsarOrderedSourceReaderTest method setupSourceReader.

private void setupSourceReader(PulsarSourceReaderBase<Integer> reader, String topicName, int partitionId, Boundedness boundedness) {
    PulsarPartitionSplit split = createPartitionSplit(topicName, partitionId, boundedness);
    reader.addSplits(Collections.singletonList(split));
    reader.notifyNoMoreSplits();
}
Also used : PulsarPartitionSplit(org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit)

Example 7 with PulsarPartitionSplit

use of org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit in project flink by apache.

the class PulsarPartitionSplitReaderTestBase method seekStartPositionAndHandleSplit.

private void seekStartPositionAndHandleSplit(PulsarPartitionSplitReaderBase<String> reader, String topicName, int partitionId, MessageId startPosition) {
    TopicPartition partition = new TopicPartition(topicName, partitionId, createFullRange());
    PulsarPartitionSplit split = new PulsarPartitionSplit(partition, StopCursor.never(), null, null);
    SplitsAddition<PulsarPartitionSplit> addition = new SplitsAddition<>(singletonList(split));
    // create consumer and seek before split changes
    try (Consumer<byte[]> consumer = reader.createPulsarConsumer(partition)) {
        // inclusive messageId
        StartCursor startCursor = StartCursor.fromMessageId(startPosition);
        startCursor.seekPosition(partition.getTopic(), partition.getPartitionId(), consumer);
    } catch (PulsarClientException e) {
        sneakyThrow(e);
    }
    reader.handleSplitsChanges(addition);
}
Also used : TopicPartition(org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition) StartCursor(org.apache.flink.connector.pulsar.source.enumerator.cursor.StartCursor) SplitsAddition(org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition) PulsarClientException(org.apache.pulsar.client.api.PulsarClientException) PulsarPartitionSplit(org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit)

Example 8 with PulsarPartitionSplit

use of org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit in project flink by apache.

the class PulsarOrderedSourceReader method snapshotState.

@Override
public List<PulsarPartitionSplit> snapshotState(long checkpointId) {
    List<PulsarPartitionSplit> splits = super.snapshotState(checkpointId);
    // Perform a snapshot for these splits.
    Map<TopicPartition, MessageId> cursors = cursorsToCommit.computeIfAbsent(checkpointId, id -> new HashMap<>());
    // Put the cursors of the active splits.
    for (PulsarPartitionSplit split : splits) {
        MessageId latestConsumedId = split.getLatestConsumedId();
        if (latestConsumedId != null) {
            cursors.put(split.getPartition(), latestConsumedId);
        }
    }
    // Put cursors of all the finished splits.
    cursors.putAll(cursorsOfFinishedSplits);
    return splits;
}
Also used : TopicPartition(org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition) PulsarPartitionSplit(org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit) MessageId(org.apache.pulsar.client.api.MessageId)

Example 9 with PulsarPartitionSplit

use of org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit in project flink by apache.

the class PulsarOrderedSourceReader method cumulativeAcknowledgmentMessage.

/**
 * Acknowledge the pulsar topic partition cursor by the last consumed message id.
 */
private void cumulativeAcknowledgmentMessage() {
    Map<TopicPartition, MessageId> cursors = new HashMap<>(cursorsOfFinishedSplits);
    // We reuse snapshotState for acquiring a consume status snapshot.
    // So the checkpoint didn't really happen, so we just pass a fake checkpoint id.
    List<PulsarPartitionSplit> splits = super.snapshotState(1L);
    for (PulsarPartitionSplit split : splits) {
        MessageId latestConsumedId = split.getLatestConsumedId();
        if (latestConsumedId != null) {
            cursors.put(split.getPartition(), latestConsumedId);
        }
    }
    try {
        ((PulsarOrderedFetcherManager<OUT>) splitFetcherManager).acknowledgeMessages(cursors);
        // Clean up the finish splits.
        cursorsOfFinishedSplits.keySet().removeAll(cursors.keySet());
    } catch (Exception e) {
        LOG.error("Fail in auto cursor commit.", e);
        cursorCommitThrowable.compareAndSet(null, e);
    }
}
Also used : HashMap(java.util.HashMap) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) TopicPartition(org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition) PulsarOrderedFetcherManager(org.apache.flink.connector.pulsar.source.reader.fetcher.PulsarOrderedFetcherManager) PulsarPartitionSplit(org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit) MessageId(org.apache.pulsar.client.api.MessageId)

Example 10 with PulsarPartitionSplit

use of org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit in project flink by apache.

the class PulsarUnorderedSourceReader method snapshotState.

@Override
public List<PulsarPartitionSplit> snapshotState(long checkpointId) {
    LOG.debug("Trigger the new transaction for downstream readers.");
    List<PulsarPartitionSplit> splits = ((PulsarUnorderedFetcherManager<OUT>) splitFetcherManager).snapshotState(checkpointId);
    if (coordinatorClient != null) {
        // Snapshot the transaction status and commit it after checkpoint finished.
        List<TxnID> txnIDs = transactionsToCommit.computeIfAbsent(checkpointId, id -> new ArrayList<>());
        for (PulsarPartitionSplit split : splits) {
            TxnID uncommittedTransactionId = split.getUncommittedTransactionId();
            if (uncommittedTransactionId != null) {
                txnIDs.add(uncommittedTransactionId);
            }
        }
    }
    return splits;
}
Also used : TxnID(org.apache.pulsar.client.api.transaction.TxnID) PulsarUnorderedFetcherManager(org.apache.flink.connector.pulsar.source.reader.fetcher.PulsarUnorderedFetcherManager) PulsarPartitionSplit(org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit)

Aggregations

PulsarPartitionSplit (org.apache.flink.connector.pulsar.source.split.PulsarPartitionSplit)14 TopicPartition (org.apache.flink.connector.pulsar.source.enumerator.topic.TopicPartition)9 HashMap (java.util.HashMap)3 Set (java.util.Set)3 SplitsAddition (org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition)3 ArrayList (java.util.ArrayList)2 List (java.util.List)2 StopCursor (org.apache.flink.connector.pulsar.source.enumerator.cursor.StopCursor)2 MessageId (org.apache.pulsar.client.api.MessageId)2 ByteArrayInputStream (java.io.ByteArrayInputStream)1 DataInput (java.io.DataInput)1 DataInputStream (java.io.DataInputStream)1 IOException (java.io.IOException)1 HashSet (java.util.HashSet)1 Map (java.util.Map)1 Objects (java.util.Objects)1 Optional (java.util.Optional)1 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)1 Internal (org.apache.flink.annotation.Internal)1 SplitEnumeratorContext (org.apache.flink.api.connector.source.SplitEnumeratorContext)1