Search in sources :

Example 16 with RawMessage

use of org.apache.pulsar.client.api.RawMessage in project incubator-pulsar by apache.

the class TwoPhaseCompactor method phaseOneLoop.

private void phaseOneLoop(RawReader reader, Optional<MessageId> firstMessageId, MessageId lastMessageId, Map<String, MessageId> latestForKey, CompletableFuture<PhaseOneResult> loopPromise) {
    if (loopPromise.isDone()) {
        return;
    }
    CompletableFuture<RawMessage> future = reader.readNextAsync();
    scheduleTimeout(future);
    future.whenCompleteAsync((m, exception) -> {
        try {
            if (exception != null) {
                loopPromise.completeExceptionally(exception);
                return;
            }
            MessageId id = m.getMessageId();
            if (RawBatchConverter.isBatch(m)) {
                try {
                    RawBatchConverter.extractIdsAndKeys(m).forEach(e -> latestForKey.put(e.getRight(), e.getLeft()));
                } catch (IOException ioe) {
                    log.info("Error decoding batch for message {}. Whole batch will be included in output", id, ioe);
                }
            } else {
                String key = extractKey(m);
                latestForKey.put(key, id);
            }
            if (id.compareTo(lastMessageId) == 0) {
                loopPromise.complete(new PhaseOneResult(firstMessageId.orElse(id), id, latestForKey));
            } else {
                phaseOneLoop(reader, Optional.of(firstMessageId.orElse(id)), lastMessageId, latestForKey, loopPromise);
            }
        } finally {
            m.close();
        }
    }, scheduler);
}
Also used : IOException(java.io.IOException) RawMessage(org.apache.pulsar.client.api.RawMessage) MessageId(org.apache.pulsar.client.api.MessageId)

Aggregations

RawMessage (org.apache.pulsar.client.api.RawMessage)16 Test (org.testng.annotations.Test)11 RawReader (org.apache.pulsar.client.api.RawReader)10 ByteBuf (io.netty.buffer.ByteBuf)8 MockedPulsarServiceBaseTest (org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest)8 MessageId (org.apache.pulsar.client.api.MessageId)7 ArrayList (java.util.ArrayList)6 HashSet (java.util.HashSet)4 CompletableFuture (java.util.concurrent.CompletableFuture)4 HashMap (java.util.HashMap)3 List (java.util.List)3 Future (java.util.concurrent.Future)3 TimeoutException (java.util.concurrent.TimeoutException)3 BookKeeper (org.apache.bookkeeper.client.BookKeeper)3 LedgerHandle (org.apache.bookkeeper.client.LedgerHandle)3 MessageIdData (org.apache.pulsar.common.api.proto.PulsarApi.MessageIdData)3 Lists (com.google.common.collect.Lists)2 Sets (com.google.common.collect.Sets)2 IOException (java.io.IOException)2 Collections (java.util.Collections)2