Search in sources :

Example 41 with LedgerHandle

use of org.apache.bookkeeper.client.LedgerHandle in project pulsar by yahoo.

the class ManagedLedgerImpl method asyncReadEntries.

void asyncReadEntries(OpReadEntry opReadEntry) {
    final State state = STATE_UPDATER.get(this);
    if (state == State.Fenced || state == State.Closed) {
        opReadEntry.readEntriesFailed(new ManagedLedgerFencedException(), opReadEntry.ctx);
        return;
    }
    long ledgerId = opReadEntry.readPosition.getLedgerId();
    LedgerHandle currentLedger = this.currentLedger;
    if (ledgerId == currentLedger.getId()) {
        // Current writing ledger is not in the cache (since we don't want
        // it to be automatically evicted), and we cannot use 2 different
        // ledger handles (read & write)for the same ledger.
        internalReadFromLedger(currentLedger, opReadEntry);
    } else {
        LedgerInfo ledgerInfo = ledgers.get(ledgerId);
        if (ledgerInfo == null || ledgerInfo.getEntries() == 0) {
            // Cursor is pointing to a empty ledger, there's no need to try opening it. Skip this ledger and
            // move to the next one
            opReadEntry.updateReadPosition(new PositionImpl(opReadEntry.readPosition.getLedgerId() + 1, 0));
            opReadEntry.checkReadCompletion();
            return;
        }
        // Get a ledger handle to read from
        getLedgerHandle(ledgerId).thenAccept(ledger -> {
            internalReadFromLedger(ledger, opReadEntry);
        }).exceptionally(ex -> {
            log.error("[{}] Error opening ledger for reading at position {} - {}", name, opReadEntry.readPosition, ex.getMessage());
            opReadEntry.readEntriesFailed(new ManagedLedgerException(ex), opReadEntry.ctx);
            return null;
        });
    }
}
Also used : ManagedLedgerInfo(org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo) LedgerInfo(org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo.LedgerInfo) OpenCallback(org.apache.bookkeeper.client.AsyncCallback.OpenCallback) CloseCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.CloseCallback) LoggerFactory(org.slf4j.LoggerFactory) Random(java.util.Random) MetaStoreCallback(org.apache.bookkeeper.mledger.impl.MetaStore.MetaStoreCallback) Pair(org.apache.bookkeeper.mledger.util.Pair) Unpooled(io.netty.buffer.Unpooled) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) OpenCursorCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.OpenCursorCallback) ManagedCursor(org.apache.bookkeeper.mledger.ManagedCursor) ManagedLedger(org.apache.bookkeeper.mledger.ManagedLedger) ManagedLedgerConfig(org.apache.bookkeeper.mledger.ManagedLedgerConfig) Map(java.util.Map) ManagedLedgerMXBean(org.apache.bookkeeper.mledger.ManagedLedgerMXBean) DeleteCursorCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteCursorCallback) Commands(com.yahoo.pulsar.common.api.Commands) LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) ManagedLedgerInfo(org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo) LedgerInfo(org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo.LedgerInfo) Range(com.google.common.collect.Range) Futures(org.apache.bookkeeper.mledger.util.Futures) Math.min(java.lang.Math.min) Position(org.apache.bookkeeper.mledger.Position) NavigableMap(java.util.NavigableMap) BookKeeper(org.apache.bookkeeper.client.BookKeeper) BKException(org.apache.bookkeeper.client.BKException) AddEntryCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) CountDownLatch(java.util.concurrent.CountDownLatch) List(java.util.List) Queues(com.google.common.collect.Queues) ManagedLedgerFencedException(org.apache.bookkeeper.mledger.ManagedLedgerException.ManagedLedgerFencedException) BoundType(com.google.common.collect.BoundType) Stat(org.apache.bookkeeper.mledger.impl.MetaStore.Stat) ConcurrentLongHashMap(com.yahoo.pulsar.common.util.collections.ConcurrentLongHashMap) Queue(java.util.Queue) BadVersionException(org.apache.bookkeeper.mledger.ManagedLedgerException.BadVersionException) ConcurrentLinkedQueue(java.util.concurrent.ConcurrentLinkedQueue) CreateCallback(org.apache.bookkeeper.client.AsyncCallback.CreateCallback) DeleteLedgerCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.DeleteLedgerCallback) Entry(org.apache.bookkeeper.mledger.Entry) CompletableFuture(java.util.concurrent.CompletableFuture) ReadEntryCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback) RateLimiter(com.google.common.util.concurrent.RateLimiter) SafeRun.safeRun(org.apache.bookkeeper.mledger.util.SafeRun.safeRun) MetaStoreException(org.apache.bookkeeper.mledger.ManagedLedgerException.MetaStoreException) Lists(com.google.common.collect.Lists) ByteBuf(io.netty.buffer.ByteBuf) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) OrderedSafeExecutor(org.apache.bookkeeper.util.OrderedSafeExecutor) java.util.concurrent.atomic(java.util.concurrent.atomic) Maps(com.google.common.collect.Maps) TimeUnit(java.util.concurrent.TimeUnit) MessageMetadata(com.yahoo.pulsar.common.api.proto.PulsarApi.MessageMetadata) ConcurrentSkipListMap(java.util.concurrent.ConcurrentSkipListMap) CallbackMutex(org.apache.bookkeeper.mledger.util.CallbackMutex) VoidCallback(org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.VoidCallback) UnboundArrayBlockingQueue(org.apache.bookkeeper.util.UnboundArrayBlockingQueue) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) ManagedLedgerFencedException(org.apache.bookkeeper.mledger.ManagedLedgerException.ManagedLedgerFencedException)

Example 42 with LedgerHandle

use of org.apache.bookkeeper.client.LedgerHandle in project pulsar by yahoo.

the class EntryCacheImpl method asyncReadEntry.

@Override
public void asyncReadEntry(LedgerHandle lh, PositionImpl position, final ReadEntryCallback callback, final Object ctx) {
    if (log.isDebugEnabled()) {
        log.debug("[{}] Reading entry ledger {}: {}", ml.getName(), lh.getId(), position.getEntryId());
    }
    EntryImpl entry = entries.get(position);
    if (entry != null) {
        EntryImpl cachedEntry = new EntryImpl(entry);
        entry.release();
        manager.mlFactoryMBean.recordCacheHit(cachedEntry.getLength());
        callback.readEntryComplete(cachedEntry, ctx);
    } else {
        ReadCallback readCallback = (rc, ledgerHandle, sequence, obj) -> {
            if (rc != BKException.Code.OK) {
                ml.invalidateLedgerHandle(ledgerHandle, rc);
                callback.readEntryFailed(new ManagedLedgerException(BKException.create(rc)), obj);
                return;
            }
            if (sequence.hasMoreElements()) {
                EntryImpl returnEntry = new EntryImpl(sequence.nextElement());
                manager.mlFactoryMBean.recordCacheMiss(1, returnEntry.getLength());
                ml.mbean.addReadEntriesSample(1, returnEntry.getLength());
                callback.readEntryComplete(returnEntry, obj);
            } else {
                // got an empty sequence
                callback.readEntryFailed(new ManagedLedgerException("Could not read given position"), obj);
            }
        };
        lh.asyncReadEntries(position.getEntryId(), position.getEntryId(), readCallback, ctx);
    }
}
Also used : Longs(com.google.common.primitives.Longs) Logger(org.slf4j.Logger) Weighter(org.apache.bookkeeper.mledger.util.RangeCache.Weighter) Collection(java.util.Collection) Preconditions.checkNotNull(com.google.common.base.Preconditions.checkNotNull) LoggerFactory(org.slf4j.LoggerFactory) ReadEntryCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback) PooledByteBufAllocator(io.netty.buffer.PooledByteBufAllocator) Pair(org.apache.bookkeeper.mledger.util.Pair) BKException(org.apache.bookkeeper.client.BKException) TooManyRequestsException(org.apache.bookkeeper.mledger.ManagedLedgerException.TooManyRequestsException) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) SafeRun.safeRun(org.apache.bookkeeper.mledger.util.SafeRun.safeRun) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) List(java.util.List) Lists(com.google.common.collect.Lists) ByteBuf(io.netty.buffer.ByteBuf) RangeCache(org.apache.bookkeeper.mledger.util.RangeCache) ReadCallback(org.apache.bookkeeper.client.AsyncCallback.ReadCallback) ReadEntriesCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback) LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) ReadCallback(org.apache.bookkeeper.client.AsyncCallback.ReadCallback)

Example 43 with LedgerHandle

use of org.apache.bookkeeper.client.LedgerHandle in project pulsar by yahoo.

the class SimpleBookKeeperTest method simpleTest.

@Test
public void simpleTest() throws Exception {
    LedgerHandle ledger = bkc.createLedger(DigestType.MAC, SECRET.getBytes());
    long ledgerId = ledger.getId();
    log.info("Writing to ledger: {}", ledgerId);
    for (int i = 0; i < 10; i++) {
        String content = "entry-" + i;
        ledger.addEntry(content.getBytes(Encoding));
    }
    ledger.close();
    ledger = bkc.openLedger(ledgerId, DigestType.MAC, SECRET.getBytes());
    Enumeration<LedgerEntry> entries = ledger.readEntries(0, 9);
    while (entries.hasMoreElements()) {
        LedgerEntry entry = entries.nextElement();
        String content = new String(entry.getEntry(), Encoding);
        log.info("Entry {}  lenght={} content='{}'", entry.getEntryId(), entry.getLength(), content);
    }
    ledger.close();
}
Also used : LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) LedgerEntry(org.apache.bookkeeper.client.LedgerEntry) Test(org.testng.annotations.Test)

Example 44 with LedgerHandle

use of org.apache.bookkeeper.client.LedgerHandle in project pulsar by yahoo.

the class EntryCacheTest method testReadMissingMultiple.

@Test(timeOut = 5000)
void testReadMissingMultiple() throws Exception {
    LedgerHandle lh = getLedgerHandle();
    when(lh.getId()).thenReturn((long) 0);
    EntryCacheManager cacheManager = factory.getEntryCacheManager();
    EntryCache entryCache = cacheManager.getEntryCache(ml);
    byte[] data = new byte[10];
    entryCache.insert(new EntryImpl(0, 0, data));
    entryCache.insert(new EntryImpl(0, 2, data));
    entryCache.insert(new EntryImpl(0, 5, data));
    entryCache.insert(new EntryImpl(0, 8, data));
    final CountDownLatch counter = new CountDownLatch(1);
    entryCache.asyncReadEntry(lh, 0, 9, false, new ReadEntriesCallback() {

        public void readEntriesComplete(List<Entry> entries, Object ctx) {
            assertEquals(entries.size(), 10);
            counter.countDown();
        }

        public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
            Assert.fail("should not have failed");
        }
    }, null);
    counter.await();
}
Also used : ReadEntriesCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback) Entry(org.apache.bookkeeper.mledger.Entry) LedgerEntry(org.apache.bookkeeper.client.LedgerEntry) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) CountDownLatch(java.util.concurrent.CountDownLatch) Test(org.testng.annotations.Test)

Example 45 with LedgerHandle

use of org.apache.bookkeeper.client.LedgerHandle in project pulsar by yahoo.

the class EntryCacheTest method testReadWithError.

@Test(timeOut = 5000)
void testReadWithError() throws Exception {
    final LedgerHandle lh = getLedgerHandle();
    when(lh.getId()).thenReturn((long) 0);
    doAnswer(new Answer<Object>() {

        public Object answer(InvocationOnMock invocation) {
            Object[] args = invocation.getArguments();
            ReadCallback callback = (ReadCallback) args[2];
            Object ctx = args[3];
            callback.readComplete(BKException.Code.NoSuchLedgerExistsException, lh, null, ctx);
            return null;
        }
    }).when(lh).asyncReadEntries(anyLong(), anyLong(), any(ReadCallback.class), any());
    EntryCacheManager cacheManager = factory.getEntryCacheManager();
    EntryCache entryCache = cacheManager.getEntryCache(ml);
    byte[] data = new byte[10];
    entryCache.insert(new EntryImpl(0, 2, data));
    final CountDownLatch counter = new CountDownLatch(1);
    entryCache.asyncReadEntry(lh, 0, 9, false, new ReadEntriesCallback() {

        public void readEntriesComplete(List<Entry> entries, Object ctx) {
            Assert.fail("should not complete");
        }

        public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
            counter.countDown();
        }
    }, null);
    counter.await();
}
Also used : ReadEntriesCallback(org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback) LedgerHandle(org.apache.bookkeeper.client.LedgerHandle) CountDownLatch(java.util.concurrent.CountDownLatch) Entry(org.apache.bookkeeper.mledger.Entry) LedgerEntry(org.apache.bookkeeper.client.LedgerEntry) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ReadCallback(org.apache.bookkeeper.client.AsyncCallback.ReadCallback) Test(org.testng.annotations.Test)

Aggregations

LedgerHandle (org.apache.bookkeeper.client.LedgerHandle)51 Test (org.junit.Test)19 ZKTransaction (com.twitter.distributedlog.zk.ZKTransaction)13 LedgerEntry (org.apache.bookkeeper.client.LedgerEntry)12 ManagedLedgerException (org.apache.bookkeeper.mledger.ManagedLedgerException)12 CountDownLatch (java.util.concurrent.CountDownLatch)10 IOException (java.io.IOException)9 Entry (org.apache.bookkeeper.mledger.Entry)8 BKException (org.apache.bookkeeper.client.BKException)7 ReadEntriesCallback (org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntriesCallback)7 Test (org.testng.annotations.Test)7 List (java.util.List)6 BookKeeper (org.apache.bookkeeper.client.BookKeeper)6 ZKDistributedLock (com.twitter.distributedlog.lock.ZKDistributedLock)5 Future (com.twitter.util.Future)5 ArrayList (java.util.ArrayList)5 lombok.val (lombok.val)5 ZKException (com.twitter.distributedlog.exceptions.ZKException)4 Preconditions.checkArgument (com.google.common.base.Preconditions.checkArgument)3 Lists (com.google.common.collect.Lists)3