Search in sources :

Example 1 with Snapshot

use of org.rocksdb.Snapshot in project flink by apache.

the class RocksDBStateBackendTest method testRunningSnapshotAfterBackendClosed.

@Test
public void testRunningSnapshotAfterBackendClosed() throws Exception {
    setupRocksKeyedStateBackend();
    RunnableFuture<KeyGroupsStateHandle> snapshot = keyedStateBackend.snapshot(0L, 0L, testStreamFactory, CheckpointOptions.forFullCheckpoint());
    RocksDB spyDB = keyedStateBackend.db;
    verify(spyDB, times(1)).getSnapshot();
    verify(spyDB, times(0)).releaseSnapshot(any(Snapshot.class));
    this.keyedStateBackend.dispose();
    verify(spyDB, times(1)).close();
    assertEquals(null, keyedStateBackend.db);
    //Ensure every RocksObjects not closed yet
    for (RocksObject rocksCloseable : allCreatedCloseables) {
        verify(rocksCloseable, times(0)).close();
    }
    Thread asyncSnapshotThread = new Thread(snapshot);
    asyncSnapshotThread.start();
    try {
        snapshot.get();
        fail();
    } catch (Exception ignored) {
    }
    asyncSnapshotThread.join();
    //Ensure every RocksObject was closed exactly once
    for (RocksObject rocksCloseable : allCreatedCloseables) {
        verify(rocksCloseable, times(1)).close();
    }
}
Also used : Snapshot(org.rocksdb.Snapshot) RocksDB(org.rocksdb.RocksDB) RocksObject(org.rocksdb.RocksObject) KeyGroupsStateHandle(org.apache.flink.runtime.state.KeyGroupsStateHandle) IOException(java.io.IOException) Test(org.junit.Test)

Example 2 with Snapshot

use of org.rocksdb.Snapshot in project flink by apache.

the class RocksDBStateBackendTest method setupRocksKeyedStateBackend.

public void setupRocksKeyedStateBackend() throws Exception {
    blocker = new OneShotLatch();
    waiter = new OneShotLatch();
    testStreamFactory = new BlockerCheckpointStreamFactory(1024 * 1024);
    testStreamFactory.setBlockerLatch(blocker);
    testStreamFactory.setWaiterLatch(waiter);
    testStreamFactory.setAfterNumberInvocations(100);
    RocksDBStateBackend backend = getStateBackend();
    Environment env = new DummyEnvironment("TestTask", 1, 0);
    keyedStateBackend = (RocksDBKeyedStateBackend<Integer>) backend.createKeyedStateBackend(env, new JobID(), "Test", IntSerializer.INSTANCE, 2, new KeyGroupRange(0, 1), mock(TaskKvStateRegistry.class));
    testState1 = keyedStateBackend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, new ValueStateDescriptor<>("TestState-1", Integer.class, 0));
    testState2 = keyedStateBackend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, new ValueStateDescriptor<>("TestState-2", String.class, ""));
    allCreatedCloseables = new ArrayList<>();
    keyedStateBackend.db = spy(keyedStateBackend.db);
    doAnswer(new Answer<Object>() {

        @Override
        public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
            RocksIterator rocksIterator = spy((RocksIterator) invocationOnMock.callRealMethod());
            allCreatedCloseables.add(rocksIterator);
            return rocksIterator;
        }
    }).when(keyedStateBackend.db).newIterator(any(ColumnFamilyHandle.class), any(ReadOptions.class));
    doAnswer(new Answer<Object>() {

        @Override
        public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
            Snapshot snapshot = spy((Snapshot) invocationOnMock.callRealMethod());
            allCreatedCloseables.add(snapshot);
            return snapshot;
        }
    }).when(keyedStateBackend.db).getSnapshot();
    doAnswer(new Answer<Object>() {

        @Override
        public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
            ColumnFamilyHandle snapshot = spy((ColumnFamilyHandle) invocationOnMock.callRealMethod());
            allCreatedCloseables.add(snapshot);
            return snapshot;
        }
    }).when(keyedStateBackend.db).createColumnFamily(any(ColumnFamilyDescriptor.class));
    for (int i = 0; i < 100; ++i) {
        keyedStateBackend.setCurrentKey(i);
        testState1.update(4200 + i);
        testState2.update("S-" + (4200 + i));
    }
}
Also used : KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) TaskKvStateRegistry(org.apache.flink.runtime.query.TaskKvStateRegistry) DummyEnvironment(org.apache.flink.runtime.operators.testutils.DummyEnvironment) RocksIterator(org.rocksdb.RocksIterator) ColumnFamilyDescriptor(org.rocksdb.ColumnFamilyDescriptor) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) ValueStateDescriptor(org.apache.flink.api.common.state.ValueStateDescriptor) Snapshot(org.rocksdb.Snapshot) ReadOptions(org.rocksdb.ReadOptions) InvocationOnMock(org.mockito.invocation.InvocationOnMock) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) BlockerCheckpointStreamFactory(org.apache.flink.runtime.util.BlockerCheckpointStreamFactory) DummyEnvironment(org.apache.flink.runtime.operators.testutils.DummyEnvironment) Environment(org.apache.flink.runtime.execution.Environment) RocksObject(org.rocksdb.RocksObject) JobID(org.apache.flink.api.common.JobID)

Example 3 with Snapshot

use of org.rocksdb.Snapshot in project flink by apache.

the class RocksDBStateBackendTest method testReleasingSnapshotAfterBackendClosed.

@Test
public void testReleasingSnapshotAfterBackendClosed() throws Exception {
    setupRocksKeyedStateBackend();
    RunnableFuture<KeyGroupsStateHandle> snapshot = keyedStateBackend.snapshot(0L, 0L, testStreamFactory, CheckpointOptions.forFullCheckpoint());
    RocksDB spyDB = keyedStateBackend.db;
    verify(spyDB, times(1)).getSnapshot();
    verify(spyDB, times(0)).releaseSnapshot(any(Snapshot.class));
    this.keyedStateBackend.dispose();
    verify(spyDB, times(1)).close();
    assertEquals(null, keyedStateBackend.db);
    //Ensure every RocksObjects not closed yet
    for (RocksObject rocksCloseable : allCreatedCloseables) {
        verify(rocksCloseable, times(0)).close();
    }
    snapshot.cancel(true);
    //Ensure every RocksObjects was closed exactly once
    for (RocksObject rocksCloseable : allCreatedCloseables) {
        verify(rocksCloseable, times(1)).close();
    }
}
Also used : Snapshot(org.rocksdb.Snapshot) RocksDB(org.rocksdb.RocksDB) RocksObject(org.rocksdb.RocksObject) KeyGroupsStateHandle(org.apache.flink.runtime.state.KeyGroupsStateHandle) Test(org.junit.Test)

Aggregations

RocksObject (org.rocksdb.RocksObject)3 Snapshot (org.rocksdb.Snapshot)3 KeyGroupsStateHandle (org.apache.flink.runtime.state.KeyGroupsStateHandle)2 Test (org.junit.Test)2 RocksDB (org.rocksdb.RocksDB)2 IOException (java.io.IOException)1 JobID (org.apache.flink.api.common.JobID)1 ValueStateDescriptor (org.apache.flink.api.common.state.ValueStateDescriptor)1 OneShotLatch (org.apache.flink.core.testutils.OneShotLatch)1 Environment (org.apache.flink.runtime.execution.Environment)1 DummyEnvironment (org.apache.flink.runtime.operators.testutils.DummyEnvironment)1 TaskKvStateRegistry (org.apache.flink.runtime.query.TaskKvStateRegistry)1 KeyGroupRange (org.apache.flink.runtime.state.KeyGroupRange)1 BlockerCheckpointStreamFactory (org.apache.flink.runtime.util.BlockerCheckpointStreamFactory)1 InvocationOnMock (org.mockito.invocation.InvocationOnMock)1 ColumnFamilyDescriptor (org.rocksdb.ColumnFamilyDescriptor)1 ColumnFamilyHandle (org.rocksdb.ColumnFamilyHandle)1 ReadOptions (org.rocksdb.ReadOptions)1 RocksIterator (org.rocksdb.RocksIterator)1