Search in sources :

Example 21 with WriteOptions

use of org.rocksdb.WriteOptions in project beam by apache.

the class SamzaTimerInternalsFactoryTest method createStore.

private KeyValueStore<ByteArray, StateValue<?>> createStore() {
    final Options options = new Options();
    options.setCreateIfMissing(true);
    RocksDbKeyValueStore rocksStore = new RocksDbKeyValueStore(temporaryFolder.getRoot(), options, new MapConfig(), false, "beamStore", new WriteOptions(), new FlushOptions(), new KeyValueStoreMetrics("beamStore", new MetricsRegistryMap()));
    return new SerializedKeyValueStore<>(rocksStore, new ByteArraySerdeFactory.ByteArraySerde(), new StateValueSerdeFactory.StateValueSerde(), new SerializedKeyValueStoreMetrics("beamStore", new MetricsRegistryMap()));
}
Also used : FlushOptions(org.rocksdb.FlushOptions) WriteOptions(org.rocksdb.WriteOptions) SamzaPipelineOptions(org.apache.beam.runners.samza.SamzaPipelineOptions) Options(org.rocksdb.Options) SerializedKeyValueStoreMetrics(org.apache.samza.storage.kv.SerializedKeyValueStoreMetrics) KeyValueStoreMetrics(org.apache.samza.storage.kv.KeyValueStoreMetrics) ByteArraySerdeFactory(org.apache.beam.runners.samza.runtime.SamzaStoreStateInternals.ByteArraySerdeFactory) FlushOptions(org.rocksdb.FlushOptions) SerializedKeyValueStoreMetrics(org.apache.samza.storage.kv.SerializedKeyValueStoreMetrics) WriteOptions(org.rocksdb.WriteOptions) SerializedKeyValueStore(org.apache.samza.storage.kv.SerializedKeyValueStore) MapConfig(org.apache.samza.config.MapConfig) StateValueSerdeFactory(org.apache.beam.runners.samza.runtime.SamzaStoreStateInternals.StateValueSerdeFactory) MetricsRegistryMap(org.apache.samza.metrics.MetricsRegistryMap) RocksDbKeyValueStore(org.apache.samza.storage.kv.RocksDbKeyValueStore)

Example 22 with WriteOptions

use of org.rocksdb.WriteOptions in project flink by apache.

the class RocksDBWriteBatchWrapperTest method testWriteBatchWrapperFlushAfterCountExceed.

/**
 * Tests that {@link RocksDBWriteBatchWrapper} flushes after the kv count exceeds the
 * preconfigured value.
 */
@Test
public void testWriteBatchWrapperFlushAfterCountExceed() throws Exception {
    try (RocksDB db = RocksDB.open(folder.newFolder().getAbsolutePath());
        WriteOptions options = new WriteOptions().setDisableWAL(true);
        ColumnFamilyHandle handle = db.createColumnFamily(new ColumnFamilyDescriptor("test".getBytes()));
        RocksDBWriteBatchWrapper writeBatchWrapper = new RocksDBWriteBatchWrapper(db, options, 100, 50000)) {
        long initBatchSize = writeBatchWrapper.getDataSize();
        byte[] dummy = new byte[2];
        ThreadLocalRandom.current().nextBytes(dummy);
        for (int i = 1; i < 100; ++i) {
            writeBatchWrapper.put(handle, dummy, dummy);
            // each kv consumes 8 bytes
            assertEquals(initBatchSize + 8 * i, writeBatchWrapper.getDataSize());
        }
        writeBatchWrapper.put(handle, dummy, dummy);
        assertEquals(initBatchSize, writeBatchWrapper.getDataSize());
    }
}
Also used : WriteOptions(org.rocksdb.WriteOptions) RocksDB(org.rocksdb.RocksDB) ColumnFamilyDescriptor(org.rocksdb.ColumnFamilyDescriptor) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) Test(org.junit.Test)

Example 23 with WriteOptions

use of org.rocksdb.WriteOptions in project flink by apache.

the class RocksDBResourceContainerTest method testFreeWriteReadOptionsAfterClose.

@Test
public void testFreeWriteReadOptionsAfterClose() throws Exception {
    RocksDBResourceContainer container = new RocksDBResourceContainer();
    WriteOptions writeOptions = container.getWriteOptions();
    ReadOptions readOptions = container.getReadOptions();
    assertThat(writeOptions.isOwningHandle(), is(true));
    assertThat(readOptions.isOwningHandle(), is(true));
    container.close();
    assertThat(writeOptions.isOwningHandle(), is(false));
    assertThat(readOptions.isOwningHandle(), is(false));
}
Also used : WriteOptions(org.rocksdb.WriteOptions) ReadOptions(org.rocksdb.ReadOptions) Test(org.junit.Test)

Aggregations

WriteOptions (org.rocksdb.WriteOptions)23 File (java.io.File)11 FlushOptions (org.rocksdb.FlushOptions)9 Options (org.rocksdb.Options)9 Test (org.junit.Test)7 RocksDB (org.rocksdb.RocksDB)6 MapConfig (org.apache.samza.config.MapConfig)5 MetricsRegistryMap (org.apache.samza.metrics.MetricsRegistryMap)5 ColumnFamilyHandle (org.rocksdb.ColumnFamilyHandle)5 IOException (java.io.IOException)4 ColumnFamilyDescriptor (org.rocksdb.ColumnFamilyDescriptor)4 ColumnFamilyOptions (org.rocksdb.ColumnFamilyOptions)4 RocksDBException (org.rocksdb.RocksDBException)4 WriteBatch (org.rocksdb.WriteBatch)4 ByteArrayOutputStream (java.io.ByteArrayOutputStream)3 RocksDBConfigSetter (org.apache.kafka.streams.state.RocksDBConfigSetter)3 Config (org.apache.samza.config.Config)3 BlockBasedTableConfig (org.rocksdb.BlockBasedTableConfig)3 DBOptions (org.rocksdb.DBOptions)3 ReadOptions (org.rocksdb.ReadOptions)3