Search in sources :

Example 11 with FlushOptions

use of org.rocksdb.FlushOptions in project samza by apache.

the class TestRocksDbKeyValueStoreJava method testIterate.

@Test
public void testIterate() throws Exception {
    Config config = new MapConfig();
    Options options = new Options();
    options.setCreateIfMissing(true);
    File dbDir = new File(System.getProperty("java.io.tmpdir") + "/dbStore" + System.currentTimeMillis());
    RocksDbKeyValueStore store = new RocksDbKeyValueStore(dbDir, options, config, false, "dbStore", new WriteOptions(), new FlushOptions(), new KeyValueStoreMetrics("dbStore", new MetricsRegistryMap()));
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    String prefix = "prefix";
    for (int i = 0; i < 100; i++) {
        store.put(genKey(outputStream, prefix, i), genValue());
    }
    byte[] firstKey = genKey(outputStream, prefix, 0);
    byte[] lastKey = genKey(outputStream, prefix, 1000);
    KeyValueSnapshot<byte[], byte[]> snapshot = store.snapshot(firstKey, lastKey);
    // Make sure the cached Iterable won't change when new elements are added
    store.put(genKey(outputStream, prefix, 200), genValue());
    KeyValueIterator<byte[], byte[]> iterator = snapshot.iterator();
    assertTrue(Iterators.size(iterator) == 100);
    iterator.close();
    List<Integer> keys = new ArrayList<>();
    KeyValueIterator<byte[], byte[]> iterator2 = snapshot.iterator();
    while (iterator2.hasNext()) {
        Entry<byte[], byte[]> entry = iterator2.next();
        int key = Ints.fromByteArray(Arrays.copyOfRange(entry.getKey(), prefix.getBytes().length, entry.getKey().length));
        keys.add(key);
    }
    assertEquals(keys, IntStream.rangeClosed(0, 99).boxed().collect(Collectors.toList()));
    iterator2.close();
    outputStream.close();
    snapshot.close();
    store.close();
}
Also used : FlushOptions(org.rocksdb.FlushOptions) WriteOptions(org.rocksdb.WriteOptions) Options(org.rocksdb.Options) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) ArrayList(java.util.ArrayList) ByteArrayOutputStream(java.io.ByteArrayOutputStream) FlushOptions(org.rocksdb.FlushOptions) WriteOptions(org.rocksdb.WriteOptions) MapConfig(org.apache.samza.config.MapConfig) MetricsRegistryMap(org.apache.samza.metrics.MetricsRegistryMap) File(java.io.File) Test(org.junit.Test)

Example 12 with FlushOptions

use of org.rocksdb.FlushOptions in project beam by apache.

the class SamzaTimerInternalsFactoryTest method createStore.

private KeyValueStore<ByteArray, StateValue<?>> createStore() {
    final Options options = new Options();
    options.setCreateIfMissing(true);
    RocksDbKeyValueStore rocksStore = new RocksDbKeyValueStore(temporaryFolder.getRoot(), options, new MapConfig(), false, "beamStore", new WriteOptions(), new FlushOptions(), new KeyValueStoreMetrics("beamStore", new MetricsRegistryMap()));
    return new SerializedKeyValueStore<>(rocksStore, new ByteArraySerdeFactory.ByteArraySerde(), new StateValueSerdeFactory.StateValueSerde(), new SerializedKeyValueStoreMetrics("beamStore", new MetricsRegistryMap()));
}
Also used : FlushOptions(org.rocksdb.FlushOptions) WriteOptions(org.rocksdb.WriteOptions) SamzaPipelineOptions(org.apache.beam.runners.samza.SamzaPipelineOptions) Options(org.rocksdb.Options) SerializedKeyValueStoreMetrics(org.apache.samza.storage.kv.SerializedKeyValueStoreMetrics) KeyValueStoreMetrics(org.apache.samza.storage.kv.KeyValueStoreMetrics) ByteArraySerdeFactory(org.apache.beam.runners.samza.runtime.SamzaStoreStateInternals.ByteArraySerdeFactory) FlushOptions(org.rocksdb.FlushOptions) SerializedKeyValueStoreMetrics(org.apache.samza.storage.kv.SerializedKeyValueStoreMetrics) WriteOptions(org.rocksdb.WriteOptions) SerializedKeyValueStore(org.apache.samza.storage.kv.SerializedKeyValueStore) MapConfig(org.apache.samza.config.MapConfig) StateValueSerdeFactory(org.apache.beam.runners.samza.runtime.SamzaStoreStateInternals.StateValueSerdeFactory) MetricsRegistryMap(org.apache.samza.metrics.MetricsRegistryMap) RocksDbKeyValueStore(org.apache.samza.storage.kv.RocksDbKeyValueStore)

Aggregations

FlushOptions (org.rocksdb.FlushOptions)12 WriteOptions (org.rocksdb.WriteOptions)9 File (java.io.File)7 Options (org.rocksdb.Options)6 MapConfig (org.apache.samza.config.MapConfig)5 MetricsRegistryMap (org.apache.samza.metrics.MetricsRegistryMap)5 IOException (java.io.IOException)3 RocksDBConfigSetter (org.apache.kafka.streams.state.RocksDBConfigSetter)3 Config (org.apache.samza.config.Config)3 Test (org.junit.Test)3 BlockBasedTableConfig (org.rocksdb.BlockBasedTableConfig)3 RocksDBException (org.rocksdb.RocksDBException)3 ByteArrayOutputStream (java.io.ByteArrayOutputStream)2 ArrayList (java.util.ArrayList)2 List (java.util.List)2 Random (java.util.Random)2 ProcessorStateException (org.apache.kafka.streams.errors.ProcessorStateException)2 Before (org.junit.Before)2 Checkpoint (org.rocksdb.Checkpoint)2 Arrays (java.util.Arrays)1