Search in sources :

Example 11 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBFoldingState method get.

@Override
public ACC get() {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return null;
        }
        return valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStreamWithPos(valueBytes)));
    } catch (IOException | RocksDBException e) {
        throw new RuntimeException("Error while retrieving data from RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 12 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBReducingState method get.

@Override
public V get() {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return null;
        }
        return valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStream(valueBytes)));
    } catch (IOException | RocksDBException e) {
        throw new RuntimeException("Error while retrieving data from RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStream(java.io.ByteArrayInputStream) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 13 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBValueState method value.

@Override
public V value() {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return stateDesc.getDefaultValue();
        }
        return valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStream(valueBytes)));
    } catch (IOException | RocksDBException e) {
        throw new RuntimeException("Error while retrieving data from RocksDB.", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStream(java.io.ByteArrayInputStream) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 14 with RocksDBException

use of org.rocksdb.RocksDBException in project kafka by apache.

the class RocksDBStore method putAll.

@Override
public void putAll(List<KeyValue<K, V>> entries) {
    try (WriteBatch batch = new WriteBatch()) {
        for (KeyValue<K, V> entry : entries) {
            final byte[] rawKey = serdes.rawKey(entry.key);
            if (entry.value == null) {
                db.delete(rawKey);
            } else {
                final byte[] value = serdes.rawValue(entry.value);
                batch.put(rawKey, value);
            }
        }
        db.write(wOptions, batch);
    } catch (RocksDBException e) {
        throw new ProcessorStateException("Error while batch writing to store " + this.name, e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) WriteBatch(org.rocksdb.WriteBatch) ProcessorStateException(org.apache.kafka.streams.errors.ProcessorStateException)

Example 15 with RocksDBException

use of org.rocksdb.RocksDBException in project jstorm by alibaba.

the class RocksDBTest method visitorAccross.

public void visitorAccross() throws RocksDBException, InterruptedException {
    DBOptions dbOptions = null;
    TtlDB ttlDB = null;
    List<ColumnFamilyDescriptor> cfNames = new ArrayList<ColumnFamilyDescriptor>();
    List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<ColumnFamilyHandle>();
    cfNames.add(new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY));
    cfNames.add(new ColumnFamilyDescriptor("new_cf".getBytes()));
    List<Integer> ttlValues = new ArrayList<Integer>();
    // new column family with 1 second ttl
    ttlValues.add(1);
    // Default column family with infinite lifetime
    ttlValues.add(0);
    try {
        System.out.println("Begin to open db");
        dbOptions = new DBOptions().setCreateMissingColumnFamilies(true).setCreateIfMissing(true);
        ttlDB = TtlDB.open(dbOptions, rootDir, cfNames, columnFamilyHandleList, ttlValues, false);
        System.out.println("Successfully open db " + rootDir);
        List<String> keys = new ArrayList<String>();
        keys.add("key");
        ttlDB.put("key".getBytes(), "key".getBytes());
        for (int i = 0; i < 2; i++) {
            String key = "key" + i;
            keys.add(key);
            ttlDB.put(columnFamilyHandleList.get(i), key.getBytes(), key.getBytes());
        }
        try {
            byte[] value = ttlDB.get("others".getBytes());
            if (value != null) {
                System.out.println("Raw get :" + new String(value));
            } else {
                System.out.println("No value of other");
            }
        } catch (Exception e) {
            System.out.println("Occur exception other");
        }
        for (String key : keys) {
            try {
                byte[] value = ttlDB.get(key.getBytes());
                if (value != null) {
                    System.out.println("Raw get :" + new String(value));
                } else {
                    System.out.println("No value of " + key);
                }
            } catch (Exception e) {
                System.out.println("Occur exception " + key + ", Raw");
            }
            for (int i = 0; i < 2; i++) {
                try {
                    byte[] value = ttlDB.get(columnFamilyHandleList.get(i), key.getBytes());
                    if (value != null) {
                        System.out.println("handler index" + i + " get :" + new String(value));
                    } else {
                        System.out.println("No value of index" + i + " get :" + key);
                    }
                } catch (Exception e) {
                    System.out.println("Occur exception " + key + ", handler index:" + i);
                }
            }
        }
    } finally {
        for (ColumnFamilyHandle columnFamilyHandle : columnFamilyHandleList) {
            columnFamilyHandle.dispose();
        }
        if (ttlDB != null) {
            ttlDB.close();
        }
        if (dbOptions != null) {
            dbOptions.dispose();
        }
    }
}
Also used : ArrayList(java.util.ArrayList) ColumnFamilyDescriptor(org.rocksdb.ColumnFamilyDescriptor) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) RocksDBException(org.rocksdb.RocksDBException) DBOptions(org.rocksdb.DBOptions) TtlDB(org.rocksdb.TtlDB)

Aggregations

RocksDBException (org.rocksdb.RocksDBException)16 IOException (java.io.IOException)8 DataInputViewStreamWrapper (org.apache.flink.core.memory.DataInputViewStreamWrapper)6 PersistenceFailureException (voldemort.store.PersistenceFailureException)4 Versioned (voldemort.versioning.Versioned)4 ByteArrayInputStream (java.io.ByteArrayInputStream)3 ByteArrayInputStreamWithPos (org.apache.flink.core.memory.ByteArrayInputStreamWithPos)3 ColumnFamilyDescriptor (org.rocksdb.ColumnFamilyDescriptor)3 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 ProcessorStateException (org.apache.kafka.streams.errors.ProcessorStateException)2 ColumnFamilyHandle (org.rocksdb.ColumnFamilyHandle)2 Map (java.util.Map)1 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)1 DataOutputViewStreamWrapper (org.apache.flink.core.memory.DataOutputViewStreamWrapper)1 RegisteredBackendStateMetaInfo (org.apache.flink.runtime.state.RegisteredBackendStateMetaInfo)1 SamzaException (org.apache.samza.SamzaException)1 Config (org.apache.samza.config.Config)1 MapConfig (org.apache.samza.config.MapConfig)1 IntegerSerdeFactory (org.apache.samza.serializers.IntegerSerdeFactory)1