Search in sources :

Example 36 with RocksDBException

use of org.rocksdb.RocksDBException in project kafka by apache.

the class RocksDBStore method putAll.

@Override
public void putAll(List<KeyValue<K, V>> entries) {
    try (WriteBatch batch = new WriteBatch()) {
        for (KeyValue<K, V> entry : entries) {
            final byte[] rawKey = serdes.rawKey(entry.key);
            if (entry.value == null) {
                db.delete(rawKey);
            } else {
                final byte[] value = serdes.rawValue(entry.value);
                batch.put(rawKey, value);
            }
        }
        db.write(wOptions, batch);
    } catch (RocksDBException e) {
        throw new ProcessorStateException("Error while batch writing to store " + this.name, e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) WriteBatch(org.rocksdb.WriteBatch) ProcessorStateException(org.apache.kafka.streams.errors.ProcessorStateException)

Example 37 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBAggregatingState method get.

@Override
public R get() throws IOException {
    try {
        // prepare the current key and namespace for RocksDB lookup
        writeCurrentKeyWithGroupAndNamespace();
        final byte[] key = keySerializationStream.toByteArray();
        // get the current value
        final byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return null;
        }
        ACC accumulator = valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStreamWithPos(valueBytes)));
        return aggFunction.getResult(accumulator);
    } catch (IOException | RocksDBException e) {
        throw new IOException("Error while retrieving value from RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 38 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBAggregatingState method add.

@Override
public void add(T value) throws IOException {
    try {
        // prepare the current key and namespace for RocksDB lookup
        writeCurrentKeyWithGroupAndNamespace();
        final byte[] key = keySerializationStream.toByteArray();
        keySerializationStream.reset();
        // get the current value
        final byte[] valueBytes = backend.db.get(columnFamily, key);
        // deserialize the current accumulator, or create a blank one
        final ACC accumulator = valueBytes == null ? aggFunction.createAccumulator() : valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStreamWithPos(valueBytes)));
        // aggregate the value into the accumulator
        aggFunction.add(value, accumulator);
        // serialize the new accumulator
        final DataOutputViewStreamWrapper out = new DataOutputViewStreamWrapper(keySerializationStream);
        valueSerializer.serialize(accumulator, out);
        // write the new value to RocksDB
        backend.db.put(columnFamily, writeOptions, key, keySerializationStream.toByteArray());
    } catch (IOException | RocksDBException e) {
        throw new IOException("Error while adding value to RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 39 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBFoldingState method get.

@Override
public ACC get() {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return null;
        }
        return valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStreamWithPos(valueBytes)));
    } catch (IOException | RocksDBException e) {
        throw new RuntimeException("Error while retrieving data from RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Example 40 with RocksDBException

use of org.rocksdb.RocksDBException in project flink by apache.

the class RocksDBReducingState method get.

@Override
public V get() {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        if (valueBytes == null) {
            return null;
        }
        return valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStream(valueBytes)));
    } catch (IOException | RocksDBException e) {
        throw new RuntimeException("Error while retrieving data from RocksDB", e);
    }
}
Also used : RocksDBException(org.rocksdb.RocksDBException) ByteArrayInputStream(java.io.ByteArrayInputStream) IOException(java.io.IOException) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper)

Aggregations

RocksDBException (org.rocksdb.RocksDBException)66 IOException (java.io.IOException)17 ColumnFamilyHandle (org.rocksdb.ColumnFamilyHandle)17 ProcessorStateException (org.apache.kafka.streams.errors.ProcessorStateException)11 ColumnFamilyDescriptor (org.rocksdb.ColumnFamilyDescriptor)11 ArrayList (java.util.ArrayList)10 WriteBatch (org.rocksdb.WriteBatch)10 HashMap (java.util.HashMap)8 Map (java.util.Map)8 MetricException (org.apache.storm.metricstore.MetricException)8 WriteOptions (org.rocksdb.WriteOptions)7 Options (org.rocksdb.Options)6 File (java.io.File)5 DBOptions (org.rocksdb.DBOptions)5 FlushOptions (org.rocksdb.FlushOptions)5 RocksDB (org.rocksdb.RocksDB)5 DataInputViewStreamWrapper (org.apache.flink.core.memory.DataInputViewStreamWrapper)4 ColumnFamilyOptions (org.rocksdb.ColumnFamilyOptions)4 ReadOptions (org.rocksdb.ReadOptions)4 RocksIterator (org.rocksdb.RocksIterator)4