Search in sources :

Example 1 with EnvOptions

use of org.rocksdb.EnvOptions in project ignite-3 by apache.

the class RocksUtils method createSstFile.

/**
 * Creates an SST file for the column family.
 *
 * @param columnFamily Column family.
 * @param snapshot     Point-in-time snapshot.
 * @param path         Directory to put the SST file in.
 */
public static void createSstFile(ColumnFamily columnFamily, Snapshot snapshot, Path path) {
    try (EnvOptions envOptions = new EnvOptions();
        Options options = new Options();
        ReadOptions readOptions = new ReadOptions().setSnapshot(snapshot);
        RocksIterator it = columnFamily.newIterator(readOptions);
        SstFileWriter sstFileWriter = new SstFileWriter(envOptions, options)) {
        Path sstFile = path.resolve(columnFamily.name());
        sstFileWriter.open(sstFile.toString());
        it.seekToFirst();
        forEach(it, sstFileWriter::put);
        sstFileWriter.finish();
    } catch (Throwable t) {
        throw new IgniteInternalException("Failed to write snapshot: " + t.getMessage(), t);
    }
}
Also used : Path(java.nio.file.Path) ReadOptions(org.rocksdb.ReadOptions) Options(org.rocksdb.Options) EnvOptions(org.rocksdb.EnvOptions) SstFileWriter(org.rocksdb.SstFileWriter) ReadOptions(org.rocksdb.ReadOptions) IgniteInternalException(org.apache.ignite.lang.IgniteInternalException) RocksIterator(org.rocksdb.RocksIterator) EnvOptions(org.rocksdb.EnvOptions)

Example 2 with EnvOptions

use of org.rocksdb.EnvOptions in project sofa-jraft by sofastack.

the class RocksRawKVStore method doCreateSstFiles.

void doCreateSstFiles(final Snapshot snapshot, final EnumMap<SstColumnFamily, File> sstFileTable, final byte[] startKey, final byte[] endKey, final CompletableFuture<Void> future) {
    final Timer.Context timeCtx = getTimeContext("CREATE_SST_FILE");
    final Lock readLock = this.readWriteLock.readLock();
    readLock.lock();
    try {
        if (!this.shutdownLock.isAvailable()) {
            // KV store has shutdown, we do not release rocksdb's snapshot
            future.completeExceptionally(new StorageException("KV store has shutdown."));
            return;
        }
        try (final ReadOptions readOptions = new ReadOptions();
            final EnvOptions envOptions = new EnvOptions();
            final Options options = new Options().setMergeOperator(new StringAppendOperator())) {
            readOptions.setSnapshot(snapshot);
            for (final Map.Entry<SstColumnFamily, File> entry : sstFileTable.entrySet()) {
                final SstColumnFamily sstColumnFamily = entry.getKey();
                final File sstFile = entry.getValue();
                final ColumnFamilyHandle columnFamilyHandle = findColumnFamilyHandle(sstColumnFamily);
                try (final RocksIterator it = this.db.newIterator(columnFamilyHandle, readOptions);
                    final SstFileWriter sstFileWriter = new SstFileWriter(envOptions, options)) {
                    if (startKey == null) {
                        it.seekToFirst();
                    } else {
                        it.seek(startKey);
                    }
                    sstFileWriter.open(sstFile.getAbsolutePath());
                    long count = 0;
                    for (; ; ) {
                        if (!it.isValid()) {
                            break;
                        }
                        final byte[] key = it.key();
                        if (endKey != null && BytesUtil.compare(key, endKey) >= 0) {
                            break;
                        }
                        sstFileWriter.put(key, it.value());
                        ++count;
                        it.next();
                    }
                    if (count == 0) {
                        sstFileWriter.close();
                    } else {
                        sstFileWriter.finish();
                    }
                    LOG.info("Finish sst file {} with {} keys.", sstFile, count);
                } catch (final RocksDBException e) {
                    throw new StorageException("Fail to create sst file at path: " + sstFile, e);
                }
            }
            future.complete(null);
        } catch (final Throwable t) {
            future.completeExceptionally(t);
        } finally {
            // Nothing to release, rocksDB never own the pointer for a snapshot.
            snapshot.close();
            // The pointer to the snapshot is released by the database instance.
            this.db.releaseSnapshot(snapshot);
        }
    } finally {
        readLock.unlock();
        timeCtx.stop();
    }
}
Also used : ColumnFamilyOptions(org.rocksdb.ColumnFamilyOptions) ReadOptions(org.rocksdb.ReadOptions) BackupableDBOptions(org.rocksdb.BackupableDBOptions) IngestExternalFileOptions(org.rocksdb.IngestExternalFileOptions) WriteOptions(org.rocksdb.WriteOptions) RestoreOptions(org.rocksdb.RestoreOptions) Options(org.rocksdb.Options) RocksDBOptions(com.alipay.sofa.jraft.rhea.options.RocksDBOptions) DBOptions(org.rocksdb.DBOptions) EnvOptions(org.rocksdb.EnvOptions) RocksDBException(org.rocksdb.RocksDBException) StringAppendOperator(org.rocksdb.StringAppendOperator) RocksIterator(org.rocksdb.RocksIterator) EnvOptions(org.rocksdb.EnvOptions) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) ReadWriteLock(java.util.concurrent.locks.ReadWriteLock) DistributedLock(com.alipay.sofa.jraft.rhea.util.concurrent.DistributedLock) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) Lock(java.util.concurrent.locks.Lock) SstFileWriter(org.rocksdb.SstFileWriter) Timer(com.codahale.metrics.Timer) ReadOptions(org.rocksdb.ReadOptions) StorageException(com.alipay.sofa.jraft.rhea.errors.StorageException) Map(java.util.Map) EnumMap(java.util.EnumMap) File(java.io.File)

Example 3 with EnvOptions

use of org.rocksdb.EnvOptions in project dingo by dingodb.

the class RocksRawKVStore method doCreateSstFiles.

void doCreateSstFiles(final Snapshot snapshot, final EnumMap<SstColumnFamily, File> sstFileTable, final byte[] startKey, final byte[] endKey, final CompletableFuture<Void> future) {
    final Timer.Context timeCtx = getTimeContext("CREATE_SST_FILE");
    final Lock readLock = this.readWriteLock.readLock();
    readLock.lock();
    try {
        if (!this.shutdownLock.isAvailable()) {
            // KV store has shutdown, we do not release rocksdb's snapshot
            future.completeExceptionally(new StorageException("KV store has shutdown."));
            return;
        }
        try (final ReadOptions readOptions = new ReadOptions();
            final EnvOptions envOptions = new EnvOptions();
            final Options options = new Options().setMergeOperator(new StringAppendOperator())) {
            readOptions.setSnapshot(snapshot);
            for (final Map.Entry<SstColumnFamily, File> entry : sstFileTable.entrySet()) {
                final SstColumnFamily sstColumnFamily = entry.getKey();
                final File sstFile = entry.getValue();
                final ColumnFamilyHandle columnFamilyHandle = findColumnFamilyHandle(sstColumnFamily);
                try (final RocksIterator it = this.db.newIterator(columnFamilyHandle, readOptions);
                    final SstFileWriter sstFileWriter = new SstFileWriter(envOptions, options)) {
                    if (startKey == null) {
                        it.seekToFirst();
                    } else {
                        it.seek(startKey);
                    }
                    sstFileWriter.open(sstFile.getAbsolutePath());
                    long count = 0;
                    for (; ; ) {
                        if (!it.isValid()) {
                            break;
                        }
                        final byte[] key = it.key();
                        if (endKey != null && BytesUtil.compare(key, endKey) >= 0) {
                            break;
                        }
                        sstFileWriter.put(key, it.value());
                        ++count;
                        it.next();
                    }
                    if (count == 0) {
                        sstFileWriter.close();
                    } else {
                        sstFileWriter.finish();
                    }
                    LOG.info("Finish sst file {} with {} keys.", sstFile, count);
                } catch (final RocksDBException e) {
                    throw new StorageException("Fail to create sst file at path: " + sstFile, e);
                }
            }
            future.complete(null);
        } catch (final Throwable t) {
            future.completeExceptionally(t);
        } finally {
            // Nothing to release, rocksDB never own the pointer for a snapshot.
            snapshot.close();
            // The pointer to the snapshot is released by the database instance.
            this.db.releaseSnapshot(snapshot);
        }
    } finally {
        readLock.unlock();
        timeCtx.stop();
    }
}
Also used : StoreDBOptions(io.dingodb.store.row.options.StoreDBOptions) ColumnFamilyOptions(org.rocksdb.ColumnFamilyOptions) ReadOptions(org.rocksdb.ReadOptions) BackupableDBOptions(org.rocksdb.BackupableDBOptions) IngestExternalFileOptions(org.rocksdb.IngestExternalFileOptions) WriteOptions(org.rocksdb.WriteOptions) RestoreOptions(org.rocksdb.RestoreOptions) Options(org.rocksdb.Options) DBOptions(org.rocksdb.DBOptions) EnvOptions(org.rocksdb.EnvOptions) RocksDBException(org.rocksdb.RocksDBException) StringAppendOperator(org.rocksdb.StringAppendOperator) RocksIterator(org.rocksdb.RocksIterator) EnvOptions(org.rocksdb.EnvOptions) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) DistributedLock(io.dingodb.store.row.util.concurrent.DistributedLock) ReadWriteLock(java.util.concurrent.locks.ReadWriteLock) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) Lock(java.util.concurrent.locks.Lock) SstFileWriter(org.rocksdb.SstFileWriter) Timer(com.codahale.metrics.Timer) ReadOptions(org.rocksdb.ReadOptions) StorageException(io.dingodb.store.row.errors.StorageException) Map(java.util.Map) EnumMap(java.util.EnumMap) File(java.io.File)

Aggregations

EnvOptions (org.rocksdb.EnvOptions)3 Options (org.rocksdb.Options)3 ReadOptions (org.rocksdb.ReadOptions)3 RocksIterator (org.rocksdb.RocksIterator)3 SstFileWriter (org.rocksdb.SstFileWriter)3 Timer (com.codahale.metrics.Timer)2 File (java.io.File)2 EnumMap (java.util.EnumMap)2 Map (java.util.Map)2 Lock (java.util.concurrent.locks.Lock)2 ReadWriteLock (java.util.concurrent.locks.ReadWriteLock)2 ReentrantReadWriteLock (java.util.concurrent.locks.ReentrantReadWriteLock)2 BackupableDBOptions (org.rocksdb.BackupableDBOptions)2 ColumnFamilyHandle (org.rocksdb.ColumnFamilyHandle)2 ColumnFamilyOptions (org.rocksdb.ColumnFamilyOptions)2 DBOptions (org.rocksdb.DBOptions)2 IngestExternalFileOptions (org.rocksdb.IngestExternalFileOptions)2 RestoreOptions (org.rocksdb.RestoreOptions)2 RocksDBException (org.rocksdb.RocksDBException)2 StringAppendOperator (org.rocksdb.StringAppendOperator)2