Search in sources :

Example 1 with VirtualInternalRecordSerializer

use of com.swirlds.jasperdb.VirtualInternalRecordSerializer in project hedera-services by hashgraph.

the class VirtualDataSourceBench method setup.

@Setup(Level.Trial)
public void setup() throws IOException {
    random = new Random(1234);
    System.out.println("dataSourcePath = " + dataSourcePath);
    if (Files.exists(dataSourcePath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db.");
        deleteDirectoryAndContents(dataSourcePath);
    }
    if (Files.exists(dataSourceSnapshotPath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db snapshot.");
        deleteDirectoryAndContents(dataSourceSnapshotPath);
    }
    // create data source
    VirtualLeafRecordSerializer<ContractKey, ContractValue> virtualLeafRecordSerializer = new VirtualLeafRecordSerializer<>((short) 1, DigestType.SHA_384, (short) 1, DataFileCommon.VARIABLE_DATA_SIZE, new ContractKeySupplier(), (short) 1, ContractValue.SERIALIZED_SIZE, new ContractValueSupplier(), true);
    JasperDbBuilder<ContractKey, ContractValue> dbBuilder = new JasperDbBuilder<>();
    dbBuilder.virtualLeafRecordSerializer(virtualLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(new ContractKeySerializer()).storageDir(dataSourcePath).maxNumOfKeys(500_000_000).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(0).mergingEnabled(false);
    dataSource = dbBuilder.build("jdb", "4dsBench");
    // populate with initial data
    System.out.printf("Creating initial data set of %,d leaves\n", initialDataSize);
    progressPercentage = 0;
    final long firstLeafPath = initialDataSize;
    final long lastLeafPath = firstLeafPath + initialDataSize;
    var internalRecordStream = LongStream.range(0, firstLeafPath).mapToObj(path -> new VirtualInternalRecord(path, hash((int) path)));
    var leafRecordStream = LongStream.range(firstLeafPath, lastLeafPath + 1).mapToObj(path -> new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path, path), new ContractValue(path))).peek(leaf -> printProgress(leaf.getPath(), lastLeafPath));
    dataSource.saveRecords(firstLeafPath, lastLeafPath, internalRecordStream, leafRecordStream, Stream.empty());
    System.out.printf("Done creating initial data set of %,d leaves\n", initialDataSize);
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Measurement(org.openjdk.jmh.annotations.Measurement) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) CommonTestUtils.deleteDirectoryAndContents(utils.CommonTestUtils.deleteDirectoryAndContents) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) Scope(org.openjdk.jmh.annotations.Scope) Random(java.util.Random) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Warmup(org.openjdk.jmh.annotations.Warmup) OutputTimeUnit(org.openjdk.jmh.annotations.OutputTimeUnit) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) CommonTestUtils.hash(utils.CommonTestUtils.hash) Setup(org.openjdk.jmh.annotations.Setup) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) Mode(org.openjdk.jmh.annotations.Mode) Files(java.nio.file.Files) ContractValue(com.hedera.services.state.virtual.ContractValue) Param(org.openjdk.jmh.annotations.Param) IOException(java.io.IOException) State(org.openjdk.jmh.annotations.State) DigestType(com.swirlds.common.crypto.DigestType) Benchmark(org.openjdk.jmh.annotations.Benchmark) TimeUnit(java.util.concurrent.TimeUnit) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) Stream(java.util.stream.Stream) Level(org.openjdk.jmh.annotations.Level) Fork(org.openjdk.jmh.annotations.Fork) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) ContractValue(com.hedera.services.state.virtual.ContractValue) Random(java.util.Random) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Setup(org.openjdk.jmh.annotations.Setup)

Example 2 with VirtualInternalRecordSerializer

use of com.swirlds.jasperdb.VirtualInternalRecordSerializer in project hedera-services by hashgraph.

the class ContractDataSourceValidator method main.

public static void main(String[] args) throws IOException {
    Path dataSourcePath = Path.of("").toAbsolutePath();
    System.out.println("dataSourcePath = " + dataSourcePath + " exists " + Files.exists(dataSourcePath));
    VirtualLeafRecordSerializer<ContractKey, ContractValue> virtualLeafRecordSerializer = new VirtualLeafRecordSerializer<>((short) 1, DigestType.SHA_384, (short) 1, DataFileCommon.VARIABLE_DATA_SIZE, new ContractKeySupplier(), (short) 1, ContractValue.SERIALIZED_SIZE, new ContractValueSupplier(), true);
    ;
    JasperDbBuilder<ContractKey, ContractValue> dbBuilder = new JasperDbBuilder<>();
    dbBuilder.virtualLeafRecordSerializer(virtualLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(new ContractKeySerializer()).storageDir(dataSourcePath).maxNumOfKeys(500_000_000).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(0).mergingEnabled(true);
    DataSourceValidator<ContractKey, ContractValue> dataSourceValidator = new DataSourceValidator<>(dbBuilder.build("jdb", "4validator"));
    dataSourceValidator.validate();
}
Also used : Path(java.nio.file.Path) DataSourceValidator(com.swirlds.jasperdb.DataSourceValidator) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier)

Example 3 with VirtualInternalRecordSerializer

use of com.swirlds.jasperdb.VirtualInternalRecordSerializer in project hedera-services by hashgraph.

the class VirtualDataSourceNewAPIBench method setup.

@Setup(Level.Trial)
public void setup() {
    System.out.println("------- Setup -----------------------------");
    storePath = Path.of("store-" + impl);
    try {
        final boolean storeExists = Files.exists(storePath);
        // get slot index suppliers
        switch(impl) {
            case "lmdb":
                dataSource = new VFCDataSourceLmdb<>(// max seralized size
                1 + 8 + 32, // max seralized size
                ContractKey::new, ContractValue.SERIALIZED_SIZE, ContractValue::new, storePath);
                break;
            // storePath);
            case "jasperdb":
                VirtualLeafRecordSerializer<ContractKey, ContractValue> virtualLeafRecordSerializer = new VirtualLeafRecordSerializer<>((short) 1, DigestType.SHA_384, (short) 1, DataFileCommon.VARIABLE_DATA_SIZE, new ContractKeySupplier(), (short) 1, ContractValue.SERIALIZED_SIZE, new ContractValueSupplier(), true);
                JasperDbBuilder<ContractKey, ContractValue> dbBuilder = new JasperDbBuilder<>();
                dbBuilder.virtualLeafRecordSerializer(virtualLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(new ContractKeySerializer()).storageDir(storePath).maxNumOfKeys(numEntities + 10_000_000).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(Long.MAX_VALUE).mergingEnabled(true);
                dataSource = dbBuilder.build("jdb", "4ApiBench");
                break;
            default:
                throw new IllegalStateException("Unexpected value: " + impl);
        }
        ;
        // create data
        if (!storeExists) {
            System.out.println("================================================================================");
            System.out.println("Creating data ...");
            // create internal nodes and leaves
            long iHaveWritten = 0;
            while (iHaveWritten < numEntities) {
                final long start = System.currentTimeMillis();
                final long batchSize = Math.min(WRITE_BATCH_SIZE, numEntities - iHaveWritten);
                dataSource.saveRecords(numEntities, numEntities * 2, LongStream.range(iHaveWritten, iHaveWritten + batchSize).mapToObj(i -> new VirtualInternalRecord(i, hash((int) i))), LongStream.range(iHaveWritten, iHaveWritten + batchSize).mapToObj(i -> new VirtualLeafRecord<>(i + numEntities, hash((int) i), new ContractKey(i, i), new ContractValue(i))), Stream.empty());
                iHaveWritten += batchSize;
                printUpdate(start, batchSize, ContractValue.SERIALIZED_SIZE, "Created " + iHaveWritten + " Nodes");
            }
            System.out.println("================================================================================");
            // set nextPath
            nextPath = numEntities;
            // let merge catch up
            try {
                System.out.println("Waiting for merge");
                Thread.sleep(30000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        } else {
            System.out.println("Loaded existing data");
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) Random(java.util.Random) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Hash(com.swirlds.common.crypto.Hash) org.openjdk.jmh.annotations(org.openjdk.jmh.annotations) ArrayList(java.util.ArrayList) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) VFCDataSourceLmdb(lmdb.VFCDataSourceLmdb) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) CommonTestUtils.printDirectorySize(utils.CommonTestUtils.printDirectorySize) Files(java.nio.file.Files) ContractValue(com.hedera.services.state.virtual.ContractValue) VirtualDataSource(com.swirlds.virtualmap.datasource.VirtualDataSource) IOException(java.io.IOException) DigestType(com.swirlds.common.crypto.DigestType) TimeUnit(java.util.concurrent.TimeUnit) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) List(java.util.List) Stream(java.util.stream.Stream) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) LongArrayList(org.eclipse.collections.impl.list.mutable.primitive.LongArrayList) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) IOException(java.io.IOException) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier)

Example 4 with VirtualInternalRecordSerializer

use of com.swirlds.jasperdb.VirtualInternalRecordSerializer in project hedera-services by hashgraph.

the class DatabaseState method setupDatabase.

@Setup(Level.Trial)
public void setupDatabase() throws IOException {
    System.out.println("dataSourcePath = " + dataSourcePath + " mergingEnabled=" + mergingEnabled);
    if (Files.exists(dataSourcePath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db.");
        deleteDirectoryAndContents(dataSourcePath);
    }
    if (Files.exists(dataSourceSnapshotPath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db snapshot.");
        deleteDirectoryAndContents(dataSourceSnapshotPath);
    }
    // create data source
    VirtualLeafRecordSerializer<ContractKey, ContractValue> virtualLeafRecordSerializer = new VirtualLeafRecordSerializer<>((short) 1, DigestType.SHA_384, (short) 1, DataFileCommon.VARIABLE_DATA_SIZE, new ContractKeySupplier(), (short) 1, ContractValue.SERIALIZED_SIZE, new ContractValueSupplier(), true);
    JasperDbBuilder<ContractKey, ContractValue> dbBuilder = new JasperDbBuilder<>();
    dbBuilder.virtualLeafRecordSerializer(virtualLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(new ContractKeySerializer()).storageDir(dataSourcePath).maxNumOfKeys(500_000_000).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(0).mergingEnabled(mergingEnabled);
    dataSource = dbBuilder.build("jdb", "4dbState");
    // populate with initial data
    System.out.printf("Creating initial data set of %,d leaves\n", initialDataSize);
    progressPercentage = 0;
    final long firstLeafPath = initialDataSize;
    final long lastLeafPath = firstLeafPath + initialDataSize;
    var internalRecordStream = LongStream.range(0, firstLeafPath).mapToObj(path -> new VirtualInternalRecord(path, hash((int) path))).peek(internalRecord -> printProgress(internalRecord.getPath(), lastLeafPath));
    var leafRecordStream = LongStream.range(firstLeafPath, lastLeafPath + 1).mapToObj(path -> new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path, path), new ContractValue(path))).peek(leaf -> printProgress(leaf.getPath(), lastLeafPath));
    dataSource.saveRecords(firstLeafPath, lastLeafPath, internalRecordStream, leafRecordStream, Stream.empty());
    System.out.printf("Done creating initial data set of %,d leaves\n", initialDataSize);
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) CommonTestUtils.deleteDirectoryAndContents(utils.CommonTestUtils.deleteDirectoryAndContents) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) Scope(org.openjdk.jmh.annotations.Scope) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) CommonTestUtils.hash(utils.CommonTestUtils.hash) Setup(org.openjdk.jmh.annotations.Setup) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) Files(java.nio.file.Files) ContractValue(com.hedera.services.state.virtual.ContractValue) Param(org.openjdk.jmh.annotations.Param) IOException(java.io.IOException) State(org.openjdk.jmh.annotations.State) DigestType(com.swirlds.common.crypto.DigestType) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) Stream(java.util.stream.Stream) Level(org.openjdk.jmh.annotations.Level) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Setup(org.openjdk.jmh.annotations.Setup)

Example 5 with VirtualInternalRecordSerializer

use of com.swirlds.jasperdb.VirtualInternalRecordSerializer in project hedera-services by hashgraph.

the class VirtualMapFactory method newVirtualizedBlobs.

public VirtualMap<VirtualBlobKey, VirtualBlobValue> newVirtualizedBlobs() {
    final var blobKeySerializer = new VirtualBlobKeySerializer();
    final VirtualLeafRecordSerializer<VirtualBlobKey, VirtualBlobValue> blobLeafRecordSerializer = new VirtualLeafRecordSerializer<>(CURRENT_SERIALIZATION_VERSION, DigestType.SHA_384, CURRENT_SERIALIZATION_VERSION, VirtualBlobKey.sizeInBytes(), new VirtualBlobKeySupplier(), CURRENT_SERIALIZATION_VERSION, VirtualBlobValue.sizeInBytes(), new VirtualBlobValueSupplier(), false);
    final JasperDbBuilder<VirtualBlobKey, VirtualBlobValue> dsBuilder = jdbBuilderFactory.newJdbBuilder();
    dsBuilder.virtualLeafRecordSerializer(blobLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(blobKeySerializer).maxNumOfKeys(MAX_BLOBS).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(MAX_IN_MEMORY_INTERNAL_HASHES);
    return new VirtualMap<>(BLOBS_VM_NAME, dsBuilder);
}
Also used : VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) VirtualMap(com.swirlds.virtualmap.VirtualMap)

Aggregations

VirtualInternalRecordSerializer (com.swirlds.jasperdb.VirtualInternalRecordSerializer)6 VirtualLeafRecordSerializer (com.swirlds.jasperdb.VirtualLeafRecordSerializer)6 ContractKey (com.hedera.services.state.virtual.ContractKey)4 ContractKeySerializer (com.hedera.services.state.virtual.ContractKeySerializer)4 ContractKeySupplier (com.hedera.services.state.virtual.ContractKeySupplier)4 ContractValue (com.hedera.services.state.virtual.ContractValue)4 ContractValueSupplier (com.hedera.services.state.virtual.ContractValueSupplier)4 JasperDbBuilder (com.swirlds.jasperdb.JasperDbBuilder)4 Path (java.nio.file.Path)4 DigestType (com.swirlds.common.crypto.DigestType)3 VirtualDataSourceJasperDB (com.swirlds.jasperdb.VirtualDataSourceJasperDB)3 DataFileCommon (com.swirlds.jasperdb.files.DataFileCommon)3 VirtualInternalRecord (com.swirlds.virtualmap.datasource.VirtualInternalRecord)3 VirtualLeafRecord (com.swirlds.virtualmap.datasource.VirtualLeafRecord)3 IOException (java.io.IOException)3 Files (java.nio.file.Files)3 LongStream (java.util.stream.LongStream)3 Stream (java.util.stream.Stream)3 VirtualMap (com.swirlds.virtualmap.VirtualMap)2 Random (java.util.Random)2