Search in sources :

Example 1 with VirtualLeafRecord

use of com.swirlds.virtualmap.datasource.VirtualLeafRecord in project hedera-services by hashgraph.

the class UpdateBench method updateLeavesAndInternals.

@Benchmark
public void updateLeavesAndInternals(DatabaseMergingState databaseState) throws IOException {
    var internalRecordStream = IntStream.range(0, NUMBER_OF_INTERNALS_CHANGED_PER_FLUSH).mapToObj(i -> new VirtualInternalRecord(randomInternalPaths[i], hash((int) randomInternalPaths[i])));
    var leafRecordStream = IntStream.range(0, NUMBER_OF_LEAVES_CHANGED_PER_FLUSH).mapToObj(i -> {
        final long path = randomLeafPaths[i];
        return new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path, path), new ContractValue(path));
    });
    databaseState.dataSource.saveRecords(databaseState.dataSource.getFirstLeafPath(), databaseState.dataSource.getLastLeafPath(), internalRecordStream, leafRecordStream, Stream.empty());
}
Also used : ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Example 2 with VirtualLeafRecord

use of com.swirlds.virtualmap.datasource.VirtualLeafRecord in project hedera-services by hashgraph.

the class ThorsFileHammer method updateAllValues.

public void updateAllValues() {
    try {
        final int batchSize = 1000;
        final int numOfBatches = (int) ((fileCollection.getMaximumValidKey() - fileCollection.getMinimumValidKey()) / batchSize);
        System.out.println("ThorsHammer.updateAllValues numOfBatches=" + numOfBatches);
        for (int batchIndex = 0; batchIndex < numOfBatches; batchIndex++) {
            final int firstLeaf = (int) (fileCollection.getMinimumValidKey() + (batchIndex * batchSize));
            final int lastLeaf = firstLeaf + batchSize;
            final var leafRecordStream = LongStream.range(firstLeaf, lastLeaf).mapToObj(path -> new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path / 1000, path), new ContractValue(RANDOM.nextLong()))).peek(leaf -> compareToMe.set((int) leaf.getPath(), (int) leaf.getValue().asLong()));
            try {
                readWriteLock.writeLock().lock();
                fileCollection.startWriting();
                final Map<Long, Long> indexUpdates = new HashMap<>();
                leafRecordStream.forEach(leaf -> {
                    try {
                        indexUpdates.put(leaf.getPath(), fileCollection.storeDataItem(leaf));
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });
                final var dataFile = fileCollection.endWriting(fileCollection.getMinimumValidKey(), fileCollection.getMaximumValidKey());
                for (var update : indexUpdates.entrySet()) {
                    index.put(update.getKey(), update.getValue());
                }
                dataFile.setFileAvailableForMerging(true);
            } finally {
                readWriteLock.writeLock().unlock();
            }
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) CommonTestUtils.hash(utils.CommonTestUtils.hash) LongStream(java.util.stream.LongStream) Files(java.nio.file.Files) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValue(com.hedera.services.state.virtual.ContractValue) CommonTestUtils.deleteDirectoryAndContents(utils.CommonTestUtils.deleteDirectoryAndContents) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) IOException(java.io.IOException) HashMap(java.util.HashMap) Random(java.util.Random) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) DataFileCollection(com.swirlds.jasperdb.files.DataFileCollection) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) DigestType(com.swirlds.common.crypto.DigestType) Map(java.util.Map) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) LongListOffHeap(com.swirlds.jasperdb.collections.LongListOffHeap) Path(java.nio.file.Path) AtomicIntegerArray(java.util.concurrent.atomic.AtomicIntegerArray) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) HashMap(java.util.HashMap) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) IOException(java.io.IOException)

Example 3 with VirtualLeafRecord

use of com.swirlds.virtualmap.datasource.VirtualLeafRecord in project hedera-services by hashgraph.

the class ThorsHammer method updateAllValues.

public void updateAllValues() {
    try {
        final int batchSize = 1000;
        final int numOfBatches = (int) ((dataSource.getLastLeafPath() - dataSource.getFirstLeafPath()) / batchSize);
        System.out.println("ThorsHammer.updateAllValues numOfBatches=" + numOfBatches);
        for (int batchIndex = 0; batchIndex < numOfBatches; batchIndex++) {
            final int firstLeaf = (int) (dataSource.getFirstLeafPath() + (batchIndex * batchSize));
            final int lastLeaf = firstLeaf + batchSize;
            readExclusionZoneFirstLeafPath.set(firstLeaf);
            readExclusionZoneLastLeafPath.set(lastLeaf);
            final var leafRecordStream = LongStream.range(firstLeaf, lastLeaf).mapToObj(path -> new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path / 1000, path), new ContractValue(RANDOM.nextLong()))).peek(leaf -> compareToMe.set((int) leaf.getPath(), (int) leaf.getValue().asLong()));
            // try {
            // readWriteLock.writeLock().lock();
            dataSource.saveRecords(dataSource.getFirstLeafPath(), dataSource.getLastLeafPath(), Stream.empty(), leafRecordStream, Stream.empty());
            // } finally {
            // readWriteLock.writeLock().unlock();
            // }
            Thread.sleep(5);
        }
    } catch (IOException | InterruptedException e) {
        e.printStackTrace();
    }
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) CommonTestUtils.deleteDirectoryAndContents(utils.CommonTestUtils.deleteDirectoryAndContents) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) HashMap(java.util.HashMap) Random(java.util.Random) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Map(java.util.Map) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) AtomicIntegerArray(java.util.concurrent.atomic.AtomicIntegerArray) CommonTestUtils.hash(utils.CommonTestUtils.hash) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) Files(java.nio.file.Files) CommonUtils.hardLinkTree(com.swirlds.common.CommonUtils.hardLinkTree) ContractValue(com.hedera.services.state.virtual.ContractValue) IOException(java.io.IOException) DigestType(com.swirlds.common.crypto.DigestType) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) AtomicLong(java.util.concurrent.atomic.AtomicLong) Stream(java.util.stream.Stream) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) IOException(java.io.IOException)

Example 4 with VirtualLeafRecord

use of com.swirlds.virtualmap.datasource.VirtualLeafRecord in project hedera-services by hashgraph.

the class VirtualDataSourceBench method setup.

@Setup(Level.Trial)
public void setup() throws IOException {
    random = new Random(1234);
    System.out.println("dataSourcePath = " + dataSourcePath);
    if (Files.exists(dataSourcePath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db.");
        deleteDirectoryAndContents(dataSourcePath);
    }
    if (Files.exists(dataSourceSnapshotPath)) {
        System.err.println("!!!!!!!!!!!!! Deleting old db snapshot.");
        deleteDirectoryAndContents(dataSourceSnapshotPath);
    }
    // create data source
    VirtualLeafRecordSerializer<ContractKey, ContractValue> virtualLeafRecordSerializer = new VirtualLeafRecordSerializer<>((short) 1, DigestType.SHA_384, (short) 1, DataFileCommon.VARIABLE_DATA_SIZE, new ContractKeySupplier(), (short) 1, ContractValue.SERIALIZED_SIZE, new ContractValueSupplier(), true);
    JasperDbBuilder<ContractKey, ContractValue> dbBuilder = new JasperDbBuilder<>();
    dbBuilder.virtualLeafRecordSerializer(virtualLeafRecordSerializer).virtualInternalRecordSerializer(new VirtualInternalRecordSerializer()).keySerializer(new ContractKeySerializer()).storageDir(dataSourcePath).maxNumOfKeys(500_000_000).preferDiskBasedIndexes(false).internalHashesRamToDiskThreshold(0).mergingEnabled(false);
    dataSource = dbBuilder.build("jdb", "4dsBench");
    // populate with initial data
    System.out.printf("Creating initial data set of %,d leaves\n", initialDataSize);
    progressPercentage = 0;
    final long firstLeafPath = initialDataSize;
    final long lastLeafPath = firstLeafPath + initialDataSize;
    var internalRecordStream = LongStream.range(0, firstLeafPath).mapToObj(path -> new VirtualInternalRecord(path, hash((int) path)));
    var leafRecordStream = LongStream.range(firstLeafPath, lastLeafPath + 1).mapToObj(path -> new VirtualLeafRecord<>(path, hash((int) path), new ContractKey(path, path), new ContractValue(path))).peek(leaf -> printProgress(leaf.getPath(), lastLeafPath));
    dataSource.saveRecords(firstLeafPath, lastLeafPath, internalRecordStream, leafRecordStream, Stream.empty());
    System.out.printf("Done creating initial data set of %,d leaves\n", initialDataSize);
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Measurement(org.openjdk.jmh.annotations.Measurement) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) CommonTestUtils.deleteDirectoryAndContents(utils.CommonTestUtils.deleteDirectoryAndContents) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) Scope(org.openjdk.jmh.annotations.Scope) Random(java.util.Random) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Warmup(org.openjdk.jmh.annotations.Warmup) OutputTimeUnit(org.openjdk.jmh.annotations.OutputTimeUnit) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) CommonTestUtils.hash(utils.CommonTestUtils.hash) Setup(org.openjdk.jmh.annotations.Setup) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) Mode(org.openjdk.jmh.annotations.Mode) Files(java.nio.file.Files) ContractValue(com.hedera.services.state.virtual.ContractValue) Param(org.openjdk.jmh.annotations.Param) IOException(java.io.IOException) State(org.openjdk.jmh.annotations.State) DigestType(com.swirlds.common.crypto.DigestType) Benchmark(org.openjdk.jmh.annotations.Benchmark) TimeUnit(java.util.concurrent.TimeUnit) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) Stream(java.util.stream.Stream) Level(org.openjdk.jmh.annotations.Level) Fork(org.openjdk.jmh.annotations.Fork) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) ContractValue(com.hedera.services.state.virtual.ContractValue) Random(java.util.Random) ContractKey(com.hedera.services.state.virtual.ContractKey) ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Setup(org.openjdk.jmh.annotations.Setup)

Example 5 with VirtualLeafRecord

use of com.swirlds.virtualmap.datasource.VirtualLeafRecord in project hedera-services by hashgraph.

the class VirtualDataSourceNewAPIBench method w3_add10kLeafValues.

/**
 * Updates the first 10k leaves with new random values
 */
@Benchmark
public void w3_add10kLeafValues() throws Exception {
    dataSource.saveRecords(numEntities, numEntities * 2, Stream.empty(), LongStream.range(nextPath, nextPath + 10_000).mapToObj(i -> new VirtualLeafRecord<>(i + numEntities, hash((int) i), new ContractKey(i, i), new ContractValue(i))), Stream.empty());
    nextPath += 10_000;
}
Also used : ContractValueSupplier(com.hedera.services.state.virtual.ContractValueSupplier) JasperDbBuilder(com.swirlds.jasperdb.JasperDbBuilder) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualInternalRecordSerializer(com.swirlds.jasperdb.VirtualInternalRecordSerializer) VirtualLeafRecordSerializer(com.swirlds.jasperdb.VirtualLeafRecordSerializer) DataFileCommon(com.swirlds.jasperdb.files.DataFileCommon) Random(java.util.Random) ContractKeySupplier(com.hedera.services.state.virtual.ContractKeySupplier) Hash(com.swirlds.common.crypto.Hash) org.openjdk.jmh.annotations(org.openjdk.jmh.annotations) ArrayList(java.util.ArrayList) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord) Path(java.nio.file.Path) VFCDataSourceLmdb(lmdb.VFCDataSourceLmdb) LongStream(java.util.stream.LongStream) VirtualDataSourceJasperDB(com.swirlds.jasperdb.VirtualDataSourceJasperDB) CommonTestUtils.printDirectorySize(utils.CommonTestUtils.printDirectorySize) Files(java.nio.file.Files) ContractValue(com.hedera.services.state.virtual.ContractValue) VirtualDataSource(com.swirlds.virtualmap.datasource.VirtualDataSource) IOException(java.io.IOException) DigestType(com.swirlds.common.crypto.DigestType) TimeUnit(java.util.concurrent.TimeUnit) VirtualInternalRecord(com.swirlds.virtualmap.datasource.VirtualInternalRecord) List(java.util.List) Stream(java.util.stream.Stream) ContractKeySerializer(com.hedera.services.state.virtual.ContractKeySerializer) LongArrayList(org.eclipse.collections.impl.list.mutable.primitive.LongArrayList) ContractValue(com.hedera.services.state.virtual.ContractValue) ContractKey(com.hedera.services.state.virtual.ContractKey) VirtualLeafRecord(com.swirlds.virtualmap.datasource.VirtualLeafRecord)

Aggregations

VirtualLeafRecord (com.swirlds.virtualmap.datasource.VirtualLeafRecord)8 ContractKey (com.hedera.services.state.virtual.ContractKey)7 ContractValue (com.hedera.services.state.virtual.ContractValue)7 ContractKeySupplier (com.hedera.services.state.virtual.ContractKeySupplier)6 ContractValueSupplier (com.hedera.services.state.virtual.ContractValueSupplier)6 DigestType (com.swirlds.common.crypto.DigestType)6 VirtualLeafRecordSerializer (com.swirlds.jasperdb.VirtualLeafRecordSerializer)6 DataFileCommon (com.swirlds.jasperdb.files.DataFileCommon)6 VirtualInternalRecord (com.swirlds.virtualmap.datasource.VirtualInternalRecord)6 IOException (java.io.IOException)6 Files (java.nio.file.Files)6 Path (java.nio.file.Path)6 LongStream (java.util.stream.LongStream)6 ContractKeySerializer (com.hedera.services.state.virtual.ContractKeySerializer)5 JasperDbBuilder (com.swirlds.jasperdb.JasperDbBuilder)5 VirtualDataSourceJasperDB (com.swirlds.jasperdb.VirtualDataSourceJasperDB)5 VirtualInternalRecordSerializer (com.swirlds.jasperdb.VirtualInternalRecordSerializer)5 Random (java.util.Random)5 Stream (java.util.stream.Stream)5 CommonTestUtils.deleteDirectoryAndContents (utils.CommonTestUtils.deleteDirectoryAndContents)4