Search in sources :

Example 1 with BonsaiWorldStateKeyValueStorage

use of org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage in project besu by hyperledger.

the class FastDownloaderFactory method create.

public static Optional<FastSyncDownloader> create(final SynchronizerConfiguration syncConfig, final Path dataDirectory, final ProtocolSchedule protocolSchedule, final ProtocolContext protocolContext, final MetricsSystem metricsSystem, final EthContext ethContext, final WorldStateStorage worldStateStorage, final SyncState syncState, final Clock clock) {
    final Path fastSyncDataDirectory = dataDirectory.resolve(FAST_SYNC_FOLDER);
    final FastSyncStateStorage fastSyncStateStorage = new FastSyncStateStorage(fastSyncDataDirectory);
    if (syncConfig.getSyncMode() != SyncMode.FAST) {
        if (fastSyncStateStorage.isFastSyncInProgress()) {
            throw new IllegalStateException("Unable to change the sync mode when fast sync is incomplete, please restart with fast sync mode");
        } else {
            return Optional.empty();
        }
    }
    ensureDirectoryExists(fastSyncDataDirectory.toFile());
    final FastSyncState fastSyncState = fastSyncStateStorage.loadState(ScheduleBasedBlockHeaderFunctions.create(protocolSchedule));
    if (fastSyncState.getPivotBlockHeader().isEmpty() && protocolContext.getBlockchain().getChainHeadBlockNumber() != BlockHeader.GENESIS_BLOCK_NUMBER) {
        LOG.info("Fast sync was requested, but cannot be enabled because the local blockchain is not empty.");
        return Optional.empty();
    }
    if (worldStateStorage instanceof BonsaiWorldStateKeyValueStorage) {
        worldStateStorage.clear();
    } else {
        final Path queueDataDir = fastSyncDataDirectory.resolve("statequeue");
        if (queueDataDir.toFile().exists()) {
            LOG.warn("Fast sync is picking up after old fast sync version. Pruning the world state and starting from scratch.");
            clearOldFastSyncWorldStateData(worldStateStorage, queueDataDir);
        }
    }
    final InMemoryTasksPriorityQueues<NodeDataRequest> taskCollection = createWorldStateDownloaderTaskCollection(metricsSystem, syncConfig.getWorldStateTaskCacheSize());
    final WorldStateDownloader worldStateDownloader = new FastWorldStateDownloader(ethContext, worldStateStorage, taskCollection, syncConfig.getWorldStateHashCountPerRequest(), syncConfig.getWorldStateRequestParallelism(), syncConfig.getWorldStateMaxRequestsWithoutProgress(), syncConfig.getWorldStateMinMillisBeforeStalling(), clock, metricsSystem);
    final FastSyncDownloader fastSyncDownloader = new FastSyncDownloader(new FastSyncActions(syncConfig, protocolSchedule, protocolContext, ethContext, syncState, metricsSystem), worldStateStorage, worldStateDownloader, fastSyncStateStorage, taskCollection, fastSyncDataDirectory, fastSyncState);
    syncState.setWorldStateDownloadStatus(worldStateDownloader);
    return Optional.of(fastSyncDownloader);
}
Also used : Path(java.nio.file.Path) FastWorldStateDownloader(org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate.FastWorldStateDownloader) BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) NodeDataRequest(org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate.NodeDataRequest) FastWorldStateDownloader(org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate.FastWorldStateDownloader) WorldStateDownloader(org.hyperledger.besu.ethereum.eth.sync.worldstate.WorldStateDownloader)

Example 2 with BonsaiWorldStateKeyValueStorage

use of org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage in project besu by hyperledger.

the class AccountTrieNodeDataRequest method getRequestsFromTrieNodeValue.

@Override
protected Stream<SnapDataRequest> getRequestsFromTrieNodeValue(final WorldStateStorage worldStateStorage, final Bytes location, final Bytes path, final Bytes value) {
    final Stream.Builder<SnapDataRequest> builder = Stream.builder();
    final StateTrieAccountValue accountValue = StateTrieAccountValue.readFrom(RLP.input(value));
    // Retrieve account hash
    final Hash accountHash = Hash.wrap(Bytes32.wrap(CompactEncoding.pathToBytes(Bytes.concatenate(getLocation(), path))));
    if (worldStateStorage instanceof BonsaiWorldStateKeyValueStorage) {
        ((BonsaiWorldStateKeyValueStorage.Updater) worldStateStorage.updater()).putAccountInfoState(accountHash, value).commit();
    }
    // Add code, if appropriate
    if (!accountValue.getCodeHash().equals(Hash.EMPTY)) {
        builder.add(createBytecodeRequest(accountHash, getRootHash(), accountValue.getCodeHash()));
    }
    // Add storage, if appropriate
    if (!accountValue.getStorageRoot().equals(MerklePatriciaTrie.EMPTY_TRIE_NODE_HASH)) {
        // If we detect an account storage we fill it with snapsync before completing with a heal
        final SnapDataRequest storageTrieRequest = createStorageTrieNodeDataRequest(accountValue.getStorageRoot(), accountHash, getRootHash(), Bytes.EMPTY);
        builder.add(storageTrieRequest);
    }
    return builder.build();
}
Also used : BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) StateTrieAccountValue(org.hyperledger.besu.ethereum.worldstate.StateTrieAccountValue) Stream(java.util.stream.Stream) Hash(org.hyperledger.besu.datatypes.Hash)

Example 3 with BonsaiWorldStateKeyValueStorage

use of org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage in project besu by hyperledger.

the class AccountTrieNodeDataRequest method getRequestsFromTrieNodeValue.

@Override
protected Stream<NodeDataRequest> getRequestsFromTrieNodeValue(final WorldStateStorage worldStateStorage, final Optional<Bytes> location, final Bytes path, final Bytes value) {
    final Stream.Builder<NodeDataRequest> builder = Stream.builder();
    final StateTrieAccountValue accountValue = StateTrieAccountValue.readFrom(RLP.input(value));
    // Add code, if appropriate
    final Optional<Hash> accountHash = Optional.of(Hash.wrap(Bytes32.wrap(CompactEncoding.pathToBytes(Bytes.concatenate(getLocation().orElse(Bytes.EMPTY), path)))));
    if (worldStateStorage instanceof BonsaiWorldStateKeyValueStorage) {
        ((BonsaiWorldStateKeyValueStorage.Updater) worldStateStorage.updater()).putAccountInfoState(accountHash.get(), value).commit();
    }
    if (!accountValue.getCodeHash().equals(Hash.EMPTY)) {
        builder.add(createCodeRequest(accountValue.getCodeHash(), accountHash));
    }
    // Add storage, if appropriate
    if (!accountValue.getStorageRoot().equals(MerklePatriciaTrie.EMPTY_TRIE_NODE_HASH)) {
        // If storage is non-empty queue download
        final NodeDataRequest storageNode = createStorageDataRequest(accountValue.getStorageRoot(), accountHash, Optional.empty());
        builder.add(storageNode);
    }
    return builder.build();
}
Also used : BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) StateTrieAccountValue(org.hyperledger.besu.ethereum.worldstate.StateTrieAccountValue) Stream(java.util.stream.Stream) Hash(org.hyperledger.besu.datatypes.Hash)

Example 4 with BonsaiWorldStateKeyValueStorage

use of org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage in project besu by hyperledger.

the class FastDownloaderFactory method create.

public static Optional<FastSyncDownloader<?>> create(final PivotBlockSelector pivotBlockSelector, final SynchronizerConfiguration syncConfig, final Path dataDirectory, final ProtocolSchedule protocolSchedule, final ProtocolContext protocolContext, final MetricsSystem metricsSystem, final EthContext ethContext, final WorldStateStorage worldStateStorage, final SyncState syncState, final Clock clock) {
    final Path fastSyncDataDirectory = dataDirectory.resolve(FAST_SYNC_FOLDER);
    final FastSyncStateStorage fastSyncStateStorage = new FastSyncStateStorage(fastSyncDataDirectory);
    if (SyncMode.isFullSync(syncConfig.getSyncMode())) {
        if (fastSyncStateStorage.isFastSyncInProgress()) {
            throw new IllegalStateException("Unable to change the sync mode when fast sync is incomplete, please restart with fast sync mode");
        } else {
            return Optional.empty();
        }
    }
    ensureDirectoryExists(fastSyncDataDirectory.toFile());
    final FastSyncState fastSyncState = fastSyncStateStorage.loadState(ScheduleBasedBlockHeaderFunctions.create(protocolSchedule));
    if (fastSyncState.getPivotBlockHeader().isEmpty() && protocolContext.getBlockchain().getChainHeadBlockNumber() != BlockHeader.GENESIS_BLOCK_NUMBER) {
        LOG.info("Fast sync was requested, but cannot be enabled because the local blockchain is not empty.");
        return Optional.empty();
    }
    if (worldStateStorage instanceof BonsaiWorldStateKeyValueStorage) {
        worldStateStorage.clearFlatDatabase();
    } else {
        final Path queueDataDir = fastSyncDataDirectory.resolve("statequeue");
        if (queueDataDir.toFile().exists()) {
            LOG.warn("Fast sync is picking up after old fast sync version. Pruning the world state and starting from scratch.");
            clearOldFastSyncWorldStateData(worldStateStorage, queueDataDir);
        }
    }
    final InMemoryTasksPriorityQueues<NodeDataRequest> taskCollection = createWorldStateDownloaderTaskCollection(metricsSystem, syncConfig.getWorldStateTaskCacheSize());
    final WorldStateDownloader worldStateDownloader = new FastWorldStateDownloader(ethContext, worldStateStorage, taskCollection, syncConfig.getWorldStateHashCountPerRequest(), syncConfig.getWorldStateRequestParallelism(), syncConfig.getWorldStateMaxRequestsWithoutProgress(), syncConfig.getWorldStateMinMillisBeforeStalling(), clock, metricsSystem);
    final FastSyncDownloader<NodeDataRequest> fastSyncDownloader = new FastSyncDownloader<>(new FastSyncActions(syncConfig, worldStateStorage, protocolSchedule, protocolContext, ethContext, syncState, pivotBlockSelector, metricsSystem), worldStateStorage, worldStateDownloader, fastSyncStateStorage, taskCollection, fastSyncDataDirectory, fastSyncState);
    syncState.setWorldStateDownloadStatus(worldStateDownloader);
    return Optional.of(fastSyncDownloader);
}
Also used : Path(java.nio.file.Path) FastSyncState(org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState) FastSyncDownloader(org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncDownloader) FastSyncActions(org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncActions) BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) FastSyncStateStorage(org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage) WorldStateDownloader(org.hyperledger.besu.ethereum.eth.sync.worldstate.WorldStateDownloader)

Example 5 with BonsaiWorldStateKeyValueStorage

use of org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage in project besu by hyperledger.

the class FastWorldDownloadStateTest method setUp.

@Before
public void setUp() {
    if (storageFormat == DataStorageFormat.BONSAI) {
        worldStateStorage = new BonsaiWorldStateKeyValueStorage(new InMemoryKeyValueStorageProvider());
    } else {
        worldStateStorage = new WorldStateKeyValueStorage(new InMemoryKeyValueStorage());
    }
    downloadState = new FastWorldDownloadState(worldStateStorage, pendingRequests, MAX_REQUESTS_WITHOUT_PROGRESS, MIN_MILLIS_BEFORE_STALLING, clock);
    assertThat(downloadState.isDownloading()).isTrue();
    downloadState.setRootNodeData(ROOT_NODE_DATA);
    future = downloadState.getDownloadFuture();
}
Also used : WorldStateKeyValueStorage(org.hyperledger.besu.ethereum.storage.keyvalue.WorldStateKeyValueStorage) BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) InMemoryKeyValueStorage(org.hyperledger.besu.services.kvstore.InMemoryKeyValueStorage) BonsaiWorldStateKeyValueStorage(org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage) InMemoryKeyValueStorageProvider(org.hyperledger.besu.ethereum.core.InMemoryKeyValueStorageProvider) Before(org.junit.Before)

Aggregations

BonsaiWorldStateKeyValueStorage (org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage)6 Path (java.nio.file.Path)2 Stream (java.util.stream.Stream)2 Hash (org.hyperledger.besu.datatypes.Hash)2 InMemoryKeyValueStorageProvider (org.hyperledger.besu.ethereum.core.InMemoryKeyValueStorageProvider)2 WorldStateDownloader (org.hyperledger.besu.ethereum.eth.sync.worldstate.WorldStateDownloader)2 WorldStateKeyValueStorage (org.hyperledger.besu.ethereum.storage.keyvalue.WorldStateKeyValueStorage)2 StateTrieAccountValue (org.hyperledger.besu.ethereum.worldstate.StateTrieAccountValue)2 InMemoryKeyValueStorage (org.hyperledger.besu.services.kvstore.InMemoryKeyValueStorage)2 Before (org.junit.Before)2 BlockHeader (org.hyperledger.besu.ethereum.core.BlockHeader)1 FastSyncActions (org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncActions)1 FastSyncDownloader (org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncDownloader)1 FastSyncState (org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState)1 FastSyncStateStorage (org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage)1 FastWorldStateDownloader (org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate.FastWorldStateDownloader)1 NodeDataRequest (org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate.NodeDataRequest)1 NoOpMetricsSystem (org.hyperledger.besu.metrics.noop.NoOpMetricsSystem)1