Search in sources :

Example 1 with FinalizedUpdater

use of tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater in project teku by ConsenSys.

the class KvStoreDatabase method updateFinalizedDataPruneMode.

private void updateFinalizedDataPruneMode(Map<Bytes32, Bytes32> finalizedChildToParentMap, final Map<Bytes32, SignedBeaconBlock> finalizedBlocks) {
    final Optional<Bytes32> initialBlockRoot = hotDao.getAnchor().map(Checkpoint::getRoot);
    final BlockProvider blockProvider = BlockProvider.withKnownBlocks(roots -> SafeFuture.completedFuture(getHotBlocks(roots)), finalizedBlocks);
    final List<Bytes32> finalizedRoots = new ArrayList<>(finalizedChildToParentMap.keySet());
    int i = 0;
    while (i < finalizedRoots.size()) {
        try (final FinalizedUpdater updater = finalizedDao.finalizedUpdater()) {
            final int start = i;
            while (i < finalizedRoots.size() && (i - start) < TX_BATCH_SIZE) {
                final Bytes32 root = finalizedRoots.get(i);
                final Optional<SignedBeaconBlock> maybeBlock = blockProvider.getBlock(root).join();
                maybeBlock.ifPresent(updater::addFinalizedBlock);
                // If block is missing and doesn't match the initial anchor, throw
                if (maybeBlock.isEmpty() && initialBlockRoot.filter(r -> r.equals(root)).isEmpty()) {
                    throw new IllegalStateException("Missing finalized block");
                }
                i++;
            }
            updater.commit();
            if (i >= TX_BATCH_SIZE) {
                STATUS_LOG.recordedFinalizedBlocks(i, finalizedRoots.size());
            }
        }
    }
}
Also used : Checkpoint(tech.pegasys.teku.spec.datastructures.state.Checkpoint) FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) ArrayList(java.util.ArrayList) BlockProvider(tech.pegasys.teku.dataproviders.lookup.BlockProvider) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock) Bytes32(org.apache.tuweni.bytes.Bytes32) Checkpoint(tech.pegasys.teku.spec.datastructures.state.Checkpoint) AnchorPoint(tech.pegasys.teku.spec.datastructures.state.AnchorPoint)

Example 2 with FinalizedUpdater

use of tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater in project teku by ConsenSys.

the class KvStoreDatabase method storeInitialAnchor.

@Override
public void storeInitialAnchor(final AnchorPoint anchor) {
    try (final HotUpdater hotUpdater = hotDao.hotUpdater();
        final FinalizedUpdater finalizedUpdater = finalizedDao.finalizedUpdater()) {
        // We should only have a single block / state / checkpoint at anchorpoint initialization
        final Checkpoint anchorCheckpoint = anchor.getCheckpoint();
        final Bytes32 anchorRoot = anchorCheckpoint.getRoot();
        final BeaconState anchorState = anchor.getState();
        final Optional<SignedBeaconBlock> anchorBlock = anchor.getSignedBeaconBlock();
        hotUpdater.setAnchor(anchor.getCheckpoint());
        hotUpdater.setGenesisTime(anchorState.getGenesis_time());
        hotUpdater.setJustifiedCheckpoint(anchorCheckpoint);
        hotUpdater.setBestJustifiedCheckpoint(anchorCheckpoint);
        hotUpdater.setFinalizedCheckpoint(anchorCheckpoint);
        hotUpdater.setLatestFinalizedState(anchorState);
        // We need to store the anchor block in both hot and cold storage so that on restart
        // we're guaranteed to have at least one block / state to load into RecentChainData.
        anchorBlock.ifPresent(block -> {
            // Save to hot storage
            hotUpdater.addHotBlock(new BlockAndCheckpointEpochs(block, new CheckpointEpochs(anchorState.getCurrent_justified_checkpoint().getEpoch(), anchorState.getFinalized_checkpoint().getEpoch())));
            // Save to cold storage
            finalizedUpdater.addFinalizedBlock(block);
        });
        putFinalizedState(finalizedUpdater, anchorRoot, anchorState);
        finalizedUpdater.commit();
        hotUpdater.commit();
    }
}
Also used : BlockAndCheckpointEpochs(tech.pegasys.teku.spec.datastructures.blocks.BlockAndCheckpointEpochs) Checkpoint(tech.pegasys.teku.spec.datastructures.state.Checkpoint) FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) BlockAndCheckpointEpochs(tech.pegasys.teku.spec.datastructures.blocks.BlockAndCheckpointEpochs) CheckpointEpochs(tech.pegasys.teku.spec.datastructures.blocks.CheckpointEpochs) HotUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreHotDao.HotUpdater) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock) Bytes32(org.apache.tuweni.bytes.Bytes32) BeaconState(tech.pegasys.teku.spec.datastructures.state.beaconstate.BeaconState)

Example 3 with FinalizedUpdater

use of tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater in project teku by ConsenSys.

the class KvStoreDatabase method storeNonCanonicalBlocks.

private void storeNonCanonicalBlocks(final Set<SignedBeaconBlock> nonCanonicalBlocks) {
    int i = 0;
    final Iterator<SignedBeaconBlock> it = nonCanonicalBlocks.iterator();
    while (it.hasNext()) {
        final Map<UInt64, Set<Bytes32>> nonCanonicalRootsBySlotBuffer = new HashMap<>();
        final int start = i;
        try (final FinalizedUpdater updater = finalizedDao.finalizedUpdater()) {
            while (it.hasNext() && (i - start) < TX_BATCH_SIZE) {
                final SignedBeaconBlock block = it.next();
                LOG.debug("Non canonical block {}:{}", block.getRoot().toHexString(), block.getSlot());
                updater.addNonCanonicalBlock(block);
                nonCanonicalRootsBySlotBuffer.computeIfAbsent(block.getSlot(), __ -> new HashSet<>()).add(block.getRoot());
                i++;
            }
            nonCanonicalRootsBySlotBuffer.forEach(updater::addNonCanonicalRootAtSlot);
            updater.commit();
        }
    }
}
Also used : SchemaFinalizedSnapshotState(tech.pegasys.teku.storage.server.kvstore.schema.SchemaFinalizedSnapshotState) BlockProvider(tech.pegasys.teku.dataproviders.lookup.BlockProvider) MinGenesisTimeBlockEvent(tech.pegasys.teku.ethereum.pow.api.MinGenesisTimeBlockEvent) SlotAndExecutionPayload(tech.pegasys.teku.spec.datastructures.execution.SlotAndExecutionPayload) StateAndBlockSummary(tech.pegasys.teku.spec.datastructures.blocks.StateAndBlockSummary) HotUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreHotDao.HotUpdater) V4SchemaHot(tech.pegasys.teku.storage.server.kvstore.schema.V4SchemaHot) FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) Map(java.util.Map) ExecutionPayload(tech.pegasys.teku.spec.datastructures.execution.ExecutionPayload) SchemaFinalizedTreeState(tech.pegasys.teku.storage.server.kvstore.schema.SchemaFinalizedTreeState) KvStoreFinalizedDao(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao) Bytes32(org.apache.tuweni.bytes.Bytes32) Collection(java.util.Collection) Set(java.util.Set) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) StateStorageMode(tech.pegasys.teku.storage.server.StateStorageMode) V4FinalizedStateTreeStorageLogic(tech.pegasys.teku.storage.server.kvstore.dataaccess.V4FinalizedStateTreeStorageLogic) List(java.util.List) Stream(java.util.stream.Stream) Logger(org.apache.logging.log4j.Logger) BlockAndCheckpointEpochs(tech.pegasys.teku.spec.datastructures.blocks.BlockAndCheckpointEpochs) KvStoreHotDao(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreHotDao) V4FinalizedStateStorageLogic(tech.pegasys.teku.storage.server.kvstore.dataaccess.V4FinalizedStateStorageLogic) Optional(java.util.Optional) WeakSubjectivityUpdate(tech.pegasys.teku.storage.events.WeakSubjectivityUpdate) V4FinalizedStateSnapshotStorageLogic(tech.pegasys.teku.storage.server.kvstore.dataaccess.V4FinalizedStateSnapshotStorageLogic) MetricsSystem(org.hyperledger.besu.plugin.services.MetricsSystem) Checkpoint(tech.pegasys.teku.spec.datastructures.state.Checkpoint) BeaconBlockSummary(tech.pegasys.teku.spec.datastructures.blocks.BeaconBlockSummary) CheckpointEpochs(tech.pegasys.teku.spec.datastructures.blocks.CheckpointEpochs) KvStoreEth1Dao(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreEth1Dao) VoteTracker(tech.pegasys.teku.spec.datastructures.forkchoice.VoteTracker) SafeFuture(tech.pegasys.teku.infrastructure.async.SafeFuture) UpdateResult(tech.pegasys.teku.storage.api.UpdateResult) HashMap(java.util.HashMap) StorageUpdate(tech.pegasys.teku.storage.events.StorageUpdate) Function(java.util.function.Function) Supplier(java.util.function.Supplier) BeaconBlockHeader(tech.pegasys.teku.spec.datastructures.blocks.BeaconBlockHeader) StoreBuilder(tech.pegasys.teku.storage.store.StoreBuilder) MustBeClosed(com.google.errorprone.annotations.MustBeClosed) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) V4HotKvStoreDao(tech.pegasys.teku.storage.server.kvstore.dataaccess.V4HotKvStoreDao) WeakSubjectivityState(tech.pegasys.teku.storage.events.WeakSubjectivityState) V4FinalizedKvStoreDao(tech.pegasys.teku.storage.server.kvstore.dataaccess.V4FinalizedKvStoreDao) UInt64(tech.pegasys.teku.infrastructure.unsigned.UInt64) DepositsFromBlockEvent(tech.pegasys.teku.ethereum.pow.api.DepositsFromBlockEvent) Spec(tech.pegasys.teku.spec.Spec) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock) SlotAndBlockRoot(tech.pegasys.teku.spec.datastructures.blocks.SlotAndBlockRoot) SchemaFinalized(tech.pegasys.teku.storage.server.kvstore.schema.SchemaFinalized) SchemaHot(tech.pegasys.teku.storage.server.kvstore.schema.SchemaHot) StateRootRecorder(tech.pegasys.teku.storage.server.state.StateRootRecorder) Iterator(java.util.Iterator) Preconditions.checkNotNull(com.google.common.base.Preconditions.checkNotNull) StoredBlockMetadata(tech.pegasys.teku.ethereum.forkchoice.StoredBlockMetadata) Consumer(java.util.function.Consumer) Database(tech.pegasys.teku.storage.server.Database) HashTree(tech.pegasys.teku.spec.datastructures.hashtree.HashTree) AnchorPoint(tech.pegasys.teku.spec.datastructures.state.AnchorPoint) STATUS_LOG(tech.pegasys.teku.infrastructure.logging.StatusLogger.STATUS_LOG) Eth1Updater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreEth1Dao.Eth1Updater) V4SchemaFinalized(tech.pegasys.teku.storage.server.kvstore.schema.V4SchemaFinalized) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Comparator(java.util.Comparator) LogManager(org.apache.logging.log4j.LogManager) BeaconState(tech.pegasys.teku.spec.datastructures.state.beaconstate.BeaconState) Set(java.util.Set) HashSet(java.util.HashSet) FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) HashMap(java.util.HashMap) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock) UInt64(tech.pegasys.teku.infrastructure.unsigned.UInt64) Checkpoint(tech.pegasys.teku.spec.datastructures.state.Checkpoint) AnchorPoint(tech.pegasys.teku.spec.datastructures.state.AnchorPoint) HashSet(java.util.HashSet)

Example 4 with FinalizedUpdater

use of tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater in project teku by ConsenSys.

the class KvStoreDatabase method storeFinalizedBlocks.

@Override
public void storeFinalizedBlocks(final Collection<SignedBeaconBlock> blocks) {
    if (blocks.isEmpty()) {
        return;
    }
    // Sort blocks and verify that they are contiguous with the oldestBlock
    final List<SignedBeaconBlock> sorted = blocks.stream().sorted(Comparator.comparing(SignedBeaconBlock::getSlot).reversed()).collect(Collectors.toList());
    // The new block should be just prior to our earliest block if available, and otherwise should
    // match our latest finalized block
    Bytes32 expectedRoot = getEarliestAvailableBlock().map(SignedBeaconBlock::getParentRoot).orElseGet(() -> this.getLatestFinalizedBlockSummary().getRoot());
    for (SignedBeaconBlock block : sorted) {
        if (!block.getRoot().equals(expectedRoot)) {
            throw new IllegalArgumentException("Blocks must be contiguous with the earliest known block.");
        }
        expectedRoot = block.getParentRoot();
    }
    try (final FinalizedUpdater updater = finalizedDao.finalizedUpdater()) {
        blocks.forEach(updater::addFinalizedBlock);
        updater.commit();
    }
}
Also used : FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock) Bytes32(org.apache.tuweni.bytes.Bytes32)

Example 5 with FinalizedUpdater

use of tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater in project teku by ConsenSys.

the class KvStoreDatabase method updateFinalizedOptimisticTransitionBlock.

private Optional<SlotAndExecutionPayload> updateFinalizedOptimisticTransitionBlock(final boolean isFinalizedOptimisticBlockRootSet, final Optional<Bytes32> finalizedOptimisticTransitionBlockRoot) {
    if (isFinalizedOptimisticBlockRootSet) {
        final Optional<SignedBeaconBlock> transitionBlock = finalizedOptimisticTransitionBlockRoot.flatMap(this::getHotBlock);
        try (final FinalizedUpdater updater = finalizedDao.finalizedUpdater()) {
            updater.setOptimisticTransitionBlockSlot(transitionBlock.map(SignedBeaconBlock::getSlot));
            updater.commit();
        }
        return transitionBlock.flatMap(SlotAndExecutionPayload::fromBlock);
    } else {
        return Optional.empty();
    }
}
Also used : SlotAndExecutionPayload(tech.pegasys.teku.spec.datastructures.execution.SlotAndExecutionPayload) FinalizedUpdater(tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater) SignedBeaconBlock(tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock)

Aggregations

SignedBeaconBlock (tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock)6 FinalizedUpdater (tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreFinalizedDao.FinalizedUpdater)6 Bytes32 (org.apache.tuweni.bytes.Bytes32)5 Checkpoint (tech.pegasys.teku.spec.datastructures.state.Checkpoint)4 BlockProvider (tech.pegasys.teku.dataproviders.lookup.BlockProvider)3 AnchorPoint (tech.pegasys.teku.spec.datastructures.state.AnchorPoint)3 ArrayList (java.util.ArrayList)2 UInt64 (tech.pegasys.teku.infrastructure.unsigned.UInt64)2 BlockAndCheckpointEpochs (tech.pegasys.teku.spec.datastructures.blocks.BlockAndCheckpointEpochs)2 CheckpointEpochs (tech.pegasys.teku.spec.datastructures.blocks.CheckpointEpochs)2 SlotAndExecutionPayload (tech.pegasys.teku.spec.datastructures.execution.SlotAndExecutionPayload)2 BeaconState (tech.pegasys.teku.spec.datastructures.state.beaconstate.BeaconState)2 HotUpdater (tech.pegasys.teku.storage.server.kvstore.dataaccess.KvStoreHotDao.HotUpdater)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Preconditions.checkNotNull (com.google.common.base.Preconditions.checkNotNull)1 MustBeClosed (com.google.errorprone.annotations.MustBeClosed)1 Instant (java.time.Instant)1 Collection (java.util.Collection)1 Comparator (java.util.Comparator)1 HashMap (java.util.HashMap)1