Search in sources :

Example 1 with Visitor

use of org.neo4j.helpers.collection.Visitor in project neo4j by neo4j.

the class StoreCopyResponsePacker method packTransactionStreamResponse.

@Override
public <T> Response<T> packTransactionStreamResponse(RequestContext context, T response) {
    final long toStartFrom = mandatoryStartTransactionId;
    final long toEndAt = transactionIdStore.getLastCommittedTransactionId();
    TransactionStream transactions = visitor -> {
        if (toStartFrom > BASE_TX_ID && toStartFrom <= toEndAt) {
            monitor.startStreamingTransactions(toStartFrom);
            extractTransactions(toStartFrom, filterVisitor(visitor, toEndAt));
            monitor.finishStreamingTransactions(toEndAt);
        }
    };
    return new TransactionStreamResponse<>(response, storeId.get(), transactions, ResourceReleaser.NO_OP);
}
Also used : Response(org.neo4j.com.Response) StoreId(org.neo4j.kernel.impl.store.StoreId) TransactionStream(org.neo4j.com.TransactionStream) TransactionIdStore(org.neo4j.kernel.impl.transaction.log.TransactionIdStore) LogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.LogicalTransactionStore) NoSuchTransactionException(org.neo4j.kernel.impl.transaction.log.NoSuchTransactionException) Supplier(java.util.function.Supplier) RequestContext(org.neo4j.com.RequestContext) TransactionStreamResponse(org.neo4j.com.TransactionStreamResponse) ResponsePacker(org.neo4j.com.storecopy.ResponsePacker) CommittedTransactionRepresentation(org.neo4j.kernel.impl.transaction.CommittedTransactionRepresentation) LogFileInformation(org.neo4j.kernel.impl.transaction.log.LogFileInformation) Monitor(org.neo4j.com.storecopy.StoreCopyServer.Monitor) StoreCopyServer(org.neo4j.com.storecopy.StoreCopyServer) ResourceReleaser(org.neo4j.com.ResourceReleaser) Visitor(org.neo4j.helpers.collection.Visitor) BASE_TX_ID(org.neo4j.kernel.impl.transaction.log.TransactionIdStore.BASE_TX_ID) TransactionStreamResponse(org.neo4j.com.TransactionStreamResponse) TransactionStream(org.neo4j.com.TransactionStream)

Example 2 with Visitor

use of org.neo4j.helpers.collection.Visitor in project neo4j by neo4j.

the class StoreCopyClient method writeTransactionsToActiveLogFile.

private void writeTransactionsToActiveLogFile(File tempStoreDir, Response<?> response) throws Exception {
    LifeSupport life = new LifeSupport();
    try {
        // Start the log and appender
        PhysicalLogFiles logFiles = new PhysicalLogFiles(tempStoreDir, fs);
        LogHeaderCache logHeaderCache = new LogHeaderCache(10);
        ReadOnlyLogVersionRepository logVersionRepository = new ReadOnlyLogVersionRepository(pageCache, tempStoreDir);
        ReadOnlyTransactionIdStore readOnlyTransactionIdStore = new ReadOnlyTransactionIdStore(pageCache, tempStoreDir);
        LogFile logFile = life.add(new PhysicalLogFile(fs, logFiles, Long.MAX_VALUE, /*don't rotate*/
        readOnlyTransactionIdStore::getLastCommittedTransactionId, logVersionRepository, new Monitors().newMonitor(PhysicalLogFile.Monitor.class), logHeaderCache));
        life.start();
        // Just write all transactions to the active log version. Remember that this is after a store copy
        // where there are no logs, and the transaction stream we're about to write will probably contain
        // transactions that goes some time back, before the last committed transaction id. So we cannot
        // use a TransactionAppender, since it has checks for which transactions one can append.
        FlushableChannel channel = logFile.getWriter();
        final TransactionLogWriter writer = new TransactionLogWriter(new LogEntryWriter(channel));
        final AtomicLong firstTxId = new AtomicLong(BASE_TX_ID);
        response.accept(new Response.Handler() {

            @Override
            public void obligation(long txId) throws IOException {
                throw new UnsupportedOperationException("Shouldn't be called");
            }

            @Override
            public Visitor<CommittedTransactionRepresentation, Exception> transactions() {
                return transaction -> {
                    long txId = transaction.getCommitEntry().getTxId();
                    if (firstTxId.compareAndSet(BASE_TX_ID, txId)) {
                        monitor.startReceivingTransactions(txId);
                    }
                    writer.append(transaction.getTransactionRepresentation(), txId);
                    return false;
                };
            }
        });
        long endTxId = firstTxId.get();
        if (endTxId != BASE_TX_ID) {
            monitor.finishReceivingTransactions(endTxId);
        }
        long currentLogVersion = logVersionRepository.getCurrentLogVersion();
        writer.checkPoint(new LogPosition(currentLogVersion, LOG_HEADER_SIZE));
        // And since we write this manually we need to set the correct transaction id in the
        // header of the log that we just wrote.
        File currentLogFile = logFiles.getLogFileForVersion(currentLogVersion);
        writeLogHeader(fs, currentLogFile, currentLogVersion, max(BASE_TX_ID, endTxId - 1));
        if (!forensics) {
            // since we just create new log and put checkpoint into it with offset equals to
            // LOG_HEADER_SIZE we need to update last transaction offset to be equal to this newly defined max
            // offset otherwise next checkpoint that use last transaction offset will be created for non
            // existing offset that is in most of the cases bigger than new log size.
            // Recovery will treat that as last checkpoint and will not try to recover store till new
            // last closed transaction offset will not overcome old one. Till that happens it will be
            // impossible for recovery process to restore the store
            File neoStore = new File(tempStoreDir, MetaDataStore.DEFAULT_NAME);
            MetaDataStore.setRecord(pageCache, neoStore, MetaDataStore.Position.LAST_CLOSED_TRANSACTION_LOG_BYTE_OFFSET, LOG_HEADER_SIZE);
        }
    } finally {
        life.shutdown();
    }
}
Also used : Visitor(org.neo4j.helpers.collection.Visitor) IOException(java.io.IOException) PhysicalLogFiles(org.neo4j.kernel.impl.transaction.log.PhysicalLogFiles) LogFile(org.neo4j.kernel.impl.transaction.log.LogFile) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) Response(org.neo4j.com.Response) AtomicLong(java.util.concurrent.atomic.AtomicLong) ReadOnlyTransactionIdStore(org.neo4j.kernel.impl.transaction.log.ReadOnlyTransactionIdStore) FlushableChannel(org.neo4j.kernel.impl.transaction.log.FlushableChannel) Monitors(org.neo4j.kernel.monitoring.Monitors) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) TransactionLogWriter(org.neo4j.kernel.impl.transaction.log.TransactionLogWriter) LogEntryWriter(org.neo4j.kernel.impl.transaction.log.entry.LogEntryWriter) ReadOnlyLogVersionRepository(org.neo4j.kernel.impl.transaction.log.ReadOnlyLogVersionRepository) LogHeaderCache(org.neo4j.kernel.impl.transaction.log.LogHeaderCache) LogFile(org.neo4j.kernel.impl.transaction.log.LogFile) File(java.io.File) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) LogPosition(org.neo4j.kernel.impl.transaction.log.LogPosition)

Example 3 with Visitor

use of org.neo4j.helpers.collection.Visitor in project neo4j by neo4j.

the class BatchInserterImpl method repopulateAllIndexes.

private void repopulateAllIndexes() throws IOException, IndexEntryConflictException {
    if (!labelsTouched) {
        return;
    }
    final IndexRule[] rules = getIndexesNeedingPopulation();
    final IndexPopulator[] populators = new IndexPopulator[rules.length];
    // the store is uncontended at this point, so creating a local LockService is safe.
    final NewIndexDescriptor[] descriptors = new NewIndexDescriptor[rules.length];
    for (int i = 0; i < rules.length; i++) {
        IndexRule rule = rules[i];
        descriptors[i] = rule.getIndexDescriptor();
        populators[i] = schemaIndexProviders.apply(rule.getProviderDescriptor()).getPopulator(rule.getId(), descriptors[i], new IndexSamplingConfig(config));
        populators[i].create();
    }
    Visitor<NodeUpdates, IOException> propertyUpdateVisitor = updates -> {
        for (int i = 0; i < descriptors.length; i++) {
            Optional<IndexEntryUpdate> update = updates.forIndex(descriptors[i].schema());
            if (update.isPresent()) {
                try {
                    populators[i].add(Collections.singletonList(update.get()));
                } catch (IndexEntryConflictException conflict) {
                    throw conflict.notAllowed(descriptors[i]);
                }
            }
        }
        return true;
    };
    List<NewIndexDescriptor> descriptorList = Arrays.asList(descriptors);
    int[] labelIds = descriptorList.stream().mapToInt(index -> index.schema().getLabelId()).toArray();
    int[] propertyKeyIds = descriptorList.stream().flatMapToInt(d -> Arrays.stream(d.schema().getPropertyIds())).toArray();
    InitialNodeLabelCreationVisitor labelUpdateVisitor = new InitialNodeLabelCreationVisitor();
    StoreScan<IOException> storeScan = indexStoreView.visitNodes(labelIds, (propertyKeyId) -> PrimitiveIntCollections.contains(propertyKeyIds, propertyKeyId), propertyUpdateVisitor, labelUpdateVisitor, true);
    storeScan.run();
    for (IndexPopulator populator : populators) {
        populator.verifyDeferredConstraints(indexStoreView);
        populator.close(true);
    }
    labelUpdateVisitor.close();
}
Also used : InternalIndexState(org.neo4j.kernel.api.index.InternalIndexState) Arrays(java.util.Arrays) SimpleKernelContext(org.neo4j.kernel.impl.spi.SimpleKernelContext) PropertyKeyTokenStore(org.neo4j.kernel.impl.store.PropertyKeyTokenStore) BatchRelationship(org.neo4j.unsafe.batchinsert.BatchRelationship) Iterators(org.neo4j.helpers.collection.Iterators) IndexPopulator(org.neo4j.kernel.api.index.IndexPopulator) RelationshipGroupGetter(org.neo4j.kernel.impl.transaction.state.RelationshipGroupGetter) PrimitiveIntCollections(org.neo4j.collection.primitive.PrimitiveIntCollections) RelationshipPropertyExistenceConstraintDefinition(org.neo4j.kernel.impl.coreapi.schema.RelationshipPropertyExistenceConstraintDefinition) NodeLabelsField.parseLabelsField(org.neo4j.kernel.impl.store.NodeLabelsField.parseLabelsField) NoOpClient(org.neo4j.kernel.impl.locking.NoOpClient) SchemaIndexProvider(org.neo4j.kernel.api.index.SchemaIndexProvider) CountsComputer(org.neo4j.kernel.impl.store.CountsComputer) PropertyStore.encodeString(org.neo4j.kernel.impl.store.PropertyStore.encodeString) Map(java.util.Map) PageCacheTracer(org.neo4j.io.pagecache.tracing.PageCacheTracer) NodeMultiPropertyDescriptor(org.neo4j.kernel.api.schema.NodeMultiPropertyDescriptor) DefaultSchemaIndexProviderMap(org.neo4j.kernel.impl.transaction.state.DefaultSchemaIndexProviderMap) KernelExtensions(org.neo4j.kernel.extension.KernelExtensions) IndexDefinitionImpl(org.neo4j.kernel.impl.coreapi.schema.IndexDefinitionImpl) NullLog(org.neo4j.logging.NullLog) NodeUpdates(org.neo4j.kernel.api.index.NodeUpdates) HighestSelectionStrategy(org.neo4j.kernel.extension.dependency.HighestSelectionStrategy) LabelTokenRecord(org.neo4j.kernel.impl.store.record.LabelTokenRecord) DefaultIdGeneratorFactory(org.neo4j.kernel.impl.store.id.DefaultIdGeneratorFactory) RecordCursors(org.neo4j.kernel.impl.store.RecordCursors) SchemaStore(org.neo4j.kernel.impl.store.SchemaStore) ConstraintDefinition(org.neo4j.graphdb.schema.ConstraintDefinition) Locks(org.neo4j.kernel.impl.locking.Locks) UniquenessConstraint(org.neo4j.kernel.api.constraints.UniquenessConstraint) KernelExtensionFactory(org.neo4j.kernel.extension.KernelExtensionFactory) LabelScanStoreProvider(org.neo4j.kernel.impl.api.scan.LabelScanStoreProvider) RelationshipType(org.neo4j.graphdb.RelationshipType) NodeStore(org.neo4j.kernel.impl.store.NodeStore) GraphDatabaseSettings(org.neo4j.graphdb.factory.GraphDatabaseSettings) StoreFactory(org.neo4j.kernel.impl.store.StoreFactory) IndexCreatorImpl(org.neo4j.kernel.impl.coreapi.schema.IndexCreatorImpl) RelationshipTypeToken(org.neo4j.kernel.impl.core.RelationshipTypeToken) RecordFormatSelector(org.neo4j.kernel.impl.store.format.RecordFormatSelector) LabelScanStore(org.neo4j.kernel.api.labelscan.LabelScanStore) PageCacheLifecycle(org.neo4j.kernel.impl.pagecache.PageCacheLifecycle) IteratorWrapper(org.neo4j.helpers.collection.IteratorWrapper) ArrayList(java.util.ArrayList) InternalSchemaActions(org.neo4j.kernel.impl.coreapi.schema.InternalSchemaActions) RelationshipStore(org.neo4j.kernel.impl.store.RelationshipStore) BatchInserter(org.neo4j.unsafe.batchinsert.BatchInserter) DirectRecordAccessSet(org.neo4j.unsafe.batchinsert.DirectRecordAccessSet) SchemaCache(org.neo4j.kernel.impl.api.store.SchemaCache) PropertyStore(org.neo4j.kernel.impl.store.PropertyStore) CreateConstraintFailureException(org.neo4j.kernel.api.exceptions.schema.CreateConstraintFailureException) PrimitiveLongCollections(org.neo4j.collection.primitive.PrimitiveLongCollections) DatabaseInfo(org.neo4j.kernel.impl.factory.DatabaseInfo) UnsatisfiedDependencyStrategies(org.neo4j.kernel.extension.UnsatisfiedDependencyStrategies) Listener(org.neo4j.kernel.impl.util.Listener) PrimitiveLongCollections.map(org.neo4j.collection.primitive.PrimitiveLongCollections.map) BaseNodeConstraintCreator(org.neo4j.kernel.impl.coreapi.schema.BaseNodeConstraintCreator) IndexRule(org.neo4j.kernel.impl.store.record.IndexRule) NewIndexDescriptorFactory(org.neo4j.kernel.api.schema_new.index.NewIndexDescriptorFactory) IOException(java.io.IOException) Label.label(org.neo4j.graphdb.Label.label) NewIndexDescriptor(org.neo4j.kernel.api.schema_new.index.NewIndexDescriptor) SchemaRule(org.neo4j.storageengine.api.schema.SchemaRule) IoPrimitiveUtils.safeCastLongToInt(org.neo4j.kernel.impl.util.IoPrimitiveUtils.safeCastLongToInt) File(java.io.File) Iterables(org.neo4j.helpers.collection.Iterables) ConstraintDescriptorFactory(org.neo4j.kernel.api.schema_new.constaints.ConstraintDescriptorFactory) IndexConfigStore(org.neo4j.kernel.impl.index.IndexConfigStore) RelationshipGroupRecord(org.neo4j.kernel.impl.store.record.RelationshipGroupRecord) Boolean.parseBoolean(java.lang.Boolean.parseBoolean) SchemaIndexProviderMap(org.neo4j.kernel.impl.api.index.SchemaIndexProviderMap) NodeRecord(org.neo4j.kernel.impl.store.record.NodeRecord) NodeLabelUpdate(org.neo4j.kernel.api.labelscan.NodeLabelUpdate) NodePropertyExistenceConstraintDefinition(org.neo4j.kernel.impl.coreapi.schema.NodePropertyExistenceConstraintDefinition) AlreadyConstrainedException(org.neo4j.kernel.api.exceptions.schema.AlreadyConstrainedException) IndexDefinition(org.neo4j.graphdb.schema.IndexDefinition) IndexSamplingConfig(org.neo4j.kernel.impl.api.index.sampling.IndexSamplingConfig) RelationshipRecord(org.neo4j.kernel.impl.store.record.RelationshipRecord) PageCursorTracerSupplier(org.neo4j.io.pagecache.tracing.cursor.PageCursorTracerSupplier) PropertyDeleter(org.neo4j.kernel.impl.transaction.state.PropertyDeleter) Log(org.neo4j.logging.Log) IdGeneratorFactory(org.neo4j.kernel.impl.store.id.IdGeneratorFactory) StoreScan(org.neo4j.kernel.impl.api.index.StoreScan) Dependencies(org.neo4j.kernel.impl.util.Dependencies) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) LabelTokenStore(org.neo4j.kernel.impl.store.LabelTokenStore) NodePropertyDescriptor(org.neo4j.kernel.api.schema.NodePropertyDescriptor) RelationshipTypeTokenStore(org.neo4j.kernel.impl.store.RelationshipTypeTokenStore) DefinedProperty(org.neo4j.kernel.api.properties.DefinedProperty) Record(org.neo4j.kernel.impl.store.record.Record) PageCache(org.neo4j.io.pagecache.PageCache) UnderlyingStorageException(org.neo4j.kernel.impl.store.UnderlyingStorageException) PropertyCreator(org.neo4j.kernel.impl.transaction.state.PropertyCreator) Collection(java.util.Collection) IndexCreator(org.neo4j.graphdb.schema.IndexCreator) IndexEntryUpdate(org.neo4j.kernel.api.index.IndexEntryUpdate) ConstraintDescriptor(org.neo4j.kernel.api.schema_new.constaints.ConstraintDescriptor) KernelException(org.neo4j.kernel.api.exceptions.KernelException) LabelScanWriter(org.neo4j.kernel.api.labelscan.LabelScanWriter) DynamicRecord(org.neo4j.kernel.impl.store.record.DynamicRecord) RecordStore(org.neo4j.kernel.impl.store.RecordStore) List(java.util.List) NamedLabelScanStoreSelectionStrategy(org.neo4j.kernel.extension.dependency.NamedLabelScanStoreSelectionStrategy) StoreLocker(org.neo4j.kernel.internal.StoreLocker) Entry(java.util.Map.Entry) Optional(java.util.Optional) EmbeddedGraphDatabase(org.neo4j.kernel.internal.EmbeddedGraphDatabase) Label(org.neo4j.graphdb.Label) SchemaDescriptorFactory(org.neo4j.kernel.api.schema_new.SchemaDescriptorFactory) LabelSchemaDescriptor(org.neo4j.kernel.api.schema_new.LabelSchemaDescriptor) LogProvider(org.neo4j.logging.LogProvider) HashMap(java.util.HashMap) Token(org.neo4j.storageengine.api.Token) NotFoundException(org.neo4j.graphdb.NotFoundException) ConfiguringPageCacheFactory(org.neo4j.kernel.impl.pagecache.ConfiguringPageCacheFactory) UniquenessConstraintDefinition(org.neo4j.kernel.impl.coreapi.schema.UniquenessConstraintDefinition) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) ConstraintViolationException(org.neo4j.graphdb.ConstraintViolationException) NodeLabels(org.neo4j.kernel.impl.store.NodeLabels) IndexEntryConflictException(org.neo4j.kernel.api.exceptions.index.IndexEntryConflictException) ConstraintCreator(org.neo4j.graphdb.schema.ConstraintCreator) NeoStores(org.neo4j.kernel.impl.store.NeoStores) StoreLogService(org.neo4j.kernel.impl.logging.StoreLogService) PropertyBlock(org.neo4j.kernel.impl.store.record.PropertyBlock) CountsTracker(org.neo4j.kernel.impl.store.counts.CountsTracker) RelationshipTypeTokenRecord(org.neo4j.kernel.impl.store.record.RelationshipTypeTokenRecord) PrintStream(java.io.PrintStream) Config(org.neo4j.kernel.configuration.Config) ConstraintRule(org.neo4j.kernel.impl.store.record.ConstraintRule) RecordProxy(org.neo4j.kernel.impl.transaction.state.RecordAccess.RecordProxy) StandardConstraintSemantics(org.neo4j.kernel.impl.constraints.StandardConstraintSemantics) LockService(org.neo4j.kernel.impl.locking.LockService) RelationshipCreator(org.neo4j.kernel.impl.transaction.state.RelationshipCreator) Iterator(java.util.Iterator) LongFunction(java.util.function.LongFunction) PropertyRecord(org.neo4j.kernel.impl.store.record.PropertyRecord) RecordAccess(org.neo4j.kernel.impl.transaction.state.RecordAccess) PropertyKeyTokenRecord(org.neo4j.kernel.impl.store.record.PropertyKeyTokenRecord) IdValidator(org.neo4j.kernel.impl.store.id.validation.IdValidator) PropertyTraverser(org.neo4j.kernel.impl.transaction.state.PropertyTraverser) PrimitiveRecord(org.neo4j.kernel.impl.store.record.PrimitiveRecord) NeoStoreIndexStoreView(org.neo4j.kernel.impl.transaction.state.storeview.NeoStoreIndexStoreView) Visitor(org.neo4j.helpers.collection.Visitor) Collections(java.util.Collections) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) IndexRule(org.neo4j.kernel.impl.store.record.IndexRule) IndexSamplingConfig(org.neo4j.kernel.impl.api.index.sampling.IndexSamplingConfig) Optional(java.util.Optional) IOException(java.io.IOException) UniquenessConstraint(org.neo4j.kernel.api.constraints.UniquenessConstraint) NodeUpdates(org.neo4j.kernel.api.index.NodeUpdates) IndexPopulator(org.neo4j.kernel.api.index.IndexPopulator) NewIndexDescriptor(org.neo4j.kernel.api.schema_new.index.NewIndexDescriptor) IndexEntryConflictException(org.neo4j.kernel.api.exceptions.index.IndexEntryConflictException)

Example 4 with Visitor

use of org.neo4j.helpers.collection.Visitor in project neo4j by neo4j.

the class DefaultRecoverySPI method startRecovery.

@Override
public Visitor<CommittedTransactionRepresentation, Exception> startRecovery() {
    // Calling this method means that recovery is required, tell storage engine about it
    // This method will be called before recovery actually starts and so will ensure that
    // each store is aware that recovery will be performed. At this point all the stores have
    // already started btw.
    // Go and read more at {@link CommonAbstractStore#deleteIdGenerator()}
    storageEngine.prepareForRecoveryRequired();
    transactionsToApply = new TransactionQueue(10_000, (first, last) -> storageEngine.apply(first, RECOVERY));
    recoveryVisitor = new RecoveryVisitor(transactionsToApply);
    return recoveryVisitor;
}
Also used : TransactionIdStore(org.neo4j.kernel.impl.transaction.log.TransactionIdStore) TransactionRepresentation(org.neo4j.kernel.impl.transaction.TransactionRepresentation) LogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.LogicalTransactionStore) StorageEngine(org.neo4j.storageengine.api.StorageEngine) LogVersionRepository(org.neo4j.kernel.impl.transaction.log.LogVersionRepository) IOException(java.io.IOException) LogEntryStart(org.neo4j.kernel.impl.transaction.log.entry.LogEntryStart) IOLimiter(org.neo4j.io.pagecache.IOLimiter) TransactionToApply(org.neo4j.kernel.impl.api.TransactionToApply) CommittedTransactionRepresentation(org.neo4j.kernel.impl.transaction.CommittedTransactionRepresentation) LogPosition(org.neo4j.kernel.impl.transaction.log.LogPosition) TransactionCursor(org.neo4j.kernel.impl.transaction.log.TransactionCursor) RECOVERY(org.neo4j.storageengine.api.TransactionApplicationMode.RECOVERY) Visitor(org.neo4j.helpers.collection.Visitor) NO_COMMITMENT(org.neo4j.kernel.impl.transaction.log.Commitment.NO_COMMITMENT) TransactionQueue(org.neo4j.kernel.impl.api.TransactionQueue) PhysicalLogFiles(org.neo4j.kernel.impl.transaction.log.PhysicalLogFiles) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) TransactionQueue(org.neo4j.kernel.impl.api.TransactionQueue)

Example 5 with Visitor

use of org.neo4j.helpers.collection.Visitor in project neo4j by neo4j.

the class RecoveryTest method shouldRecoverExistingData.

@Test
public void shouldRecoverExistingData() throws Exception {
    final PhysicalLogFiles logFiles = new PhysicalLogFiles(directory.directory(), "log", fileSystemRule.get());
    File file = logFiles.getLogFileForVersion(logVersion);
    writeSomeData(file, new Visitor<Pair<LogEntryWriter, Consumer<LogPositionMarker>>, IOException>() {

        @Override
        public boolean visit(Pair<LogEntryWriter, Consumer<LogPositionMarker>> pair) throws IOException {
            LogEntryWriter writer = pair.first();
            Consumer<LogPositionMarker> consumer = pair.other();
            LogPositionMarker marker = new LogPositionMarker();
            // last committed tx
            consumer.accept(marker);
            LogPosition lastCommittedTxPosition = marker.newPosition();
            writer.writeStartEntry(0, 1, 2L, 3L, new byte[0]);
            lastCommittedTxStartEntry = new LogEntryStart(0, 1, 2L, 3L, new byte[0], lastCommittedTxPosition);
            writer.writeCommitEntry(4L, 5L);
            lastCommittedTxCommitEntry = new OnePhaseCommit(4L, 5L);
            // check point pointing to the previously committed transaction
            writer.writeCheckPointEntry(lastCommittedTxPosition);
            expectedCheckPointEntry = new CheckPoint(lastCommittedTxPosition);
            // tx committed after checkpoint
            consumer.accept(marker);
            writer.writeStartEntry(0, 1, 6L, 4L, new byte[0]);
            expectedStartEntry = new LogEntryStart(0, 1, 6L, 4L, new byte[0], marker.newPosition());
            writer.writeCommitEntry(5L, 7L);
            expectedCommitEntry = new OnePhaseCommit(5L, 7L);
            return true;
        }
    });
    LifeSupport life = new LifeSupport();
    Recovery.Monitor monitor = mock(Recovery.Monitor.class);
    final AtomicBoolean recoveryRequired = new AtomicBoolean();
    try {
        StorageEngine storageEngine = mock(StorageEngine.class);
        final LogEntryReader<ReadableClosablePositionAwareChannel> reader = new VersionAwareLogEntryReader<>();
        LatestCheckPointFinder finder = new LatestCheckPointFinder(logFiles, fileSystemRule.get(), reader);
        LogHeaderCache logHeaderCache = new LogHeaderCache(10);
        TransactionMetadataCache metadataCache = new TransactionMetadataCache(100);
        LogFile logFile = life.add(new PhysicalLogFile(fileSystemRule.get(), logFiles, 50, () -> transactionIdStore.getLastCommittedTransactionId(), logVersionRepository, mock(PhysicalLogFile.Monitor.class), logHeaderCache));
        LogicalTransactionStore txStore = new PhysicalLogicalTransactionStore(logFile, metadataCache, reader);
        life.add(new Recovery(new DefaultRecoverySPI(storageEngine, logFiles, fileSystemRule.get(), logVersionRepository, finder, transactionIdStore, txStore, NO_MONITOR) {

            private int nr = 0;

            @Override
            public Visitor<CommittedTransactionRepresentation, Exception> startRecovery() {
                recoveryRequired.set(true);
                final Visitor<CommittedTransactionRepresentation, Exception> actual = super.startRecovery();
                return new Visitor<CommittedTransactionRepresentation, Exception>() {

                    @Override
                    public boolean visit(CommittedTransactionRepresentation tx) throws Exception {
                        actual.visit(tx);
                        switch(nr++) {
                            case 0:
                                assertEquals(lastCommittedTxStartEntry, tx.getStartEntry());
                                assertEquals(lastCommittedTxCommitEntry, tx.getCommitEntry());
                                break;
                            case 1:
                                assertEquals(expectedStartEntry, tx.getStartEntry());
                                assertEquals(expectedCommitEntry, tx.getCommitEntry());
                                break;
                            default:
                                fail("Too many recovered transactions");
                        }
                        return false;
                    }
                };
            }
        }, monitor));
        life.start();
        InOrder order = inOrder(monitor);
        order.verify(monitor, times(1)).recoveryRequired(any(LogPosition.class));
        order.verify(monitor, times(1)).recoveryCompleted(2);
        assertTrue(recoveryRequired.get());
    } finally {
        life.shutdown();
    }
}
Also used : DefaultRecoverySPI(org.neo4j.kernel.recovery.DefaultRecoverySPI) Visitor(org.neo4j.helpers.collection.Visitor) PhysicalLogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.PhysicalLogicalTransactionStore) LogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.LogicalTransactionStore) StorageEngine(org.neo4j.storageengine.api.StorageEngine) Recovery(org.neo4j.kernel.recovery.Recovery) PhysicalLogFiles(org.neo4j.kernel.impl.transaction.log.PhysicalLogFiles) LogPositionMarker(org.neo4j.kernel.impl.transaction.log.LogPositionMarker) ReadableClosablePositionAwareChannel(org.neo4j.kernel.impl.transaction.log.ReadableClosablePositionAwareChannel) LogFile(org.neo4j.kernel.impl.transaction.log.LogFile) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) Consumer(java.util.function.Consumer) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) VersionAwareLogEntryReader(org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader) LogEntryWriter(org.neo4j.kernel.impl.transaction.log.entry.LogEntryWriter) OnePhaseCommit(org.neo4j.kernel.impl.transaction.log.entry.OnePhaseCommit) Pair(org.neo4j.helpers.collection.Pair) LogPosition(org.neo4j.kernel.impl.transaction.log.LogPosition) CheckPoint(org.neo4j.kernel.impl.transaction.log.entry.CheckPoint) LogEntryStart(org.neo4j.kernel.impl.transaction.log.entry.LogEntryStart) PhysicalLogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.PhysicalLogicalTransactionStore) LatestCheckPointFinder(org.neo4j.kernel.recovery.LatestCheckPointFinder) CommittedTransactionRepresentation(org.neo4j.kernel.impl.transaction.CommittedTransactionRepresentation) InOrder(org.mockito.InOrder) TransactionMetadataCache(org.neo4j.kernel.impl.transaction.log.TransactionMetadataCache) IOException(java.io.IOException) IOException(java.io.IOException) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) LogFile(org.neo4j.kernel.impl.transaction.log.LogFile) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) File(java.io.File) LogHeaderCache(org.neo4j.kernel.impl.transaction.log.LogHeaderCache) PhysicalLogFile(org.neo4j.kernel.impl.transaction.log.PhysicalLogFile) Test(org.junit.Test)

Aggregations

Visitor (org.neo4j.helpers.collection.Visitor)12 IOException (java.io.IOException)7 CommittedTransactionRepresentation (org.neo4j.kernel.impl.transaction.CommittedTransactionRepresentation)7 LogicalTransactionStore (org.neo4j.kernel.impl.transaction.log.LogicalTransactionStore)7 Response (org.neo4j.com.Response)5 TransactionIdStore (org.neo4j.kernel.impl.transaction.log.TransactionIdStore)5 File (java.io.File)4 Test (org.junit.Test)4 LogPosition (org.neo4j.kernel.impl.transaction.log.LogPosition)4 PhysicalLogFiles (org.neo4j.kernel.impl.transaction.log.PhysicalLogFiles)4 LifeSupport (org.neo4j.kernel.lifecycle.LifeSupport)4 RequestContext (org.neo4j.com.RequestContext)3 FileSystemAbstraction (org.neo4j.io.fs.FileSystemAbstraction)3 NeoStores (org.neo4j.kernel.impl.store.NeoStores)3 LogFile (org.neo4j.kernel.impl.transaction.log.LogFile)3 LogHeaderCache (org.neo4j.kernel.impl.transaction.log.LogHeaderCache)3 PhysicalLogFile (org.neo4j.kernel.impl.transaction.log.PhysicalLogFile)3 LogEntryStart (org.neo4j.kernel.impl.transaction.log.entry.LogEntryStart)3 LogEntryWriter (org.neo4j.kernel.impl.transaction.log.entry.LogEntryWriter)3 StorageEngine (org.neo4j.storageengine.api.StorageEngine)3