Search in sources :

Example 1 with IdGenerator

use of org.neo4j.internal.id.IdGenerator in project neo4j by neo4j.

the class IndexedIdGeneratorRecoverabilityTest method resetUsabilityOnRestart.

@Test
void resetUsabilityOnRestart() throws IOException {
    // Create the freelist
    try (IdGenerator freelist = instantiateFreelist()) {
        freelist.checkpoint(NULL);
    }
    final long id1;
    final long id2;
    try (IdGenerator freelist = instantiateFreelist()) {
        id1 = freelist.nextId(NULL);
        id2 = freelist.nextId(NULL);
        markUsed(freelist, id1, id2);
        markDeleted(freelist, id1, id2);
        freelist.checkpoint(NULL);
    }
    try (IdGenerator freelist = instantiateFreelist()) {
        freelist.start(NO_FREE_IDS, NULL);
        final ImmutableLongSet reused = LongSets.immutable.of(freelist.nextId(NULL), freelist.nextId(NULL));
        assertEquals(LongSets.immutable.of(id1, id2), reused, "IDs are not reused");
    }
}
Also used : ImmutableLongSet(org.eclipse.collections.api.set.primitive.ImmutableLongSet) IdGenerator(org.neo4j.internal.id.IdGenerator) Test(org.junit.jupiter.api.Test)

Example 2 with IdGenerator

use of org.neo4j.internal.id.IdGenerator in project neo4j by neo4j.

the class IndexedIdGeneratorRecoverabilityTest method doNotPersistHighIdBetweenCleanRestartsWithoutCheckpoint.

@Test
void doNotPersistHighIdBetweenCleanRestartsWithoutCheckpoint() {
    try (IdGenerator freelist = instantiateFreelist()) {
        freelist.nextId(NULL);
        assertEquals(1, freelist.getHighId());
        freelist.nextId(NULL);
        assertEquals(2, freelist.getHighId());
    }
    try (IdGenerator freelist = instantiateFreelist()) {
        assertEquals(0, freelist.getHighId());
    }
}
Also used : IdGenerator(org.neo4j.internal.id.IdGenerator) Test(org.junit.jupiter.api.Test)

Example 3 with IdGenerator

use of org.neo4j.internal.id.IdGenerator in project neo4j by neo4j.

the class RecoveryIT method idGeneratorIsDirty.

private boolean idGeneratorIsDirty(Path path, IdType idType) throws IOException {
    DefaultIdGeneratorFactory idGeneratorFactory = new DefaultIdGeneratorFactory(fileSystem, immediate(), "my db");
    try (IdGenerator idGenerator = idGeneratorFactory.open(pageCache, path, idType, () -> 0L, /*will not be used*/
    10_000, readOnly(), Config.defaults(), NULL, Sets.immutable.empty())) {
        MutableBoolean dirtyOnStartup = new MutableBoolean();
        InvocationHandler invocationHandler = (proxy, method, args) -> {
            if (method.getName().equals("dirtyOnStartup")) {
                dirtyOnStartup.setTrue();
            }
            return null;
        };
        ReporterFactory reporterFactory = new ReporterFactory(invocationHandler);
        idGenerator.consistencyCheck(reporterFactory, NULL);
        return dirtyOnStartup.booleanValue();
    }
}
Also used : LifecycleAdapter(org.neo4j.kernel.lifecycle.LifecycleAdapter) MetaDataStore.getRecord(org.neo4j.kernel.impl.store.MetaDataStore.getRecord) DatabaseReadOnlyChecker.readOnly(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker.readOnly) GraphDatabaseSettings.fail_on_missing_files(org.neo4j.configuration.GraphDatabaseSettings.fail_on_missing_files) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) ReporterFactory(org.neo4j.annotations.documented.ReporterFactory) LogFilesBuilder(org.neo4j.kernel.impl.transaction.log.files.LogFilesBuilder) Config(org.neo4j.configuration.Config) DefaultPageCacheTracer(org.neo4j.io.pagecache.tracing.DefaultPageCacheTracer) InternalTransaction(org.neo4j.kernel.impl.coreapi.InternalTransaction) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) Recovery.performRecovery(org.neo4j.kernel.recovery.Recovery.performRecovery) DEFAULT_DATABASE_NAME(org.neo4j.configuration.GraphDatabaseSettings.DEFAULT_DATABASE_NAME) GraphDatabaseSettings.logical_log_rotation_threshold(org.neo4j.configuration.GraphDatabaseSettings.logical_log_rotation_threshold) LogCheckPointEvent(org.neo4j.kernel.impl.transaction.tracing.LogCheckPointEvent) Assertions.assertFalse(org.junit.jupiter.api.Assertions.assertFalse) AssertableLogProvider(org.neo4j.logging.AssertableLogProvider) Monitors(org.neo4j.monitoring.Monitors) CHECKPOINT_LOG_VERSION(org.neo4j.kernel.impl.store.MetaDataStore.Position.CHECKPOINT_LOG_VERSION) NULL(org.neo4j.io.pagecache.context.CursorContext.NULL) Transaction(org.neo4j.graphdb.Transaction) Path(java.nio.file.Path) PageCache(org.neo4j.io.pagecache.PageCache) LAST_MISSING_STORE_FILES_RECOVERY_TIMESTAMP(org.neo4j.kernel.impl.store.MetaDataStore.Position.LAST_MISSING_STORE_FILES_RECOVERY_TIMESTAMP) DEFAULT_NAME(org.neo4j.kernel.impl.transaction.log.files.TransactionLogFilesHelper.DEFAULT_NAME) LegacyTransactionLogsLocator(org.neo4j.kernel.impl.storemigration.LegacyTransactionLogsLocator) DatabaseReadOnlyChecker.writable(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker.writable) CompositeDatabaseAvailabilityGuard(org.neo4j.kernel.availability.CompositeDatabaseAvailabilityGuard) RelationshipType.withName(org.neo4j.graphdb.RelationshipType.withName) Instant(java.time.Instant) ArrayUtil(org.neo4j.internal.helpers.ArrayUtil) Services(org.neo4j.service.Services) Neo4jLayoutExtension(org.neo4j.test.extension.Neo4jLayoutExtension) GraphDatabaseAPI(org.neo4j.kernel.internal.GraphDatabaseAPI) Test(org.junit.jupiter.api.Test) IdType(org.neo4j.internal.id.IdType) IndexQueryConstraints.unconstrained(org.neo4j.internal.kernel.api.IndexQueryConstraints.unconstrained) DatabaseStateService(org.neo4j.dbms.DatabaseStateService) INSTANCE(org.neo4j.memory.EmptyMemoryTracker.INSTANCE) ExtensionContext(org.neo4j.kernel.extension.context.ExtensionContext) RandomStringUtils.randomAlphanumeric(org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) MetaDataStore(org.neo4j.kernel.impl.store.MetaDataStore) Assertions.assertTrue(org.junit.jupiter.api.Assertions.assertTrue) Optional(java.util.Optional) DatabaseStartAbortedException(org.neo4j.dbms.database.DatabaseStartAbortedException) RelationshipType(org.neo4j.graphdb.RelationshipType) IndexDescriptor(org.neo4j.internal.schema.IndexDescriptor) MutableBoolean(org.apache.commons.lang3.mutable.MutableBoolean) DatabaseManagementService(org.neo4j.dbms.api.DatabaseManagementService) Assertions.assertThrows(org.junit.jupiter.api.Assertions.assertThrows) LogFiles(org.neo4j.kernel.impl.transaction.log.files.LogFiles) Assertions.assertNotNull(org.junit.jupiter.api.Assertions.assertNotNull) Label(org.neo4j.graphdb.Label) Config.defaults(org.neo4j.configuration.Config.defaults) IndexReadSession(org.neo4j.internal.kernel.api.IndexReadSession) DatabaseTracer(org.neo4j.kernel.impl.transaction.tracing.DatabaseTracer) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) MINUTES(java.util.concurrent.TimeUnit.MINUTES) Node(org.neo4j.graphdb.Node) IndexType(org.neo4j.graphdb.schema.IndexType) GraphDatabaseSettings.preallocate_logical_logs(org.neo4j.configuration.GraphDatabaseSettings.preallocate_logical_logs) LogPosition(org.neo4j.kernel.impl.transaction.log.LogPosition) PropertyIndexQuery.fulltextSearch(org.neo4j.internal.kernel.api.PropertyIndexQuery.fulltextSearch) TestDatabaseManagementServiceBuilder(org.neo4j.test.TestDatabaseManagementServiceBuilder) GraphDatabaseService(org.neo4j.graphdb.GraphDatabaseService) Inject(org.neo4j.test.extension.Inject) StorageEngineFactory.defaultStorageEngine(org.neo4j.storageengine.api.StorageEngineFactory.defaultStorageEngine) Iterables(org.neo4j.internal.helpers.collection.Iterables) IdGenerator(org.neo4j.internal.id.IdGenerator) GraphDatabaseInternalSettings(org.neo4j.configuration.GraphDatabaseInternalSettings) Assertions.assertEquals(org.junit.jupiter.api.Assertions.assertEquals) EMPTY(org.neo4j.kernel.database.DatabaseTracers.EMPTY) Sets(org.eclipse.collections.api.factory.Sets) Lifecycle(org.neo4j.kernel.lifecycle.Lifecycle) NullLogProvider.nullLogProvider(org.neo4j.logging.NullLogProvider.nullLogProvider) BASE_TX_LOG_BYTE_OFFSET(org.neo4j.storageengine.api.LogVersionRepository.BASE_TX_LOG_BYTE_OFFSET) DetachedCheckpointAppender(org.neo4j.kernel.impl.transaction.log.checkpoint.DetachedCheckpointAppender) ByteUnit(org.neo4j.io.ByteUnit) LockTracer(org.neo4j.lock.LockTracer) PageCacheExtension(org.neo4j.test.extension.pagecache.PageCacheExtension) Iterables.count(org.neo4j.internal.helpers.collection.Iterables.count) RelationshipValueIndexCursor(org.neo4j.internal.kernel.api.RelationshipValueIndexCursor) ExceptionUtils.getRootCause(org.apache.commons.lang3.exception.ExceptionUtils.getRootCause) IOException(java.io.IOException) Neo4jLayout(org.neo4j.io.layout.Neo4jLayout) DatabaseTracers(org.neo4j.kernel.database.DatabaseTracers) StorageEngineFactory(org.neo4j.storageengine.api.StorageEngineFactory) RecoveryCleanupWorkCollector.immediate(org.neo4j.index.internal.gbptree.RecoveryCleanupWorkCollector.immediate) DefaultFileSystemAbstraction(org.neo4j.io.fs.DefaultFileSystemAbstraction) ExtensionFactory(org.neo4j.kernel.extension.ExtensionFactory) Relationship(org.neo4j.graphdb.Relationship) String.valueOf(java.lang.String.valueOf) KernelTransaction(org.neo4j.kernel.api.KernelTransaction) Clock(java.time.Clock) InvocationHandler(java.lang.reflect.InvocationHandler) ReporterFactory(org.neo4j.annotations.documented.ReporterFactory) MutableBoolean(org.apache.commons.lang3.mutable.MutableBoolean) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) IdGenerator(org.neo4j.internal.id.IdGenerator) InvocationHandler(java.lang.reflect.InvocationHandler)

Example 4 with IdGenerator

use of org.neo4j.internal.id.IdGenerator in project neo4j by neo4j.

the class RecoveryIT method shouldForceRecoveryEvenThoughNotSeeminglyRequired.

@Test
void shouldForceRecoveryEvenThoughNotSeeminglyRequired() throws Exception {
    // given
    GraphDatabaseAPI db = createDatabase();
    generateSomeData(db);
    DatabaseLayout layout = db.databaseLayout();
    managementService.shutdown();
    assertFalse(isRecoveryRequired(layout));
    // Make an ID generator, say for the node store, dirty
    DefaultIdGeneratorFactory idGeneratorFactory = new DefaultIdGeneratorFactory(fileSystem, immediate(), "my db");
    try (IdGenerator idGenerator = idGeneratorFactory.open(pageCache, layout.idNodeStore(), IdType.NODE, () -> 0L, /*will not be used*/
    10_000, writable(), Config.defaults(), NULL, Sets.immutable.empty())) {
        // Merely opening a marker will make the backing GBPTree dirty
        idGenerator.marker(NULL).close();
    }
    assertFalse(isRecoveryRequired(layout));
    assertTrue(idGeneratorIsDirty(layout.idNodeStore(), IdType.NODE));
    // when
    MutableBoolean recoveryRunEvenThoughNoCommitsAfterLastCheckpoint = new MutableBoolean();
    RecoveryStartInformationProvider.Monitor monitor = new RecoveryStartInformationProvider.Monitor() {

        @Override
        public void noCommitsAfterLastCheckPoint(LogPosition logPosition) {
            recoveryRunEvenThoughNoCommitsAfterLastCheckpoint.setTrue();
        }
    };
    Monitors monitors = new Monitors();
    monitors.addMonitorListener(monitor);
    Recovery.performRecovery(fileSystem, pageCache, EMPTY, Config.defaults(), layout, defaultStorageEngine(), true, nullLogProvider(), monitors, Iterables.cast(Services.loadAll(ExtensionFactory.class)), Optional.empty(), null, INSTANCE, Clock.systemUTC());
    // then
    assertFalse(idGeneratorIsDirty(layout.idNodeStore(), IdType.NODE));
    assertTrue(recoveryRunEvenThoughNoCommitsAfterLastCheckpoint.booleanValue());
}
Also used : GraphDatabaseAPI(org.neo4j.kernel.internal.GraphDatabaseAPI) MutableBoolean(org.apache.commons.lang3.mutable.MutableBoolean) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) Monitors(org.neo4j.monitoring.Monitors) IdGenerator(org.neo4j.internal.id.IdGenerator) LogPosition(org.neo4j.kernel.impl.transaction.log.LogPosition) Test(org.junit.jupiter.api.Test)

Example 5 with IdGenerator

use of org.neo4j.internal.id.IdGenerator in project neo4j by neo4j.

the class BigStoreIT method setHighId.

private void setHighId(IdType type, long highId) {
    IdGeneratorFactory idGeneratorFactory = ((GraphDatabaseAPI) db).getDependencyResolver().resolveDependency(IdGeneratorFactory.class);
    IdGenerator idGenerator = idGeneratorFactory.get(type);
    idGenerator.setHighId(highId);
    idGenerator.markHighestWrittenAtHighId();
}
Also used : IdGeneratorFactory(org.neo4j.internal.id.IdGeneratorFactory) IdGenerator(org.neo4j.internal.id.IdGenerator)

Aggregations

IdGenerator (org.neo4j.internal.id.IdGenerator)12 Test (org.junit.jupiter.api.Test)9 ImmutableLongSet (org.eclipse.collections.api.set.primitive.ImmutableLongSet)3 Path (java.nio.file.Path)2 MutableBoolean (org.apache.commons.lang3.mutable.MutableBoolean)2 Config (org.neo4j.configuration.Config)2 DefaultIdGeneratorFactory (org.neo4j.internal.id.DefaultIdGeneratorFactory)2 DatabaseLayout (org.neo4j.io.layout.DatabaseLayout)2 LogPosition (org.neo4j.kernel.impl.transaction.log.LogPosition)2 GraphDatabaseAPI (org.neo4j.kernel.internal.GraphDatabaseAPI)2 Monitors (org.neo4j.monitoring.Monitors)2 IOException (java.io.IOException)1 String.valueOf (java.lang.String.valueOf)1 InvocationHandler (java.lang.reflect.InvocationHandler)1 Clock (java.time.Clock)1 Instant (java.time.Instant)1 Optional (java.util.Optional)1 MINUTES (java.util.concurrent.TimeUnit.MINUTES)1 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)1 RandomStringUtils.randomAlphanumeric (org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric)1