Search in sources :

Example 6 with LogService

use of org.neo4j.logging.internal.LogService in project neo4j by neo4j.

the class RecordStorageMigratorIT method shouldBeAbleToResumeMigrationOnRebuildingCounts.

@ParameterizedTest
@MethodSource("versions")
void shouldBeAbleToResumeMigrationOnRebuildingCounts(String version, LogPosition expectedLogPosition, Function<TransactionId, Boolean> txIdComparator) throws Exception {
    // GIVEN a legacy database
    Path prepare = testDirectory.directory("prepare");
    var fs = testDirectory.getFileSystem();
    MigrationTestUtils.prepareSampleLegacyDatabase(version, fs, databaseLayout.databaseDirectory(), prepare);
    // and a state of the migration saying that it has done the actual migration
    LogService logService = NullLogService.getInstance();
    RecordStoreVersionCheck check = getVersionCheck(pageCache, databaseLayout);
    String versionToMigrateFrom = getVersionToMigrateFrom(check);
    MigrationProgressMonitor progressMonitor = SILENT;
    RecordStorageMigrator migrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, PageCacheTracer.NULL, batchImporterFactory, INSTANCE);
    migrator.migrate(databaseLayout, migrationLayout, progressMonitor.startSection("section"), versionToMigrateFrom, getVersionToMigrateTo(check), EMPTY);
    // WHEN simulating resuming the migration
    migrator.moveMigratedFiles(migrationLayout, databaseLayout, versionToMigrateFrom, getVersionToMigrateTo(check));
    // THEN starting the new store should be successful
    StoreFactory storeFactory = new StoreFactory(databaseLayout, CONFIG, new ScanOnOpenOverwritingIdGeneratorFactory(fs, databaseLayout.getDatabaseName()), pageCache, fs, logService.getInternalLogProvider(), PageCacheTracer.NULL, writable());
    storeFactory.openAllNeoStores().close();
}
Also used : Path(java.nio.file.Path) MigrationProgressMonitor(org.neo4j.storageengine.migration.MigrationProgressMonitor) StoreFactory(org.neo4j.kernel.impl.store.StoreFactory) NullLogService(org.neo4j.logging.internal.NullLogService) SimpleLogService(org.neo4j.logging.internal.SimpleLogService) LogService(org.neo4j.logging.internal.LogService) ScanOnOpenOverwritingIdGeneratorFactory(org.neo4j.internal.id.ScanOnOpenOverwritingIdGeneratorFactory) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 7 with LogService

use of org.neo4j.logging.internal.LogService in project neo4j by neo4j.

the class RecordStorageMigratorIT method shouldBeAbleToResumeMigrationOnMoving.

@ParameterizedTest
@MethodSource("versions")
void shouldBeAbleToResumeMigrationOnMoving(String version, LogPosition expectedLogPosition, Function<TransactionId, Boolean> txIdComparator) throws Exception {
    // GIVEN a legacy database
    Path prepare = testDirectory.directory("prepare");
    var fs = testDirectory.getFileSystem();
    MigrationTestUtils.prepareSampleLegacyDatabase(version, fs, databaseLayout.databaseDirectory(), prepare);
    // and a state of the migration saying that it has done the actual migration
    LogService logService = NullLogService.getInstance();
    RecordStoreVersionCheck check = getVersionCheck(pageCache, databaseLayout);
    String versionToMigrateFrom = getVersionToMigrateFrom(check);
    MigrationProgressMonitor progressMonitor = SILENT;
    RecordStorageMigrator migrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, PageCacheTracer.NULL, batchImporterFactory, INSTANCE);
    migrator.migrate(databaseLayout, migrationLayout, progressMonitor.startSection("section"), versionToMigrateFrom, getVersionToMigrateTo(check), EMPTY);
    // WHEN simulating resuming the migration
    migrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, PageCacheTracer.NULL, batchImporterFactory, INSTANCE);
    migrator.moveMigratedFiles(migrationLayout, databaseLayout, versionToMigrateFrom, getVersionToMigrateTo(check));
    // THEN starting the new store should be successful
    StoreFactory storeFactory = new StoreFactory(databaseLayout, CONFIG, new ScanOnOpenOverwritingIdGeneratorFactory(fs, databaseLayout.getDatabaseName()), pageCache, fs, logService.getInternalLogProvider(), PageCacheTracer.NULL, writable());
    storeFactory.openAllNeoStores().close();
}
Also used : Path(java.nio.file.Path) MigrationProgressMonitor(org.neo4j.storageengine.migration.MigrationProgressMonitor) StoreFactory(org.neo4j.kernel.impl.store.StoreFactory) NullLogService(org.neo4j.logging.internal.NullLogService) SimpleLogService(org.neo4j.logging.internal.SimpleLogService) LogService(org.neo4j.logging.internal.LogService) ScanOnOpenOverwritingIdGeneratorFactory(org.neo4j.internal.id.ScanOnOpenOverwritingIdGeneratorFactory) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 8 with LogService

use of org.neo4j.logging.internal.LogService in project neo4j by neo4j.

the class RecordStorageMigratorIT method shouldComputeTheLastTxInfoCorrectly.

@ParameterizedTest
@MethodSource("versions")
void shouldComputeTheLastTxInfoCorrectly(String version, LogPosition expectedLogPosition, Function<TransactionId, Boolean> txIdComparator) throws Exception {
    // given
    Path prepare = testDirectory.directory("prepare");
    var fs = testDirectory.getFileSystem();
    MigrationTestUtils.prepareSampleLegacyDatabase(version, fs, databaseLayout.databaseDirectory(), prepare);
    // and a state of the migration saying that it has done the actual migration
    LogService logService = NullLogService.getInstance();
    RecordStoreVersionCheck check = getVersionCheck(pageCache, databaseLayout);
    String versionToMigrateFrom = getVersionToMigrateFrom(check);
    MigrationProgressMonitor progressMonitor = SILENT;
    RecordStorageMigrator migrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, PageCacheTracer.NULL, batchImporterFactory, INSTANCE);
    // when
    migrator.migrate(databaseLayout, migrationLayout, progressMonitor.startSection("section"), versionToMigrateFrom, getVersionToMigrateTo(check), EMPTY);
    // then
    assertTrue(txIdComparator.apply(migrator.readLastTxInformation(migrationLayout)));
}
Also used : Path(java.nio.file.Path) MigrationProgressMonitor(org.neo4j.storageengine.migration.MigrationProgressMonitor) NullLogService(org.neo4j.logging.internal.NullLogService) SimpleLogService(org.neo4j.logging.internal.SimpleLogService) LogService(org.neo4j.logging.internal.LogService) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 9 with LogService

use of org.neo4j.logging.internal.LogService in project neo4j by neo4j.

the class Recovery method instantiateRecoveryExtensions.

private static DatabaseExtensions instantiateRecoveryExtensions(DatabaseLayout databaseLayout, FileSystemAbstraction fileSystem, Config config, LogService logService, PageCache pageCache, JobScheduler jobScheduler, DbmsInfo dbmsInfo, Monitors monitors, TokenHolders tokenHolders, RecoveryCleanupWorkCollector recoveryCleanupCollector, DatabaseReadOnlyChecker readOnlyChecker, Iterable<ExtensionFactory<?>> extensionFactories, PageCacheTracer pageCacheTracer) {
    List<ExtensionFactory<?>> recoveryExtensions = stream(extensionFactories).filter(extension -> extension.getClass().isAnnotationPresent(RecoveryExtension.class)).collect(toList());
    Dependencies deps = new Dependencies();
    NonListenableMonitors nonListenableMonitors = new NonListenableMonitors(monitors, logService.getInternalLogProvider());
    deps.satisfyDependencies(fileSystem, config, logService, pageCache, nonListenableMonitors, jobScheduler, tokenHolders, recoveryCleanupCollector, pageCacheTracer, databaseLayout, readOnlyChecker);
    DatabaseExtensionContext extensionContext = new DatabaseExtensionContext(databaseLayout, dbmsInfo, deps);
    return new DatabaseExtensions(extensionContext, recoveryExtensions, deps, ExtensionFailureStrategies.fail());
}
Also used : DatabaseExtensionContext(org.neo4j.kernel.extension.context.DatabaseExtensionContext) DefaultIdController(org.neo4j.internal.id.DefaultIdController) LogFilesBuilder(org.neo4j.kernel.impl.transaction.log.files.LogFilesBuilder) Config(org.neo4j.configuration.Config) NullLogProvider(org.neo4j.logging.NullLogProvider) VersionAwareLogEntryReader(org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader) IndexStatisticsStore(org.neo4j.kernel.impl.api.index.stats.IndexStatisticsStore) DatabasePageCache(org.neo4j.dbms.database.DatabasePageCache) ReadOnlyTokenCreator(org.neo4j.token.ReadOnlyTokenCreator) PageCacheTracer(org.neo4j.io.pagecache.tracing.PageCacheTracer) RecoveryCleanupWorkCollector(org.neo4j.index.internal.gbptree.RecoveryCleanupWorkCollector) NullLog(org.neo4j.logging.NullLog) Path(java.nio.file.Path) INDEX_CLEANUP_WORK(org.neo4j.scheduler.Group.INDEX_CLEANUP_WORK) EMPTY_CHECKER(org.neo4j.kernel.recovery.RecoveryStartupChecker.EMPTY_CHECKER) TransactionLogFilesHelper(org.neo4j.kernel.impl.transaction.log.files.TransactionLogFilesHelper) StorageFilesState(org.neo4j.storageengine.api.StorageFilesState) Dependencies(org.neo4j.collection.Dependencies) IOController(org.neo4j.io.pagecache.IOController) FullScanStoreView(org.neo4j.kernel.impl.transaction.state.storeview.FullScanStoreView) Services(org.neo4j.service.Services) SimpleTriggerInfo(org.neo4j.kernel.impl.transaction.log.checkpoint.SimpleTriggerInfo) Database(org.neo4j.kernel.database.Database) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) Tracers(org.neo4j.kernel.monitoring.tracing.Tracers) VersionContextSupplier(org.neo4j.io.pagecache.context.VersionContextSupplier) IndexingService(org.neo4j.kernel.impl.api.index.IndexingService) RecoveryThreshold(org.neo4j.kernel.impl.transaction.log.checkpoint.RecoveryThreshold) DefaultForceOperation(org.neo4j.kernel.database.DefaultForceOperation) DelegatingTokenHolder(org.neo4j.token.DelegatingTokenHolder) TYPE_PROPERTY_KEY(org.neo4j.token.api.TokenHolder.TYPE_PROPERTY_KEY) DatabaseExtensions(org.neo4j.kernel.extension.DatabaseExtensions) DefaultIndexProviderMap(org.neo4j.kernel.impl.transaction.state.DefaultIndexProviderMap) MemoryTracker(org.neo4j.memory.MemoryTracker) Lifecycle(org.neo4j.kernel.lifecycle.Lifecycle) PanicEventGenerator(org.neo4j.monitoring.PanicEventGenerator) TransactionMetadataCache(org.neo4j.kernel.impl.transaction.log.TransactionMetadataCache) IOException(java.io.IOException) DatabaseTracers(org.neo4j.kernel.database.DatabaseTracers) ExtensionFactory(org.neo4j.kernel.extension.ExtensionFactory) TokenHolders(org.neo4j.token.TokenHolders) TYPE_LABEL(org.neo4j.token.api.TokenHolder.TYPE_LABEL) StoreCopyCheckPointMutex(org.neo4j.kernel.impl.transaction.log.checkpoint.StoreCopyCheckPointMutex) LifecycleAdapter(org.neo4j.kernel.lifecycle.LifecycleAdapter) Log(org.neo4j.logging.Log) BatchingTransactionAppender(org.neo4j.kernel.impl.transaction.log.BatchingTransactionAppender) LifeSupport(org.neo4j.kernel.lifecycle.LifeSupport) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) LogRotation(org.neo4j.kernel.impl.transaction.log.rotation.LogRotation) RecoveryState(org.neo4j.storageengine.api.RecoveryState) Monitors(org.neo4j.monitoring.Monitors) LogVersionRepository(org.neo4j.storageengine.api.LogVersionRepository) CheckPointerImpl(org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl) DatabaseSchemaState(org.neo4j.kernel.impl.api.DatabaseSchemaState) JobSchedulerFactory(org.neo4j.kernel.impl.scheduler.JobSchedulerFactory) PageCache(org.neo4j.io.pagecache.PageCache) LegacyTransactionLogsLocator(org.neo4j.kernel.impl.storemigration.LegacyTransactionLogsLocator) StorageEngine(org.neo4j.storageengine.api.StorageEngine) DatabaseReadOnlyChecker.writable(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker.writable) LogProgressReporter(org.neo4j.kernel.impl.util.monitoring.LogProgressReporter) String.format(java.lang.String.format) List(java.util.List) NO_LOCK_SERVICE(org.neo4j.lock.LockService.NO_LOCK_SERVICE) SimpleLogService(org.neo4j.logging.internal.SimpleLogService) Optional(java.util.Optional) DatabaseReadOnlyChecker(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker) TYPE_RELATIONSHIP_TYPE(org.neo4j.token.api.TokenHolder.TYPE_RELATIONSHIP_TYPE) PhysicalLogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.PhysicalLogicalTransactionStore) LogFiles(org.neo4j.kernel.impl.transaction.log.files.LogFiles) LogicalTransactionStore(org.neo4j.kernel.impl.transaction.log.LogicalTransactionStore) Config.defaults(org.neo4j.configuration.Config.defaults) ProgressReporter(org.neo4j.common.ProgressReporter) GraphDatabaseSettings(org.neo4j.configuration.GraphDatabaseSettings) LogProvider(org.neo4j.logging.LogProvider) ConfiguringPageCacheFactory(org.neo4j.kernel.impl.pagecache.ConfiguringPageCacheFactory) GroupingRecoveryCleanupWorkCollector(org.neo4j.index.internal.gbptree.GroupingRecoveryCleanupWorkCollector) Clocks.systemClock(org.neo4j.time.Clocks.systemClock) Iterables(org.neo4j.internal.helpers.collection.Iterables) Objects.requireNonNull(java.util.Objects.requireNonNull) StoreId(org.neo4j.storageengine.api.StoreId) GraphDatabaseInternalSettings(org.neo4j.configuration.GraphDatabaseInternalSettings) DbmsInfo(org.neo4j.kernel.impl.factory.DbmsInfo) LogPruning(org.neo4j.kernel.impl.transaction.log.pruning.LogPruning) MemoryPools(org.neo4j.memory.MemoryPools) IndexStoreViewFactory(org.neo4j.kernel.impl.transaction.state.storeview.IndexStoreViewFactory) JobScheduler(org.neo4j.scheduler.JobScheduler) Iterables.stream(org.neo4j.internal.helpers.collection.Iterables.stream) DatabaseHealth(org.neo4j.monitoring.DatabaseHealth) NO_LOCKS(org.neo4j.kernel.impl.locking.Locks.NO_LOCKS) ConstraintSemantics.getConstraintSemantics(org.neo4j.kernel.impl.constraints.ConstraintSemantics.getConstraintSemantics) LogService(org.neo4j.logging.internal.LogService) StorageEngineFactory(org.neo4j.storageengine.api.StorageEngineFactory) INDEX_CLEANUP(org.neo4j.scheduler.Group.INDEX_CLEANUP) MetadataProvider(org.neo4j.storageengine.api.MetadataProvider) DefaultFileSystemAbstraction(org.neo4j.io.fs.DefaultFileSystemAbstraction) Collectors.toList(java.util.stream.Collectors.toList) StorageEngineFactory.selectStorageEngine(org.neo4j.storageengine.api.StorageEngineFactory.selectStorageEngine) Clock(java.time.Clock) ExtensionFailureStrategies(org.neo4j.kernel.extension.ExtensionFailureStrategies) TransactionIdStore(org.neo4j.storageengine.api.TransactionIdStore) Clocks(org.neo4j.time.Clocks) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) EmptyVersionContextSupplier(org.neo4j.io.pagecache.context.EmptyVersionContextSupplier) ExtensionFactory(org.neo4j.kernel.extension.ExtensionFactory) DatabaseExtensionContext(org.neo4j.kernel.extension.context.DatabaseExtensionContext) DatabaseExtensions(org.neo4j.kernel.extension.DatabaseExtensions) Dependencies(org.neo4j.collection.Dependencies)

Example 10 with LogService

use of org.neo4j.logging.internal.LogService in project neo4j by neo4j.

the class StoreUpgraderInterruptionTestIT method shouldSucceedWithUpgradeAfterPreviousAttemptDiedDuringMigration.

@Test
public void shouldSucceedWithUpgradeAfterPreviousAttemptDiedDuringMigration() throws IOException, ConsistencyCheckIncompleteException {
    MigrationTestUtils.prepareSampleLegacyDatabase(version, fs, workingDatabaseLayout.databaseDirectory(), prepareDirectory);
    RecordStoreVersionCheck versionCheck = new RecordStoreVersionCheck(fs, pageCache, workingDatabaseLayout, NullLogProvider.getInstance(), Config.defaults(), NULL);
    MigrationProgressMonitor progressMonitor = MigrationProgressMonitor.SILENT;
    LogService logService = NullLogService.getInstance();
    RecordStorageMigrator failingStoreMigrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, NULL, batchImporterFactory, INSTANCE) {

        @Override
        public void migrate(DatabaseLayout directoryLayout, DatabaseLayout migrationLayout, ProgressReporter progressReporter, String versionToMigrateFrom, String versionToMigrateTo, IndexImporterFactory indexImporterFactory) throws IOException, KernelException {
            super.migrate(directoryLayout, migrationLayout, progressReporter, versionToMigrateFrom, versionToMigrateTo, indexImporterFactory);
            throw new RuntimeException("This upgrade is failing");
        }
    };
    try {
        newUpgrader(versionCheck, progressMonitor, createIndexMigrator(), failingStoreMigrator).migrateIfNeeded(workingDatabaseLayout, false);
        fail("Should throw exception");
    } catch (RuntimeException e) {
        assertEquals("This upgrade is failing", e.getMessage());
    }
    assertTrue(checkNeoStoreHasFormatVersion(versionCheck, baselineFormat));
    RecordStorageMigrator migrator = new RecordStorageMigrator(fs, pageCache, CONFIG, logService, jobScheduler, NULL, batchImporterFactory, INSTANCE);
    IdGeneratorMigrator idMigrator = new IdGeneratorMigrator(fs, pageCache, CONFIG, NULL);
    SchemaIndexMigrator indexMigrator = createIndexMigrator();
    newUpgrader(versionCheck, progressMonitor, indexMigrator, migrator, idMigrator).migrateIfNeeded(workingDatabaseLayout, false);
    assertTrue(checkNeoStoreHasFormatVersion(versionCheck, successorFormat));
    // Since consistency checker is in read only mode we need to start/stop db to generate label scan store.
    startStopDatabase(neo4jLayout.homeDirectory());
    assertConsistentStore(workingDatabaseLayout);
}
Also used : MigrationProgressMonitor(org.neo4j.storageengine.migration.MigrationProgressMonitor) RecordStoreVersionCheck(org.neo4j.kernel.impl.storemigration.RecordStoreVersionCheck) RecordStorageMigrator(org.neo4j.kernel.impl.storemigration.RecordStorageMigrator) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) IndexImporterFactory(org.neo4j.internal.batchimport.IndexImporterFactory) ProgressReporter(org.neo4j.common.ProgressReporter) IdGeneratorMigrator(org.neo4j.kernel.impl.storemigration.IdGeneratorMigrator) SchemaIndexMigrator(org.neo4j.storageengine.migration.SchemaIndexMigrator) NullLogService(org.neo4j.logging.internal.NullLogService) LogService(org.neo4j.logging.internal.LogService) Test(org.junit.Test)

Aggregations

LogService (org.neo4j.logging.internal.LogService)20 NullLogService (org.neo4j.logging.internal.NullLogService)15 Path (java.nio.file.Path)13 SimpleLogService (org.neo4j.logging.internal.SimpleLogService)11 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)8 MethodSource (org.junit.jupiter.params.provider.MethodSource)8 MigrationProgressMonitor (org.neo4j.storageengine.migration.MigrationProgressMonitor)8 Config (org.neo4j.configuration.Config)6 Log (org.neo4j.logging.Log)5 Test (org.junit.jupiter.api.Test)4 DatabaseManagementService (org.neo4j.dbms.api.DatabaseManagementService)4 ScanOnOpenOverwritingIdGeneratorFactory (org.neo4j.internal.id.ScanOnOpenOverwritingIdGeneratorFactory)4 StoreFactory (org.neo4j.kernel.impl.store.StoreFactory)4 Test (org.junit.Test)3 ProgressReporter (org.neo4j.common.ProgressReporter)3 DatabaseLayout (org.neo4j.io.layout.DatabaseLayout)3 EmbeddedChannel (io.netty.channel.embedded.EmbeddedChannel)2 IOException (java.io.IOException)2 Optional (java.util.Optional)2 BoltGraphDatabaseManagementServiceSPI (org.neo4j.bolt.dbapi.BoltGraphDatabaseManagementServiceSPI)2