Search in sources :

Example 36 with RecordFormats

use of org.neo4j.kernel.impl.store.format.RecordFormats in project neo4j by neo4j.

the class MultipleIndexPopulationStressIT method createRandomData.

private void createRandomData(long nodeCount, long relCount) throws Exception {
    Config config = Config.defaults(neo4j_home, directory.homePath());
    RecordFormats recordFormats = RecordFormatSelector.selectForConfig(config, NullLogProvider.getInstance());
    try (RandomDataInput input = new RandomDataInput(nodeCount, relCount);
        JobScheduler jobScheduler = new ThreadPoolJobScheduler()) {
        DatabaseLayout layout = Neo4jLayout.of(directory.homePath()).databaseLayout(DEFAULT_DATABASE_NAME);
        IndexImporterFactory indexImporterFactory = new IndexImporterFactoryImpl(config);
        BatchImporter importer = new ParallelBatchImporter(layout, fileSystemAbstraction, PageCacheTracer.NULL, DEFAULT, NullLogService.getInstance(), ExecutionMonitor.INVISIBLE, EMPTY, config, recordFormats, NO_MONITOR, jobScheduler, Collector.EMPTY, TransactionLogInitializer.getLogFilesInitializer(), indexImporterFactory, INSTANCE);
        importer.doImport(input);
    }
}
Also used : JobScheduler(org.neo4j.scheduler.JobScheduler) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler) ParallelBatchImporter(org.neo4j.internal.batchimport.ParallelBatchImporter) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) BatchImporter(org.neo4j.internal.batchimport.BatchImporter) ParallelBatchImporter(org.neo4j.internal.batchimport.ParallelBatchImporter) Config(org.neo4j.configuration.Config) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) IndexImporterFactory(org.neo4j.internal.batchimport.IndexImporterFactory) IndexImporterFactoryImpl(org.neo4j.kernel.impl.index.schema.IndexImporterFactoryImpl) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler)

Example 37 with RecordFormats

use of org.neo4j.kernel.impl.store.format.RecordFormats in project neo4j by neo4j.

the class IdGeneratorMigrator method createEmptyPlaceHolderStoreFiles.

private Set<Path> createEmptyPlaceHolderStoreFiles(DatabaseLayout layout, RecordFormats format) {
    Set<Path> createdStores = new HashSet<>();
    StoreType[] storesToCreate = Stream.of(StoreType.values()).filter(t -> {
        Path file = layout.file(t.getDatabaseFile());
        boolean exists = fileSystem.fileExists(file);
        if (!exists) {
            createdStores.add(file);
        }
        return !exists;
    }).toArray(StoreType[]::new);
    createStoreFactory(layout, format, new ScanOnOpenReadOnlyIdGeneratorFactory()).openNeoStores(true, storesToCreate).close();
    return createdStores;
}
Also used : Path(java.nio.file.Path) StoreType(org.neo4j.kernel.impl.store.StoreType) Sets.immutable(org.eclipse.collections.api.factory.Sets.immutable) LongSupplier(java.util.function.LongSupplier) IdGeneratorFactory(org.neo4j.internal.id.IdGeneratorFactory) ProgressReporter(org.neo4j.common.ProgressReporter) CursorContext(org.neo4j.io.pagecache.context.CursorContext) Config(org.neo4j.configuration.Config) NullLogProvider(org.neo4j.logging.NullLogProvider) DatabaseLayout(org.neo4j.io.layout.DatabaseLayout) ArrayList(java.util.ArrayList) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) HashSet(java.util.HashSet) ScanOnOpenReadOnlyIdGeneratorFactory(org.neo4j.internal.id.ScanOnOpenReadOnlyIdGeneratorFactory) NeoStores(org.neo4j.kernel.impl.store.NeoStores) Iterables(org.neo4j.internal.helpers.collection.Iterables) IdGenerator(org.neo4j.internal.id.IdGenerator) PageCacheTracer(org.neo4j.io.pagecache.tracing.PageCacheTracer) Path(java.nio.file.Path) AbstractStoreMigrationParticipant(org.neo4j.storageengine.migration.AbstractStoreMigrationParticipant) DatabaseFile(org.neo4j.io.layout.DatabaseFile) PageCache(org.neo4j.io.pagecache.PageCache) OpenOption(java.nio.file.OpenOption) DatabaseReadOnlyChecker.writable(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker.writable) Set(java.util.Set) RecordFormatSelector.selectForVersion(org.neo4j.kernel.impl.store.format.RecordFormatSelector.selectForVersion) GBPTREE_ID_FILES(org.neo4j.kernel.impl.store.format.RecordStorageCapability.GBPTREE_ID_FILES) IOException(java.io.IOException) MOVE(org.neo4j.kernel.impl.storemigration.FileOperation.MOVE) StoreMigratorFileOperation.fileOperation(org.neo4j.kernel.impl.storemigration.StoreMigratorFileOperation.fileOperation) IndexImporterFactory(org.neo4j.internal.batchimport.IndexImporterFactory) IdType(org.neo4j.internal.id.IdType) RecoveryCleanupWorkCollector.immediate(org.neo4j.index.internal.gbptree.RecoveryCleanupWorkCollector.immediate) List(java.util.List) Stream(java.util.stream.Stream) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) StoreType(org.neo4j.kernel.impl.store.StoreType) ImmutableSet(org.eclipse.collections.api.set.ImmutableSet) DatabaseReadOnlyChecker(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker) StoreFactory(org.neo4j.kernel.impl.store.StoreFactory) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) ScanOnOpenReadOnlyIdGeneratorFactory(org.neo4j.internal.id.ScanOnOpenReadOnlyIdGeneratorFactory) HashSet(java.util.HashSet)

Example 38 with RecordFormats

use of org.neo4j.kernel.impl.store.format.RecordFormats in project neo4j by neo4j.

the class EstimationSanityCheckerTest method shouldWarnAboutCountHigherThanCapacity.

@Test
void shouldWarnAboutCountHigherThanCapacity() {
    // given
    RecordFormats formats = Standard.LATEST_RECORD_FORMATS;
    ImportLogic.Monitor monitor = mock(ImportLogic.Monitor.class);
    Input.Estimates estimates = Input.knownEstimates(formats.node().getMaxId() * 2, formats.relationship().getMaxId() * 2, 0, 0, 0, 0, // we don't care about the rest of the estimates in this checking
    0);
    // when
    new EstimationSanityChecker(formats, monitor).sanityCheck(estimates);
    // then
    verify(monitor).mayExceedNodeIdCapacity(formats.node().getMaxId(), estimates.numberOfNodes());
    verify(monitor).mayExceedRelationshipIdCapacity(formats.relationship().getMaxId(), estimates.numberOfRelationships());
}
Also used : ImportLogic(org.neo4j.internal.batchimport.ImportLogic) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) Test(org.junit.jupiter.api.Test)

Example 39 with RecordFormats

use of org.neo4j.kernel.impl.store.format.RecordFormats in project neo4j by neo4j.

the class BatchingNeoStoresTest method shouldNotOpenStoreWithNodesOrRelationshipsInIt.

@Test
void shouldNotOpenStoreWithNodesOrRelationshipsInIt() throws Throwable {
    Config config = Config.defaults();
    // GIVEN
    someDataInTheDatabase(config);
    // WHEN
    DirectoryNotEmptyException exception = assertThrows(DirectoryNotEmptyException.class, () -> {
        try (JobScheduler jobScheduler = new ThreadPoolJobScheduler()) {
            RecordFormats recordFormats = selectForConfig(Config.defaults(), NullLogProvider.getInstance());
            try (BatchingNeoStores store = batchingNeoStores(fileSystem, databaseLayout, recordFormats, Configuration.DEFAULT, NullLogService.getInstance(), EMPTY, Config.defaults(), jobScheduler, PageCacheTracer.NULL, INSTANCE)) {
                store.createNew();
            }
        }
    });
    assertThat(exception.getMessage()).contains("already contains");
}
Also used : JobScheduler(org.neo4j.scheduler.JobScheduler) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) ForcedSecondaryUnitRecordFormats(org.neo4j.kernel.impl.store.format.ForcedSecondaryUnitRecordFormats) Config(org.neo4j.configuration.Config) RecordFormatSelector.selectForConfig(org.neo4j.kernel.impl.store.format.RecordFormatSelector.selectForConfig) DirectoryNotEmptyException(java.nio.file.DirectoryNotEmptyException) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler) Test(org.junit.jupiter.api.Test)

Example 40 with RecordFormats

use of org.neo4j.kernel.impl.store.format.RecordFormats in project neo4j by neo4j.

the class BatchingNeoStoresTest method shouldRespectDbConfig.

@Test
void shouldRespectDbConfig() throws Exception {
    // GIVEN
    int size = 10;
    Config config = Config.newBuilder().set(GraphDatabaseInternalSettings.array_block_size, size).set(GraphDatabaseInternalSettings.string_block_size, size).build();
    // WHEN
    RecordFormats recordFormats = LATEST_RECORD_FORMATS;
    int headerSize = recordFormats.dynamic().getRecordHeaderSize();
    try (JobScheduler jobScheduler = new ThreadPoolJobScheduler();
        BatchingNeoStores store = batchingNeoStores(fileSystem, databaseLayout, recordFormats, Configuration.DEFAULT, NullLogService.getInstance(), EMPTY, config, jobScheduler, PageCacheTracer.NULL, INSTANCE)) {
        store.createNew();
        // THEN
        assertEquals(size + headerSize, store.getPropertyStore().getArrayStore().getRecordSize());
        assertEquals(size + headerSize, store.getPropertyStore().getStringStore().getRecordSize());
    }
}
Also used : JobScheduler(org.neo4j.scheduler.JobScheduler) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler) RecordFormats(org.neo4j.kernel.impl.store.format.RecordFormats) ForcedSecondaryUnitRecordFormats(org.neo4j.kernel.impl.store.format.ForcedSecondaryUnitRecordFormats) Config(org.neo4j.configuration.Config) RecordFormatSelector.selectForConfig(org.neo4j.kernel.impl.store.format.RecordFormatSelector.selectForConfig) ThreadPoolJobScheduler(org.neo4j.test.scheduler.ThreadPoolJobScheduler) Test(org.junit.jupiter.api.Test)

Aggregations

RecordFormats (org.neo4j.kernel.impl.store.format.RecordFormats)43 Test (org.junit.jupiter.api.Test)13 IOException (java.io.IOException)10 Config (org.neo4j.configuration.Config)8 PageCache (org.neo4j.io.pagecache.PageCache)8 DefaultIdGeneratorFactory (org.neo4j.internal.id.DefaultIdGeneratorFactory)7 NullLogProvider (org.neo4j.logging.NullLogProvider)7 Config (org.neo4j.kernel.configuration.Config)6 StoreFactory (org.neo4j.kernel.impl.store.StoreFactory)6 File (java.io.File)5 Path (java.nio.file.Path)5 FileSystemAbstraction (org.neo4j.io.fs.FileSystemAbstraction)5 NeoStores (org.neo4j.kernel.impl.store.NeoStores)5 ForcedSecondaryUnitRecordFormats (org.neo4j.kernel.impl.store.format.ForcedSecondaryUnitRecordFormats)5 ArrayList (java.util.ArrayList)4 Test (org.junit.Test)4 IdGeneratorFactory (org.neo4j.internal.id.IdGeneratorFactory)4 PagedFile (org.neo4j.io.pagecache.PagedFile)4 LogProvider (org.neo4j.logging.LogProvider)4 NoSuchFileException (java.nio.file.NoSuchFileException)3