Search in sources :

Example 1 with NamedThreadFactory

use of org.neo4j.internal.helpers.NamedThreadFactory in project neo4j by neo4j.

the class ParallelExecution method run.

private void run(String taskName, int numberOfThreads, ThrowingRunnable... runnables) throws Exception {
    var pool = Executors.newFixedThreadPool(numberOfThreads, new NamedThreadFactory(getClass().getSimpleName() + "-" + taskName));
    try {
        List<InternalTask> tasks = Arrays.stream(runnables).map(InternalTask::new).collect(Collectors.toList());
        Futures.getAllResults(pool.invokeAll(tasks));
    } finally {
        pool.shutdown();
    }
}
Also used : NamedThreadFactory(org.neo4j.internal.helpers.NamedThreadFactory)

Example 2 with NamedThreadFactory

use of org.neo4j.internal.helpers.NamedThreadFactory in project neo4j by neo4j.

the class DataImporter method importData.

private static long importData(String title, Configuration configuration, InputIterable data, BatchingNeoStores stores, Supplier<EntityImporter> visitors, ExecutionMonitor executionMonitor, StatsProvider memoryStatsProvider) throws IOException {
    LongAdder roughEntityCountProgress = new LongAdder();
    int numRunners = configuration.maxNumberOfProcessors();
    ExecutorService pool = Executors.newFixedThreadPool(numRunners, new NamedThreadFactory(title + "Importer"));
    IoMonitor writeMonitor = new IoMonitor(stores.getIoTracer());
    ControllableStep step = new ControllableStep(title, roughEntityCountProgress, configuration, writeMonitor, memoryStatsProvider);
    StageExecution execution = new StageExecution(title, null, configuration, Collections.singletonList(step), 0);
    long startTime = currentTimeMillis();
    List<EntityImporter> importers = new ArrayList<>();
    try (InputIterator dataIterator = data.iterator()) {
        executionMonitor.start(execution);
        for (int i = 0; i < numRunners; i++) {
            EntityImporter importer = visitors.get();
            importers.add(importer);
            pool.submit(new ExhaustingEntityImporterRunnable(execution, dataIterator, importer, roughEntityCountProgress));
        }
        pool.shutdown();
        long nextWait = 0;
        try {
            while (!pool.awaitTermination(nextWait, TimeUnit.MILLISECONDS)) {
                executionMonitor.check(execution);
                nextWait = executionMonitor.nextCheckTime() - currentTimeMillis();
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IOException(e);
        }
    }
    execution.assertHealthy();
    stores.markHighIds();
    importers.forEach(EntityImporter::freeUnusedIds);
    step.markAsCompleted();
    writeMonitor.stop();
    executionMonitor.end(execution, currentTimeMillis() - startTime);
    execution.assertHealthy();
    return roughEntityCountProgress.sum();
}
Also used : StageExecution(org.neo4j.internal.batchimport.staging.StageExecution) NamedThreadFactory(org.neo4j.internal.helpers.NamedThreadFactory) ArrayList(java.util.ArrayList) IOException(java.io.IOException) LongAdder(java.util.concurrent.atomic.LongAdder) ExecutorService(java.util.concurrent.ExecutorService) IoMonitor(org.neo4j.internal.batchimport.store.io.IoMonitor)

Aggregations

NamedThreadFactory (org.neo4j.internal.helpers.NamedThreadFactory)2 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 ExecutorService (java.util.concurrent.ExecutorService)1 LongAdder (java.util.concurrent.atomic.LongAdder)1 StageExecution (org.neo4j.internal.batchimport.staging.StageExecution)1 IoMonitor (org.neo4j.internal.batchimport.store.io.IoMonitor)1