Search in sources :

Example 1 with Pool

use of org.openstreetmap.atlas.utilities.threads.Pool in project atlas-checks by osmlab.

the class IntegrityCheckSparkJob method executeChecks.

/**
 * Executes all {@link BaseCheck}s on the given {@link Atlas}. Each check runs in a separate
 * thread. The checks go over all {@link AtlasEntity}s and {@link Relation}s.
 * {@link ComplexEntity}s can be processed by using the appropriate {@link Finder} and adding
 * them to the {@link Iterable} of objects.
 *
 * @param atlas
 *            the {@link Atlas} on which the checks will be run
 * @param checksToRun
 *            the set of {@link BaseCheck}s to execute
 * @param configuration
 *            {@link MapRouletteConfiguration} to create a new {@link MapRouletteClient}s
 */
@SuppressWarnings("rawtypes")
private static void executeChecks(final String country, final Atlas atlas, final Set<BaseCheck> checksToRun, final MapRouletteConfiguration configuration) {
    final Pool checkExecutionPool = new Pool(checksToRun.size(), "Check execution pool", POOL_DURATION_BEFORE_KILL);
    checksToRun.stream().filter(check -> check.validCheckForCountry(country)).forEach(check -> checkExecutionPool.queue(new RunnableCheck(country, check, new MultiIterable<>(atlas.items(), atlas.relations(), findComplexEntities(check, atlas)), MapRouletteClient.instance(configuration))));
    checkExecutionPool.close();
}
Also used : SparkJob(org.openstreetmap.atlas.generator.tools.spark.SparkJob) Atlas(org.openstreetmap.atlas.geography.atlas.Atlas) CommonConstants(org.openstreetmap.atlas.checks.constants.CommonConstants) CheckResourceLoader(org.openstreetmap.atlas.checks.base.CheckResourceLoader) SparkFilePath(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFilePath) LoggerFactory(org.slf4j.LoggerFactory) MapRouletteConfiguration(org.openstreetmap.atlas.checks.maproulette.MapRouletteConfiguration) Relation(org.openstreetmap.atlas.geography.atlas.items.Relation) Finder(org.openstreetmap.atlas.geography.atlas.items.complex.Finder) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) BaseCheck(org.openstreetmap.atlas.checks.base.BaseCheck) MapRouletteClient(org.openstreetmap.atlas.checks.maproulette.MapRouletteClient) AtlasEntity(org.openstreetmap.atlas.geography.atlas.items.AtlasEntity) Configuration(org.openstreetmap.atlas.utilities.configuration.Configuration) ComplexEntity(org.openstreetmap.atlas.geography.atlas.items.complex.ComplexEntity) StringConverter(org.openstreetmap.atlas.utilities.conversion.StringConverter) Map(java.util.Map) ConfigurationResolver(org.openstreetmap.atlas.checks.configuration.ConfigurationResolver) CheckFlagGeoJsonProcessor(org.openstreetmap.atlas.checks.event.CheckFlagGeoJsonProcessor) Duration(org.openstreetmap.atlas.utilities.scalars.Duration) Logger(org.slf4j.Logger) TaskContext(org.apache.spark.TaskContext) CoreException(org.openstreetmap.atlas.exception.CoreException) SparkFileOutput(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFileOutput) Set(java.util.Set) CommandMap(org.openstreetmap.atlas.utilities.runtime.CommandMap) CheckFlagFileProcessor(org.openstreetmap.atlas.checks.event.CheckFlagFileProcessor) MetricFileGenerator(org.openstreetmap.atlas.checks.event.MetricFileGenerator) Tuple2(scala.Tuple2) Collectors(java.util.stream.Collectors) JavaPairRDD(org.apache.spark.api.java.JavaPairRDD) Pool(org.openstreetmap.atlas.utilities.threads.Pool) Objects(java.util.Objects) Consumer(java.util.function.Consumer) FileSuffix(org.openstreetmap.atlas.streaming.resource.FileSuffix) List(java.util.List) Stream(java.util.stream.Stream) Iterables(org.openstreetmap.atlas.utilities.collections.Iterables) EventService(org.openstreetmap.atlas.checks.event.EventService) SparkFileHelper(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFileHelper) MultiIterable(org.openstreetmap.atlas.utilities.collections.MultiIterable) Optional(java.util.Optional) Rectangle(org.openstreetmap.atlas.geography.Rectangle) StringList(org.openstreetmap.atlas.utilities.collections.StringList) Time(org.openstreetmap.atlas.utilities.time.Time) Collections(java.util.Collections) Pool(org.openstreetmap.atlas.utilities.threads.Pool)

Example 2 with Pool

use of org.openstreetmap.atlas.utilities.threads.Pool in project atlas-checks by osmlab.

the class EventBusTest method testProcessCount.

private void testProcessCount(final int threadCount, final int eventCount) {
    final TestProcessor testProcessor = new TestProcessor();
    final EventBus eventBus = new EventBus();
    eventBus.register(testProcessor);
    // Send events
    final Pool threadPool = new Pool(threadCount, "Test event pool", Duration.ONE_MINUTE);
    for (int threadIndex = 0; threadIndex < threadCount; threadIndex++) {
        threadPool.queue(() -> {
            for (int index = 0; index < eventCount; index++) {
                eventBus.post(new TestEvent("test " + index));
            }
        });
    }
    threadPool.close();
    // Complete
    eventBus.post(new ShutdownEvent());
    // Validate
    Assert.assertEquals(eventCount * threadCount, testProcessor.getProcessCount());
    Assert.assertEquals(1, testProcessor.getCompleteCount());
}
Also used : Pool(org.openstreetmap.atlas.utilities.threads.Pool) EventBus(com.google.common.eventbus.EventBus)

Example 3 with Pool

use of org.openstreetmap.atlas.utilities.threads.Pool in project atlas-checks by osmlab.

the class EventServiceTest method testProcessCount.

private void testProcessCount(final int threadCount, final int eventCount) {
    final TestProcessor testProcessor = new TestProcessor();
    final EventService eventService = EventService.get("Test service for " + threadCount + "-" + eventCount);
    eventService.register(testProcessor);
    // Send events
    final Pool threadPool = new Pool(threadCount, "Test pool for " + threadCount + "-" + eventCount, Duration.ONE_MINUTE);
    for (int threadIndex = 0; threadIndex < threadCount; threadIndex++) {
        threadPool.queue(() -> {
            for (int index = 0; index < eventCount; index++) {
                eventService.post(new TestEvent("test " + index));
            }
        });
    }
    threadPool.close();
    // Complete
    eventService.complete();
    // Validate
    Assert.assertEquals(eventCount * threadCount, testProcessor.getProcessCount());
    Assert.assertEquals(1, testProcessor.getCompleteCount());
}
Also used : Pool(org.openstreetmap.atlas.utilities.threads.Pool)

Aggregations

Pool (org.openstreetmap.atlas.utilities.threads.Pool)3 EventBus (com.google.common.eventbus.EventBus)1 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Map (java.util.Map)1 Objects (java.util.Objects)1 Optional (java.util.Optional)1 Set (java.util.Set)1 Consumer (java.util.function.Consumer)1 Collectors (java.util.stream.Collectors)1 Stream (java.util.stream.Stream)1 TaskContext (org.apache.spark.TaskContext)1 JavaPairRDD (org.apache.spark.api.java.JavaPairRDD)1 BaseCheck (org.openstreetmap.atlas.checks.base.BaseCheck)1 CheckResourceLoader (org.openstreetmap.atlas.checks.base.CheckResourceLoader)1 ConfigurationResolver (org.openstreetmap.atlas.checks.configuration.ConfigurationResolver)1 CommonConstants (org.openstreetmap.atlas.checks.constants.CommonConstants)1 CheckFlagFileProcessor (org.openstreetmap.atlas.checks.event.CheckFlagFileProcessor)1