Search in sources :

Example 1 with MultiIterable

use of org.openstreetmap.atlas.utilities.collections.MultiIterable in project atlas-checks by osmlab.

the class IntegrityCheckSparkJob method executeChecks.

/**
 * Executes all {@link BaseCheck}s on the given {@link Atlas}. Each check runs in a separate
 * thread. The checks go over all {@link AtlasEntity}s and {@link Relation}s.
 * {@link ComplexEntity}s can be processed by using the appropriate {@link Finder} and adding
 * them to the {@link Iterable} of objects.
 *
 * @param atlas
 *            the {@link Atlas} on which the checks will be run
 * @param checksToRun
 *            the set of {@link BaseCheck}s to execute
 * @param configuration
 *            {@link MapRouletteConfiguration} to create a new {@link MapRouletteClient}s
 */
@SuppressWarnings("rawtypes")
private static void executeChecks(final String country, final Atlas atlas, final Set<BaseCheck> checksToRun, final MapRouletteConfiguration configuration) {
    final Pool checkExecutionPool = new Pool(checksToRun.size(), "Check execution pool", POOL_DURATION_BEFORE_KILL);
    checksToRun.stream().filter(check -> check.validCheckForCountry(country)).forEach(check -> checkExecutionPool.queue(new RunnableCheck(country, check, new MultiIterable<>(atlas.items(), atlas.relations(), findComplexEntities(check, atlas)), MapRouletteClient.instance(configuration))));
    checkExecutionPool.close();
}
Also used : SparkJob(org.openstreetmap.atlas.generator.tools.spark.SparkJob) Atlas(org.openstreetmap.atlas.geography.atlas.Atlas) CommonConstants(org.openstreetmap.atlas.checks.constants.CommonConstants) CheckResourceLoader(org.openstreetmap.atlas.checks.base.CheckResourceLoader) SparkFilePath(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFilePath) LoggerFactory(org.slf4j.LoggerFactory) MapRouletteConfiguration(org.openstreetmap.atlas.checks.maproulette.MapRouletteConfiguration) Relation(org.openstreetmap.atlas.geography.atlas.items.Relation) Finder(org.openstreetmap.atlas.geography.atlas.items.complex.Finder) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) BaseCheck(org.openstreetmap.atlas.checks.base.BaseCheck) MapRouletteClient(org.openstreetmap.atlas.checks.maproulette.MapRouletteClient) AtlasEntity(org.openstreetmap.atlas.geography.atlas.items.AtlasEntity) Configuration(org.openstreetmap.atlas.utilities.configuration.Configuration) ComplexEntity(org.openstreetmap.atlas.geography.atlas.items.complex.ComplexEntity) StringConverter(org.openstreetmap.atlas.utilities.conversion.StringConverter) Map(java.util.Map) ConfigurationResolver(org.openstreetmap.atlas.checks.configuration.ConfigurationResolver) CheckFlagGeoJsonProcessor(org.openstreetmap.atlas.checks.event.CheckFlagGeoJsonProcessor) Duration(org.openstreetmap.atlas.utilities.scalars.Duration) Logger(org.slf4j.Logger) TaskContext(org.apache.spark.TaskContext) CoreException(org.openstreetmap.atlas.exception.CoreException) SparkFileOutput(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFileOutput) Set(java.util.Set) CommandMap(org.openstreetmap.atlas.utilities.runtime.CommandMap) CheckFlagFileProcessor(org.openstreetmap.atlas.checks.event.CheckFlagFileProcessor) MetricFileGenerator(org.openstreetmap.atlas.checks.event.MetricFileGenerator) Tuple2(scala.Tuple2) Collectors(java.util.stream.Collectors) JavaPairRDD(org.apache.spark.api.java.JavaPairRDD) Pool(org.openstreetmap.atlas.utilities.threads.Pool) Objects(java.util.Objects) Consumer(java.util.function.Consumer) FileSuffix(org.openstreetmap.atlas.streaming.resource.FileSuffix) List(java.util.List) Stream(java.util.stream.Stream) Iterables(org.openstreetmap.atlas.utilities.collections.Iterables) EventService(org.openstreetmap.atlas.checks.event.EventService) SparkFileHelper(org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFileHelper) MultiIterable(org.openstreetmap.atlas.utilities.collections.MultiIterable) Optional(java.util.Optional) Rectangle(org.openstreetmap.atlas.geography.Rectangle) StringList(org.openstreetmap.atlas.utilities.collections.StringList) Time(org.openstreetmap.atlas.utilities.time.Time) Collections(java.util.Collections) Pool(org.openstreetmap.atlas.utilities.threads.Pool)

Aggregations

ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Map (java.util.Map)1 Objects (java.util.Objects)1 Optional (java.util.Optional)1 Set (java.util.Set)1 Consumer (java.util.function.Consumer)1 Collectors (java.util.stream.Collectors)1 Stream (java.util.stream.Stream)1 TaskContext (org.apache.spark.TaskContext)1 JavaPairRDD (org.apache.spark.api.java.JavaPairRDD)1 BaseCheck (org.openstreetmap.atlas.checks.base.BaseCheck)1 CheckResourceLoader (org.openstreetmap.atlas.checks.base.CheckResourceLoader)1 ConfigurationResolver (org.openstreetmap.atlas.checks.configuration.ConfigurationResolver)1 CommonConstants (org.openstreetmap.atlas.checks.constants.CommonConstants)1 CheckFlagFileProcessor (org.openstreetmap.atlas.checks.event.CheckFlagFileProcessor)1 CheckFlagGeoJsonProcessor (org.openstreetmap.atlas.checks.event.CheckFlagGeoJsonProcessor)1 EventService (org.openstreetmap.atlas.checks.event.EventService)1