Search in sources :

Example 1 with DBScanIterationsJobRunner

use of org.locationtech.geowave.analytic.mapreduce.dbscan.DBScanIterationsJobRunner in project geowave by locationtech.

the class DBScanCommand method computeResults.

@Override
public Void computeResults(final OperationParams params) throws Exception {
    final String inputStoreName = parameters.get(0);
    // Config file
    final File configFile = getGeoWaveConfigFile(params);
    if (commonOptions.getMapReduceHdfsHostPort() == null) {
        final Properties configProperties = ConfigOptions.loadProperties(configFile);
        final String hdfsFSUrl = ConfigHDFSCommand.getHdfsUrl(configProperties);
        commonOptions.setMapReduceHdfsHostPort(hdfsFSUrl);
    }
    // Attempt to load store.
    inputStoreOptions = CLIUtils.loadStore(inputStoreName, configFile, params.getConsole());
    // Save a reference to the store in the property management.
    final PersistableStore persistedStore = new PersistableStore(inputStoreOptions);
    final PropertyManagement properties = new PropertyManagement();
    properties.store(StoreParameters.StoreParam.INPUT_STORE, persistedStore);
    // Convert properties from DBScanOptions and CommonOptions
    final PropertyManagementConverter converter = new PropertyManagementConverter(properties);
    converter.readProperties(commonOptions);
    converter.readProperties(dbScanOptions);
    properties.store(Extract.QUERY, commonOptions.buildQuery());
    final DBScanIterationsJobRunner runner = new DBScanIterationsJobRunner();
    final int status = runner.run(properties);
    if (status != 0) {
        throw new RuntimeException("Failed to execute: " + status);
    }
    return null;
}
Also used : DBScanIterationsJobRunner(org.locationtech.geowave.analytic.mapreduce.dbscan.DBScanIterationsJobRunner) PropertyManagementConverter(org.locationtech.geowave.analytic.mapreduce.operations.options.PropertyManagementConverter) PersistableStore(org.locationtech.geowave.analytic.store.PersistableStore) PropertyManagement(org.locationtech.geowave.analytic.PropertyManagement) Properties(java.util.Properties) File(java.io.File)

Example 2 with DBScanIterationsJobRunner

use of org.locationtech.geowave.analytic.mapreduce.dbscan.DBScanIterationsJobRunner in project geowave by locationtech.

the class DBScanIT method runScan.

private void runScan(final QueryConstraints query) throws Exception {
    final DBScanIterationsJobRunner jobRunner = new DBScanIterationsJobRunner();
    final Configuration conf = MapReduceTestUtils.getConfiguration();
    final int res = jobRunner.run(conf, new PropertyManagement(new ParameterEnum[] { ExtractParameters.Extract.QUERY, ExtractParameters.Extract.MIN_INPUT_SPLIT, ExtractParameters.Extract.MAX_INPUT_SPLIT, PartitionParameters.Partition.MAX_DISTANCE, PartitionParameters.Partition.PARTITIONER_CLASS, ClusteringParameters.Clustering.MINIMUM_SIZE, StoreParam.INPUT_STORE, MapReduceParameters.MRConfig.HDFS_BASE_DIR, OutputParameters.Output.REDUCER_COUNT, InputParameters.Input.INPUT_FORMAT, GlobalParameters.Global.BATCH_ID, PartitionParameters.Partition.PARTITION_DECREASE_RATE, PartitionParameters.Partition.PARTITION_PRECISION }, new Object[] { QueryBuilder.newBuilder().constraints(query).build(), Integer.toString(MapReduceTestUtils.MIN_INPUT_SPLITS), Integer.toString(MapReduceTestUtils.MAX_INPUT_SPLITS), 10000.0, OrthodromicDistancePartitioner.class, 10, new PersistableStore(dataStorePluginOptions), TestUtils.TEMP_DIR + File.separator + MapReduceTestEnvironment.HDFS_BASE_DIRECTORY + "/t1", 2, GeoWaveInputFormatConfiguration.class, "bx5", 0.15, 0.95 }));
    Assert.assertEquals(0, res);
    Assert.assertTrue(readHulls() > 2);
    // for travis-ci to run, we want to limit the memory consumption
    System.gc();
}
Also used : ParameterEnum(org.locationtech.geowave.analytic.param.ParameterEnum) GeoWaveInputFormatConfiguration(org.locationtech.geowave.analytic.mapreduce.GeoWaveInputFormatConfiguration) DBScanIterationsJobRunner(org.locationtech.geowave.analytic.mapreduce.dbscan.DBScanIterationsJobRunner) GeoWaveInputFormatConfiguration(org.locationtech.geowave.analytic.mapreduce.GeoWaveInputFormatConfiguration) Configuration(org.apache.hadoop.conf.Configuration) OrthodromicDistancePartitioner(org.locationtech.geowave.analytic.partitioner.OrthodromicDistancePartitioner) PersistableStore(org.locationtech.geowave.analytic.store.PersistableStore) PropertyManagement(org.locationtech.geowave.analytic.PropertyManagement) Point(org.locationtech.jts.geom.Point)

Aggregations

PropertyManagement (org.locationtech.geowave.analytic.PropertyManagement)2 DBScanIterationsJobRunner (org.locationtech.geowave.analytic.mapreduce.dbscan.DBScanIterationsJobRunner)2 PersistableStore (org.locationtech.geowave.analytic.store.PersistableStore)2 File (java.io.File)1 Properties (java.util.Properties)1 Configuration (org.apache.hadoop.conf.Configuration)1 GeoWaveInputFormatConfiguration (org.locationtech.geowave.analytic.mapreduce.GeoWaveInputFormatConfiguration)1 PropertyManagementConverter (org.locationtech.geowave.analytic.mapreduce.operations.options.PropertyManagementConverter)1 ParameterEnum (org.locationtech.geowave.analytic.param.ParameterEnum)1 OrthodromicDistancePartitioner (org.locationtech.geowave.analytic.partitioner.OrthodromicDistancePartitioner)1 Point (org.locationtech.jts.geom.Point)1