Search in sources :

Example 1 with HadoopDruidDetermineConfigurationJob

use of org.apache.druid.indexer.HadoopDruidDetermineConfigurationJob in project druid by druid-io.

the class CliInternalHadoopIndexer method run.

@Override
public void run() {
    try {
        Injector injector = makeInjector();
        config = getHadoopDruidIndexerConfig();
        MetadataStorageUpdaterJobSpec metadataSpec = config.getSchema().getIOConfig().getMetadataUpdateSpec();
        // override metadata storage type based on HadoopIOConfig
        Preconditions.checkNotNull(metadataSpec.getType(), "type in metadataUpdateSpec must not be null");
        injector.getInstance(Properties.class).setProperty("druid.metadata.storage.type", metadataSpec.getType());
        HadoopIngestionSpec.updateSegmentListIfDatasourcePathSpecIsUsed(config.getSchema(), HadoopDruidIndexerConfig.JSON_MAPPER, new MetadataStoreBasedUsedSegmentsRetriever(injector.getInstance(IndexerMetadataStorageCoordinator.class)));
        List<Jobby> jobs = new ArrayList<>();
        HadoopDruidIndexerJob indexerJob = new HadoopDruidIndexerJob(config, injector.getInstance(MetadataStorageUpdaterJobHandler.class));
        jobs.add(new HadoopDruidDetermineConfigurationJob(config));
        jobs.add(indexerJob);
        boolean jobsSucceeded = JobHelper.runJobs(jobs);
        JobHelper.renameIndexFilesForSegments(config.getSchema(), indexerJob.getPublishedSegmentAndIndexZipFilePaths());
        JobHelper.maybeDeleteIntermediatePath(jobsSucceeded, config.getSchema());
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}
Also used : ArrayList(java.util.ArrayList) Properties(java.util.Properties) URISyntaxException(java.net.URISyntaxException) MetadataStorageUpdaterJobHandler(org.apache.druid.indexer.MetadataStorageUpdaterJobHandler) Jobby(org.apache.druid.indexer.Jobby) Injector(com.google.inject.Injector) HadoopDruidIndexerJob(org.apache.druid.indexer.HadoopDruidIndexerJob) MetadataStorageUpdaterJobSpec(org.apache.druid.indexer.updater.MetadataStorageUpdaterJobSpec) MetadataStoreBasedUsedSegmentsRetriever(org.apache.druid.indexer.path.MetadataStoreBasedUsedSegmentsRetriever) HadoopDruidDetermineConfigurationJob(org.apache.druid.indexer.HadoopDruidDetermineConfigurationJob)

Aggregations

Injector (com.google.inject.Injector)1 URISyntaxException (java.net.URISyntaxException)1 ArrayList (java.util.ArrayList)1 Properties (java.util.Properties)1 HadoopDruidDetermineConfigurationJob (org.apache.druid.indexer.HadoopDruidDetermineConfigurationJob)1 HadoopDruidIndexerJob (org.apache.druid.indexer.HadoopDruidIndexerJob)1 Jobby (org.apache.druid.indexer.Jobby)1 MetadataStorageUpdaterJobHandler (org.apache.druid.indexer.MetadataStorageUpdaterJobHandler)1 MetadataStoreBasedUsedSegmentsRetriever (org.apache.druid.indexer.path.MetadataStoreBasedUsedSegmentsRetriever)1 MetadataStorageUpdaterJobSpec (org.apache.druid.indexer.updater.MetadataStorageUpdaterJobSpec)1