Search in sources :

Example 1 with RedisDataBus

use of org.icij.datashare.com.RedisDataBus in project datashare by ICIJ.

the class CommonMode method configure.

@Override
protected void configure() {
    bind(PropertiesProvider.class).toInstance(propertiesProvider);
    bind(LanguageGuesser.class).to(OptimaizeLanguageGuesser.class);
    String batchQueueType = propertiesProvider.get("batchQueueType").orElse("org.icij.datashare.extract.MemoryBlockingQueue");
    bind(new TypeLiteral<BlockingQueue<String>>() {
    }).toInstance(getBlockingQueue(propertiesProvider, batchQueueType, "ds:batchsearch:queue"));
    bind(new TypeLiteral<BlockingQueue<BatchDownload>>() {
    }).toInstance(getBlockingQueue(propertiesProvider, batchQueueType, "ds:batchdownload:queue"));
    RestHighLevelClient esClient = createESClient(propertiesProvider);
    bind(RestHighLevelClient.class).toInstance(esClient);
    bind(Indexer.class).to(ElasticsearchIndexer.class).asEagerSingleton();
    bind(TaskManagerMemory.class).toInstance(new TaskManagerMemory(propertiesProvider));
    install(new FactoryModuleBuilder().build(TaskFactory.class));
    if ("memory".equals(propertiesProvider.getProperties().get("queueType"))) {
        bind(DocumentCollectionFactory.class).to(MemoryDocumentCollectionFactory.class).asEagerSingleton();
    } else {
        install(new FactoryModuleBuilder().implement(DocumentQueue.class, RedisUserDocumentQueue.class).implement(ReportMap.class, RedisUserReportMap.class).build(DocumentCollectionFactory.class));
    }
    DataBus dataBus;
    if ("memory".equals(propertiesProvider.getProperties().get("busType"))) {
        dataBus = new MemoryDataBus();
    } else {
        dataBus = new RedisDataBus(propertiesProvider);
    }
    bind(DataBus.class).toInstance(dataBus);
    bind(Publisher.class).toInstance(dataBus);
    PipelineRegistry pipelineRegistry = new PipelineRegistry(propertiesProvider);
    pipelineRegistry.register(EmailPipeline.class);
    pipelineRegistry.register(Pipeline.Type.CORENLP);
    try {
        pipelineRegistry.load();
    } catch (FileNotFoundException e) {
        LoggerFactory.getLogger(getClass()).info("extensions dir not found " + e.getMessage());
    }
    bind(PipelineRegistry.class).toInstance(pipelineRegistry);
}
Also used : BatchDownload(org.icij.datashare.batch.BatchDownload) FactoryModuleBuilder(com.google.inject.assistedinject.FactoryModuleBuilder) FileNotFoundException(java.io.FileNotFoundException) MemoryDataBus(org.icij.datashare.com.MemoryDataBus) RedisUserDocumentQueue(org.icij.datashare.extract.RedisUserDocumentQueue) DocumentQueue(org.icij.extract.queue.DocumentQueue) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) DataBus(org.icij.datashare.com.DataBus) MemoryDataBus(org.icij.datashare.com.MemoryDataBus) RedisDataBus(org.icij.datashare.com.RedisDataBus) Publisher(org.icij.datashare.com.Publisher) TaskManagerMemory(org.icij.datashare.tasks.TaskManagerMemory) PipelineRegistry(org.icij.datashare.extension.PipelineRegistry) PropertiesProvider(org.icij.datashare.PropertiesProvider) MemoryDocumentCollectionFactory(org.icij.datashare.tasks.MemoryDocumentCollectionFactory) TypeLiteral(com.google.inject.TypeLiteral) MemoryDocumentCollectionFactory(org.icij.datashare.tasks.MemoryDocumentCollectionFactory) DocumentCollectionFactory(org.icij.datashare.tasks.DocumentCollectionFactory) TaskFactory(org.icij.datashare.tasks.TaskFactory) RedisDataBus(org.icij.datashare.com.RedisDataBus) OptimaizeLanguageGuesser(org.icij.datashare.nlp.OptimaizeLanguageGuesser) LanguageGuesser(org.icij.datashare.text.indexing.LanguageGuesser) ElasticsearchIndexer(org.icij.datashare.text.indexing.elasticsearch.ElasticsearchIndexer) RedisUserDocumentQueue(org.icij.datashare.extract.RedisUserDocumentQueue)

Aggregations

TypeLiteral (com.google.inject.TypeLiteral)1 FactoryModuleBuilder (com.google.inject.assistedinject.FactoryModuleBuilder)1 FileNotFoundException (java.io.FileNotFoundException)1 RestHighLevelClient (org.elasticsearch.client.RestHighLevelClient)1 PropertiesProvider (org.icij.datashare.PropertiesProvider)1 BatchDownload (org.icij.datashare.batch.BatchDownload)1 DataBus (org.icij.datashare.com.DataBus)1 MemoryDataBus (org.icij.datashare.com.MemoryDataBus)1 Publisher (org.icij.datashare.com.Publisher)1 RedisDataBus (org.icij.datashare.com.RedisDataBus)1 PipelineRegistry (org.icij.datashare.extension.PipelineRegistry)1 RedisUserDocumentQueue (org.icij.datashare.extract.RedisUserDocumentQueue)1 OptimaizeLanguageGuesser (org.icij.datashare.nlp.OptimaizeLanguageGuesser)1 DocumentCollectionFactory (org.icij.datashare.tasks.DocumentCollectionFactory)1 MemoryDocumentCollectionFactory (org.icij.datashare.tasks.MemoryDocumentCollectionFactory)1 TaskFactory (org.icij.datashare.tasks.TaskFactory)1 TaskManagerMemory (org.icij.datashare.tasks.TaskManagerMemory)1 LanguageGuesser (org.icij.datashare.text.indexing.LanguageGuesser)1 ElasticsearchIndexer (org.icij.datashare.text.indexing.elasticsearch.ElasticsearchIndexer)1 DocumentQueue (org.icij.extract.queue.DocumentQueue)1