Search in sources :

Example 6 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class ApplicationPluginDriver method invokeDataParser.

@Override
public Publisher<Void> invokeDataParser(Datasource datasource, Date fromDate, Date toDate) {
    JsonNode jsonNode = _jsonFactory.fromJsonToJsonNode(datasource.getJsonConfig());
    if (jsonNode.isArray()) {
        ArrayNode arrayJson = jsonNode.toArrayNode();
        for (int i = 0; i < arrayJson.size(); i++) {
            JsonNode node = arrayJson.get(i);
            if (node.hasNonNull("title")) {
                node = node.toObjectNode().set("applicationName", node.get("title"));
            }
            _ingestionLogicSender.send(IngestionPayload.builder().ingestionId(UUID.randomUUID().toString()).datasourceId(datasource.getDatasourceId()).rawContent(node.toString()).contentId(Integer.toString(i)).tenantId(datasource.getTenantId()).datasourcePayload(_jsonFactory.createObjectNode().set(getName(), node.toObjectNode()).toMap()).parsingDate(toDate.getTime()).documentTypes(new String[] { getName() }).build());
        }
    }
    return Mono.empty();
}
Also used : JsonNode(io.openk9.json.api.JsonNode) ArrayNode(io.openk9.json.api.ArrayNode)

Example 7 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class DatasourceProcessor method process.

@Incoming("ingestion")
@Outgoing("ingestion-datasource")
public Uni<IngestionDatasourcePayload> process(Object obj) {
    JsonObject jsonObject = obj instanceof JsonObject ? (JsonObject) obj : new JsonObject(new String((byte[]) obj));
    long datasourceId = jsonObject.getLong("datasourceId");
    Uni<Datasource> datasourceUni = Datasource.findById(datasourceId);
    return Panache.withTransaction(() -> datasourceUni.flatMap(datasource -> EnrichPipeline.findByDatasourceId(datasource.getDatasourceId()).onItem().ifNull().continueWith(EnrichPipeline::new).flatMap(enrichPipeline -> {
        Uni<List<EnrichItem>> enrichItemUni;
        if (enrichPipeline.getEnrichPipelineId() != null) {
            enrichItemUni = EnrichItem.findByEnrichPipelineId(enrichPipeline.getEnrichPipelineId()).onItem().ifNull().continueWith(List::of);
        } else {
            enrichItemUni = Uni.createFrom().item(List.of());
        }
        return Uni.combine().all().unis(Tenant.findById(datasource.getTenantId()), enrichItemUni).combinedWith((tenantObj, enrichItemList) -> {
            Tenant tenant = (Tenant) tenantObj;
            IngestionPayload ingestionPayload = jsonObject.mapTo(IngestionPayload.class);
            ingestionPayload.setTenantId(tenant.getTenantId());
            DatasourceContext datasourceContext = DatasourceContext.of(datasource, tenant, enrichPipeline, enrichItemList);
            return IngestionDatasourcePayload.of(ingestionPayload, datasourceContext);
        });
    })).eventually(() -> Datasource.<Datasource>findById(datasourceId).flatMap(datasource -> {
        datasource.setLastIngestionDate(Instant.ofEpochMilli(jsonObject.getLong("parsingDate")));
        return datasource.persist();
    })));
}
Also used : Datasource(io.openk9.datasource.model.Datasource) Outgoing(org.eclipse.microprofile.reactive.messaging.Outgoing) Datasource(io.openk9.datasource.model.Datasource) EnrichItem(io.openk9.datasource.model.EnrichItem) Tenant(io.openk9.datasource.model.Tenant) Instant(java.time.Instant) Uni(io.smallrye.mutiny.Uni) DatasourceContext(io.openk9.datasource.processor.payload.DatasourceContext) List(java.util.List) IngestionDatasourcePayload(io.openk9.datasource.processor.payload.IngestionDatasourcePayload) IngestionPayload(io.openk9.datasource.processor.payload.IngestionPayload) EnrichPipeline(io.openk9.datasource.model.EnrichPipeline) JsonObject(io.vertx.core.json.JsonObject) ApplicationScoped(javax.enterprise.context.ApplicationScoped) Panache(io.quarkus.hibernate.reactive.panache.Panache) Incoming(org.eclipse.microprofile.reactive.messaging.Incoming) JsonObject(io.vertx.core.json.JsonObject) EnrichItem(io.openk9.datasource.model.EnrichItem) Uni(io.smallrye.mutiny.Uni) Tenant(io.openk9.datasource.model.Tenant) IngestionPayload(io.openk9.datasource.processor.payload.IngestionPayload) DatasourceContext(io.openk9.datasource.processor.payload.DatasourceContext) EnrichPipeline(io.openk9.datasource.model.EnrichPipeline) List(java.util.List) Incoming(org.eclipse.microprofile.reactive.messaging.Incoming) Outgoing(org.eclipse.microprofile.reactive.messaging.Outgoing)

Example 8 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class SuggestionsHTTPHandler method searchHitToResponseMono.

@Override
protected Mono<Object> searchHitToResponseMono(Tenant tenant, List<Datasource> datasourceList, PluginDriverDTOList pluginDriverDTOList, HttpServerRequest httpServerRequest, SearchRequest searchRequest, SearchResponse searchResponse) {
    Long suggestionCategoryId = searchRequest.getSuggestionCategoryId();
    Mono<List<SuggestionCategoryField>> suggestionCategoryFields;
    if (suggestionCategoryId == null) {
        suggestionCategoryFields = _datasourceClient.findSuggestionCategoryFieldsByTenantId(tenant.getTenantId());
    } else {
        suggestionCategoryFields = _datasourceClient.findSuggestionCategoryFieldsByTenantIdAndCategoryId(tenant.getTenantId(), suggestionCategoryId);
    }
    if (_enableEntityAggregation) {
        return _search.search(factory -> {
            org.elasticsearch.action.search.SearchRequest searchRequestEntity = factory.createSearchRequestEntity(tenant.getTenantId());
            Aggregations aggregations = searchResponse.getAggregations();
            CompositeAggregation compositeAggregation = aggregations.get("composite");
            List<? extends CompositeAggregation.Bucket> buckets = compositeAggregation.getBuckets();
            BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
            buckets.stream().map(bucket -> (String) bucket.getKey().get("entities.id")).filter(Objects::nonNull).distinct().forEach(entityId -> boolQueryBuilder.should(QueryBuilders.matchQuery("id", entityId)));
            if (_log.isDebugEnabled()) {
                _log.debug("entities query: " + boolQueryBuilder);
            }
            SearchSourceBuilder ssb = new SearchSourceBuilder();
            ssb.query(boolQueryBuilder);
            ssb.size(1000);
            ssb.fetchSource(new String[] { "name", "id", "type" }, null);
            return searchRequestEntity.source(ssb);
        }).flatMap(entityResponse -> suggestionCategoryFields.map(fields -> {
            Map<String, String[]> entityMap = new HashMap<>();
            for (SearchHit hit : entityResponse.getHits()) {
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                String name = (String) sourceAsMap.get("name");
                String type = (String) sourceAsMap.get("type");
                String entityId = (String) sourceAsMap.get("id");
                entityMap.put(entityId, new String[] { type, name });
            }
            return _getSuggestionsResponse(datasourceList, pluginDriverDTOList, searchRequest, searchResponse, fields, entityMap);
        }));
    } else {
        return suggestionCategoryFields.map(fields -> _getSuggestionsResponse(datasourceList, pluginDriverDTOList, searchRequest, searchResponse, fields, Map.of()));
    }
}
Also used : PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) HttpUtil(io.openk9.http.util.HttpUtil) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) LoggerFactory(org.slf4j.LoggerFactory) SuggestionsResponse(io.openk9.search.query.internal.response.SuggestionsResponse) QueryParser(io.openk9.search.api.query.QueryParser) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) SearchToken(io.openk9.search.api.query.SearchToken) JsonFactory(io.openk9.json.api.JsonFactory) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) SearchHit(org.elasticsearch.search.SearchHit) Tenant(io.openk9.model.Tenant) Aggregations(org.elasticsearch.search.aggregations.Aggregations) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) ReferencePolicy(org.osgi.service.component.annotations.ReferencePolicy) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) Objects(java.util.Objects) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) Base64(java.util.Base64) List(java.util.List) ReferencePolicyOption(org.osgi.service.component.annotations.ReferencePolicyOption) SuggestionCategoryField(io.openk9.model.SuggestionCategoryField) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) Search(io.openk9.search.client.api.Search) Tuple2(reactor.util.function.Tuple2) HashMap(java.util.HashMap) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) Component(org.osgi.service.component.annotations.Component) CompositeAggregation(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregation) CompositeAggregationBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) BiConsumer(java.util.function.BiConsumer) Aggregation(org.elasticsearch.search.aggregations.Aggregation) Activate(org.osgi.service.component.annotations.Activate) LinkedList(java.util.LinkedList) RouterHandler(io.openk9.http.web.RouterHandler) CompositeValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSourceBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Suggestions(io.openk9.search.query.internal.response.suggestions.Suggestions) AggregationBuilders(org.elasticsearch.search.aggregations.AggregationBuilders) Mono(reactor.core.publisher.Mono) TermsValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) ReferenceCardinality(org.osgi.service.component.annotations.ReferenceCardinality) SearchRequest(io.openk9.search.api.query.SearchRequest) SearchTokenizer(io.openk9.search.api.query.SearchTokenizer) Modified(org.osgi.service.component.annotations.Modified) Reference(org.osgi.service.component.annotations.Reference) SearchRequest(io.openk9.search.api.query.SearchRequest) CompositeAggregation(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregation) SearchHit(org.elasticsearch.search.SearchHit) Aggregations(org.elasticsearch.search.aggregations.Aggregations) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) List(java.util.List) LinkedList(java.util.LinkedList) Map(java.util.Map) HashMap(java.util.HashMap)

Example 9 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class SuggestionsHTTPHandler method customizeSearchSourceBuilderMono.

@Override
protected Mono<org.elasticsearch.action.search.SearchRequest> customizeSearchSourceBuilderMono(Tenant tenant, List<Datasource> datasources, SearchRequest searchRequest, List<PluginDriverDTO> documentTypeList, SearchSourceBuilder searchSourceBuilder, org.elasticsearch.action.search.SearchRequest elasticSearchQuery) {
    return Mono.defer(() -> {
        Long suggestionCategoryId = searchRequest.getSuggestionCategoryId();
        if (suggestionCategoryId == null) {
            return _datasourceClient.findSuggestionCategoryFieldsByTenantId(tenant.getTenantId());
        } else {
            return _datasourceClient.findSuggestionCategoryFieldsByTenantIdAndCategoryId(tenant.getTenantId(), suggestionCategoryId);
        }
    }).map(fields -> {
        if (!(fields == null || fields.isEmpty())) {
            Function<String, CompositeValuesSourceBuilder<?>> fieldToTerms = nameField -> new TermsValuesSourceBuilder(nameField).field(nameField).missingBucket(true);
            CompositeAggregationBuilder compositeAggregation = fields.stream().map(SuggestionCategoryField::getFieldName).map(fieldToTerms).collect(Collectors.collectingAndThen(Collectors.toList(), list -> AggregationBuilders.composite("composite", list)));
            String afterKey = searchRequest.getAfterKey();
            if (afterKey != null) {
                byte[] afterKeyDecoded = Base64.getDecoder().decode(afterKey);
                Map<String, Object> map = _jsonFactory.fromJsonMap(new String(afterKeyDecoded), Object.class);
                compositeAggregation.aggregateAfter(map);
            }
            int[] range = searchRequest.getRange();
            if (range != null && range.length == 2) {
                int size = range[1];
                compositeAggregation.size(size);
            }
            searchSourceBuilder.aggregation(compositeAggregation);
        }
        searchSourceBuilder.from(0);
        searchSourceBuilder.size(0);
        searchSourceBuilder.highlighter(null);
        return elasticSearchQuery.source(searchSourceBuilder);
    });
}
Also used : PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) HttpUtil(io.openk9.http.util.HttpUtil) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) LoggerFactory(org.slf4j.LoggerFactory) SuggestionsResponse(io.openk9.search.query.internal.response.SuggestionsResponse) QueryParser(io.openk9.search.api.query.QueryParser) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) SearchToken(io.openk9.search.api.query.SearchToken) JsonFactory(io.openk9.json.api.JsonFactory) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) SearchHit(org.elasticsearch.search.SearchHit) Tenant(io.openk9.model.Tenant) Aggregations(org.elasticsearch.search.aggregations.Aggregations) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) ReferencePolicy(org.osgi.service.component.annotations.ReferencePolicy) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) Objects(java.util.Objects) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) Base64(java.util.Base64) List(java.util.List) ReferencePolicyOption(org.osgi.service.component.annotations.ReferencePolicyOption) SuggestionCategoryField(io.openk9.model.SuggestionCategoryField) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) Search(io.openk9.search.client.api.Search) Tuple2(reactor.util.function.Tuple2) HashMap(java.util.HashMap) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) Component(org.osgi.service.component.annotations.Component) CompositeAggregation(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregation) CompositeAggregationBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) BiConsumer(java.util.function.BiConsumer) Aggregation(org.elasticsearch.search.aggregations.Aggregation) Activate(org.osgi.service.component.annotations.Activate) LinkedList(java.util.LinkedList) RouterHandler(io.openk9.http.web.RouterHandler) CompositeValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSourceBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Suggestions(io.openk9.search.query.internal.response.suggestions.Suggestions) AggregationBuilders(org.elasticsearch.search.aggregations.AggregationBuilders) Mono(reactor.core.publisher.Mono) TermsValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) ReferenceCardinality(org.osgi.service.component.annotations.ReferenceCardinality) SearchRequest(io.openk9.search.api.query.SearchRequest) SearchTokenizer(io.openk9.search.api.query.SearchTokenizer) Modified(org.osgi.service.component.annotations.Modified) Reference(org.osgi.service.component.annotations.Reference) CompositeAggregationBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) TermsValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) CompositeValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSourceBuilder) SuggestionCategoryField(io.openk9.model.SuggestionCategoryField)

Example 10 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class DatasourceIdQueryParser method apply.

@Override
public Mono<Consumer<BoolQueryBuilder>> apply(Context context) {
    return Mono.fromSupplier(() -> {
        List<Datasource> datasources = context.getDatasourceList();
        long[] ids = new long[datasources.size()];
        for (int i = 0; i < datasources.size(); i++) {
            Datasource datasource = datasources.get(i);
            ids[i] = datasource.getDatasourceId();
        }
        return (bool) -> bool.filter(QueryBuilders.termsQuery("datasourceId", ids));
    });
}
Also used : Datasource(io.openk9.model.Datasource) Consumer(java.util.function.Consumer) Component(org.osgi.service.component.annotations.Component) List(java.util.List) Datasource(io.openk9.model.Datasource) Mono(reactor.core.publisher.Mono) QueryParser(io.openk9.search.api.query.QueryParser) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) QueryBuilders(org.elasticsearch.index.query.QueryBuilders)

Aggregations

Datasource (io.openk9.model.Datasource)12 Mono (reactor.core.publisher.Mono)11 List (java.util.List)9 Activate (org.osgi.service.component.annotations.Activate)9 Component (org.osgi.service.component.annotations.Component)9 Logger (org.slf4j.Logger)9 LoggerFactory (org.slf4j.LoggerFactory)9 JsonFactory (io.openk9.json.api.JsonFactory)8 PluginDriverManagerClient (io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient)8 Map (java.util.Map)8 Reference (org.osgi.service.component.annotations.Reference)8 PluginDriverDTO (io.openk9.plugin.driver.manager.model.PluginDriverDTO)7 Collectors (java.util.stream.Collectors)7 DatasourceClient (io.openk9.datasource.client.api.DatasourceClient)6 RouterHandler (io.openk9.http.web.RouterHandler)6 Tenant (io.openk9.model.Tenant)6 HashMap (java.util.HashMap)6 BoolQueryBuilder (org.elasticsearch.index.query.BoolQueryBuilder)6 QueryBuilders (org.elasticsearch.index.query.QueryBuilders)6 HttpResponseWriter (io.openk9.http.util.HttpResponseWriter)5