Search in sources :

Example 11 with PluginDriverDTO

use of io.openk9.plugin.driver.manager.model.PluginDriverDTO in project openk9 by smclab.

the class SuggestionsHTTPHandler method customizeSearchSourceBuilderMono.

@Override
protected Mono<org.elasticsearch.action.search.SearchRequest> customizeSearchSourceBuilderMono(Tenant tenant, List<Datasource> datasources, SearchRequest searchRequest, List<PluginDriverDTO> documentTypeList, SearchSourceBuilder searchSourceBuilder, org.elasticsearch.action.search.SearchRequest elasticSearchQuery) {
    return Mono.defer(() -> {
        Long suggestionCategoryId = searchRequest.getSuggestionCategoryId();
        if (suggestionCategoryId == null) {
            return _datasourceClient.findSuggestionCategoryFieldsByTenantId(tenant.getTenantId());
        } else {
            return _datasourceClient.findSuggestionCategoryFieldsByTenantIdAndCategoryId(tenant.getTenantId(), suggestionCategoryId);
        }
    }).map(fields -> {
        if (!(fields == null || fields.isEmpty())) {
            Function<String, CompositeValuesSourceBuilder<?>> fieldToTerms = nameField -> new TermsValuesSourceBuilder(nameField).field(nameField).missingBucket(true);
            CompositeAggregationBuilder compositeAggregation = fields.stream().map(SuggestionCategoryField::getFieldName).map(fieldToTerms).collect(Collectors.collectingAndThen(Collectors.toList(), list -> AggregationBuilders.composite("composite", list)));
            String afterKey = searchRequest.getAfterKey();
            if (afterKey != null) {
                byte[] afterKeyDecoded = Base64.getDecoder().decode(afterKey);
                Map<String, Object> map = _jsonFactory.fromJsonMap(new String(afterKeyDecoded), Object.class);
                compositeAggregation.aggregateAfter(map);
            }
            int[] range = searchRequest.getRange();
            if (range != null && range.length == 2) {
                int size = range[1];
                compositeAggregation.size(size);
            }
            searchSourceBuilder.aggregation(compositeAggregation);
        }
        searchSourceBuilder.from(0);
        searchSourceBuilder.size(0);
        searchSourceBuilder.highlighter(null);
        return elasticSearchQuery.source(searchSourceBuilder);
    });
}
Also used : PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) HttpUtil(io.openk9.http.util.HttpUtil) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) LoggerFactory(org.slf4j.LoggerFactory) SuggestionsResponse(io.openk9.search.query.internal.response.SuggestionsResponse) QueryParser(io.openk9.search.api.query.QueryParser) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) SearchToken(io.openk9.search.api.query.SearchToken) JsonFactory(io.openk9.json.api.JsonFactory) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) SearchHit(org.elasticsearch.search.SearchHit) Tenant(io.openk9.model.Tenant) Aggregations(org.elasticsearch.search.aggregations.Aggregations) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) ReferencePolicy(org.osgi.service.component.annotations.ReferencePolicy) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) Objects(java.util.Objects) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) Base64(java.util.Base64) List(java.util.List) ReferencePolicyOption(org.osgi.service.component.annotations.ReferencePolicyOption) SuggestionCategoryField(io.openk9.model.SuggestionCategoryField) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) Search(io.openk9.search.client.api.Search) Tuple2(reactor.util.function.Tuple2) HashMap(java.util.HashMap) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) Component(org.osgi.service.component.annotations.Component) CompositeAggregation(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregation) CompositeAggregationBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) BiConsumer(java.util.function.BiConsumer) Aggregation(org.elasticsearch.search.aggregations.Aggregation) Activate(org.osgi.service.component.annotations.Activate) LinkedList(java.util.LinkedList) RouterHandler(io.openk9.http.web.RouterHandler) CompositeValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSourceBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Suggestions(io.openk9.search.query.internal.response.suggestions.Suggestions) AggregationBuilders(org.elasticsearch.search.aggregations.AggregationBuilders) Mono(reactor.core.publisher.Mono) TermsValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) ReferenceCardinality(org.osgi.service.component.annotations.ReferenceCardinality) SearchRequest(io.openk9.search.api.query.SearchRequest) SearchTokenizer(io.openk9.search.api.query.SearchTokenizer) Modified(org.osgi.service.component.annotations.Modified) Reference(org.osgi.service.component.annotations.Reference) CompositeAggregationBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) TermsValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) CompositeValuesSourceBuilder(org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSourceBuilder) SuggestionCategoryField(io.openk9.model.SuggestionCategoryField)

Example 12 with PluginDriverDTO

use of io.openk9.plugin.driver.manager.model.PluginDriverDTO in project openk9 by smclab.

the class JsEnrichProcessor method prepareRequestRawContent.

protected ObjectNode prepareRequestRawContent(ObjectNode objectNode, ObjectNode datasourceConfiguration, DatasourceContext context, PluginDriverDTO pluginDriverDTO) {
    JsonNode rawContentNode = objectNode.get(Constants.RAW_CONTENT);
    JsonNode codeNode = datasourceConfiguration.get(Constants.CODE);
    ObjectNode request = _jsonFactory.createObjectNode();
    request.put(Constants.CODE, codeNode);
    request.put(Constants.CONTENT, rawContentNode);
    JsonNode typeNode = objectNode.get(Constants.TYPE);
    ObjectNode datasourcePayload = _jsonFactory.createObjectNode();
    if (typeNode != null && typeNode.isArray()) {
        ArrayNode types = typeNode.toArrayNode();
        for (JsonNode typeJsonNode : types) {
            String type = typeJsonNode.asText();
            datasourcePayload.put(type, objectNode.get(type));
        }
    }
    request.put(Constants.DATASOURCE_PAYLOAD, datasourcePayload);
    request.put(Constants.TENANT_ID, context.getTenant().getTenantId());
    request.put(Constants.DATASOURCE_ID, context.getDatasource().getDatasourceId());
    request.put(Constants.CONTENT_ID, objectNode.get(Constants.CONTENT_ID));
    return request;
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode) JsonNode(io.openk9.json.api.JsonNode) ArrayNode(io.openk9.json.api.ArrayNode)

Example 13 with PluginDriverDTO

use of io.openk9.plugin.driver.manager.model.PluginDriverDTO in project openk9 by smclab.

the class IndexWriterEndpoins method _deleteDataDocuments.

private Publisher<Void> _deleteDataDocuments(HttpServerRequest httpServerRequest, HttpServerResponse httpServerResponse) {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    Mono<byte[]> body = ReactorNettyUtils.aggregateBodyAsByteArray(httpServerRequest);
    Mono<String> responseMono = body.map(bytes -> _jsonFactory.fromJson(bytes, DeleteEntitiesRequest.class)).flatMap(deleteEntitiesRequest -> {
        if (!deleteEntitiesRequest.getContentIds().isEmpty()) {
            BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
            for (String contentId : deleteEntitiesRequest.getContentIds()) {
                boolQuery.mustNot(matchQuery("contentId", contentId));
            }
            Mono<Datasource> datasourceMono = _datasourceClient.findDatasource(deleteEntitiesRequest.getDatasourceId());
            return datasourceMono.flatMap(datasource -> _pluginDriverManagerClient.getPluginDriver(datasource.getDriverServiceName()).map(pluginDriverDTO -> datasource.getTenantId() + "-" + pluginDriverDTO.getName() + "-data").map(DeleteByQueryRequest::new).map(deleteByQueryRequest -> deleteByQueryRequest.setQuery(boolQuery)).flatMap(deleteByQueryRequest -> Mono.<BulkByScrollResponse>create(sink -> restHighLevelClient.deleteByQueryAsync(deleteByQueryRequest, RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).map(Object::toString).doOnNext(_log::info).flatMap(response -> Mono.fromRunnable(() -> _sendCleanOrphanEntitiesRequest(datasource.getTenantId())).thenReturn(response)));
        }
        return Mono.empty();
    });
    return _httpResponseWriter.write(httpServerResponse, responseMono);
}
Also used : LoggerFactory(org.slf4j.LoggerFactory) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) JsonFactory(io.openk9.json.api.JsonFactory) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) IndexRequest(org.elasticsearch.action.index.IndexRequest) EventBus(io.vertx.core.eventbus.EventBus) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) CountRequest(org.elasticsearch.client.core.CountRequest) TimeValue(org.elasticsearch.core.TimeValue) SearchHit(org.elasticsearch.search.SearchHit) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Message(io.vertx.core.eventbus.Message) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Scroll(org.elasticsearch.search.Scroll) Collectors(java.util.stream.Collectors) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) QueryBuilders.matchQuery(org.elasticsearch.index.query.QueryBuilders.matchQuery) Builder(lombok.Builder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) MatchQueryBuilder(org.elasticsearch.index.query.MatchQueryBuilder) XContentType(org.elasticsearch.common.xcontent.XContentType) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) CountResponse(org.elasticsearch.client.core.CountResponse) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) WriteRequest(org.elasticsearch.action.support.WriteRequest) Component(org.osgi.service.component.annotations.Component) Operator(org.elasticsearch.index.query.Operator) Cancellable(org.elasticsearch.client.Cancellable) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) Activate(org.osgi.service.component.annotations.Activate) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse) RouterHandler(io.openk9.http.web.RouterHandler) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Mono(reactor.core.publisher.Mono) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Data(lombok.Data) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) AllArgsConstructor(lombok.AllArgsConstructor) Reference(org.osgi.service.component.annotations.Reference) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) NoArgsConstructor(lombok.NoArgsConstructor) Datasource(io.openk9.model.Datasource) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient)

Example 14 with PluginDriverDTO

use of io.openk9.plugin.driver.manager.model.PluginDriverDTO in project openk9 by smclab.

the class EnrichPipelineProcessor method _mapToEnrichProcessorContext.

private ObjectNode _mapToEnrichProcessorContext(Tuple3<DatasourceContext, ObjectNode, PluginDriverDTO> t3) {
    List<String> dependencies = t3.getT1().getEnrichItems().stream().map(EnrichItem::getServiceName).collect(Collectors.toList());
    ObjectNode objectNode = _jsonFactory.createObjectNode();
    objectNode.put("pluginDriver", _jsonFactory.fromObjectToJsonNode(t3.getT3()));
    objectNode.put("dependencies", _jsonFactory.fromObjectToJsonNode(dependencies));
    objectNode.put("datasourceContext", _jsonFactory.fromObjectToJsonNode(t3.getT1()));
    objectNode.put("payload", t3.getT2());
    return objectNode;
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode)

Example 15 with PluginDriverDTO

use of io.openk9.plugin.driver.manager.model.PluginDriverDTO in project openk9 by smclab.

the class EnrichPipelineProcessor method _adaptIngestionPayload.

private Tuple3<DatasourceContext, ObjectNode, PluginDriverDTO> _adaptIngestionPayload(DatasourceContext datasourceContext, ObjectNode ingestionPayload, PluginDriverDTO pluginDriverDTO) {
    ObjectNode newIngestionPayload = ingestionPayload;
    if (newIngestionPayload.hasNonNull(io.openk9.core.api.constant.Constants.DATASOURCE_PAYLOAD)) {
        JsonNode datasourcePayload = newIngestionPayload.remove(io.openk9.core.api.constant.Constants.DATASOURCE_PAYLOAD);
        if (datasourcePayload.isObject()) {
            ObjectNode jsonNodes = datasourcePayload.toObjectNode();
            for (Map.Entry<String, JsonNode> field : jsonNodes.fields()) {
                newIngestionPayload.set(field.getKey(), field.getValue());
            }
        }
    }
    if (!ingestionPayload.hasNonNull(Constants.DATASOURCE_NAME)) {
        newIngestionPayload = newIngestionPayload.deepCopy().put(Constants.DATASOURCE_NAME, pluginDriverDTO.getName());
    }
    if (ingestionPayload.hasNonNull(io.openk9.core.api.constant.Constants.TYPE) && !ingestionPayload.get(io.openk9.core.api.constant.Constants.TYPE).toArrayNode().isEmpty()) {
        return Tuples.of(datasourceContext, ingestionPayload, pluginDriverDTO);
    }
    DocumentTypeDTO documentType = pluginDriverDTO.getDefaultDocumentType();
    return Tuples.of(datasourceContext, newIngestionPayload.set(io.openk9.core.api.constant.Constants.TYPE, _jsonFactory.createArrayNode().add(documentType.getName())), pluginDriverDTO);
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode) JsonNode(io.openk9.json.api.JsonNode) Map(java.util.Map) DocumentTypeDTO(io.openk9.plugin.driver.manager.model.DocumentTypeDTO)

Aggregations

PluginDriverDTO (io.openk9.plugin.driver.manager.model.PluginDriverDTO)10 Collection (java.util.Collection)9 BoolQueryBuilder (org.elasticsearch.index.query.BoolQueryBuilder)8 SearchKeywordDTO (io.openk9.plugin.driver.manager.model.SearchKeywordDTO)7 List (java.util.List)7 Map (java.util.Map)7 Component (org.osgi.service.component.annotations.Component)7 Mono (reactor.core.publisher.Mono)7 ObjectNode (io.openk9.json.api.ObjectNode)6 Datasource (io.openk9.model.Datasource)6 DocumentTypeDTO (io.openk9.plugin.driver.manager.model.DocumentTypeDTO)6 Collectors (java.util.stream.Collectors)6 QueryBuilders (org.elasticsearch.index.query.QueryBuilders)6 JsonNode (io.openk9.json.api.JsonNode)5 PluginDriverManagerClient (io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient)5 QueryParser (io.openk9.search.api.query.QueryParser)5 HashMap (java.util.HashMap)5 DatasourceClient (io.openk9.datasource.client.api.DatasourceClient)4 RouterHandler (io.openk9.http.web.RouterHandler)4 JsonFactory (io.openk9.json.api.JsonFactory)4