Search in sources :

Example 1 with Response

use of io.openk9.search.query.internal.response.Response in project openk9 by smclab.

the class GetOrAddEntitiesConsumer method apply.

public Mono<ObjectNode> apply(ObjectNode objectNode) {
    return Mono.defer(() -> {
        ObjectNode datasourceContextJson = objectNode.get("datasourceContext").toObjectNode();
        long datasourceId = datasourceContextJson.get("datasource").get("datasourceId").asLong();
        long tenantId = datasourceContextJson.get("tenant").get("tenantId").asLong();
        JsonNode entities = objectNode.remove("entities");
        Mono<ArrayNode> entitiesField;
        if (entities.size() == 0) {
            entitiesField = Mono.just(_jsonFactory.createArrayNode());
        } else {
            ObjectNode responseJson = _jsonFactory.createObjectNode();
            responseJson.put("entities", entities);
            responseJson.put("tenantId", tenantId);
            responseJson.put("datasourceId", datasourceId);
            Request request = _jsonFactory.fromJson(responseJson.toString(), Request.class);
            List<RequestContext> requestContextList = request.getEntities().stream().map(entityRequest -> RequestContext.builder().current(entityRequest).tenantId(request.getTenantId()).datasourceId(request.getDatasourceId()).rest(request.getEntities().stream().filter(er -> er != entityRequest).collect(Collectors.toList())).build()).collect(Collectors.toList());
            Mono<List<EntityContext>> disambiguateListMono = GetOrAddEntities.stopWatch("disambiguate-all-entities", Flux.fromIterable(requestContextList).flatMap(requestContext -> GetOrAddEntities.stopWatch("disambiguate-" + requestContext.getCurrent().getName(), Mono.<EntityContext>create(fluxSink -> _startDisambiguation.disambiguate(requestContext, fluxSink)))).collectList());
            Mono<ResponseList> writeRelations = disambiguateListMono.flatMap(entityContexts -> GetOrAddEntities.stopWatch("write-relations", writeRelations(entityContexts)));
            Mono<ResponseList> responseListWrapper = _transactional ? _graphClient.makeTransactional(writeRelations) : writeRelations;
            entitiesField = responseListWrapper.map(responseListDTO -> {
                List<Response> responseList = responseListDTO.getResponse();
                ArrayNode entitiesArrayNode = entities.toArrayNode();
                ArrayNode arrayNode = _jsonFactory.createArrayNode();
                for (JsonNode node : entitiesArrayNode) {
                    Optional<Response> responseOptional = responseList.stream().filter(response -> node.get("tmpId").asLong() == response.getTmpId()).findFirst();
                    if (responseOptional.isPresent()) {
                        Entity entity = responseOptional.get().getEntity();
                        ObjectNode result = _jsonFactory.createObjectNode();
                        result.put("entityType", entity.getType());
                        result.put("id", entity.getId());
                        result.put("context", node.get("context"));
                        arrayNode.add(result);
                    }
                }
                return arrayNode;
            });
        }
        return entitiesField.map(entitiesArray -> {
            ObjectNode payload = objectNode.get("payload").toObjectNode();
            payload.set("entities", entitiesArray);
            return objectNode;
        }).timeout(Duration.ofSeconds(_timeout), Mono.error(new TimeoutException("timeout on entities count: " + entities.size() + " (Did not observe any item or terminal signal within " + Duration.ofSeconds(_timeout).toMillis() + "ms)")));
    });
}
Also used : Response(io.openk9.entity.manager.model.payload.Response) SenderReactor(io.openk9.ingestion.api.SenderReactor) Disposable(reactor.core.Disposable) JsonNode(io.openk9.json.api.JsonNode) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Tuple2(reactor.util.function.Tuple2) TimeoutException(java.util.concurrent.TimeoutException) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) GraphClient(io.openk9.relationship.graph.api.client.GraphClient) ArrayNode(io.openk9.json.api.ArrayNode) JsonFactory(io.openk9.json.api.JsonFactory) ArrayList(java.util.ArrayList) Component(org.osgi.service.component.annotations.Component) Statement(org.neo4j.cypherdsl.core.Statement) Duration(java.time.Duration) OutboundMessageFactory(io.openk9.ingestion.api.OutboundMessageFactory) Schedulers(reactor.core.scheduler.Schedulers) Request(io.openk9.entity.manager.model.payload.Request) Cypher(org.neo4j.cypherdsl.core.Cypher) Activate(org.osgi.service.component.annotations.Activate) Node(org.neo4j.cypherdsl.core.Node) Binding(io.openk9.ingestion.api.Binding) Logger(org.slf4j.Logger) Deactivate(org.osgi.service.component.annotations.Deactivate) Functions(org.neo4j.cypherdsl.core.Functions) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) Mono(reactor.core.publisher.Mono) Entity(io.openk9.entity.manager.model.Entity) Collectors(java.util.stream.Collectors) Flux(reactor.core.publisher.Flux) List(java.util.List) Stream(java.util.stream.Stream) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ReceiverReactor(io.openk9.ingestion.api.ReceiverReactor) Modified(org.osgi.service.component.annotations.Modified) Optional(java.util.Optional) ObjectNode(io.openk9.json.api.ObjectNode) Reference(org.osgi.service.component.annotations.Reference) Cypher.literalOf(org.neo4j.cypherdsl.core.Cypher.literalOf) Entity(io.openk9.entity.manager.model.Entity) ObjectNode(io.openk9.json.api.ObjectNode) Optional(java.util.Optional) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) Request(io.openk9.entity.manager.model.payload.Request) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) JsonNode(io.openk9.json.api.JsonNode) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ArrayList(java.util.ArrayList) List(java.util.List) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ArrayNode(io.openk9.json.api.ArrayNode) TimeoutException(java.util.concurrent.TimeoutException)

Example 2 with Response

use of io.openk9.search.query.internal.response.Response in project openk9 by smclab.

the class GetOrAddEntitiesConsumer method writeRelations.

public Mono<ResponseList> writeRelations(List<EntityContext> entityContext) {
    return Mono.defer(() -> {
        List<Statement> statementList = new ArrayList<>();
        for (EntityContext context : entityContext) {
            EntityRequest entityRequest = context.getEntityRequest();
            List<RelationRequest> relations = entityRequest.getRelations();
            if (relations == null || relations.isEmpty()) {
                continue;
            }
            Entity currentEntity = context.getEntity();
            List<Tuple2<String, Entity>> entityRelations = entityContext.stream().flatMap(entry -> {
                for (RelationRequest relation : relations) {
                    if (entry.getEntityRequest().getTmpId() == relation.getTo()) {
                        return Stream.of(Tuples.of(relation.getName(), entry.getEntity()));
                    }
                }
                return Stream.empty();
            }).collect(Collectors.toList());
            Node currentEntityNode = Cypher.node(currentEntity.getType()).named("a");
            List<Statement> currentStatementList = entityRelations.stream().map(t2 -> {
                Entity entityRelation = t2.getT2();
                Node entityRelationNode = Cypher.node(entityRelation.getType()).named("b");
                return Cypher.match(currentEntityNode, entityRelationNode).where(Functions.id(currentEntityNode).eq(literalOf(currentEntity.getId())).and(Functions.id(entityRelationNode).eq(literalOf(entityRelation.getId())))).merge(currentEntityNode.relationshipTo(entityRelationNode, t2.getT1())).build();
            }).collect(Collectors.toList());
            statementList.addAll(currentStatementList);
        }
        List<Response> response = entityContext.stream().map(context -> Response.builder().entity(Entity.builder().name(context.getEntity().getName()).id(context.getEntity().getId()).tenantId(context.getEntity().getTenantId()).type(context.getEntity().getType()).build()).tmpId(context.getEntityRequest().getTmpId()).build()).collect(Collectors.toList());
        if (statementList.size() > 1) {
            return _graphClient.write(Cypher.unionAll(statementList.toArray(new Statement[0]))).then(Mono.just(ResponseList.of("", response)));
        } else if (statementList.size() == 1) {
            return _graphClient.write(statementList.get(0)).then(Mono.just(ResponseList.of("", response)));
        } else {
            return Mono.just(ResponseList.of("", response));
        }
    });
}
Also used : Response(io.openk9.entity.manager.model.payload.Response) SenderReactor(io.openk9.ingestion.api.SenderReactor) Disposable(reactor.core.Disposable) JsonNode(io.openk9.json.api.JsonNode) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Tuple2(reactor.util.function.Tuple2) TimeoutException(java.util.concurrent.TimeoutException) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) GraphClient(io.openk9.relationship.graph.api.client.GraphClient) ArrayNode(io.openk9.json.api.ArrayNode) JsonFactory(io.openk9.json.api.JsonFactory) ArrayList(java.util.ArrayList) Component(org.osgi.service.component.annotations.Component) Statement(org.neo4j.cypherdsl.core.Statement) Duration(java.time.Duration) OutboundMessageFactory(io.openk9.ingestion.api.OutboundMessageFactory) Schedulers(reactor.core.scheduler.Schedulers) Request(io.openk9.entity.manager.model.payload.Request) Cypher(org.neo4j.cypherdsl.core.Cypher) Activate(org.osgi.service.component.annotations.Activate) Node(org.neo4j.cypherdsl.core.Node) Binding(io.openk9.ingestion.api.Binding) Logger(org.slf4j.Logger) Deactivate(org.osgi.service.component.annotations.Deactivate) Functions(org.neo4j.cypherdsl.core.Functions) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) Mono(reactor.core.publisher.Mono) Entity(io.openk9.entity.manager.model.Entity) Collectors(java.util.stream.Collectors) Flux(reactor.core.publisher.Flux) List(java.util.List) Stream(java.util.stream.Stream) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ReceiverReactor(io.openk9.ingestion.api.ReceiverReactor) Modified(org.osgi.service.component.annotations.Modified) Optional(java.util.Optional) ObjectNode(io.openk9.json.api.ObjectNode) Reference(org.osgi.service.component.annotations.Reference) Cypher.literalOf(org.neo4j.cypherdsl.core.Cypher.literalOf) Entity(io.openk9.entity.manager.model.Entity) Statement(org.neo4j.cypherdsl.core.Statement) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) JsonNode(io.openk9.json.api.JsonNode) ArrayNode(io.openk9.json.api.ArrayNode) Node(org.neo4j.cypherdsl.core.Node) ObjectNode(io.openk9.json.api.ObjectNode) ArrayList(java.util.ArrayList) Response(io.openk9.entity.manager.model.payload.Response) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) Tuple2(reactor.util.function.Tuple2)

Example 3 with Response

use of io.openk9.search.query.internal.response.Response in project openk9 by smclab.

the class ReindexDatasourceConsumer method activate.

@Activate
void activate() {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    _disposable = _datasourceEventConsumer.datasourceUpdateEvents().flatMap(datasource -> _pluginDriverManagerClient.getPluginDriver(datasource.getDriverServiceName()).onErrorResume(throwable -> {
        if (_log.isErrorEnabled()) {
            _log.error(throwable.getMessage());
        }
        return Mono.empty();
    }).map(pluginDriverDTO -> Tuples.of(datasource, datasource.getTenantId() + "-" + pluginDriverDTO.getName() + "-data"))).filterWhen(t2 -> Mono.create(sink -> restHighLevelClient.indices().existsAsync(new GetIndexRequest(t2.getT2()), RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).flatMap(t2 -> Mono.<GetSettingsResponse>create(sink -> restHighLevelClient.indices().getSettingsAsync(new GetSettingsRequest().indices(t2.getT2()).names("index.creation_date"), RequestOptions.DEFAULT, new ReactorActionListener<>(sink))).map(response -> Tuples.of(t2.getT1(), t2.getT2(), Instant.ofEpochMilli(Long.parseLong(response.getSetting(t2.getT2(), "index.creation_date")))))).log().filter(t3 -> t3.getT1().getLastIngestionDate().isBefore(t3.getT3())).flatMap(t3 -> Mono.<AcknowledgedResponse>create(sink -> restHighLevelClient.indices().deleteAsync(new DeleteIndexRequest(t3.getT2()), RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).onErrorContinue((throwable, ignore) -> {
        if (_log.isErrorEnabled()) {
            _log.error(throwable.getMessage());
        }
    }).subscribe();
}
Also used : ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) DatasourceEventConsumer(io.openk9.datasource.event.consumer.api.DatasourceEventConsumer) Logger(org.slf4j.Logger) Disposable(reactor.core.Disposable) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) Deactivate(org.osgi.service.component.annotations.Deactivate) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Mono(reactor.core.publisher.Mono) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Instant(java.time.Instant) Component(org.osgi.service.component.annotations.Component) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) Activate(org.osgi.service.component.annotations.Activate) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) Reference(org.osgi.service.component.annotations.Reference) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Activate(org.osgi.service.component.annotations.Activate)

Example 4 with Response

use of io.openk9.search.query.internal.response.Response in project openk9 by smclab.

the class DocumentTypesEndpoint method _mapToResponse.

private Object _mapToResponse(List<Tuple2<Datasource, PluginDriverDTO>> list) {
    Map<String, Collection<String>> response = new HashMap<>();
    for (Tuple2<Datasource, PluginDriverDTO> t2 : list) {
        PluginDriverDTO pluginDriver = t2.getT2();
        for (DocumentTypeDTO documentType : pluginDriver.getDocumentTypes()) {
            String name = documentType.getName();
            List<SearchKeywordDTO> searchKeywords = documentType.getSearchKeywords();
            List<String> keywords = searchKeywords.stream().map(SearchKeywordDTO::getKeyword).collect(Collectors.toList());
            Collection<String> prevKeywords = response.get(name);
            Set<String> combine = new HashSet<>(keywords);
            if (prevKeywords != null) {
                combine.addAll(prevKeywords);
            }
            response.put(name, combine);
        }
    }
    return response;
}
Also used : Datasource(io.openk9.model.Datasource) HashMap(java.util.HashMap) SearchKeywordDTO(io.openk9.plugin.driver.manager.model.SearchKeywordDTO) Collection(java.util.Collection) DocumentTypeDTO(io.openk9.plugin.driver.manager.model.DocumentTypeDTO) PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) HashSet(java.util.HashSet)

Example 5 with Response

use of io.openk9.search.query.internal.response.Response in project openk9 by smclab.

the class EntityNameHTTPHandler method _searchHitToResponse.

private Response _searchHitToResponse(SearchHits searchHits) {
    SearchHit[] hits = searchHits.getHits();
    List<Map<String, Object>> result = new ArrayList<>(hits.length);
    for (SearchHit hit : hits) {
        Map<String, Object> sourceAsMap = hit.getSourceAsMap();
        result.add(Map.of("name", sourceAsMap.get("name")));
    }
    TotalHits totalHits = searchHits.getTotalHits();
    return new Response(result, totalHits.value);
}
Also used : TotalHits(org.apache.lucene.search.TotalHits) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) SearchResponse(org.elasticsearch.action.search.SearchResponse) Response(io.openk9.search.query.internal.response.Response) SearchHit(org.elasticsearch.search.SearchHit) ArrayList(java.util.ArrayList) Map(java.util.Map)

Aggregations

ArrayList (java.util.ArrayList)7 Mono (reactor.core.publisher.Mono)7 JsonFactory (io.openk9.json.api.JsonFactory)6 List (java.util.List)6 Map (java.util.Map)6 Activate (org.osgi.service.component.annotations.Activate)6 Component (org.osgi.service.component.annotations.Component)6 Reference (org.osgi.service.component.annotations.Reference)6 HttpServerResponse (reactor.netty.http.server.HttpServerResponse)6 SearchHit (org.elasticsearch.search.SearchHit)5 Logger (org.slf4j.Logger)5 LoggerFactory (org.slf4j.LoggerFactory)5 Entity (io.openk9.entity.manager.model.Entity)4 Request (io.openk9.entity.manager.model.payload.Request)4 Response (io.openk9.entity.manager.model.payload.Response)4 PluginDriverDTO (io.openk9.plugin.driver.manager.model.PluginDriverDTO)4 Collectors (java.util.stream.Collectors)4 EntityRequest (io.openk9.entity.manager.model.payload.EntityRequest)3 RelationRequest (io.openk9.entity.manager.model.payload.RelationRequest)3 RouterHandler (io.openk9.http.web.RouterHandler)3