Search in sources :

Example 1 with Request

use of io.openk9.entity.manager.model.payload.Request in project openk9 by smclab.

the class GetOrAddEntitiesConsumer method apply.

public Mono<ObjectNode> apply(ObjectNode objectNode) {
    return Mono.defer(() -> {
        ObjectNode datasourceContextJson = objectNode.get("datasourceContext").toObjectNode();
        long datasourceId = datasourceContextJson.get("datasource").get("datasourceId").asLong();
        long tenantId = datasourceContextJson.get("tenant").get("tenantId").asLong();
        JsonNode entities = objectNode.remove("entities");
        Mono<ArrayNode> entitiesField;
        if (entities.size() == 0) {
            entitiesField = Mono.just(_jsonFactory.createArrayNode());
        } else {
            ObjectNode responseJson = _jsonFactory.createObjectNode();
            responseJson.put("entities", entities);
            responseJson.put("tenantId", tenantId);
            responseJson.put("datasourceId", datasourceId);
            Request request = _jsonFactory.fromJson(responseJson.toString(), Request.class);
            List<RequestContext> requestContextList = request.getEntities().stream().map(entityRequest -> RequestContext.builder().current(entityRequest).tenantId(request.getTenantId()).datasourceId(request.getDatasourceId()).rest(request.getEntities().stream().filter(er -> er != entityRequest).collect(Collectors.toList())).build()).collect(Collectors.toList());
            Mono<List<EntityContext>> disambiguateListMono = GetOrAddEntities.stopWatch("disambiguate-all-entities", Flux.fromIterable(requestContextList).flatMap(requestContext -> GetOrAddEntities.stopWatch("disambiguate-" + requestContext.getCurrent().getName(), Mono.<EntityContext>create(fluxSink -> _startDisambiguation.disambiguate(requestContext, fluxSink)))).collectList());
            Mono<ResponseList> writeRelations = disambiguateListMono.flatMap(entityContexts -> GetOrAddEntities.stopWatch("write-relations", writeRelations(entityContexts)));
            Mono<ResponseList> responseListWrapper = _transactional ? _graphClient.makeTransactional(writeRelations) : writeRelations;
            entitiesField = responseListWrapper.map(responseListDTO -> {
                List<Response> responseList = responseListDTO.getResponse();
                ArrayNode entitiesArrayNode = entities.toArrayNode();
                ArrayNode arrayNode = _jsonFactory.createArrayNode();
                for (JsonNode node : entitiesArrayNode) {
                    Optional<Response> responseOptional = responseList.stream().filter(response -> node.get("tmpId").asLong() == response.getTmpId()).findFirst();
                    if (responseOptional.isPresent()) {
                        Entity entity = responseOptional.get().getEntity();
                        ObjectNode result = _jsonFactory.createObjectNode();
                        result.put("entityType", entity.getType());
                        result.put("id", entity.getId());
                        result.put("context", node.get("context"));
                        arrayNode.add(result);
                    }
                }
                return arrayNode;
            });
        }
        return entitiesField.map(entitiesArray -> {
            ObjectNode payload = objectNode.get("payload").toObjectNode();
            payload.set("entities", entitiesArray);
            return objectNode;
        }).timeout(Duration.ofSeconds(_timeout), Mono.error(new TimeoutException("timeout on entities count: " + entities.size() + " (Did not observe any item or terminal signal within " + Duration.ofSeconds(_timeout).toMillis() + "ms)")));
    });
}
Also used : Response(io.openk9.entity.manager.model.payload.Response) SenderReactor(io.openk9.ingestion.api.SenderReactor) Disposable(reactor.core.Disposable) JsonNode(io.openk9.json.api.JsonNode) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Tuple2(reactor.util.function.Tuple2) TimeoutException(java.util.concurrent.TimeoutException) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) GraphClient(io.openk9.relationship.graph.api.client.GraphClient) ArrayNode(io.openk9.json.api.ArrayNode) JsonFactory(io.openk9.json.api.JsonFactory) ArrayList(java.util.ArrayList) Component(org.osgi.service.component.annotations.Component) Statement(org.neo4j.cypherdsl.core.Statement) Duration(java.time.Duration) OutboundMessageFactory(io.openk9.ingestion.api.OutboundMessageFactory) Schedulers(reactor.core.scheduler.Schedulers) Request(io.openk9.entity.manager.model.payload.Request) Cypher(org.neo4j.cypherdsl.core.Cypher) Activate(org.osgi.service.component.annotations.Activate) Node(org.neo4j.cypherdsl.core.Node) Binding(io.openk9.ingestion.api.Binding) Logger(org.slf4j.Logger) Deactivate(org.osgi.service.component.annotations.Deactivate) Functions(org.neo4j.cypherdsl.core.Functions) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) Mono(reactor.core.publisher.Mono) Entity(io.openk9.entity.manager.model.Entity) Collectors(java.util.stream.Collectors) Flux(reactor.core.publisher.Flux) List(java.util.List) Stream(java.util.stream.Stream) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ReceiverReactor(io.openk9.ingestion.api.ReceiverReactor) Modified(org.osgi.service.component.annotations.Modified) Optional(java.util.Optional) ObjectNode(io.openk9.json.api.ObjectNode) Reference(org.osgi.service.component.annotations.Reference) Cypher.literalOf(org.neo4j.cypherdsl.core.Cypher.literalOf) Entity(io.openk9.entity.manager.model.Entity) ObjectNode(io.openk9.json.api.ObjectNode) Optional(java.util.Optional) EntityRequest(io.openk9.entity.manager.model.payload.EntityRequest) Request(io.openk9.entity.manager.model.payload.Request) RelationRequest(io.openk9.entity.manager.model.payload.RelationRequest) JsonNode(io.openk9.json.api.JsonNode) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ArrayList(java.util.ArrayList) List(java.util.List) ResponseList(io.openk9.entity.manager.model.payload.ResponseList) ArrayNode(io.openk9.json.api.ArrayNode) TimeoutException(java.util.concurrent.TimeoutException)

Example 2 with Request

use of io.openk9.entity.manager.model.payload.Request in project openk9 by smclab.

the class IndexWriterEndpoins method _insertEntity.

private Publisher<Void> _insertEntity(HttpServerRequest httpRequest, HttpServerResponse httpResponse) {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    Mono<List<DocumentEntityRequest>> request = Mono.from(ReactorNettyUtils.aggregateBodyAsByteArray(httpRequest)).map(json -> _jsonFactory.fromJsonList(json, DocumentEntityRequest.class));
    Mono<BulkResponse> elasticResponse = request.flatMapIterable(Function.identity()).map(entity -> {
        IndexRequest indexRequest = new IndexRequest(entity.getTenantId() + "-entity");
        return indexRequest.source(_jsonFactory.toJson(entity), XContentType.JSON);
    }).reduce(new BulkRequest(), BulkRequest::add).flatMap(bulkRequest -> Mono.create(sink -> {
        bulkRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
        Cancellable cancellable = restHighLevelClient.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ReactorActionListener<>(sink));
        sink.onCancel(cancellable::cancel);
    }));
    return _httpResponseWriter.write(httpResponse, elasticResponse.thenReturn("{}"));
}
Also used : LoggerFactory(org.slf4j.LoggerFactory) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) JsonFactory(io.openk9.json.api.JsonFactory) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) IndexRequest(org.elasticsearch.action.index.IndexRequest) EventBus(io.vertx.core.eventbus.EventBus) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) CountRequest(org.elasticsearch.client.core.CountRequest) TimeValue(org.elasticsearch.core.TimeValue) SearchHit(org.elasticsearch.search.SearchHit) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Message(io.vertx.core.eventbus.Message) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Scroll(org.elasticsearch.search.Scroll) Collectors(java.util.stream.Collectors) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) QueryBuilders.matchQuery(org.elasticsearch.index.query.QueryBuilders.matchQuery) Builder(lombok.Builder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) MatchQueryBuilder(org.elasticsearch.index.query.MatchQueryBuilder) XContentType(org.elasticsearch.common.xcontent.XContentType) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) CountResponse(org.elasticsearch.client.core.CountResponse) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) WriteRequest(org.elasticsearch.action.support.WriteRequest) Component(org.osgi.service.component.annotations.Component) Operator(org.elasticsearch.index.query.Operator) Cancellable(org.elasticsearch.client.Cancellable) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) Activate(org.osgi.service.component.annotations.Activate) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse) RouterHandler(io.openk9.http.web.RouterHandler) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Mono(reactor.core.publisher.Mono) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Data(lombok.Data) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) AllArgsConstructor(lombok.AllArgsConstructor) Reference(org.osgi.service.component.annotations.Reference) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) NoArgsConstructor(lombok.NoArgsConstructor) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) Cancellable(org.elasticsearch.client.Cancellable) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) List(java.util.List) ArrayList(java.util.ArrayList) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) IndexRequest(org.elasticsearch.action.index.IndexRequest)

Example 3 with Request

use of io.openk9.entity.manager.model.payload.Request in project openk9 by smclab.

the class EntityManagerBus method run.

@SneakyThrows
public void run() {
    while (true) {
        Payload request = _entityManagerQueue.take();
        TransactionContext transactionContext = _hazelcastInstance.newTransactionContext();
        transactionContext.beginTransaction();
        try {
            TransactionalMap<EntityKey, Entity> entityTransactionalMap = transactionContext.getMap("entityMap");
            TransactionalMap<EntityRelationKey, EntityRelation> transactionalEntityRelationMap = transactionContext.getMap("entityRelationMap");
            TransactionalMultiMap<DocumentKey, String> documentEntityMap = transactionContext.getMultiMap("documentEntityMap");
            EntityManagerRequest payload = request.getPayload();
            _loggerAggregator.emitLog("process ingestionId", payload.getIngestionId());
            long tenantId = payload.getTenantId();
            String ingestionId = payload.getIngestionId();
            List<EntityRequest> entities = request.getEntities();
            Map<EntityKey, Entity> localEntityMap = new HashMap<>(entities.size());
            for (EntityRequest entityRequest : entities) {
                String name = entityRequest.getName();
                String type = entityRequest.getType();
                String cacheId = Long.toString(_entityFlakeId.newId());
                EntityKey key = EntityKey.of(tenantId, name, type, cacheId, ingestionId);
                Entity entity = new Entity(null, cacheId, tenantId, name, type, null, ingestionId, false, true, entityRequest.getContext());
                entityTransactionalMap.set(key, entity);
                localEntityMap.put(key, entity);
                for (EntityRequest entityRequest2 : entities) {
                    for (RelationRequest relation : entityRequest2.getRelations()) {
                        if (relation.getTo().equals(entityRequest.getTmpId())) {
                            relation.setTo(entity.getCacheId());
                        }
                    }
                }
            }
            for (EntityRequest entity : entities) {
                List<RelationRequest> relations = entity.getRelations();
                if (relations == null || relations.isEmpty()) {
                    continue;
                }
                Collection<Entity> values = localEntityMap.values();
                Entity current = values.stream().filter(e -> e.getName().equals(entity.getName()) && e.getType().equals(entity.getType())).findFirst().orElse(null);
                if (current == null) {
                    continue;
                }
                for (RelationRequest relation : relations) {
                    String to = relation.getTo();
                    String name = relation.getName();
                    for (Entity value : values) {
                        if (value.getCacheId().equals(to)) {
                            long entityRelationId = _entityRelationFlakeId.newId();
                            EntityRelation entityRelation = new EntityRelation(entityRelationId, current.getCacheId(), ingestionId, name, value.getCacheId());
                            transactionalEntityRelationMap.set(EntityRelationKey.of(entityRelationId, current.getCacheId(), ingestionId), entityRelation);
                        }
                    }
                }
            }
            if (!localEntityMap.isEmpty()) {
                DocumentKey key = DocumentKey.of(payload.getDatasourceId(), payload.getContentId(), tenantId);
                for (Entity value : localEntityMap.values()) {
                    documentEntityMap.put(key, value.getCacheId());
                }
            }
        } catch (Exception e) {
            _log.error(e.getMessage(), e);
            transactionContext.rollbackTransaction();
        } finally {
            transactionContext.commitTransaction();
        }
    }
}
Also used : Entity(io.openk9.entity.manager.cache.model.Entity) HashMap(java.util.HashMap) RelationRequest(io.openk9.entity.manager.dto.RelationRequest) EntityRelationKey(io.openk9.entity.manager.cache.model.EntityRelationKey) EntityKey(io.openk9.entity.manager.cache.model.EntityKey) EntityRelation(io.openk9.entity.manager.cache.model.EntityRelation) EntityRequest(io.openk9.entity.manager.dto.EntityRequest) TransactionContext(com.hazelcast.transaction.TransactionContext) DocumentKey(io.openk9.entity.manager.cache.model.DocumentKey) Payload(io.openk9.entity.manager.dto.Payload) EntityManagerRequest(io.openk9.entity.manager.dto.EntityManagerRequest) SneakyThrows(lombok.SneakyThrows)

Example 4 with Request

use of io.openk9.entity.manager.model.payload.Request in project openk9 by smclab.

the class BaseNerEnrichProcessor method prepareRequestRawContent.

protected ObjectNode prepareRequestRawContent(ObjectNode objectNode, ObjectNode datasourceConfiguration, DatasourceContext context, PluginDriverDTO pluginDriverDTO) {
    JsonNode entitiesNode = datasourceConfiguration.get(entitiesField());
    JsonNode relationsNode = datasourceConfiguration.get(relationsField());
    JsonNode rawContentNode = objectNode.get(Constants.RAW_CONTENT);
    JsonNode confidenceNode = datasourceConfiguration.get(Constants.CONFIDENCE);
    ObjectNode request = _jsonFactory.createObjectNode();
    request.put(entitiesField(), entitiesNode);
    request.put(relationsField(), relationsNode);
    request.put(Constants.CONFIDENCE, confidenceNode);
    request.put(Constants.CONTENT, rawContentNode);
    JsonNode typeNode = objectNode.get(Constants.TYPE);
    ObjectNode datasourcePayload = _jsonFactory.createObjectNode();
    if (typeNode != null && typeNode.isArray()) {
        ArrayNode types = typeNode.toArrayNode();
        for (JsonNode typeJsonNode : types) {
            String type = typeJsonNode.asText();
            datasourcePayload.put(type, objectNode.get(type));
        }
    }
    request.put(Constants.DATASOURCE_PAYLOAD, datasourcePayload);
    request.put(Constants.TENANT_ID, context.getTenant().getTenantId());
    request.put(Constants.DATASOURCE_ID, context.getDatasource().getDatasourceId());
    request.put(Constants.CONTENT_ID, objectNode.get(Constants.CONTENT_ID));
    return request;
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode) JsonNode(io.openk9.json.api.JsonNode) ArrayNode(io.openk9.json.api.ArrayNode)

Example 5 with Request

use of io.openk9.entity.manager.model.payload.Request in project openk9 by smclab.

the class JsEnrichProcessor method process.

@Override
public Mono<ObjectNode> process(ObjectNode objectNode, DatasourceContext context, EnrichItem enrichItem, PluginDriverDTO pluginDriverName) {
    return Mono.defer(() -> {
        JsonNode datasourceConfiguration = _jsonFactory.fromJsonToJsonNode(enrichItem.getJsonConfig());
        if (!datasourceConfiguration.isObject()) {
            return Mono.error(new RuntimeException("jsonConfig must be an instance of ObjectNode " + datasourceConfiguration.toString()));
        }
        ObjectNode request = prepareRequestRawContent(objectNode, datasourceConfiguration.toObjectNode(), context, pluginDriverName);
        return Mono.from(_httpClient.request(getMethod(), getPath(), request.toString(), getHeaders())).map(_jsonFactory::fromJsonToJsonNode).map(JsonNode::toObjectNode).map(objectNode::merge);
    });
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode) JsonNode(io.openk9.json.api.JsonNode)

Aggregations

JsonNode (io.openk9.json.api.JsonNode)6 ObjectNode (io.openk9.json.api.ObjectNode)6 Tenant (io.openk9.api.aggregator.model.Tenant)3 ArrayNode (io.openk9.json.api.ArrayNode)3 JsonFactory (io.openk9.json.api.JsonFactory)3 HttpException (io.vertx.ext.web.handler.HttpException)3 Collectors (java.util.stream.Collectors)3 PermitAll (javax.annotation.security.PermitAll)3 POST (javax.ws.rs.POST)3 Path (javax.ws.rs.Path)3 Mono (reactor.core.publisher.Mono)3 Payload (io.openk9.entity.manager.dto.Payload)2 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 List (java.util.List)2 Activate (org.osgi.service.component.annotations.Activate)2 Component (org.osgi.service.component.annotations.Component)2 Reference (org.osgi.service.component.annotations.Reference)2 Logger (org.slf4j.Logger)2 LoggerFactory (org.slf4j.LoggerFactory)2