Search in sources :

Example 1 with IngestionPayload

use of io.openk9.datasource.processor.payload.IngestionPayload in project openk9 by smclab.

the class DatasourceProcessor method process.

@Incoming("ingestion")
@Outgoing("ingestion-datasource")
public Uni<IngestionDatasourcePayload> process(Object obj) {
    JsonObject jsonObject = obj instanceof JsonObject ? (JsonObject) obj : new JsonObject(new String((byte[]) obj));
    long datasourceId = jsonObject.getLong("datasourceId");
    Uni<Datasource> datasourceUni = Datasource.findById(datasourceId);
    return Panache.withTransaction(() -> datasourceUni.flatMap(datasource -> EnrichPipeline.findByDatasourceId(datasource.getDatasourceId()).onItem().ifNull().continueWith(EnrichPipeline::new).flatMap(enrichPipeline -> {
        Uni<List<EnrichItem>> enrichItemUni;
        if (enrichPipeline.getEnrichPipelineId() != null) {
            enrichItemUni = EnrichItem.findByEnrichPipelineId(enrichPipeline.getEnrichPipelineId()).onItem().ifNull().continueWith(List::of);
        } else {
            enrichItemUni = Uni.createFrom().item(List.of());
        }
        return Uni.combine().all().unis(Tenant.findById(datasource.getTenantId()), enrichItemUni).combinedWith((tenantObj, enrichItemList) -> {
            Tenant tenant = (Tenant) tenantObj;
            IngestionPayload ingestionPayload = jsonObject.mapTo(IngestionPayload.class);
            ingestionPayload.setTenantId(tenant.getTenantId());
            DatasourceContext datasourceContext = DatasourceContext.of(datasource, tenant, enrichPipeline, enrichItemList);
            return IngestionDatasourcePayload.of(ingestionPayload, datasourceContext);
        });
    })).eventually(() -> Datasource.<Datasource>findById(datasourceId).flatMap(datasource -> {
        datasource.setLastIngestionDate(Instant.ofEpochMilli(jsonObject.getLong("parsingDate")));
        return datasource.persist();
    })));
}
Also used : Datasource(io.openk9.datasource.model.Datasource) Outgoing(org.eclipse.microprofile.reactive.messaging.Outgoing) Datasource(io.openk9.datasource.model.Datasource) EnrichItem(io.openk9.datasource.model.EnrichItem) Tenant(io.openk9.datasource.model.Tenant) Instant(java.time.Instant) Uni(io.smallrye.mutiny.Uni) DatasourceContext(io.openk9.datasource.processor.payload.DatasourceContext) List(java.util.List) IngestionDatasourcePayload(io.openk9.datasource.processor.payload.IngestionDatasourcePayload) IngestionPayload(io.openk9.datasource.processor.payload.IngestionPayload) EnrichPipeline(io.openk9.datasource.model.EnrichPipeline) JsonObject(io.vertx.core.json.JsonObject) ApplicationScoped(javax.enterprise.context.ApplicationScoped) Panache(io.quarkus.hibernate.reactive.panache.Panache) Incoming(org.eclipse.microprofile.reactive.messaging.Incoming) JsonObject(io.vertx.core.json.JsonObject) EnrichItem(io.openk9.datasource.model.EnrichItem) Uni(io.smallrye.mutiny.Uni) Tenant(io.openk9.datasource.model.Tenant) IngestionPayload(io.openk9.datasource.processor.payload.IngestionPayload) DatasourceContext(io.openk9.datasource.processor.payload.DatasourceContext) EnrichPipeline(io.openk9.datasource.model.EnrichPipeline) List(java.util.List) Incoming(org.eclipse.microprofile.reactive.messaging.Incoming) Outgoing(org.eclipse.microprofile.reactive.messaging.Outgoing)

Example 2 with IngestionPayload

use of io.openk9.datasource.processor.payload.IngestionPayload in project openk9 by smclab.

the class EnrichPipelineProcessor method _adaptIngestionPayload.

private Tuple3<DatasourceContext, ObjectNode, PluginDriverDTO> _adaptIngestionPayload(DatasourceContext datasourceContext, ObjectNode ingestionPayload, PluginDriverDTO pluginDriverDTO) {
    ObjectNode newIngestionPayload = ingestionPayload;
    if (newIngestionPayload.hasNonNull(io.openk9.core.api.constant.Constants.DATASOURCE_PAYLOAD)) {
        JsonNode datasourcePayload = newIngestionPayload.remove(io.openk9.core.api.constant.Constants.DATASOURCE_PAYLOAD);
        if (datasourcePayload.isObject()) {
            ObjectNode jsonNodes = datasourcePayload.toObjectNode();
            for (Map.Entry<String, JsonNode> field : jsonNodes.fields()) {
                newIngestionPayload.set(field.getKey(), field.getValue());
            }
        }
    }
    if (!ingestionPayload.hasNonNull(Constants.DATASOURCE_NAME)) {
        newIngestionPayload = newIngestionPayload.deepCopy().put(Constants.DATASOURCE_NAME, pluginDriverDTO.getName());
    }
    if (ingestionPayload.hasNonNull(io.openk9.core.api.constant.Constants.TYPE) && !ingestionPayload.get(io.openk9.core.api.constant.Constants.TYPE).toArrayNode().isEmpty()) {
        return Tuples.of(datasourceContext, ingestionPayload, pluginDriverDTO);
    }
    DocumentTypeDTO documentType = pluginDriverDTO.getDefaultDocumentType();
    return Tuples.of(datasourceContext, newIngestionPayload.set(io.openk9.core.api.constant.Constants.TYPE, _jsonFactory.createArrayNode().add(documentType.getName())), pluginDriverDTO);
}
Also used : ObjectNode(io.openk9.json.api.ObjectNode) JsonNode(io.openk9.json.api.JsonNode) Map(java.util.Map) DocumentTypeDTO(io.openk9.plugin.driver.manager.model.DocumentTypeDTO)

Aggregations

Datasource (io.openk9.datasource.model.Datasource)1 EnrichItem (io.openk9.datasource.model.EnrichItem)1 EnrichPipeline (io.openk9.datasource.model.EnrichPipeline)1 Tenant (io.openk9.datasource.model.Tenant)1 DatasourceContext (io.openk9.datasource.processor.payload.DatasourceContext)1 IngestionDatasourcePayload (io.openk9.datasource.processor.payload.IngestionDatasourcePayload)1 IngestionPayload (io.openk9.datasource.processor.payload.IngestionPayload)1 JsonNode (io.openk9.json.api.JsonNode)1 ObjectNode (io.openk9.json.api.ObjectNode)1 DocumentTypeDTO (io.openk9.plugin.driver.manager.model.DocumentTypeDTO)1 Panache (io.quarkus.hibernate.reactive.panache.Panache)1 Uni (io.smallrye.mutiny.Uni)1 JsonObject (io.vertx.core.json.JsonObject)1 Instant (java.time.Instant)1 List (java.util.List)1 Map (java.util.Map)1 ApplicationScoped (javax.enterprise.context.ApplicationScoped)1 Incoming (org.eclipse.microprofile.reactive.messaging.Incoming)1 Outgoing (org.eclipse.microprofile.reactive.messaging.Outgoing)1