Search in sources :

Example 11 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class IndexWriterEndpoins method _deleteDataDocuments.

private Publisher<Void> _deleteDataDocuments(HttpServerRequest httpServerRequest, HttpServerResponse httpServerResponse) {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    Mono<byte[]> body = ReactorNettyUtils.aggregateBodyAsByteArray(httpServerRequest);
    Mono<String> responseMono = body.map(bytes -> _jsonFactory.fromJson(bytes, DeleteEntitiesRequest.class)).flatMap(deleteEntitiesRequest -> {
        if (!deleteEntitiesRequest.getContentIds().isEmpty()) {
            BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
            for (String contentId : deleteEntitiesRequest.getContentIds()) {
                boolQuery.mustNot(matchQuery("contentId", contentId));
            }
            Mono<Datasource> datasourceMono = _datasourceClient.findDatasource(deleteEntitiesRequest.getDatasourceId());
            return datasourceMono.flatMap(datasource -> _pluginDriverManagerClient.getPluginDriver(datasource.getDriverServiceName()).map(pluginDriverDTO -> datasource.getTenantId() + "-" + pluginDriverDTO.getName() + "-data").map(DeleteByQueryRequest::new).map(deleteByQueryRequest -> deleteByQueryRequest.setQuery(boolQuery)).flatMap(deleteByQueryRequest -> Mono.<BulkByScrollResponse>create(sink -> restHighLevelClient.deleteByQueryAsync(deleteByQueryRequest, RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).map(Object::toString).doOnNext(_log::info).flatMap(response -> Mono.fromRunnable(() -> _sendCleanOrphanEntitiesRequest(datasource.getTenantId())).thenReturn(response)));
        }
        return Mono.empty();
    });
    return _httpResponseWriter.write(httpServerResponse, responseMono);
}
Also used : LoggerFactory(org.slf4j.LoggerFactory) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) JsonFactory(io.openk9.json.api.JsonFactory) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) IndexRequest(org.elasticsearch.action.index.IndexRequest) EventBus(io.vertx.core.eventbus.EventBus) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) CountRequest(org.elasticsearch.client.core.CountRequest) TimeValue(org.elasticsearch.core.TimeValue) SearchHit(org.elasticsearch.search.SearchHit) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Message(io.vertx.core.eventbus.Message) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Scroll(org.elasticsearch.search.Scroll) Collectors(java.util.stream.Collectors) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) QueryBuilders.matchQuery(org.elasticsearch.index.query.QueryBuilders.matchQuery) Builder(lombok.Builder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) MatchQueryBuilder(org.elasticsearch.index.query.MatchQueryBuilder) XContentType(org.elasticsearch.common.xcontent.XContentType) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) CountResponse(org.elasticsearch.client.core.CountResponse) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) WriteRequest(org.elasticsearch.action.support.WriteRequest) Component(org.osgi.service.component.annotations.Component) Operator(org.elasticsearch.index.query.Operator) Cancellable(org.elasticsearch.client.Cancellable) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) Activate(org.osgi.service.component.annotations.Activate) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse) RouterHandler(io.openk9.http.web.RouterHandler) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Mono(reactor.core.publisher.Mono) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Data(lombok.Data) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) AllArgsConstructor(lombok.AllArgsConstructor) Reference(org.osgi.service.component.annotations.Reference) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) NoArgsConstructor(lombok.NoArgsConstructor) Datasource(io.openk9.model.Datasource) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient)

Example 12 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class DriverManagerActivator method activate.

@Activate
public void activate(Config config) throws SchedulerError {
    Disposable disposable1 = _datasourceRepository.findAll(true).concatMap(this::_schedule).subscribe();
    Disposable disposable2 = _entityEventBus.stream().filter(e -> e.getEntityClass() == Datasource.class).concatMap(entityEvent -> {
        Datasource datasource = (Datasource) entityEvent.getValue();
        if (entityEvent instanceof EntityEvent.UpdateEvent || entityEvent instanceof EntityEvent.InsertEvent) {
            return _schedule(datasource);
        } else {
            return Mono.fromRunnable(() -> _scheduler.unschedule(_PREFIX + datasource.getName()));
        }
    }).subscribe();
    _autoClosableSafe = AutoCloseables.mergeAutoCloseableToSafe(disposable1::dispose, disposable2::dispose);
}
Also used : Disposable(reactor.core.Disposable) Datasource(io.openk9.model.Datasource) SchedulerEnabledDTO(io.openk9.plugin.driver.manager.model.SchedulerEnabledDTO) Logger(org.slf4j.Logger) Disposable(reactor.core.Disposable) Datasource(io.openk9.model.Datasource) Deactivate(org.osgi.service.component.annotations.Deactivate) Date(java.util.Date) LoggerFactory(org.slf4j.LoggerFactory) Mono(reactor.core.publisher.Mono) Scheduler(org.apache.karaf.scheduler.Scheduler) AutoCloseables(io.openk9.osgi.util.AutoCloseables) ScheduleOptions(org.apache.karaf.scheduler.ScheduleOptions) Component(org.osgi.service.component.annotations.Component) SchedulerError(org.apache.karaf.scheduler.SchedulerError) DatasourceRepository(io.openk9.datasource.repository.DatasourceRepository) Map(java.util.Map) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) Activate(org.osgi.service.component.annotations.Activate) Reference(org.osgi.service.component.annotations.Reference) EntityEvent(io.openk9.sql.api.event.EntityEvent) EntityEventBus(io.openk9.sql.api.event.EntityEventBus) Job(org.apache.karaf.scheduler.Job) Activate(org.osgi.service.component.annotations.Activate)

Example 13 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class BaseSearchHTTPHandler method _toQuerySearchRequest.

private Mono<SearchResponse> _toQuerySearchRequest(Tenant tenant, List<Datasource> datasources, PluginDriverDTOList pdDTOList, SearchRequest searchRequest, HttpServerRequest httpRequest) {
    return Mono.defer(() -> {
        List<PluginDriverDTO> pluginDriverDTOList = pdDTOList.getPluginDriverDTOList();
        Map<String, List<SearchToken>> tokenTypeGroup = searchRequest.getSearchQuery().stream().collect(Collectors.groupingBy(SearchToken::getTokenType));
        List<SearchToken> datasource = tokenTypeGroup.get("DATASOURCE");
        Stream<PluginDriverDTO> documentTypeStream = pluginDriverDTOList.stream();
        if (datasource != null) {
            List<String> datasourceValues = datasource.stream().map(SearchToken::getValues).flatMap(Arrays::stream).distinct().collect(Collectors.toList());
            documentTypeStream = documentTypeStream.filter(entry -> datasourceValues.contains(entry.getName()));
        }
        List<PluginDriverDTO> documentTypeList = documentTypeStream.collect(Collectors.toList());
        QueryParser queryParser = _queryParsers.stream().reduce(QueryParser.NOTHING, QueryParser::andThen);
        return queryParser.apply(createQueryParserContext(tenant, datasources, httpRequest, tokenTypeGroup, documentTypeList)).flatMap(boolQueryBuilderConsumer -> _search.flatMapSearch(factory -> {
            long tenantId = tenant.getTenantId();
            if (documentTypeList.isEmpty()) {
                return Mono.just(SearchUtil.EMPTY_SEARCH_REQUEST);
            }
            BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
            boolQueryBuilderConsumer.accept(boolQuery);
            org.elasticsearch.action.search.SearchRequest elasticSearchQuery;
            if (datasource != null) {
                String[] indexNames = documentTypeList.stream().map(PluginDriverDTO::getName).distinct().toArray(String[]::new);
                elasticSearchQuery = factory.createSearchRequestData(tenantId, indexNames);
            } else {
                elasticSearchQuery = factory.createSearchRequestData(tenantId, "*");
            }
            SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
            searchSourceBuilder.fetchSource(includeFields(), excludeFields());
            searchSourceBuilder.query(boolQuery);
            searchSourceBuilder.trackTotalHits(true);
            return customizeSearchSourceBuilderMono(tenant, datasources, searchRequest, documentTypeList, searchSourceBuilder, elasticSearchQuery);
        }));
    });
}
Also used : HttpServerResponse(reactor.netty.http.server.HttpServerResponse) PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) Arrays(java.util.Arrays) SearchUtil(io.openk9.search.client.api.util.SearchUtil) SearchHits(org.elasticsearch.search.SearchHits) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Search(io.openk9.search.client.api.Search) Tuple2(reactor.util.function.Tuple2) HashMap(java.util.HashMap) QueryParser(io.openk9.search.api.query.QueryParser) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) SearchToken(io.openk9.search.api.query.SearchToken) JsonFactory(io.openk9.json.api.JsonFactory) ArrayList(java.util.ArrayList) HighlightBuilder(org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) Text(org.elasticsearch.common.text.Text) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) HighlightField(org.elasticsearch.search.fetch.subphase.highlight.HighlightField) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) RouterHandler(io.openk9.http.web.RouterHandler) SearchHit(org.elasticsearch.search.SearchHit) Tenant(io.openk9.model.Tenant) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Collection(java.util.Collection) DocumentTypeDTO(io.openk9.plugin.driver.manager.model.DocumentTypeDTO) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Mono(reactor.core.publisher.Mono) SearchKeywordDTO(io.openk9.plugin.driver.manager.model.SearchKeywordDTO) Collectors(java.util.stream.Collectors) TotalHits(org.apache.lucene.search.TotalHits) SearchRequest(io.openk9.search.api.query.SearchRequest) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) HttpHandler(io.openk9.http.web.HttpHandler) Stream(java.util.stream.Stream) SearchTokenizer(io.openk9.search.api.query.SearchTokenizer) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) Response(io.openk9.search.query.internal.response.Response) SearchRequest(io.openk9.search.api.query.SearchRequest) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) QueryParser(io.openk9.search.api.query.QueryParser) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) SearchToken(io.openk9.search.api.query.SearchToken) PluginDriverDTOList(io.openk9.plugin.driver.manager.model.PluginDriverDTOList) ArrayList(java.util.ArrayList) List(java.util.List) Arrays(java.util.Arrays) PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO)

Example 14 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class BaseSearchHTTPHandler method customizeSearchSourceBuilder.

protected void customizeSearchSourceBuilder(Tenant tenant, List<Datasource> datasources, SearchRequest searchRequest, List<PluginDriverDTO> documentTypeList, SearchSourceBuilder searchSourceBuilder, org.elasticsearch.action.search.SearchRequest elasticSearchQuery) {
    int[] range = searchRequest.getRange();
    if (range != null) {
        searchSourceBuilder.from(range[0]);
        searchSourceBuilder.size(range[1]);
    }
    HighlightBuilder highlightBuilder = new HighlightBuilder();
    documentTypeList.stream().map(PluginDriverDTO::getDocumentTypes).flatMap(Collection::stream).map(DocumentTypeDTO::getSearchKeywords).flatMap(Collection::stream).filter(SearchKeywordDTO::isText).map(SearchKeywordDTO::getKeyword).distinct().forEach(highlightBuilder::field);
    highlightBuilder.forceSource(true);
    highlightBuilder.tagsSchema("default");
    searchSourceBuilder.highlighter(highlightBuilder);
}
Also used : Collection(java.util.Collection) HighlightBuilder(org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder) SearchKeywordDTO(io.openk9.plugin.driver.manager.model.SearchKeywordDTO)

Example 15 with Datasource

use of io.openk9.model.Datasource in project openk9 by smclab.

the class ResourcesHttpHandler method _sendResource.

private Mono<Void> _sendResource(Tenant tenant, Datasource datasource, PluginDriverDTO pluginDriverDTO, long datasourceId, String documentId, String resourceId, HttpServerRequest httpRequest, HttpServerResponse httpResponse) {
    return _queryParser.apply(QueryParser.Context.of(tenant, null, null, null, httpRequest, QueryParser.QueryCondition.DEFAULT)).flatMap(consumer -> _search.search(factory -> {
        SearchRequest searchRequest = factory.createSearchRequestData(tenant.getTenantId(), pluginDriverDTO.getName());
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(QueryBuilders.idsQuery().addIds(documentId));
        boolQueryBuilder.must(QueryBuilders.nestedQuery(_RESOURCES_BINARIES, QueryBuilders.matchQuery(_RESOURCES_BINARIES_ID, resourceId), ScoreMode.Max));
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        consumer.accept(boolQueryBuilder);
        searchSourceBuilder.query(boolQueryBuilder);
        searchSourceBuilder.fetchSource(new String[] { _RESOURCES_BINARIES_DATA, _RESOURCES_BINARIES_CONTENT_TYPE }, null);
        return searchRequest.source(searchSourceBuilder);
    })).flatMap(response -> {
        SearchHits searchHits = response.getHits();
        SearchHit[] hits = searchHits.getHits();
        if (hits.length == 0) {
            return httpResponse.sendNotFound();
        } else if (hits.length > 1) {
            _log.warn("found more than one resource (datasourceId: " + datasourceId + " resourceId: " + resourceId + " tenantId: " + tenant.getTenantId() + " documentId: " + documentId + ")");
        }
        SearchHit hit = hits[0];
        String source = hit.getSourceAsString();
        Resources resources = _jsonFactory.fromJson(source, Resources.class);
        ResourcesPayload resourcesPayload = resources.getResources();
        BinaryPayload binaryPayload = resourcesPayload.getBinaries().get(0);
        String data = binaryPayload.getData();
        String contentType = binaryPayload.getContentType();
        _manageCache(httpRequest, httpResponse);
        if (contentType != null && !contentType.isBlank()) {
            httpResponse.header(_CONTENT_TYPE, contentType);
        }
        byte[] decode = Base64.getDecoder().decode(data);
        return Mono.from(httpResponse.sendByteArray(Mono.just(decode)));
    });
}
Also used : HttpServerResponse(reactor.netty.http.server.HttpServerResponse) PluginDriverDTO(io.openk9.plugin.driver.manager.model.PluginDriverDTO) HttpUtil(io.openk9.http.util.HttpUtil) SearchHits(org.elasticsearch.search.SearchHits) ZonedDateTime(java.time.ZonedDateTime) LoggerFactory(org.slf4j.LoggerFactory) Search(io.openk9.search.client.api.Search) QueryParser(io.openk9.search.api.query.QueryParser) SearchRequest(org.elasticsearch.action.search.SearchRequest) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) ResourcesPayload(io.openk9.model.ResourcesPayload) JsonFactory(io.openk9.json.api.JsonFactory) Component(org.osgi.service.component.annotations.Component) Map(java.util.Map) BinaryPayload(io.openk9.model.BinaryPayload) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) Activate(org.osgi.service.component.annotations.Activate) ZoneOffset(java.time.ZoneOffset) RouterHandler(io.openk9.http.web.RouterHandler) SearchHit(org.elasticsearch.search.SearchHit) Tenant(io.openk9.model.Tenant) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Mono(reactor.core.publisher.Mono) ScoreMode(org.apache.lucene.search.join.ScoreMode) Instant(java.time.Instant) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) Base64(java.util.Base64) List(java.util.List) ChronoUnit(java.time.temporal.ChronoUnit) Builder(lombok.Builder) DateTimeFormatter(java.time.format.DateTimeFormatter) Data(lombok.Data) NumberUtils(org.apache.commons.lang3.math.NumberUtils) AllArgsConstructor(lombok.AllArgsConstructor) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) Reference(org.osgi.service.component.annotations.Reference) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) NoArgsConstructor(lombok.NoArgsConstructor) SearchRequest(org.elasticsearch.action.search.SearchRequest) SearchHit(org.elasticsearch.search.SearchHit) BinaryPayload(io.openk9.model.BinaryPayload) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) SearchHits(org.elasticsearch.search.SearchHits) ResourcesPayload(io.openk9.model.ResourcesPayload) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder)

Aggregations

Datasource (io.openk9.model.Datasource)12 Mono (reactor.core.publisher.Mono)11 List (java.util.List)9 Activate (org.osgi.service.component.annotations.Activate)9 Component (org.osgi.service.component.annotations.Component)9 Logger (org.slf4j.Logger)9 LoggerFactory (org.slf4j.LoggerFactory)9 JsonFactory (io.openk9.json.api.JsonFactory)8 PluginDriverManagerClient (io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient)8 Map (java.util.Map)8 Reference (org.osgi.service.component.annotations.Reference)8 PluginDriverDTO (io.openk9.plugin.driver.manager.model.PluginDriverDTO)7 Collectors (java.util.stream.Collectors)7 DatasourceClient (io.openk9.datasource.client.api.DatasourceClient)6 RouterHandler (io.openk9.http.web.RouterHandler)6 Tenant (io.openk9.model.Tenant)6 HashMap (java.util.HashMap)6 BoolQueryBuilder (org.elasticsearch.index.query.BoolQueryBuilder)6 QueryBuilders (org.elasticsearch.index.query.QueryBuilders)6 HttpResponseWriter (io.openk9.http.util.HttpResponseWriter)5