Search in sources :

Example 1 with ReactorActionListener

use of io.openk9.search.client.api.ReactorActionListener in project openk9 by smclab.

the class IndexWriterEndpoins method _insertEntity.

private Publisher<Void> _insertEntity(HttpServerRequest httpRequest, HttpServerResponse httpResponse) {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    Mono<List<DocumentEntityRequest>> request = Mono.from(ReactorNettyUtils.aggregateBodyAsByteArray(httpRequest)).map(json -> _jsonFactory.fromJsonList(json, DocumentEntityRequest.class));
    Mono<BulkResponse> elasticResponse = request.flatMapIterable(Function.identity()).map(entity -> {
        IndexRequest indexRequest = new IndexRequest(entity.getTenantId() + "-entity");
        return indexRequest.source(_jsonFactory.toJson(entity), XContentType.JSON);
    }).reduce(new BulkRequest(), BulkRequest::add).flatMap(bulkRequest -> Mono.create(sink -> {
        bulkRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
        Cancellable cancellable = restHighLevelClient.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ReactorActionListener<>(sink));
        sink.onCancel(cancellable::cancel);
    }));
    return _httpResponseWriter.write(httpResponse, elasticResponse.thenReturn("{}"));
}
Also used : LoggerFactory(org.slf4j.LoggerFactory) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) JsonFactory(io.openk9.json.api.JsonFactory) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) IndexRequest(org.elasticsearch.action.index.IndexRequest) EventBus(io.vertx.core.eventbus.EventBus) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) CountRequest(org.elasticsearch.client.core.CountRequest) TimeValue(org.elasticsearch.core.TimeValue) SearchHit(org.elasticsearch.search.SearchHit) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Message(io.vertx.core.eventbus.Message) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Scroll(org.elasticsearch.search.Scroll) Collectors(java.util.stream.Collectors) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) QueryBuilders.matchQuery(org.elasticsearch.index.query.QueryBuilders.matchQuery) Builder(lombok.Builder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) MatchQueryBuilder(org.elasticsearch.index.query.MatchQueryBuilder) XContentType(org.elasticsearch.common.xcontent.XContentType) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) CountResponse(org.elasticsearch.client.core.CountResponse) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) WriteRequest(org.elasticsearch.action.support.WriteRequest) Component(org.osgi.service.component.annotations.Component) Operator(org.elasticsearch.index.query.Operator) Cancellable(org.elasticsearch.client.Cancellable) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) Activate(org.osgi.service.component.annotations.Activate) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse) RouterHandler(io.openk9.http.web.RouterHandler) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Mono(reactor.core.publisher.Mono) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Data(lombok.Data) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) AllArgsConstructor(lombok.AllArgsConstructor) Reference(org.osgi.service.component.annotations.Reference) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) NoArgsConstructor(lombok.NoArgsConstructor) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) Cancellable(org.elasticsearch.client.Cancellable) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) List(java.util.List) ArrayList(java.util.ArrayList) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) IndexRequest(org.elasticsearch.action.index.IndexRequest)

Example 2 with ReactorActionListener

use of io.openk9.search.client.api.ReactorActionListener in project openk9 by smclab.

the class ReindexDatasourceConsumer method activate.

@Activate
void activate() {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    _disposable = _datasourceEventConsumer.datasourceUpdateEvents().flatMap(datasource -> _pluginDriverManagerClient.getPluginDriver(datasource.getDriverServiceName()).onErrorResume(throwable -> {
        if (_log.isErrorEnabled()) {
            _log.error(throwable.getMessage());
        }
        return Mono.empty();
    }).map(pluginDriverDTO -> Tuples.of(datasource, datasource.getTenantId() + "-" + pluginDriverDTO.getName() + "-data"))).filterWhen(t2 -> Mono.create(sink -> restHighLevelClient.indices().existsAsync(new GetIndexRequest(t2.getT2()), RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).flatMap(t2 -> Mono.<GetSettingsResponse>create(sink -> restHighLevelClient.indices().getSettingsAsync(new GetSettingsRequest().indices(t2.getT2()).names("index.creation_date"), RequestOptions.DEFAULT, new ReactorActionListener<>(sink))).map(response -> Tuples.of(t2.getT1(), t2.getT2(), Instant.ofEpochMilli(Long.parseLong(response.getSetting(t2.getT2(), "index.creation_date")))))).log().filter(t3 -> t3.getT1().getLastIngestionDate().isBefore(t3.getT3())).flatMap(t3 -> Mono.<AcknowledgedResponse>create(sink -> restHighLevelClient.indices().deleteAsync(new DeleteIndexRequest(t3.getT2()), RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).onErrorContinue((throwable, ignore) -> {
        if (_log.isErrorEnabled()) {
            _log.error(throwable.getMessage());
        }
    }).subscribe();
}
Also used : ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) DatasourceEventConsumer(io.openk9.datasource.event.consumer.api.DatasourceEventConsumer) Logger(org.slf4j.Logger) Disposable(reactor.core.Disposable) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) Deactivate(org.osgi.service.component.annotations.Deactivate) LoggerFactory(org.slf4j.LoggerFactory) Tuples(reactor.util.function.Tuples) Mono(reactor.core.publisher.Mono) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Instant(java.time.Instant) Component(org.osgi.service.component.annotations.Component) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) Activate(org.osgi.service.component.annotations.Activate) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) Reference(org.osgi.service.component.annotations.Reference) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Activate(org.osgi.service.component.annotations.Activate)

Example 3 with ReactorActionListener

use of io.openk9.search.client.api.ReactorActionListener in project openk9 by smclab.

the class IndexWriterEndpoins method _deleteDataDocuments.

private Publisher<Void> _deleteDataDocuments(HttpServerRequest httpServerRequest, HttpServerResponse httpServerResponse) {
    RestHighLevelClient restHighLevelClient = _restHighLevelClientProvider.get();
    Mono<byte[]> body = ReactorNettyUtils.aggregateBodyAsByteArray(httpServerRequest);
    Mono<String> responseMono = body.map(bytes -> _jsonFactory.fromJson(bytes, DeleteEntitiesRequest.class)).flatMap(deleteEntitiesRequest -> {
        if (!deleteEntitiesRequest.getContentIds().isEmpty()) {
            BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
            for (String contentId : deleteEntitiesRequest.getContentIds()) {
                boolQuery.mustNot(matchQuery("contentId", contentId));
            }
            Mono<Datasource> datasourceMono = _datasourceClient.findDatasource(deleteEntitiesRequest.getDatasourceId());
            return datasourceMono.flatMap(datasource -> _pluginDriverManagerClient.getPluginDriver(datasource.getDriverServiceName()).map(pluginDriverDTO -> datasource.getTenantId() + "-" + pluginDriverDTO.getName() + "-data").map(DeleteByQueryRequest::new).map(deleteByQueryRequest -> deleteByQueryRequest.setQuery(boolQuery)).flatMap(deleteByQueryRequest -> Mono.<BulkByScrollResponse>create(sink -> restHighLevelClient.deleteByQueryAsync(deleteByQueryRequest, RequestOptions.DEFAULT, new ReactorActionListener<>(sink)))).map(Object::toString).doOnNext(_log::info).flatMap(response -> Mono.fromRunnable(() -> _sendCleanOrphanEntitiesRequest(datasource.getTenantId())).thenReturn(response)));
        }
        return Mono.empty();
    });
    return _httpResponseWriter.write(httpServerResponse, responseMono);
}
Also used : LoggerFactory(org.slf4j.LoggerFactory) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) JsonFactory(io.openk9.json.api.JsonFactory) DocumentEntityRequest(io.openk9.index.writer.entity.model.DocumentEntityRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) ReactorNettyUtils(io.openk9.reactor.netty.util.ReactorNettyUtils) IndexRequest(org.elasticsearch.action.index.IndexRequest) EventBus(io.vertx.core.eventbus.EventBus) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) PluginDriverManagerClient(io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) CountRequest(org.elasticsearch.client.core.CountRequest) TimeValue(org.elasticsearch.core.TimeValue) SearchHit(org.elasticsearch.search.SearchHit) ReactorActionListener(io.openk9.search.client.api.ReactorActionListener) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Message(io.vertx.core.eventbus.Message) DatasourceClient(io.openk9.datasource.client.api.DatasourceClient) Scroll(org.elasticsearch.search.Scroll) Collectors(java.util.stream.Collectors) BundleContext(org.osgi.framework.BundleContext) HttpServerRequest(reactor.netty.http.server.HttpServerRequest) List(java.util.List) QueryBuilders.matchQuery(org.elasticsearch.index.query.QueryBuilders.matchQuery) Builder(lombok.Builder) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) RestHighLevelClientProvider(io.openk9.search.client.api.RestHighLevelClientProvider) HttpServerRoutes(reactor.netty.http.server.HttpServerRoutes) HttpServerResponse(reactor.netty.http.server.HttpServerResponse) MatchQueryBuilder(org.elasticsearch.index.query.MatchQueryBuilder) XContentType(org.elasticsearch.common.xcontent.XContentType) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) HttpResponseWriter(io.openk9.http.util.HttpResponseWriter) Function(java.util.function.Function) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) CountResponse(org.elasticsearch.client.core.CountResponse) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) WriteRequest(org.elasticsearch.action.support.WriteRequest) Component(org.osgi.service.component.annotations.Component) Operator(org.elasticsearch.index.query.Operator) Cancellable(org.elasticsearch.client.Cancellable) SearchSourceBuilder(org.elasticsearch.search.builder.SearchSourceBuilder) Activate(org.osgi.service.component.annotations.Activate) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse) RouterHandler(io.openk9.http.web.RouterHandler) QueryBuilder(org.elasticsearch.index.query.QueryBuilder) Logger(org.slf4j.Logger) Datasource(io.openk9.model.Datasource) Publisher(org.reactivestreams.Publisher) Mono(reactor.core.publisher.Mono) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Data(lombok.Data) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) AllArgsConstructor(lombok.AllArgsConstructor) Reference(org.osgi.service.component.annotations.Reference) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) NoArgsConstructor(lombok.NoArgsConstructor) Datasource(io.openk9.model.Datasource) BoolQueryBuilder(org.elasticsearch.index.query.BoolQueryBuilder) DeleteByQueryRequest(org.elasticsearch.index.reindex.DeleteByQueryRequest) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient)

Aggregations

PluginDriverManagerClient (io.openk9.plugin.driver.manager.client.api.PluginDriverManagerClient)3 ReactorActionListener (io.openk9.search.client.api.ReactorActionListener)3 RestHighLevelClientProvider (io.openk9.search.client.api.RestHighLevelClientProvider)3 RequestOptions (org.elasticsearch.client.RequestOptions)3 RestHighLevelClient (org.elasticsearch.client.RestHighLevelClient)3 DatasourceClient (io.openk9.datasource.client.api.DatasourceClient)2 HttpResponseWriter (io.openk9.http.util.HttpResponseWriter)2 RouterHandler (io.openk9.http.web.RouterHandler)2 DocumentEntityRequest (io.openk9.index.writer.entity.model.DocumentEntityRequest)2 JsonFactory (io.openk9.json.api.JsonFactory)2 Datasource (io.openk9.model.Datasource)2 ReactorNettyUtils (io.openk9.reactor.netty.util.ReactorNettyUtils)2 EventBus (io.vertx.core.eventbus.EventBus)2 Message (io.vertx.core.eventbus.Message)2 ArrayList (java.util.ArrayList)2 Collection (java.util.Collection)2 HashMap (java.util.HashMap)2 HashSet (java.util.HashSet)2 List (java.util.List)2 Map (java.util.Map)2