Search in sources :

Example 6 with SliceBuilder

use of org.elasticsearch.search.slice.SliceBuilder in project elasticsearch by elastic.

the class ReindexRequestTests method testNoSliceWithWorkers.

public void testNoSliceWithWorkers() {
    ReindexRequest reindex = newRequest();
    reindex.getSearchRequest().source().slice(new SliceBuilder(0, 4));
    reindex.setSlices(between(2, Integer.MAX_VALUE));
    ActionRequestValidationException e = reindex.validate();
    assertEquals("Validation Failed: 1: can't specify both slice and workers;", e.getMessage());
}
Also used : SliceBuilder(org.elasticsearch.search.slice.SliceBuilder) ActionRequestValidationException(org.elasticsearch.action.ActionRequestValidationException)

Example 7 with SliceBuilder

use of org.elasticsearch.search.slice.SliceBuilder in project hazelcast by hazelcast.

the class ElasticSourceP method init.

@Override
protected void init(@Nonnull Context context) throws Exception {
    super.init(context);
    logger = context.logger();
    logger.fine("init");
    client = configuration.clientFn().get();
    SearchRequest sr = configuration.searchRequestFn().get();
    sr.scroll(configuration.scrollKeepAlive());
    if (configuration.isSlicingEnabled()) {
        if (configuration.isCoLocatedReadingEnabled()) {
            int sliceId = context.localProcessorIndex();
            int totalSlices = context.localParallelism();
            if (totalSlices > 1) {
                logger.fine("Slice id=" + sliceId + ", max=" + totalSlices);
                sr.source().slice(new SliceBuilder(sliceId, totalSlices));
            }
        } else {
            int sliceId = context.globalProcessorIndex();
            int totalSlices = context.totalParallelism();
            if (totalSlices > 1) {
                logger.fine("Slice id=" + sliceId + ", max=" + totalSlices);
                sr.source().slice(new SliceBuilder(sliceId, totalSlices));
            }
        }
    }
    if (configuration.isCoLocatedReadingEnabled()) {
        logger.fine("Assigned shards: " + shards);
        if (shards.isEmpty()) {
            traverser = Traversers.empty();
            return;
        }
        Node node = createLocalElasticNode();
        client.getLowLevelClient().setNodes(singleton(node));
        String preference = "_shards:" + shards.stream().map(shard -> String.valueOf(shard.getShard())).collect(joining(",")) + "|_only_local";
        sr.preference(preference);
    }
    scrollTraverser = new ElasticScrollTraverser(configuration, client, sr, logger);
    traverser = scrollTraverser.map(configuration.mapToItemFn());
}
Also used : AbstractProcessor(com.hazelcast.jet.core.AbstractProcessor) Traverser(com.hazelcast.jet.Traverser) SearchHits(org.elasticsearch.search.SearchHits) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) SearchRequest(org.elasticsearch.action.search.SearchRequest) Node(org.elasticsearch.client.Node) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) JetException(com.hazelcast.jet.JetException) Collections.singleton(java.util.Collections.singleton) ILogger(com.hazelcast.logging.ILogger) Objects.requireNonNull(java.util.Objects.requireNonNull) SearchResponse(org.elasticsearch.action.search.SearchResponse) RequestOptions(org.elasticsearch.client.RequestOptions) Nonnull(javax.annotation.Nonnull) RetryUtils.withRetry(com.hazelcast.jet.elastic.impl.RetryUtils.withRetry) SearchHit(org.elasticsearch.search.SearchHit) FunctionEx(com.hazelcast.function.FunctionEx) ActionRequest(org.elasticsearch.action.ActionRequest) SliceBuilder(org.elasticsearch.search.slice.SliceBuilder) Traversers(com.hazelcast.jet.Traversers) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) Collectors.joining(java.util.stream.Collectors.joining) TotalHits(org.apache.lucene.search.TotalHits) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) HttpHost(org.apache.http.HttpHost) SearchRequest(org.elasticsearch.action.search.SearchRequest) SliceBuilder(org.elasticsearch.search.slice.SliceBuilder) Node(org.elasticsearch.client.Node)

Aggregations

SliceBuilder (org.elasticsearch.search.slice.SliceBuilder)7 SearchRequest (org.elasticsearch.action.search.SearchRequest)4 TotalHits (org.apache.lucene.search.TotalHits)3 SearchHit (org.elasticsearch.search.SearchHit)3 SearchHits (org.elasticsearch.search.SearchHits)3 TestSupport (com.hazelcast.jet.core.test.TestSupport)2 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)2 QuickTest (com.hazelcast.test.annotation.QuickTest)2 ArrayList (java.util.ArrayList)2 SearchSourceBuilder (org.elasticsearch.search.builder.SearchSourceBuilder)2 Test (org.junit.Test)2 FunctionEx (com.hazelcast.function.FunctionEx)1 JetException (com.hazelcast.jet.JetException)1 Traverser (com.hazelcast.jet.Traverser)1 Traversers (com.hazelcast.jet.Traversers)1 AbstractProcessor (com.hazelcast.jet.core.AbstractProcessor)1 RetryUtils.withRetry (com.hazelcast.jet.elastic.impl.RetryUtils.withRetry)1 ILogger (com.hazelcast.logging.ILogger)1 IOException (java.io.IOException)1 Collections.singleton (java.util.Collections.singleton)1