Search in sources :

Example 46 with Request

use of org.elasticsearch.client.Request in project sonarqube by SonarSource.

the class EsClient method indicesStats.

// https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html
public IndicesStatsResponse indicesStats(String... indices) {
    return execute(() -> {
        Request request = new Request("GET", "/" + (indices.length > 0 ? (String.join(",", indices) + "/") : "") + "_stats");
        request.addParameter("level", "shards");
        Response response = restHighLevelClient.getLowLevelClient().performRequest(request);
        return IndicesStatsResponse.toIndicesStatsResponse(gson.fromJson(EntityUtils.toString(response.getEntity()), JsonObject.class));
    }, () -> computeDetailsAsString(indices));
}
Also used : GetResponse(org.elasticsearch.action.get.GetResponse) ClearIndicesCacheResponse(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheResponse) ForceMergeResponse(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeResponse) GetIndexResponse(org.elasticsearch.client.indices.GetIndexResponse) SearchResponse(org.elasticsearch.action.search.SearchResponse) DeleteResponse(org.elasticsearch.action.delete.DeleteResponse) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Response(org.elasticsearch.client.Response) CreateIndexResponse(org.elasticsearch.client.indices.CreateIndexResponse) ClusterStatsResponse(org.sonar.server.es.response.ClusterStatsResponse) GetMappingsResponse(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) IndexResponse(org.elasticsearch.action.index.IndexResponse) NodeStatsResponse(org.sonar.server.es.response.NodeStatsResponse) ClusterHealthResponse(org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) RefreshResponse(org.elasticsearch.action.admin.indices.refresh.RefreshResponse) IndicesStatsResponse(org.sonar.server.es.response.IndicesStatsResponse) GetMappingsRequest(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) IndexRequest(org.elasticsearch.action.index.IndexRequest) GetRequest(org.elasticsearch.action.get.GetRequest) ClusterHealthRequest(org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest) ClearIndicesCacheRequest(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheRequest) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) SearchRequest(org.elasticsearch.action.search.SearchRequest) UpdateSettingsRequest(org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) RefreshRequest(org.elasticsearch.action.admin.indices.refresh.RefreshRequest) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) CreateIndexRequest(org.elasticsearch.client.indices.CreateIndexRequest) Request(org.elasticsearch.client.Request) PutMappingRequest(org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) JsonObject(com.google.gson.JsonObject)

Example 47 with Request

use of org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIOTestCommon method testMaxParallelRequestsPerWindow.

void testMaxParallelRequestsPerWindow() throws Exception {
    List<Document> data = ElasticsearchIOTestUtils.createDocuments(numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS).stream().map(doc -> Document.create().withInputDoc(doc).withTimestamp(Instant.now())).collect(Collectors.toList());
    Write write = ElasticsearchIO.write().withConnectionConfiguration(connectionConfiguration).withMaxParallelRequestsPerWindow(1);
    PCollection<KV<Integer, Iterable<Document>>> batches = pipeline.apply(Create.of(data)).apply(StatefulBatching.fromSpec(write.getBulkIO()));
    PCollection<Integer> keyValues = batches.apply(MapElements.into(integers()).via((SerializableFunction<KV<Integer, Iterable<Document>>, Integer>) KV::getKey));
    // Number of unique keys produced should be number of maxParallelRequestsPerWindow * numWindows
    // There is only 1 request (key) per window, and 1 (global) window ie. one key total where
    // key value is 0
    PAssert.that(keyValues).containsInAnyOrder(0);
    PAssert.that(batches).satisfies(new AssertThatHasExpectedContents(0, data));
    pipeline.run();
}
Also used : Read(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Read) Count(org.apache.beam.sdk.transforms.Count) Arrays(java.util.Arrays) BiFunction(java.util.function.BiFunction) LoggerFactory(org.slf4j.LoggerFactory) SerializableFunction(org.apache.beam.sdk.transforms.SerializableFunction) InjectionMode(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.InjectionMode) ElasticsearchIOTestUtils.countByMatch(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByMatch) DoFnTester(org.apache.beam.sdk.transforms.DoFnTester) FAMOUS_SCIENTISTS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.FAMOUS_SCIENTISTS) PipedInputStream(java.io.PipedInputStream) Create(org.apache.beam.sdk.transforms.Create) Map(java.util.Map) Is.is(org.hamcrest.core.Is.is) PCollectionTuple(org.apache.beam.sdk.values.PCollectionTuple) Assert.fail(org.junit.Assert.fail) JsonNode(com.fasterxml.jackson.databind.JsonNode) Document(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Document) NStringEntity(org.apache.http.nio.entity.NStringEntity) ElasticsearchIO.getBackendVersion(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion) ValueProvider(org.apache.beam.sdk.options.ValueProvider) DocToBulk(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.DocToBulk) MapElements(org.apache.beam.sdk.transforms.MapElements) ElasticsearchIOTestUtils.mapToInputId(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.mapToInputId) SourceTestUtils.readFromSource(org.apache.beam.sdk.testing.SourceTestUtils.readFromSource) CustomMatcher(org.hamcrest.CustomMatcher) ConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration) HttpEntity(org.apache.http.HttpEntity) ContentType(org.apache.http.entity.ContentType) Set(java.util.Set) ElasticsearchIOTestUtils.flushAndRefreshAllIndices(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.flushAndRefreshAllIndices) Collectors(java.util.stream.Collectors) TypeSafeMatcher(org.hamcrest.TypeSafeMatcher) StandardCharsets(java.nio.charset.StandardCharsets) Serializable(java.io.Serializable) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) Response(org.elasticsearch.client.Response) State(org.apache.beam.sdk.PipelineResult.State) Matchers.greaterThan(org.hamcrest.Matchers.greaterThan) SCRIPT_SOURCE(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.SCRIPT_SOURCE) BulkIO(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BulkIO) IntStream(java.util.stream.IntStream) RestClient(org.elasticsearch.client.RestClient) TypeDescriptors.integers(org.apache.beam.sdk.values.TypeDescriptors.integers) KV(org.apache.beam.sdk.values.KV) ElasticsearchIOTestUtils.refreshIndexAndGetCurrentNumDocs(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.refreshIndexAndGetCurrentNumDocs) Duration(org.joda.time.Duration) RetryPredicate(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.RetryPredicate) ElasticsearchIOTestUtils.insertTestDocuments(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.insertTestDocuments) PipelineOptionsFactory(org.apache.beam.sdk.options.PipelineOptionsFactory) INVALID_DOCS_IDS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.INVALID_DOCS_IDS) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) ArrayList(java.util.ArrayList) Write(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write) DefaultRetryPredicate(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DefaultRetryPredicate) ThreadLocalRandom(java.util.concurrent.ThreadLocalRandom) TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Matchers.lessThan(org.hamcrest.Matchers.lessThan) MatcherAssert.assertThat(org.hamcrest.MatcherAssert.assertThat) PipelineOptions(org.apache.beam.sdk.options.PipelineOptions) ExpectedException(org.junit.rules.ExpectedException) BoundedElasticsearchSource(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BoundedElasticsearchSource) Description(org.hamcrest.Description) SourceTestUtils(org.apache.beam.sdk.testing.SourceTestUtils) Logger(org.slf4j.Logger) StatefulBatching(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BulkIO.StatefulBatching) PAssert(org.apache.beam.sdk.testing.PAssert) ElasticsearchIOTestUtils.countByScientistName(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByScientistName) DocumentCoder(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.DocumentCoder) Assert.assertTrue(org.junit.Assert.assertTrue) IOException(java.io.IOException) PipedOutputStream(java.io.PipedOutputStream) PCollection(org.apache.beam.sdk.values.PCollection) Request(org.elasticsearch.client.Request) Is.isA(org.hamcrest.core.Is.isA) NUM_SCIENTISTS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.NUM_SCIENTISTS) BoundedSource(org.apache.beam.sdk.io.BoundedSource) DEFAULT_RETRY_PREDICATE(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DEFAULT_RETRY_PREDICATE) Matcher(org.hamcrest.Matcher) Instant(org.joda.time.Instant) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) IsIterableContainingInAnyOrder(org.hamcrest.collection.IsIterableContainingInAnyOrder) Write(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write) SerializableFunction(org.apache.beam.sdk.transforms.SerializableFunction) KV(org.apache.beam.sdk.values.KV) Document(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Document)

Example 48 with Request

use of org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIOTestCommon method testDefaultRetryPredicate.

/**
 * Test that the default predicate correctly parses chosen error code.
 */
void testDefaultRetryPredicate(RestClient restClient) throws IOException {
    HttpEntity entity1 = new NStringEntity(BAD_REQUEST, ContentType.APPLICATION_JSON);
    Request request = new Request("POST", "/_bulk");
    request.addParameters(Collections.emptyMap());
    request.setEntity(entity1);
    Response response1 = restClient.performRequest(request);
    assertTrue(CUSTOM_RETRY_PREDICATE.test(response1.getEntity()));
    HttpEntity entity2 = new NStringEntity(OK_REQUEST, ContentType.APPLICATION_JSON);
    request.setEntity(entity2);
    Response response2 = restClient.performRequest(request);
    assertFalse(DEFAULT_RETRY_PREDICATE.test(response2.getEntity()));
}
Also used : Response(org.elasticsearch.client.Response) NStringEntity(org.apache.http.nio.entity.NStringEntity) HttpEntity(org.apache.http.HttpEntity) Request(org.elasticsearch.client.Request)

Example 49 with Request

use of org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIO method getBackendVersion.

static int getBackendVersion(ConnectionConfiguration connectionConfiguration) {
    try (RestClient restClient = connectionConfiguration.createClient()) {
        Request request = new Request("GET", "");
        Response response = restClient.performRequest(request);
        JsonNode jsonNode = parseResponse(response.getEntity());
        int backendVersion = Integer.parseInt(jsonNode.path("version").path("number").asText().substring(0, 1));
        checkArgument(VALID_CLUSTER_VERSIONS.contains(backendVersion), "The Elasticsearch version to connect to is %s.x. " + "This version of the ElasticsearchIO is only compatible with " + "Elasticsearch v7.x, v6.x, v5.x and v2.x", backendVersion);
        return backendVersion;
    } catch (IOException e) {
        throw new IllegalArgumentException("Cannot get Elasticsearch version", e);
    }
}
Also used : Response(org.elasticsearch.client.Response) RestClient(org.elasticsearch.client.RestClient) Request(org.elasticsearch.client.Request) JsonNode(com.fasterxml.jackson.databind.JsonNode) IOException(java.io.IOException)

Example 50 with Request

use of org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIOTestUtils method deleteIndex.

static void deleteIndex(RestClient restClient, String index) throws IOException {
    try {
        closeIndex(restClient, index);
        Request request = new Request("DELETE", String.format("/%s", index));
        restClient.performRequest(request);
    } catch (IOException e) {
        // so when the first tests is run, the index does not exist yet
        if (!e.getMessage().contains("index_not_found_exception")) {
            throw e;
        }
    }
}
Also used : Request(org.elasticsearch.client.Request) IOException(java.io.IOException)

Aggregations

Request (org.elasticsearch.client.Request)55 Response (org.elasticsearch.client.Response)35 JsonNode (com.fasterxml.jackson.databind.JsonNode)14 IOException (java.io.IOException)12 HttpEntity (org.apache.http.HttpEntity)12 Request (org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request)12 NStringEntity (org.apache.http.nio.entity.NStringEntity)11 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)9 Map (java.util.Map)9 Collectors (java.util.stream.Collectors)9 StringEntity (org.apache.http.entity.StringEntity)7 RestClient (org.elasticsearch.client.RestClient)7 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)6 InputStream (java.io.InputStream)6 ContentType (org.apache.http.entity.ContentType)5 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)4 ImmutableMap (com.google.common.collect.ImmutableMap)4 UncheckedIOException (java.io.UncheckedIOException)4 ArrayList (java.util.ArrayList)4 HashMap (java.util.HashMap)4