Search in sources :

Example 61 with Request

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request in project hazelcast by hazelcast.

the class ElasticCatClient method master.

/**
 * Returns current master of the ES cluster
 */
public Master master() {
    try {
        Request r = new Request("GET", "/_cat/master");
        r.addParameter("format", "json");
        Response res = withRetry(() -> client.performRequest(r), retries);
        try (InputStreamReader reader = new InputStreamReader(res.getEntity().getContent(), UTF_8)) {
            JsonArray array = Json.parse(reader).asArray();
            JsonObject object = array.get(0).asObject();
            return new Master(object.get("host").asString(), object.get("id").asString(), object.get("ip").asString(), object.get("node").asString());
        }
    } catch (IOException e) {
        throw new JetException("Could not get ES cluster master", e);
    }
}
Also used : Response(org.elasticsearch.client.Response) JsonArray(com.hazelcast.internal.json.JsonArray) InputStreamReader(java.io.InputStreamReader) Request(org.elasticsearch.client.Request) JsonObject(com.hazelcast.internal.json.JsonObject) IOException(java.io.IOException) JetException(com.hazelcast.jet.JetException)

Example 62 with Request

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request in project sonarqube by SonarSource.

the class EsClient method clusterStats.

// https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html
public ClusterStatsResponse clusterStats() {
    return execute(() -> {
        Request request = new Request("GET", "/_cluster/stats");
        Response response = restHighLevelClient.getLowLevelClient().performRequest(request);
        return ClusterStatsResponse.toClusterStatsResponse(gson.fromJson(EntityUtils.toString(response.getEntity()), JsonObject.class));
    });
}
Also used : GetResponse(org.elasticsearch.action.get.GetResponse) ClearIndicesCacheResponse(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheResponse) ForceMergeResponse(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeResponse) GetIndexResponse(org.elasticsearch.client.indices.GetIndexResponse) SearchResponse(org.elasticsearch.action.search.SearchResponse) DeleteResponse(org.elasticsearch.action.delete.DeleteResponse) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Response(org.elasticsearch.client.Response) CreateIndexResponse(org.elasticsearch.client.indices.CreateIndexResponse) ClusterStatsResponse(org.sonar.server.es.response.ClusterStatsResponse) GetMappingsResponse(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) IndexResponse(org.elasticsearch.action.index.IndexResponse) NodeStatsResponse(org.sonar.server.es.response.NodeStatsResponse) ClusterHealthResponse(org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) RefreshResponse(org.elasticsearch.action.admin.indices.refresh.RefreshResponse) IndicesStatsResponse(org.sonar.server.es.response.IndicesStatsResponse) GetMappingsRequest(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) IndexRequest(org.elasticsearch.action.index.IndexRequest) GetRequest(org.elasticsearch.action.get.GetRequest) ClusterHealthRequest(org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest) ClearIndicesCacheRequest(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheRequest) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) SearchRequest(org.elasticsearch.action.search.SearchRequest) UpdateSettingsRequest(org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) RefreshRequest(org.elasticsearch.action.admin.indices.refresh.RefreshRequest) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) CreateIndexRequest(org.elasticsearch.client.indices.CreateIndexRequest) Request(org.elasticsearch.client.Request) PutMappingRequest(org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) JsonObject(com.google.gson.JsonObject)

Example 63 with Request

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request in project sonarqube by SonarSource.

the class EsClient method indicesStats.

// https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html
public IndicesStatsResponse indicesStats(String... indices) {
    return execute(() -> {
        Request request = new Request("GET", "/" + (indices.length > 0 ? (String.join(",", indices) + "/") : "") + "_stats");
        request.addParameter("level", "shards");
        Response response = restHighLevelClient.getLowLevelClient().performRequest(request);
        return IndicesStatsResponse.toIndicesStatsResponse(gson.fromJson(EntityUtils.toString(response.getEntity()), JsonObject.class));
    }, () -> computeDetailsAsString(indices));
}
Also used : GetResponse(org.elasticsearch.action.get.GetResponse) ClearIndicesCacheResponse(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheResponse) ForceMergeResponse(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeResponse) GetIndexResponse(org.elasticsearch.client.indices.GetIndexResponse) SearchResponse(org.elasticsearch.action.search.SearchResponse) DeleteResponse(org.elasticsearch.action.delete.DeleteResponse) GetSettingsResponse(org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) Response(org.elasticsearch.client.Response) CreateIndexResponse(org.elasticsearch.client.indices.CreateIndexResponse) ClusterStatsResponse(org.sonar.server.es.response.ClusterStatsResponse) GetMappingsResponse(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse) ClearScrollResponse(org.elasticsearch.action.search.ClearScrollResponse) IndexResponse(org.elasticsearch.action.index.IndexResponse) NodeStatsResponse(org.sonar.server.es.response.NodeStatsResponse) ClusterHealthResponse(org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse) AcknowledgedResponse(org.elasticsearch.action.support.master.AcknowledgedResponse) RefreshResponse(org.elasticsearch.action.admin.indices.refresh.RefreshResponse) IndicesStatsResponse(org.sonar.server.es.response.IndicesStatsResponse) GetMappingsRequest(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest) DeleteRequest(org.elasticsearch.action.delete.DeleteRequest) IndexRequest(org.elasticsearch.action.index.IndexRequest) GetRequest(org.elasticsearch.action.get.GetRequest) ClusterHealthRequest(org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest) ClearIndicesCacheRequest(org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheRequest) GetSettingsRequest(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest) GetIndexRequest(org.elasticsearch.client.indices.GetIndexRequest) ClearScrollRequest(org.elasticsearch.action.search.ClearScrollRequest) SearchRequest(org.elasticsearch.action.search.SearchRequest) UpdateSettingsRequest(org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) RefreshRequest(org.elasticsearch.action.admin.indices.refresh.RefreshRequest) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) CreateIndexRequest(org.elasticsearch.client.indices.CreateIndexRequest) Request(org.elasticsearch.client.Request) PutMappingRequest(org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest) SearchScrollRequest(org.elasticsearch.action.search.SearchScrollRequest) BulkRequest(org.elasticsearch.action.bulk.BulkRequest) JsonObject(com.google.gson.JsonObject)

Example 64 with Request

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIOTestCommon method testMaxParallelRequestsPerWindow.

void testMaxParallelRequestsPerWindow() throws Exception {
    List<Document> data = ElasticsearchIOTestUtils.createDocuments(numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS).stream().map(doc -> Document.create().withInputDoc(doc).withTimestamp(Instant.now())).collect(Collectors.toList());
    Write write = ElasticsearchIO.write().withConnectionConfiguration(connectionConfiguration).withMaxParallelRequestsPerWindow(1);
    PCollection<KV<Integer, Iterable<Document>>> batches = pipeline.apply(Create.of(data)).apply(StatefulBatching.fromSpec(write.getBulkIO()));
    PCollection<Integer> keyValues = batches.apply(MapElements.into(integers()).via((SerializableFunction<KV<Integer, Iterable<Document>>, Integer>) KV::getKey));
    // Number of unique keys produced should be number of maxParallelRequestsPerWindow * numWindows
    // There is only 1 request (key) per window, and 1 (global) window ie. one key total where
    // key value is 0
    PAssert.that(keyValues).containsInAnyOrder(0);
    PAssert.that(batches).satisfies(new AssertThatHasExpectedContents(0, data));
    pipeline.run();
}
Also used : Read(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Read) Count(org.apache.beam.sdk.transforms.Count) Arrays(java.util.Arrays) BiFunction(java.util.function.BiFunction) LoggerFactory(org.slf4j.LoggerFactory) SerializableFunction(org.apache.beam.sdk.transforms.SerializableFunction) InjectionMode(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.InjectionMode) ElasticsearchIOTestUtils.countByMatch(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByMatch) DoFnTester(org.apache.beam.sdk.transforms.DoFnTester) FAMOUS_SCIENTISTS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.FAMOUS_SCIENTISTS) PipedInputStream(java.io.PipedInputStream) Create(org.apache.beam.sdk.transforms.Create) Map(java.util.Map) Is.is(org.hamcrest.core.Is.is) PCollectionTuple(org.apache.beam.sdk.values.PCollectionTuple) Assert.fail(org.junit.Assert.fail) JsonNode(com.fasterxml.jackson.databind.JsonNode) Document(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Document) NStringEntity(org.apache.http.nio.entity.NStringEntity) ElasticsearchIO.getBackendVersion(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion) ValueProvider(org.apache.beam.sdk.options.ValueProvider) DocToBulk(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.DocToBulk) MapElements(org.apache.beam.sdk.transforms.MapElements) ElasticsearchIOTestUtils.mapToInputId(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.mapToInputId) SourceTestUtils.readFromSource(org.apache.beam.sdk.testing.SourceTestUtils.readFromSource) CustomMatcher(org.hamcrest.CustomMatcher) ConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration) HttpEntity(org.apache.http.HttpEntity) ContentType(org.apache.http.entity.ContentType) Set(java.util.Set) ElasticsearchIOTestUtils.flushAndRefreshAllIndices(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.flushAndRefreshAllIndices) Collectors(java.util.stream.Collectors) TypeSafeMatcher(org.hamcrest.TypeSafeMatcher) StandardCharsets(java.nio.charset.StandardCharsets) Serializable(java.io.Serializable) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) Response(org.elasticsearch.client.Response) State(org.apache.beam.sdk.PipelineResult.State) Matchers.greaterThan(org.hamcrest.Matchers.greaterThan) SCRIPT_SOURCE(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.SCRIPT_SOURCE) BulkIO(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BulkIO) IntStream(java.util.stream.IntStream) RestClient(org.elasticsearch.client.RestClient) TypeDescriptors.integers(org.apache.beam.sdk.values.TypeDescriptors.integers) KV(org.apache.beam.sdk.values.KV) ElasticsearchIOTestUtils.refreshIndexAndGetCurrentNumDocs(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.refreshIndexAndGetCurrentNumDocs) Duration(org.joda.time.Duration) RetryPredicate(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.RetryPredicate) ElasticsearchIOTestUtils.insertTestDocuments(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.insertTestDocuments) PipelineOptionsFactory(org.apache.beam.sdk.options.PipelineOptionsFactory) INVALID_DOCS_IDS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.INVALID_DOCS_IDS) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) ArrayList(java.util.ArrayList) Write(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write) DefaultRetryPredicate(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DefaultRetryPredicate) ThreadLocalRandom(java.util.concurrent.ThreadLocalRandom) TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Matchers.lessThan(org.hamcrest.Matchers.lessThan) MatcherAssert.assertThat(org.hamcrest.MatcherAssert.assertThat) PipelineOptions(org.apache.beam.sdk.options.PipelineOptions) ExpectedException(org.junit.rules.ExpectedException) BoundedElasticsearchSource(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BoundedElasticsearchSource) Description(org.hamcrest.Description) SourceTestUtils(org.apache.beam.sdk.testing.SourceTestUtils) Logger(org.slf4j.Logger) StatefulBatching(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BulkIO.StatefulBatching) PAssert(org.apache.beam.sdk.testing.PAssert) ElasticsearchIOTestUtils.countByScientistName(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByScientistName) DocumentCoder(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.DocumentCoder) Assert.assertTrue(org.junit.Assert.assertTrue) IOException(java.io.IOException) PipedOutputStream(java.io.PipedOutputStream) PCollection(org.apache.beam.sdk.values.PCollection) Request(org.elasticsearch.client.Request) Is.isA(org.hamcrest.core.Is.isA) NUM_SCIENTISTS(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.NUM_SCIENTISTS) BoundedSource(org.apache.beam.sdk.io.BoundedSource) DEFAULT_RETRY_PREDICATE(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DEFAULT_RETRY_PREDICATE) Matcher(org.hamcrest.Matcher) Instant(org.joda.time.Instant) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) IsIterableContainingInAnyOrder(org.hamcrest.collection.IsIterableContainingInAnyOrder) Write(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write) SerializableFunction(org.apache.beam.sdk.transforms.SerializableFunction) KV(org.apache.beam.sdk.values.KV) Document(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Document)

Example 65 with Request

use of org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request in project beam by apache.

the class ElasticsearchIOTestCommon method testDefaultRetryPredicate.

/**
 * Test that the default predicate correctly parses chosen error code.
 */
void testDefaultRetryPredicate(RestClient restClient) throws IOException {
    HttpEntity entity1 = new NStringEntity(BAD_REQUEST, ContentType.APPLICATION_JSON);
    Request request = new Request("POST", "/_bulk");
    request.addParameters(Collections.emptyMap());
    request.setEntity(entity1);
    Response response1 = restClient.performRequest(request);
    assertTrue(CUSTOM_RETRY_PREDICATE.test(response1.getEntity()));
    HttpEntity entity2 = new NStringEntity(OK_REQUEST, ContentType.APPLICATION_JSON);
    request.setEntity(entity2);
    Response response2 = restClient.performRequest(request);
    assertFalse(DEFAULT_RETRY_PREDICATE.test(response2.getEntity()));
}
Also used : Response(org.elasticsearch.client.Response) NStringEntity(org.apache.http.nio.entity.NStringEntity) HttpEntity(org.apache.http.HttpEntity) Request(org.elasticsearch.client.Request)

Aggregations

Request (org.elasticsearch.client.Request)54 Response (org.elasticsearch.client.Response)34 JsonNode (com.fasterxml.jackson.databind.JsonNode)15 IOException (java.io.IOException)12 Request (org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Request)12 HttpEntity (org.apache.http.HttpEntity)11 NStringEntity (org.apache.http.nio.entity.NStringEntity)11 Map (java.util.Map)10 Collectors (java.util.stream.Collectors)10 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)9 SearchRequest (org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest)7 Response (org.graylog.shaded.elasticsearch7.org.elasticsearch.client.Response)7 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)6 InputStream (java.io.InputStream)6 StringEntity (org.apache.http.entity.StringEntity)6 RestClient (org.elasticsearch.client.RestClient)6 ImmutableMap (com.google.common.collect.ImmutableMap)5 HashMap (java.util.HashMap)5 Locale (java.util.Locale)5 ContentType (org.apache.http.entity.ContentType)5