Search in sources :

Example 31 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class ElasticSearchTypeImpl method search.

@Override
public SearchResponse<T> search(SearchRequest request) {
    validate(request);
    StopWatch watch = new StopWatch();
    long esTookTime = 0;
    String index = request.index == null ? this.index : request.index;
    long hits = 0;
    try {
        SearchRequestBuilder builder = client().prepareSearch(index).setQuery(request.query);
        if (request.type != null)
            builder.setSearchType(request.type);
        request.aggregations.forEach(builder::addAggregation);
        request.sorts.forEach(builder::addSort);
        if (request.skip != null)
            builder.setFrom(request.skip);
        if (request.limit != null)
            builder.setSize(request.limit);
        logger.debug("search, index={}, type={}, request={}", index, type, builder);
        org.elasticsearch.action.search.SearchResponse searchResponse = builder.get();
        hits = searchResponse.getHits().getTotalHits();
        esTookTime = searchResponse.getTook().nanos();
        if (searchResponse.getFailedShards() > 0)
            logger.warn("some shard failed, response={}", searchResponse);
        return searchResponse(searchResponse);
    } catch (ElasticsearchException e) {
        // due to elastic search uses async executor to run, we have to wrap the exception to retain the original place caused the exception
        throw new SearchException(e);
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("elasticsearch", elapsedTime, (int) hits, 0);
        logger.debug("search, hits={}, esTookTime={}, elapsedTime={}", hits, esTookTime, elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : SearchRequestBuilder(org.elasticsearch.action.search.SearchRequestBuilder) SearchException(core.framework.search.SearchException) ElasticsearchException(org.elasticsearch.ElasticsearchException) StopWatch(core.framework.util.StopWatch)

Example 32 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class ElasticSearchTypeImpl method deleteByQuery.

@Override
public long deleteByQuery(DeleteByQueryRequest request) {
    if (request.query == null)
        throw Exceptions.error("request.query must not be null");
    StopWatch watch = new StopWatch();
    String index = request.index == null ? this.index : request.index;
    long esTookTime = 0;
    long deleted = 0;
    try {
        DeleteByQueryRequestBuilder builder = DeleteByQueryAction.INSTANCE.newRequestBuilder(client());
        BulkByScrollResponse response = builder.filter(request.query).source(index).get();
        esTookTime = response.getTook().nanos();
        deleted = response.getDeleted();
        return deleted;
    } catch (ElasticsearchException e) {
        // due to elastic search uses async executor to run, we have to wrap the exception to retain the original place caused the exception
        throw new SearchException(e);
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("elasticsearch", elapsedTime, 0, (int) deleted);
        logger.debug("deleteByQuery, index={}, type={}, deleted={}, esTookTime={}, elapsedTime={}", index, type, deleted, esTookTime, elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : SearchException(core.framework.search.SearchException) DeleteByQueryRequestBuilder(org.elasticsearch.index.reindex.DeleteByQueryRequestBuilder) ElasticsearchException(org.elasticsearch.ElasticsearchException) StopWatch(core.framework.util.StopWatch) BulkByScrollResponse(org.elasticsearch.index.reindex.BulkByScrollResponse)

Example 33 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class ElasticSearchTypeImpl method index.

@Override
public void index(IndexRequest<T> request) {
    StopWatch watch = new StopWatch();
    String index = request.index == null ? this.index : request.index;
    validator.validate(request.source);
    byte[] document = writer.toJSON(request.source);
    try {
        client().prepareIndex(index, type, request.id).setSource(document, XContentType.JSON).get();
    } catch (ElasticsearchException e) {
        // due to elastic search uses async executor to run, we have to wrap the exception to retain the original place caused the exception
        throw new SearchException(e);
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("elasticsearch", elapsedTime, 0, 1);
        logger.debug("index, index={}, type={}, id={}, elapsedTime={}", index, type, request.id, elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : SearchException(core.framework.search.SearchException) ElasticsearchException(org.elasticsearch.ElasticsearchException) StopWatch(core.framework.util.StopWatch)

Example 34 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class HTTPClientImpl method execute.

@Override
public HTTPResponse execute(HTTPRequest request) {
    StopWatch watch = new StopWatch();
    HttpUriRequest httpRequest = httpRequest(request);
    try (CloseableHttpResponse httpResponse = client.execute(httpRequest)) {
        int statusCode = httpResponse.getStatusLine().getStatusCode();
        logger.debug("[response] status={}", statusCode);
        Map<String, String> headers = Maps.newHashMap();
        for (Header header : httpResponse.getAllHeaders()) {
            logger.debug("[response:header] {}={}", header.getName(), header.getValue());
            headers.putIfAbsent(header.getName(), header.getValue());
        }
        HttpEntity entity = httpResponse.getEntity();
        byte[] body = responseBody(entity);
        HTTPResponse response = new HTTPResponse(parseHTTPStatus(statusCode), headers, body);
        logResponseText(response);
        return response;
    } catch (IOException | UncheckedIOException e) {
        throw new HTTPClientException(e.getMessage(), "HTTP_COMMUNICATION_FAILED", e);
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("http", elapsedTime);
        logger.debug("execute, elapsedTime={}", elapsedTime);
        if (elapsedTime > slowOperationThresholdInNanos) {
            logger.warn(Markers.errorCode("SLOW_HTTP"), "slow http operation, elapsedTime={}", elapsedTime);
        }
    }
}
Also used : HttpUriRequest(org.apache.http.client.methods.HttpUriRequest) HttpEntity(org.apache.http.HttpEntity) HTTPResponse(core.framework.http.HTTPResponse) HTTPClientException(core.framework.http.HTTPClientException) UncheckedIOException(java.io.UncheckedIOException) IOException(java.io.IOException) UncheckedIOException(java.io.UncheckedIOException) StopWatch(core.framework.util.StopWatch) Header(org.apache.http.Header) CloseableHttpResponse(org.apache.http.client.methods.CloseableHttpResponse)

Example 35 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class Kafka method createProducer.

protected Producer<String, byte[]> createProducer() {
    if (uri == null)
        throw new Error("uri must not be null");
    StopWatch watch = new StopWatch();
    try {
        Map<String, Object> config = Maps.newHashMap();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, uri);
        // metadata update timeout
        config.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, Duration.ofSeconds(30).toMillis());
        Producer<String, byte[]> producer = new KafkaProducer<>(config, new StringSerializer(), new ByteArraySerializer());
        producerMetrics.set(producer.metrics());
        return producer;
    } finally {
        logger.info("create kafka producer, uri={}, name={}, elapsedTime={}", uri, name, watch.elapsedTime());
    }
}
Also used : KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) ByteArraySerializer(org.apache.kafka.common.serialization.ByteArraySerializer) StopWatch(core.framework.util.StopWatch)

Aggregations

StopWatch (core.framework.util.StopWatch)79 IOException (java.io.IOException)21 UncheckedIOException (java.io.UncheckedIOException)21 SearchException (core.framework.search.SearchException)10 ElasticsearchException (org.elasticsearch.ElasticsearchException)10 ArrayList (java.util.ArrayList)6 HTMLTemplate (core.framework.impl.template.HTMLTemplate)4 BytesParam (core.framework.impl.log.filter.BytesParam)3 Map (java.util.Map)3 BsonDocument (org.bson.BsonDocument)3 BulkWriteOptions (com.mongodb.client.model.BulkWriteOptions)2 UpdateOptions (com.mongodb.client.model.UpdateOptions)2 DeleteResult (com.mongodb.client.result.DeleteResult)2 TemplateContext (core.framework.impl.template.TemplateContext)2 Headers (org.apache.kafka.common.header.Headers)2 Bson (org.bson.conversions.Bson)2 BulkRequestBuilder (org.elasticsearch.action.bulk.BulkRequestBuilder)2 BulkResponse (org.elasticsearch.action.bulk.BulkResponse)2 Settings (org.elasticsearch.common.settings.Settings)2 MongoClient (com.mongodb.MongoClient)1