Search in sources :

Example 36 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class Kafka method consumer.

public Consumer<String, byte[]> consumer(String group, Set<String> topics) {
    if (uri == null)
        throw new Error("uri must not be null");
    StopWatch watch = new StopWatch();
    try {
        Map<String, Object> config = Maps.newHashMap();
        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, uri);
        config.put(ConsumerConfig.GROUP_ID_CONFIG, group);
        config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
        config.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, (int) maxProcessTime.toMillis());
        config.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, (int) maxProcessTime.plusSeconds(5).toMillis());
        config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecords);
        config.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, maxPollBytes);
        config.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, minPollBytes);
        config.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, (int) minPollMaxWaitTime.toMillis());
        Consumer<String, byte[]> consumer = new KafkaConsumer<>(config, new StringDeserializer(), new ByteArrayDeserializer());
        consumer.subscribe(topics);
        consumerMetrics.add(consumer.metrics());
        return consumer;
    } finally {
        logger.info("create kafka consumer, uri={}, name={}, topics={}, elapsedTime={}", uri, name, topics, watch.elapsedTime());
    }
}
Also used : StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) KafkaConsumer(org.apache.kafka.clients.consumer.KafkaConsumer) ByteArrayDeserializer(org.apache.kafka.common.serialization.ByteArrayDeserializer) StopWatch(core.framework.util.StopWatch)

Example 37 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class MongoCollectionImpl method find.

@Override
public List<T> find(Query query) {
    StopWatch watch = new StopWatch();
    List<T> results = query.limit == null ? Lists.newArrayList() : new ArrayList<>(query.limit);
    try {
        FindIterable<T> mongoQuery = mongoQuery(query).maxTime(mongo.timeoutInMs, TimeUnit.MILLISECONDS);
        fetch(mongoQuery, results);
        checkTooManyRowsReturned(results.size());
        return results;
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("mongoDB", elapsedTime, results.size(), 0);
        logger.debug("find, collection={}, filter={}, projection={}, sort={}, skip={}, limit={}, readPref={}, returnedRows={}, elapsedTime={}", collectionName, new BsonParam(query.filter, mongo.registry), new BsonParam(query.projection, mongo.registry), new BsonParam(query.sort, mongo.registry), query.skip, query.limit, query.readPreference == null ? null : query.readPreference.getName(), results.size(), elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : StopWatch(core.framework.util.StopWatch)

Example 38 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class MongoCollectionImpl method bulkReplace.

@Override
public void bulkReplace(List<T> entities) {
    StopWatch watch = new StopWatch();
    if (entities == null || entities.isEmpty())
        throw Exceptions.error("entities must not be empty");
    for (T entity : entities) {
        validator.validate(entity);
    }
    try {
        List<ReplaceOneModel<T>> models = new ArrayList<>(entities.size());
        for (T entity : entities) {
            Object id = mongo.codecs.id(entity);
            if (id == null)
                throw Exceptions.error("entity must have id, entityClass={}", entityClass.getCanonicalName());
            models.add(new ReplaceOneModel<>(Filters.eq("_id", id), entity, new UpdateOptions().upsert(true)));
        }
        collection().bulkWrite(models, new BulkWriteOptions().ordered(false));
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("mongoDB", elapsedTime, 0, entities.size());
        logger.debug("bulkReplace, collection={}, size={}, elapsedTime={}", collectionName, entities.size(), elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : BulkWriteOptions(com.mongodb.client.model.BulkWriteOptions) ReplaceOneModel(com.mongodb.client.model.ReplaceOneModel) ArrayList(java.util.ArrayList) UpdateOptions(com.mongodb.client.model.UpdateOptions) StopWatch(core.framework.util.StopWatch)

Example 39 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class MongoCollectionImpl method findOne.

@Override
public Optional<T> findOne(FindOne findOne) {
    StopWatch watch = new StopWatch();
    Bson filter = findOne.filter == null ? new BsonDocument() : findOne.filter;
    int returnedRows = 0;
    try {
        List<T> results = new ArrayList<>(2);
        FindIterable<T> query = collection().find(filter).limit(2).maxTime(mongo.timeoutInMs, TimeUnit.MILLISECONDS);
        fetch(query, results);
        if (results.isEmpty())
            return Optional.empty();
        if (results.size() > 1)
            throw Exceptions.error("more than one row returned, size={}", results.size());
        returnedRows = 1;
        return Optional.of(results.get(0));
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("mongoDB", elapsedTime, returnedRows, 0);
        logger.debug("findOne, collection={}, filter={}, readPref={}, elapsedTime={}", collectionName, new BsonParam(filter, mongo.registry), findOne.readPreference == null ? null : findOne.readPreference.getName(), elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : BsonDocument(org.bson.BsonDocument) ArrayList(java.util.ArrayList) StopWatch(core.framework.util.StopWatch) Bson(org.bson.conversions.Bson)

Example 40 with StopWatch

use of core.framework.util.StopWatch in project core-ng-project by neowu.

the class MongoCollectionImpl method count.

@Override
public long count(Count count) {
    StopWatch watch = new StopWatch();
    Bson filter = count.filter == null ? new BsonDocument() : count.filter;
    try {
        return collection(count.readPreference).count(filter, new CountOptions().maxTime(mongo.timeoutInMs, TimeUnit.MILLISECONDS));
    } finally {
        long elapsedTime = watch.elapsedTime();
        ActionLogContext.track("mongoDB", elapsedTime, 1, 0);
        logger.debug("count, collection={}, filter={}, readPref={}, elapsedTime={}", collectionName, new BsonParam(filter, mongo.registry), count.readPreference == null ? null : count.readPreference.getName(), elapsedTime);
        checkSlowOperation(elapsedTime);
    }
}
Also used : BsonDocument(org.bson.BsonDocument) CountOptions(com.mongodb.client.model.CountOptions) StopWatch(core.framework.util.StopWatch) Bson(org.bson.conversions.Bson)

Aggregations

StopWatch (core.framework.util.StopWatch)79 IOException (java.io.IOException)21 UncheckedIOException (java.io.UncheckedIOException)21 SearchException (core.framework.search.SearchException)10 ElasticsearchException (org.elasticsearch.ElasticsearchException)10 ArrayList (java.util.ArrayList)6 HTMLTemplate (core.framework.impl.template.HTMLTemplate)4 BytesParam (core.framework.impl.log.filter.BytesParam)3 Map (java.util.Map)3 BsonDocument (org.bson.BsonDocument)3 BulkWriteOptions (com.mongodb.client.model.BulkWriteOptions)2 UpdateOptions (com.mongodb.client.model.UpdateOptions)2 DeleteResult (com.mongodb.client.result.DeleteResult)2 TemplateContext (core.framework.impl.template.TemplateContext)2 Headers (org.apache.kafka.common.header.Headers)2 Bson (org.bson.conversions.Bson)2 BulkRequestBuilder (org.elasticsearch.action.bulk.BulkRequestBuilder)2 BulkResponse (org.elasticsearch.action.bulk.BulkResponse)2 Settings (org.elasticsearch.common.settings.Settings)2 MongoClient (com.mongodb.MongoClient)1