Search in sources :

Example 6 with MetricTimeSeries

use of io.cdap.cdap.api.metrics.MetricTimeSeries in project cdap by caskdata.

the class SupportBundleRuntimeInfoTask method queryMetrics.

public JsonObject queryMetrics(String runId, String configuration, long startTs, long stopTs) {
    // startTs from run time but metrics already starts before that time
    int startQueryTime = (int) (startTs - 5000);
    JsonObject metrics = new JsonObject();
    try {
        // This program type tag can be null
        String typeTag = getMetricsTag(programType);
        Map<String, String> queryTags = new HashMap<>();
        queryTags.put("namespace", namespaceId.getNamespace());
        queryTags.put("app", appId.getApplication());
        queryTags.put("run", runId);
        if (typeTag != null) {
            queryTags.put(typeTag, programName.getProgram());
        }
        Collection<String> metricsNameList = remoteMetricsSystemClient.search(queryTags);
        List<MetricTimeSeries> metricTimeSeriesList = new ArrayList<>(remoteMetricsSystemClient.query(startQueryTime, (int) (stopTs), queryTags, metricsNameList));
        for (MetricTimeSeries timeSeries : metricTimeSeriesList) {
            if (!metrics.has(timeSeries.getMetricName())) {
                metrics.add(timeSeries.getMetricName(), new JsonArray());
            }
            for (TimeValue timeValue : timeSeries.getTimeValues()) {
                JsonObject time = new JsonObject();
                time.addProperty("time", timeValue.getTimestamp());
                time.addProperty("value", timeValue.getValue());
                metrics.getAsJsonArray(timeSeries.getMetricName()).add(time);
            }
        }
    } catch (IOException e) {
        LOG.warn("Failed to find metrics with run {} ", runId, e);
        return null;
    }
    return metrics;
}
Also used : JsonArray(com.google.gson.JsonArray) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) JsonObject(com.google.gson.JsonObject) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) IOException(java.io.IOException) TimeValue(io.cdap.cdap.api.dataset.lib.cube.TimeValue)

Example 7 with MetricTimeSeries

use of io.cdap.cdap.api.metrics.MetricTimeSeries in project cdap by caskdata.

the class RemoteMetricsSystemClient method query.

@Override
public Collection<MetricTimeSeries> query(int start, int end, int resolution, Map<String, String> tags, Collection<String> metrics, Collection<String> groupByTags) throws IOException {
    String metricsParam = Joiner.on("&metric=").join(metrics);
    if (!metricsParam.isEmpty()) {
        metricsParam = "&metric=" + metricsParam;
    }
    String tagsParam = Joiner.on("&tag=").withKeyValueSeparator(":").join(tags);
    if (!tagsParam.isEmpty()) {
        tagsParam = "&tag=" + tagsParam;
    }
    String groupByParam = Joiner.on("&groupBy=").join(groupByTags);
    if (!groupByParam.isEmpty()) {
        groupByParam = "&groupBy=" + groupByParam;
    }
    // Only query for aggregate metrics. Currently that's the only use case.
    String queryString = "aggregate=true&start=" + start + "&end=" + end + "&resolution=" + resolution + "s" + metricsParam + tagsParam + groupByParam;
    URL url = getBaseURI().resolve("query?" + queryString).toURL();
    HttpResponse response = HttpRequests.execute(HttpRequest.post(url).build(), REQUEST_CONFIG);
    if (response.getResponseCode() != HttpURLConnection.HTTP_OK) {
        throw new IOException("Failed to query for metrics " + metrics + ", with tags " + tags + ". Error code " + response.getResponseCode() + ", message " + response.getResponseBodyAsString());
    }
    MetricQueryResult queryResult = GSON.fromJson(response.getResponseBodyAsString(), MetricQueryResult.class);
    List<MetricTimeSeries> result = new ArrayList<>();
    for (MetricQueryResult.TimeSeries timeSeries : queryResult.getSeries()) {
        List<TimeValue> timeValues = Arrays.stream(timeSeries.getData()).map(tv -> new TimeValue(tv.getTime(), tv.getValue())).collect(Collectors.toList());
        result.add(new MetricTimeSeries(timeSeries.getMetricName(), timeSeries.getGrouping(), timeValues));
    }
    return result;
}
Also used : HttpURLConnection(java.net.HttpURLConnection) Retries(io.cdap.cdap.common.service.Retries) RetryStrategy(io.cdap.cdap.common.service.RetryStrategy) Arrays(java.util.Arrays) TypeToken(com.google.gson.reflect.TypeToken) RandomEndpointStrategy(io.cdap.cdap.common.discovery.RandomEndpointStrategy) NamespaceId(io.cdap.cdap.proto.id.NamespaceId) URL(java.net.URL) Inject(com.google.inject.Inject) HttpResponse(io.cdap.common.http.HttpResponse) EndpointStrategy(io.cdap.cdap.common.discovery.EndpointStrategy) RetryStrategies(io.cdap.cdap.common.service.RetryStrategies) DefaultHttpRequestConfig(io.cdap.cdap.common.http.DefaultHttpRequestConfig) ArrayList(java.util.ArrayList) Discoverable(org.apache.twill.discovery.Discoverable) DiscoveryServiceClient(org.apache.twill.discovery.DiscoveryServiceClient) Gson(com.google.gson.Gson) Map(java.util.Map) MetricDeleteQuery(io.cdap.cdap.api.metrics.MetricDeleteQuery) HttpRequests(io.cdap.common.http.HttpRequests) URI(java.net.URI) URIScheme(io.cdap.cdap.common.discovery.URIScheme) Collection(java.util.Collection) MessagingService(io.cdap.cdap.messaging.MessagingService) IOException(java.io.IOException) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) Collectors(java.util.stream.Collectors) TopicNotFoundException(io.cdap.cdap.api.messaging.TopicNotFoundException) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) CConfiguration(io.cdap.cdap.common.conf.CConfiguration) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) TimeValue(io.cdap.cdap.api.dataset.lib.cube.TimeValue) MetricsSystemClient(io.cdap.cdap.api.metrics.MetricsSystemClient) Constants(io.cdap.cdap.common.conf.Constants) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) MetricQueryResult(io.cdap.cdap.proto.MetricQueryResult) Joiner(com.google.common.base.Joiner) HttpRequest(io.cdap.common.http.HttpRequest) ArrayList(java.util.ArrayList) HttpResponse(io.cdap.common.http.HttpResponse) MetricQueryResult(io.cdap.cdap.proto.MetricQueryResult) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) IOException(java.io.IOException) URL(java.net.URL) TimeValue(io.cdap.cdap.api.dataset.lib.cube.TimeValue)

Example 8 with MetricTimeSeries

use of io.cdap.cdap.api.metrics.MetricTimeSeries in project cdap by caskdata.

the class DefaultMetricStore method query.

@Override
public Collection<MetricTimeSeries> query(MetricDataQuery query) {
    Collection<TimeSeries> cubeResult = cube.get().query(buildCubeQuery(query));
    List<MetricTimeSeries> result = Lists.newArrayList();
    for (TimeSeries timeSeries : cubeResult) {
        result.add(new MetricTimeSeries(timeSeries.getMeasureName(), timeSeries.getDimensionValues(), timeSeries.getTimeValues()));
    }
    return result;
}
Also used : TimeSeries(io.cdap.cdap.api.dataset.lib.cube.TimeSeries) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries)

Example 9 with MetricTimeSeries

use of io.cdap.cdap.api.metrics.MetricTimeSeries in project cdap by caskdata.

the class MetricsQueryHelper method executeQuery.

private MetricQueryResult executeQuery(MetricQueryRequest queryRequest) throws Exception {
    if (queryRequest.getMetrics().size() == 0) {
        throw new IllegalArgumentException("Missing metrics parameter in the query");
    }
    MetricQueryRequest.TimeRange timeRange = queryRequest.getTimeRange();
    AggregationOption aggregation = timeRange.getAggregation();
    if (timeRange.getCount() <= 0) {
        throw new IllegalArgumentException("Invalid metrics aggregation request, the limit must be greater than 0");
    }
    Map<String, String> tagsSliceBy = humanToTagNames(transformTagMap(queryRequest.getTags()));
    MetricDataQuery query = new MetricDataQuery(timeRange.getStart(), timeRange.getEnd(), timeRange.getResolutionInSeconds(), timeRange.getCount(), toMetrics(queryRequest.getMetrics()), tagsSliceBy, transformGroupByTags(queryRequest.getGroupBy()), aggregation, timeRange.getInterpolate());
    Collection<MetricTimeSeries> queryResult = metricStore.query(query);
    long endTime = timeRange.getEnd();
    if (timeRange.getResolutionInSeconds() == Integer.MAX_VALUE && endTime == 0) {
        // for aggregate query, we set the end time to be query time (current time)
        endTime = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis());
    }
    return decorate(queryResult, timeRange.getStart(), endTime, timeRange.getResolutionInSeconds());
}
Also used : MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) MetricDataQuery(io.cdap.cdap.api.metrics.MetricDataQuery) MetricQueryRequest(io.cdap.cdap.proto.MetricQueryRequest) AggregationOption(io.cdap.cdap.api.dataset.lib.cube.AggregationOption)

Example 10 with MetricTimeSeries

use of io.cdap.cdap.api.metrics.MetricTimeSeries in project cdap by caskdata.

the class MetricsQueryHelper method decorate.

private MetricQueryResult decorate(Collection<MetricTimeSeries> series, long startTs, long endTs, int resolution) {
    MetricQueryResult.TimeSeries[] serieses = new MetricQueryResult.TimeSeries[series.size()];
    int i = 0;
    for (MetricTimeSeries timeSeries : series) {
        MetricQueryResult.TimeValue[] timeValues = decorate(timeSeries.getTimeValues());
        serieses[i++] = new MetricQueryResult.TimeSeries(timeSeries.getMetricName(), tagNamesToHuman(timeSeries.getTagValues()), timeValues);
    }
    return new MetricQueryResult(startTs, endTs, serieses, resolution);
}
Also used : MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) MetricQueryResult(io.cdap.cdap.proto.MetricQueryResult) TimeValue(io.cdap.cdap.api.dataset.lib.cube.TimeValue)

Aggregations

MetricTimeSeries (io.cdap.cdap.api.metrics.MetricTimeSeries)52 MetricDataQuery (io.cdap.cdap.api.metrics.MetricDataQuery)30 TimeValue (io.cdap.cdap.api.dataset.lib.cube.TimeValue)28 Test (org.junit.Test)14 IOException (java.io.IOException)10 ArrayList (java.util.ArrayList)10 Collection (java.util.Collection)8 KeyValueTable (io.cdap.cdap.api.dataset.lib.KeyValueTable)6 Map (java.util.Map)6 NamespaceId (io.cdap.cdap.proto.id.NamespaceId)5 ImmutableMap (com.google.common.collect.ImmutableMap)4 MetricDeleteQuery (io.cdap.cdap.api.metrics.MetricDeleteQuery)4 MetricsSystemClient (io.cdap.cdap.api.metrics.MetricsSystemClient)4 CConfiguration (io.cdap.cdap.common.conf.CConfiguration)3 Constants (io.cdap.cdap.common.conf.Constants)3 ApplicationWithPrograms (io.cdap.cdap.internal.app.deploy.pipeline.ApplicationWithPrograms)3 MessagingService (io.cdap.cdap.messaging.MessagingService)3 Arrays (java.util.Arrays)3 TimeUnit (java.util.concurrent.TimeUnit)3 Function (com.google.common.base.Function)2