Search in sources :

Example 1 with ParsedStats

use of org.opensearch.search.aggregations.metrics.ParsedStats in project kestra by kestra-io.

the class ElasticSearchExecutionRepository method dailyStatistics.

@Override
public List<DailyExecutionStatistics> dailyStatistics(@Nullable String query, @Nullable ZonedDateTime startDate, @Nullable ZonedDateTime endDate, boolean isTaskRun) {
    if (startDate == null) {
        startDate = ZonedDateTime.now().minusDays(30);
    }
    if (endDate == null) {
        endDate = ZonedDateTime.now();
    }
    AggregationBuilder agg = dailyExecutionStatisticsFinalAgg(startDate, endDate, isTaskRun);
    if (isTaskRun) {
        agg = AggregationBuilders.nested(NESTED_AGG, "taskRunList").subAggregation(agg);
    }
    SearchSourceBuilder sourceBuilder = this.searchSource(this.dateFilters(query, startDate, endDate), Optional.of(Collections.singletonList(agg)), null);
    try {
        SearchRequest searchRequest = searchRequest(INDEX_NAME, sourceBuilder, false);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        ParsedDateHistogram groupAgg = isTaskRun ? ((ParsedNested) searchResponse.getAggregations().get(NESTED_AGG)).getAggregations().get(DATE_AGG) : searchResponse.getAggregations().get(DATE_AGG);
        List<DailyExecutionStatistics> result = new ArrayList<>();
        groupAgg.getBuckets().forEach(bucket -> {
            ParsedStringTerms countAgg = bucket.getAggregations().get(COUNT_AGG);
            ParsedStats durationAgg = bucket.getAggregations().get(DURATION_AGG);
            final LocalDate currentStartDate = LocalDate.parse(bucket.getKeyAsString(), DateTimeFormatter.ISO_LOCAL_DATE);
            result.add(dailyExecutionStatisticsMap(countAgg, durationAgg, currentStartDate));
        });
        return result;
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}
Also used : ParsedStats(org.opensearch.search.aggregations.metrics.ParsedStats) SearchRequest(org.opensearch.action.search.SearchRequest) DateHistogramAggregationBuilder(org.opensearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) TermsAggregationBuilder(org.opensearch.search.aggregations.bucket.terms.TermsAggregationBuilder) NestedAggregationBuilder(org.opensearch.search.aggregations.bucket.nested.NestedAggregationBuilder) ParsedDateHistogram(org.opensearch.search.aggregations.bucket.histogram.ParsedDateHistogram) ParsedNested(org.opensearch.search.aggregations.bucket.nested.ParsedNested) IOException(java.io.IOException) DailyExecutionStatistics(io.kestra.core.models.executions.statistics.DailyExecutionStatistics) ParsedStringTerms(org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms) LocalDate(java.time.LocalDate) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) SearchResponse(org.opensearch.action.search.SearchResponse)

Example 2 with ParsedStats

use of org.opensearch.search.aggregations.metrics.ParsedStats in project kestra by kestra-io.

the class ElasticSearchExecutionRepository method parseDateAgg.

private <T extends Terms.Bucket> void parseDateAgg(T bucket, Map<String, Map<String, List<DailyExecutionStatistics>>> result, String namespace, String flowId) {
    ((ParsedDateHistogram) bucket.getAggregations().get(DATE_AGG)).getBuckets().forEach(dateBucket -> {
        final LocalDate currentStartDate = LocalDate.parse(dateBucket.getKeyAsString(), DateTimeFormatter.ISO_LOCAL_DATE);
        ParsedStringTerms countAgg = dateBucket.getAggregations().get(COUNT_AGG);
        ParsedStats durationAgg = dateBucket.getAggregations().get(DURATION_AGG);
        result.compute(namespace, (namespaceKey, namespaceMap) -> {
            if (namespaceMap == null) {
                namespaceMap = new HashMap<>();
            }
            namespaceMap.compute(flowId, (flowKey, flowList) -> {
                if (flowList == null) {
                    flowList = new ArrayList<>();
                }
                flowList.add(dailyExecutionStatisticsMap(countAgg, durationAgg, currentStartDate));
                return flowList;
            });
            return namespaceMap;
        });
    });
}
Also used : ParsedStats(org.opensearch.search.aggregations.metrics.ParsedStats) ParsedStringTerms(org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms) LocalDate(java.time.LocalDate)

Example 3 with ParsedStats

use of org.opensearch.search.aggregations.metrics.ParsedStats in project kestra by kestra-io.

the class ElasticSearchExecutionRepository method dailyExecutionStatisticsMap.

private static DailyExecutionStatistics dailyExecutionStatisticsMap(ParsedStringTerms countAgg, ParsedStats durationAgg, LocalDate startDate) {
    DailyExecutionStatistics build = DailyExecutionStatistics.builder().startDate(startDate).duration(DailyExecutionStatistics.Duration.builder().avg(durationFromDouble(durationAgg.getAvg())).min(durationFromDouble(durationAgg.getMin())).max(durationFromDouble(durationAgg.getMax())).sum(durationFromDouble(durationAgg.getSum())).count(durationAgg.getCount()).build()).build();
    countAgg.getBuckets().forEach(item -> build.getExecutionCounts().compute(State.Type.valueOf(item.getKeyAsString()), (type, current) -> item.getDocCount()));
    return build;
}
Also used : TaskRun(io.kestra.core.models.executions.TaskRun) RequestOptions(org.opensearch.client.RequestOptions) java.util(java.util) ExecutionRepositoryInterface(io.kestra.core.repositories.ExecutionRepositoryInterface) Flow(io.kestra.core.models.executions.statistics.Flow) ParsedStringTerms(org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms) ZonedDateTime(java.time.ZonedDateTime) DateHistogramAggregationBuilder(org.opensearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder) DateHistogramInterval(org.opensearch.search.aggregations.bucket.histogram.DateHistogramInterval) ExecutionCount(io.kestra.core.models.executions.statistics.ExecutionCount) ParsedDateHistogram(org.opensearch.search.aggregations.bucket.histogram.ParsedDateHistogram) State(io.kestra.core.models.flows.State) LongBounds(org.opensearch.search.aggregations.bucket.histogram.LongBounds) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) FiltersAggregator(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator) Nullable(io.micronaut.core.annotation.Nullable) Duration(java.time.Duration) Pageable(io.micronaut.data.model.Pageable) SearchRequest(org.opensearch.action.search.SearchRequest) SearchResponse(org.opensearch.action.search.SearchResponse) RestHighLevelClient(org.opensearch.client.RestHighLevelClient) ParsedFilters(org.opensearch.search.aggregations.bucket.filter.ParsedFilters) QueryBuilders(org.opensearch.index.query.QueryBuilders) Filter(org.opensearch.search.aggregations.bucket.filter.Filter) TopHits(org.opensearch.search.aggregations.metrics.TopHits) Singleton(jakarta.inject.Singleton) Settings(org.opensearch.common.settings.Settings) IOException(java.io.IOException) ScoreMode(org.apache.lucene.search.join.ScoreMode) Terms(org.opensearch.search.aggregations.bucket.terms.Terms) Execution(io.kestra.core.models.executions.Execution) Collectors(java.util.stream.Collectors) DailyExecutionStatistics(io.kestra.core.models.executions.statistics.DailyExecutionStatistics) ZoneId(java.time.ZoneId) AggregationBuilders(org.opensearch.search.aggregations.AggregationBuilders) ParsedStats(org.opensearch.search.aggregations.metrics.ParsedStats) ArrayListTotal(io.kestra.core.repositories.ArrayListTotal) ExecutorsUtils(io.kestra.core.utils.ExecutorsUtils) TermsAggregationBuilder(org.opensearch.search.aggregations.bucket.terms.TermsAggregationBuilder) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) LocalDate(java.time.LocalDate) DateTimeFormatter(java.time.format.DateTimeFormatter) IndexSettings(org.opensearch.index.IndexSettings) ModelValidator(io.kestra.core.models.validations.ModelValidator) NestedAggregationBuilder(org.opensearch.search.aggregations.bucket.nested.NestedAggregationBuilder) Inject(jakarta.inject.Inject) ParsedNested(org.opensearch.search.aggregations.bucket.nested.ParsedNested) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) DailyExecutionStatistics(io.kestra.core.models.executions.statistics.DailyExecutionStatistics)

Aggregations

LocalDate (java.time.LocalDate)3 DailyExecutionStatistics (io.kestra.core.models.executions.statistics.DailyExecutionStatistics)2 IOException (java.io.IOException)2 SearchRequest (org.opensearch.action.search.SearchRequest)2 SearchResponse (org.opensearch.action.search.SearchResponse)2 AggregationBuilder (org.opensearch.search.aggregations.AggregationBuilder)2 DateHistogramAggregationBuilder (org.opensearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder)2 ParsedDateHistogram (org.opensearch.search.aggregations.bucket.histogram.ParsedDateHistogram)2 ParsedStringTerms (org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms)2 ParsedStats (org.opensearch.search.aggregations.metrics.ParsedStats)2 Execution (io.kestra.core.models.executions.Execution)1 TaskRun (io.kestra.core.models.executions.TaskRun)1 ExecutionCount (io.kestra.core.models.executions.statistics.ExecutionCount)1 Flow (io.kestra.core.models.executions.statistics.Flow)1 State (io.kestra.core.models.flows.State)1 ModelValidator (io.kestra.core.models.validations.ModelValidator)1 ArrayListTotal (io.kestra.core.repositories.ArrayListTotal)1 ExecutionRepositoryInterface (io.kestra.core.repositories.ExecutionRepositoryInterface)1 ExecutorsUtils (io.kestra.core.utils.ExecutorsUtils)1 Nullable (io.micronaut.core.annotation.Nullable)1