Search in sources :

Example 1 with ExecutionCount

use of io.kestra.core.models.executions.statistics.ExecutionCount in project kestra by kestra-io.

the class Counts method run.

@Override
public Output run(RunContext runContext) throws Exception {
    Logger logger = runContext.logger();
    ExecutionRepositoryInterface executionRepository = runContext.getApplicationContext().getBean(ExecutionRepositoryInterface.class);
    String query = null;
    if (this.states != null) {
        query = "state.current:(" + this.states.stream().map(Enum::name).collect(Collectors.joining(" OR ")) + ")";
    }
    List<ExecutionCount> executionCounts = executionRepository.executionCounts(flows, query, startDate != null ? ZonedDateTime.parse(runContext.render(startDate)) : null, endDate != null ? ZonedDateTime.parse(runContext.render(endDate)) : null);
    logger.trace("{} flows matching filters", executionCounts.size());
    List<Result> count = executionCounts.stream().filter(throwPredicate(item -> runContext.render(this.expression, ImmutableMap.of("count", item.getCount().intValue())).equals("true"))).map(item -> Result.builder().namespace(item.getNamespace()).flowId(item.getFlowId()).count(item.getCount()).build()).collect(Collectors.toList());
    logger.debug("{} flows matching the expression", count.size());
    return Output.builder().results(count).build();
}
Also used : Schema(io.swagger.v3.oas.annotations.media.Schema) Rethrow.throwPredicate(io.kestra.core.utils.Rethrow.throwPredicate) ExecutionRepositoryInterface(io.kestra.core.repositories.ExecutionRepositoryInterface) Logger(org.slf4j.Logger) ImmutableMap(com.google.common.collect.ImmutableMap) Plugin(io.kestra.core.models.annotations.Plugin) Flow(io.kestra.core.models.executions.statistics.Flow) ZonedDateTime(java.time.ZonedDateTime) SuperBuilder(lombok.experimental.SuperBuilder) NotNull(javax.validation.constraints.NotNull) ExecutionCount(io.kestra.core.models.executions.statistics.ExecutionCount) Collectors(java.util.stream.Collectors) lombok(lombok) PluginProperty(io.kestra.core.models.annotations.PluginProperty) State(io.kestra.core.models.flows.State) NotEmpty(javax.validation.constraints.NotEmpty) Task(io.kestra.core.models.tasks.Task) List(java.util.List) RunContext(io.kestra.core.runners.RunContext) Example(io.kestra.core.models.annotations.Example) RunnableTask(io.kestra.core.models.tasks.RunnableTask) ExecutionCount(io.kestra.core.models.executions.statistics.ExecutionCount) ExecutionRepositoryInterface(io.kestra.core.repositories.ExecutionRepositoryInterface) Logger(org.slf4j.Logger)

Example 2 with ExecutionCount

use of io.kestra.core.models.executions.statistics.ExecutionCount in project kestra by kestra-io.

the class ElasticSearchExecutionRepository method executionCounts.

@Override
public List<ExecutionCount> executionCounts(List<Flow> flows, String query, ZonedDateTime startDate, ZonedDateTime endDate) {
    if (startDate == null) {
        startDate = ZonedDateTime.now().minusDays(30);
    }
    if (endDate == null) {
        endDate = ZonedDateTime.now();
    }
    SearchSourceBuilder sourceBuilder = this.searchSource(this.dateFilters(query, startDate, endDate), Optional.of(Collections.singletonList(AggregationBuilders.filters("FILTERS", flows.stream().map(flow -> new FiltersAggregator.KeyedFilter(flow.getNamespace() + "/" + flow.getFlowId(), QueryBuilders.boolQuery().must(QueryBuilders.termQuery("namespace", flow.getNamespace())).must(QueryBuilders.termQuery("flowId", flow.getFlowId())))).toArray(FiltersAggregator.KeyedFilter[]::new)))), null);
    try {
        SearchRequest searchRequest = searchRequest(INDEX_NAME, sourceBuilder, false);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        List<ExecutionCount> result = new ArrayList<>();
        ((ParsedFilters) searchResponse.getAggregations().get("FILTERS")).getBuckets().forEach(filtersBuckets -> {
            final String key = filtersBuckets.getKeyAsString();
            result.add(new ExecutionCount(key.split("/")[0], key.split("/")[1], filtersBuckets.getDocCount()));
        });
        return result;
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}
Also used : TaskRun(io.kestra.core.models.executions.TaskRun) RequestOptions(org.opensearch.client.RequestOptions) java.util(java.util) ExecutionRepositoryInterface(io.kestra.core.repositories.ExecutionRepositoryInterface) Flow(io.kestra.core.models.executions.statistics.Flow) ParsedStringTerms(org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms) ZonedDateTime(java.time.ZonedDateTime) DateHistogramAggregationBuilder(org.opensearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder) DateHistogramInterval(org.opensearch.search.aggregations.bucket.histogram.DateHistogramInterval) ExecutionCount(io.kestra.core.models.executions.statistics.ExecutionCount) ParsedDateHistogram(org.opensearch.search.aggregations.bucket.histogram.ParsedDateHistogram) State(io.kestra.core.models.flows.State) LongBounds(org.opensearch.search.aggregations.bucket.histogram.LongBounds) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) FiltersAggregator(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator) Nullable(io.micronaut.core.annotation.Nullable) Duration(java.time.Duration) Pageable(io.micronaut.data.model.Pageable) SearchRequest(org.opensearch.action.search.SearchRequest) SearchResponse(org.opensearch.action.search.SearchResponse) RestHighLevelClient(org.opensearch.client.RestHighLevelClient) ParsedFilters(org.opensearch.search.aggregations.bucket.filter.ParsedFilters) QueryBuilders(org.opensearch.index.query.QueryBuilders) Filter(org.opensearch.search.aggregations.bucket.filter.Filter) TopHits(org.opensearch.search.aggregations.metrics.TopHits) Singleton(jakarta.inject.Singleton) Settings(org.opensearch.common.settings.Settings) IOException(java.io.IOException) ScoreMode(org.apache.lucene.search.join.ScoreMode) Terms(org.opensearch.search.aggregations.bucket.terms.Terms) Execution(io.kestra.core.models.executions.Execution) Collectors(java.util.stream.Collectors) DailyExecutionStatistics(io.kestra.core.models.executions.statistics.DailyExecutionStatistics) ZoneId(java.time.ZoneId) AggregationBuilders(org.opensearch.search.aggregations.AggregationBuilders) ParsedStats(org.opensearch.search.aggregations.metrics.ParsedStats) ArrayListTotal(io.kestra.core.repositories.ArrayListTotal) ExecutorsUtils(io.kestra.core.utils.ExecutorsUtils) TermsAggregationBuilder(org.opensearch.search.aggregations.bucket.terms.TermsAggregationBuilder) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) LocalDate(java.time.LocalDate) DateTimeFormatter(java.time.format.DateTimeFormatter) IndexSettings(org.opensearch.index.IndexSettings) ModelValidator(io.kestra.core.models.validations.ModelValidator) NestedAggregationBuilder(org.opensearch.search.aggregations.bucket.nested.NestedAggregationBuilder) Inject(jakarta.inject.Inject) ParsedNested(org.opensearch.search.aggregations.bucket.nested.ParsedNested) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) SearchRequest(org.opensearch.action.search.SearchRequest) ExecutionCount(io.kestra.core.models.executions.statistics.ExecutionCount) FiltersAggregator(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator) IOException(java.io.IOException) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) SearchResponse(org.opensearch.action.search.SearchResponse)

Aggregations

ExecutionCount (io.kestra.core.models.executions.statistics.ExecutionCount)2 Flow (io.kestra.core.models.executions.statistics.Flow)2 State (io.kestra.core.models.flows.State)2 ExecutionRepositoryInterface (io.kestra.core.repositories.ExecutionRepositoryInterface)2 ZonedDateTime (java.time.ZonedDateTime)2 Collectors (java.util.stream.Collectors)2 ImmutableMap (com.google.common.collect.ImmutableMap)1 Example (io.kestra.core.models.annotations.Example)1 Plugin (io.kestra.core.models.annotations.Plugin)1 PluginProperty (io.kestra.core.models.annotations.PluginProperty)1 Execution (io.kestra.core.models.executions.Execution)1 TaskRun (io.kestra.core.models.executions.TaskRun)1 DailyExecutionStatistics (io.kestra.core.models.executions.statistics.DailyExecutionStatistics)1 RunnableTask (io.kestra.core.models.tasks.RunnableTask)1 Task (io.kestra.core.models.tasks.Task)1 ModelValidator (io.kestra.core.models.validations.ModelValidator)1 ArrayListTotal (io.kestra.core.repositories.ArrayListTotal)1 RunContext (io.kestra.core.runners.RunContext)1 ExecutorsUtils (io.kestra.core.utils.ExecutorsUtils)1 Rethrow.throwPredicate (io.kestra.core.utils.Rethrow.throwPredicate)1