Search in sources :

Example 16 with Input

use of org.opensearch.ml.common.parameter.Input in project ml-commons by opensearch-project.

the class AnomalyLocalizerImpl method newSearchRequestForEntry.

private SearchRequest newSearchRequestForEntry(AnomalyLocalizationInput input, AggregationBuilder agg, AnomalyLocalizationOutput.Bucket bucket, Optional<Map<String, Object>> afterKey) {
    RangeQueryBuilder timeRangeFilter = new RangeQueryBuilder(input.getTimeFieldName()).from(bucket.getStartTime(), true).to(bucket.getEndTime(), true);
    BoolQueryBuilder filter = QueryBuilders.boolQuery().filter(timeRangeFilter);
    input.getFilterQuery().ifPresent(q -> filter.filter(q));
    CompositeAggregationBuilder compositeAgg = new CompositeAggregationBuilder(agg.getName(), input.getAttributeFieldNames().stream().map(name -> new TermsValuesSourceBuilder(name).field(name)).collect(Collectors.toList())).size(MAX_BUCKET_SETTING.get(this.settings));
    compositeAgg.subAggregation(agg);
    if (afterKey.isPresent()) {
        compositeAgg.aggregateAfter(afterKey.get());
    }
    SearchSourceBuilder search = new SearchSourceBuilder().size(0).query(filter).aggregation(compositeAgg);
    SearchRequest searchRequest = new SearchRequest(new String[] { input.getIndexName() }, search);
    return searchRequest;
}
Also used : IntStream(java.util.stream.IntStream) Arrays(java.util.Arrays) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SneakyThrows(lombok.SneakyThrows) PriorityQueue(java.util.PriorityQueue) NotifyOnceListener(org.opensearch.action.NotifyOnceListener) Executable(org.opensearch.ml.engine.Executable) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Input(org.opensearch.ml.common.parameter.Input) ArrayList(java.util.ArrayList) CompositeAggregationBuilder(org.opensearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) Output(org.opensearch.ml.common.parameter.Output) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) LatchedActionListener(org.opensearch.action.LatchedActionListener) KeyedFilter(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator.KeyedFilter) Map(java.util.Map) SearchRequest(org.opensearch.action.search.SearchRequest) ActionListener(org.opensearch.action.ActionListener) SearchResponse(org.opensearch.action.search.SearchResponse) SimpleEntry(java.util.AbstractMap.SimpleEntry) QueryBuilders(org.opensearch.index.query.QueryBuilders) Client(org.opensearch.client.Client) MAX_BUCKET_SETTING(org.opensearch.search.aggregations.MultiBucketConsumerService.MAX_BUCKET_SETTING) RangeQueryBuilder(org.opensearch.index.query.RangeQueryBuilder) Filters(org.opensearch.search.aggregations.bucket.filter.Filters) TermQueryBuilder(org.opensearch.index.query.TermQueryBuilder) Settings(org.opensearch.common.settings.Settings) FiltersAggregationBuilder(org.opensearch.search.aggregations.bucket.filter.FiltersAggregationBuilder) Collectors(java.util.stream.Collectors) CompositeAggregation(org.opensearch.search.aggregations.bucket.composite.CompositeAggregation) MultiSearchResponse(org.opensearch.action.search.MultiSearchResponse) CountDownLatch(java.util.concurrent.CountDownLatch) AggregationBuilders(org.opensearch.search.aggregations.AggregationBuilders) List(java.util.List) SingleValue(org.opensearch.search.aggregations.metrics.NumericMetricsAggregation.SingleValue) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) Data(lombok.Data) Log4j2(lombok.extern.log4j.Log4j2) Optional(java.util.Optional) ActionListener.wrap(org.opensearch.action.ActionListener.wrap) Collections(java.util.Collections) TermsValuesSourceBuilder(org.opensearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SearchRequest(org.opensearch.action.search.SearchRequest) TermsValuesSourceBuilder(org.opensearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) CompositeAggregationBuilder(org.opensearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) RangeQueryBuilder(org.opensearch.index.query.RangeQueryBuilder) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder)

Example 17 with Input

use of org.opensearch.ml.common.parameter.Input in project ml-commons by opensearch-project.

the class AnomalyLocalizerImpl method processNewEntry.

private void processNewEntry(AnomalyLocalizationInput input, AggregationBuilder agg, AnomalyLocalizationOutput.Result result, AnomalyLocalizationOutput.Bucket bucket, Optional<Map<String, Object>> afterKey, PriorityQueue<AnomalyLocalizationOutput.Entity> queue, AnomalyLocalizationOutput output, ActionListener<AnomalyLocalizationOutput> listener) {
    SearchRequest request = newSearchRequestForEntry(input, agg, bucket, afterKey);
    client.search(request, wrap(r -> onNewEntryResponse(r, input, agg, result, bucket, queue, output, listener), listener::onFailure));
}
Also used : IntStream(java.util.stream.IntStream) Arrays(java.util.Arrays) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SneakyThrows(lombok.SneakyThrows) PriorityQueue(java.util.PriorityQueue) NotifyOnceListener(org.opensearch.action.NotifyOnceListener) Executable(org.opensearch.ml.engine.Executable) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Input(org.opensearch.ml.common.parameter.Input) ArrayList(java.util.ArrayList) CompositeAggregationBuilder(org.opensearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) Output(org.opensearch.ml.common.parameter.Output) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) LatchedActionListener(org.opensearch.action.LatchedActionListener) KeyedFilter(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator.KeyedFilter) Map(java.util.Map) SearchRequest(org.opensearch.action.search.SearchRequest) ActionListener(org.opensearch.action.ActionListener) SearchResponse(org.opensearch.action.search.SearchResponse) SimpleEntry(java.util.AbstractMap.SimpleEntry) QueryBuilders(org.opensearch.index.query.QueryBuilders) Client(org.opensearch.client.Client) MAX_BUCKET_SETTING(org.opensearch.search.aggregations.MultiBucketConsumerService.MAX_BUCKET_SETTING) RangeQueryBuilder(org.opensearch.index.query.RangeQueryBuilder) Filters(org.opensearch.search.aggregations.bucket.filter.Filters) TermQueryBuilder(org.opensearch.index.query.TermQueryBuilder) Settings(org.opensearch.common.settings.Settings) FiltersAggregationBuilder(org.opensearch.search.aggregations.bucket.filter.FiltersAggregationBuilder) Collectors(java.util.stream.Collectors) CompositeAggregation(org.opensearch.search.aggregations.bucket.composite.CompositeAggregation) MultiSearchResponse(org.opensearch.action.search.MultiSearchResponse) CountDownLatch(java.util.concurrent.CountDownLatch) AggregationBuilders(org.opensearch.search.aggregations.AggregationBuilders) List(java.util.List) SingleValue(org.opensearch.search.aggregations.metrics.NumericMetricsAggregation.SingleValue) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) Data(lombok.Data) Log4j2(lombok.extern.log4j.Log4j2) Optional(java.util.Optional) ActionListener.wrap(org.opensearch.action.ActionListener.wrap) Collections(java.util.Collections) TermsValuesSourceBuilder(org.opensearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SearchRequest(org.opensearch.action.search.SearchRequest)

Example 18 with Input

use of org.opensearch.ml.common.parameter.Input in project ml-commons by opensearch-project.

the class AnomalyLocalizerImpl method newSearchRequestForOverallAggregates.

private MultiSearchRequest newSearchRequestForOverallAggregates(AnomalyLocalizationInput input, AggregationBuilder agg, LocalizationTimeBuckets timeBuckets) {
    MultiSearchRequest multiSearchRequest = new MultiSearchRequest();
    timeBuckets.getAllIntervals().stream().map(i -> {
        RangeQueryBuilder timeRangeFilter = new RangeQueryBuilder(input.getTimeFieldName()).from(i.getKey(), true).to(i.getValue(), true);
        BoolQueryBuilder filter = QueryBuilders.boolQuery().filter(timeRangeFilter);
        input.getFilterQuery().ifPresent(q -> filter.filter(q));
        SearchSourceBuilder search = new SearchSourceBuilder().size(0).query(filter).aggregation(agg);
        SearchRequest searchRequest = new SearchRequest(new String[] { input.getIndexName() }, search);
        return searchRequest;
    }).forEach(multiSearchRequest::add);
    return multiSearchRequest;
}
Also used : IntStream(java.util.stream.IntStream) Arrays(java.util.Arrays) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SneakyThrows(lombok.SneakyThrows) PriorityQueue(java.util.PriorityQueue) NotifyOnceListener(org.opensearch.action.NotifyOnceListener) Executable(org.opensearch.ml.engine.Executable) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Input(org.opensearch.ml.common.parameter.Input) ArrayList(java.util.ArrayList) CompositeAggregationBuilder(org.opensearch.search.aggregations.bucket.composite.CompositeAggregationBuilder) Output(org.opensearch.ml.common.parameter.Output) AggregationBuilder(org.opensearch.search.aggregations.AggregationBuilder) LatchedActionListener(org.opensearch.action.LatchedActionListener) KeyedFilter(org.opensearch.search.aggregations.bucket.filter.FiltersAggregator.KeyedFilter) Map(java.util.Map) SearchRequest(org.opensearch.action.search.SearchRequest) ActionListener(org.opensearch.action.ActionListener) SearchResponse(org.opensearch.action.search.SearchResponse) SimpleEntry(java.util.AbstractMap.SimpleEntry) QueryBuilders(org.opensearch.index.query.QueryBuilders) Client(org.opensearch.client.Client) MAX_BUCKET_SETTING(org.opensearch.search.aggregations.MultiBucketConsumerService.MAX_BUCKET_SETTING) RangeQueryBuilder(org.opensearch.index.query.RangeQueryBuilder) Filters(org.opensearch.search.aggregations.bucket.filter.Filters) TermQueryBuilder(org.opensearch.index.query.TermQueryBuilder) Settings(org.opensearch.common.settings.Settings) FiltersAggregationBuilder(org.opensearch.search.aggregations.bucket.filter.FiltersAggregationBuilder) Collectors(java.util.stream.Collectors) CompositeAggregation(org.opensearch.search.aggregations.bucket.composite.CompositeAggregation) MultiSearchResponse(org.opensearch.action.search.MultiSearchResponse) CountDownLatch(java.util.concurrent.CountDownLatch) AggregationBuilders(org.opensearch.search.aggregations.AggregationBuilders) List(java.util.List) SingleValue(org.opensearch.search.aggregations.metrics.NumericMetricsAggregation.SingleValue) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder) Data(lombok.Data) Log4j2(lombok.extern.log4j.Log4j2) Optional(java.util.Optional) ActionListener.wrap(org.opensearch.action.ActionListener.wrap) Collections(java.util.Collections) TermsValuesSourceBuilder(org.opensearch.search.aggregations.bucket.composite.TermsValuesSourceBuilder) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) SearchRequest(org.opensearch.action.search.SearchRequest) BoolQueryBuilder(org.opensearch.index.query.BoolQueryBuilder) MultiSearchRequest(org.opensearch.action.search.MultiSearchRequest) RangeQueryBuilder(org.opensearch.index.query.RangeQueryBuilder) SearchSourceBuilder(org.opensearch.search.builder.SearchSourceBuilder)

Example 19 with Input

use of org.opensearch.ml.common.parameter.Input in project ml-commons by opensearch-project.

the class MLEngineTest method predictWithoutModel.

@Test
public void predictWithoutModel() {
    exceptionRule.expect(IllegalArgumentException.class);
    exceptionRule.expectMessage("No model found for linear regression prediction.");
    MLInputDataset inputDataset = DataFrameInputDataset.builder().dataFrame(constructLinearRegressionPredictionDataFrame()).build();
    Input mlInput = MLInput.builder().algorithm(FunctionName.LINEAR_REGRESSION).inputDataset(inputDataset).build();
    MLEngine.predict(mlInput, null);
}
Also used : MLInput(org.opensearch.ml.common.parameter.MLInput) Input(org.opensearch.ml.common.parameter.Input) LocalSampleCalculatorInput(org.opensearch.ml.common.parameter.LocalSampleCalculatorInput) MLInputDataset(org.opensearch.ml.common.dataset.MLInputDataset) Test(org.junit.Test)

Example 20 with Input

use of org.opensearch.ml.common.parameter.Input in project ml-commons by opensearch-project.

the class MLEngineTest method predictLinearRegression.

@Test
public void predictLinearRegression() {
    Model model = trainLinearRegressionModel();
    DataFrame predictionDataFrame = constructLinearRegressionPredictionDataFrame();
    MLInputDataset inputDataset = DataFrameInputDataset.builder().dataFrame(predictionDataFrame).build();
    Input mlInput = MLInput.builder().algorithm(FunctionName.LINEAR_REGRESSION).inputDataset(inputDataset).build();
    MLPredictionOutput output = (MLPredictionOutput) MLEngine.predict(mlInput, model);
    DataFrame predictions = output.getPredictionResult();
    Assert.assertEquals(2, predictions.size());
}
Also used : MLInput(org.opensearch.ml.common.parameter.MLInput) Input(org.opensearch.ml.common.parameter.Input) LocalSampleCalculatorInput(org.opensearch.ml.common.parameter.LocalSampleCalculatorInput) Model(org.opensearch.ml.common.parameter.Model) MLInputDataset(org.opensearch.ml.common.dataset.MLInputDataset) MLPredictionOutput(org.opensearch.ml.common.parameter.MLPredictionOutput) LinearRegressionHelper.constructLinearRegressionPredictionDataFrame(org.opensearch.ml.engine.helper.LinearRegressionHelper.constructLinearRegressionPredictionDataFrame) KMeansHelper.constructKMeansDataFrame(org.opensearch.ml.engine.helper.KMeansHelper.constructKMeansDataFrame) LinearRegressionHelper.constructLinearRegressionTrainDataFrame(org.opensearch.ml.engine.helper.LinearRegressionHelper.constructLinearRegressionTrainDataFrame) DataFrame(org.opensearch.ml.common.dataframe.DataFrame) Test(org.junit.Test)

Aggregations

Input (org.opensearch.ml.common.parameter.Input)23 LocalSampleCalculatorInput (org.opensearch.ml.common.parameter.LocalSampleCalculatorInput)13 MLInput (org.opensearch.ml.common.parameter.MLInput)12 Test (org.junit.Test)10 Output (org.opensearch.ml.common.parameter.Output)10 Client (org.opensearch.client.Client)9 Settings (org.opensearch.common.settings.Settings)9 ArrayList (java.util.ArrayList)8 List (java.util.List)8 Data (lombok.Data)8 ActionListener (org.opensearch.action.ActionListener)8 SearchRequest (org.opensearch.action.search.SearchRequest)8 SearchResponse (org.opensearch.action.search.SearchResponse)8 MLInputDataset (org.opensearch.ml.common.dataset.MLInputDataset)8 Executable (org.opensearch.ml.engine.Executable)8 SimpleEntry (java.util.AbstractMap.SimpleEntry)7 Arrays (java.util.Arrays)7 Collections (java.util.Collections)7 Map (java.util.Map)7 Optional (java.util.Optional)7