Search in sources :

Example 11 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class RebuildIndexRangesJob method execute.

@Override
public void execute() {
    info("Recalculating index ranges.");
    // for each index set we know about
    final ListMultimap<IndexSet, String> indexSets = MultimapBuilder.hashKeys().arrayListValues().build();
    for (IndexSet indexSet : this.indexSets) {
        final String[] managedIndicesNames = indexSet.getManagedIndices();
        for (String name : managedIndicesNames) {
            indexSets.put(indexSet, name);
        }
    }
    if (indexSets.size() == 0) {
        info("No indices, nothing to calculate.");
        return;
    }
    indicesToCalculate = indexSets.values().size();
    Stopwatch sw = Stopwatch.createStarted();
    for (IndexSet indexSet : indexSets.keySet()) {
        LOG.info("Recalculating index ranges for index set {} ({}): {} indices affected.", indexSet.getConfig().title(), indexSet.getIndexWildcard(), indexSets.get(indexSet).size());
        for (String index : indexSets.get(indexSet)) {
            try {
                if (index.equals(indexSet.getActiveWriteIndex())) {
                    LOG.debug("{} is current write target, do not calculate index range for it", index);
                    final IndexRange emptyRange = indexRangeService.createUnknownRange(index);
                    try {
                        final IndexRange indexRange = indexRangeService.get(index);
                        if (indexRange.begin().getMillis() != 0 || indexRange.end().getMillis() != 0) {
                            LOG.info("Invalid date ranges for write index {}, resetting it.", index);
                            indexRangeService.save(emptyRange);
                        }
                    } catch (NotFoundException e) {
                        LOG.info("No index range found for write index {}, recreating it.", index);
                        indexRangeService.save(emptyRange);
                    }
                    indicesCalculated.incrementAndGet();
                    continue;
                }
            } catch (TooManyAliasesException e) {
                LOG.error("Multiple write alias targets found, this is a bug.");
                indicesCalculated.incrementAndGet();
                continue;
            }
            if (cancelRequested) {
                info("Stop requested. Not calculating next index range, not updating ranges.");
                sw.stop();
                return;
            }
            try {
                final IndexRange indexRange = indexRangeService.calculateRange(index);
                indexRangeService.save(indexRange);
                LOG.info("Created ranges for index {}: {}", index, indexRange);
            } catch (Exception e) {
                LOG.info("Could not calculate range of index [" + index + "]. Skipping.", e);
            } finally {
                indicesCalculated.incrementAndGet();
            }
        }
    }
    info("Done calculating index ranges for " + indicesToCalculate + " indices. Took " + sw.stop().elapsed(TimeUnit.MILLISECONDS) + "ms.");
}
Also used : Stopwatch(com.google.common.base.Stopwatch) NotFoundException(org.graylog2.database.NotFoundException) TooManyAliasesException(org.graylog2.indexer.indices.TooManyAliasesException) IndexSet(org.graylog2.indexer.IndexSet) TooManyAliasesException(org.graylog2.indexer.indices.TooManyAliasesException) NotFoundException(org.graylog2.database.NotFoundException)

Example 12 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class MongoIndexSet method addDeflectorIndexRange.

private void addDeflectorIndexRange(String indexName) {
    final IndexRange deflectorRange = indexRangeService.createUnknownRange(indexName);
    indexRangeService.save(deflectorRange);
}
Also used : IndexRange(org.graylog2.indexer.ranges.IndexRange)

Example 13 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class FieldContentValueAlertConditionTest method testRunNoMatchingMessages.

@Test
public void testRunNoMatchingMessages() throws Exception {
    final SearchHits searchHits = mock(SearchHits.class);
    when(searchHits.iterator()).thenReturn(Collections.<SearchHit>emptyIterator());
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final IndexRange indexRange = MongoIndexRange.create("graylog_test", now.minusDays(1), now, now, 0);
    final Set<IndexRange> indexRanges = Sets.newHashSet(indexRange);
    final SearchResult searchResult = spy(new SearchResult(searchHits, indexRanges, "message:something", null, new TimeValue(100, TimeUnit.MILLISECONDS)));
    when(searches.search(anyString(), anyString(), any(RelativeRange.class), anyInt(), anyInt(), any(Sorting.class))).thenReturn(searchResult);
    final FieldContentValueAlertCondition condition = getCondition(getParametersMap(0, "message", "something"), alertConditionTitle);
    final AlertCondition.CheckResult result = condition.runCheck();
    assertNotTriggered(result);
}
Also used : IndexRange(org.graylog2.indexer.ranges.IndexRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) RelativeRange(org.graylog2.plugin.indexer.searches.timeranges.RelativeRange) AbstractAlertCondition(org.graylog2.alerts.AbstractAlertCondition) AlertCondition(org.graylog2.plugin.alarms.AlertCondition) SearchResult(org.graylog2.indexer.results.SearchResult) SearchHits(org.elasticsearch.search.SearchHits) DateTime(org.joda.time.DateTime) TimeValue(org.elasticsearch.common.unit.TimeValue) Sorting(org.graylog2.indexer.searches.Sorting) Test(org.junit.Test) AlertConditionTest(org.graylog2.alerts.AlertConditionTest)

Example 14 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class FieldContentValueAlertConditionTest method testRunMatchingMessagesInStream.

@Test
public void testRunMatchingMessagesInStream() throws Exception {
    final SearchHits searchHits = mock(SearchHits.class);
    final SearchHit searchHit = mock(SearchHit.class);
    final HashMap<String, Object> source = Maps.newHashMap();
    source.put("message", "something is in here");
    when(searchHit.getId()).thenReturn("some id");
    when(searchHit.getSource()).thenReturn(source);
    when(searchHit.getIndex()).thenReturn("graylog_test");
    when(searchHits.iterator()).thenReturn(Iterators.singletonIterator(searchHit));
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final IndexRange indexRange = MongoIndexRange.create("graylog_test", now.minusDays(1), now, now, 0);
    final Set<IndexRange> indexRanges = Sets.newHashSet(indexRange);
    final SearchResult searchResult = spy(new SearchResult(searchHits, indexRanges, "message:something", null, new TimeValue(100, TimeUnit.MILLISECONDS)));
    when(searchResult.getTotalResults()).thenReturn(1L);
    when(searches.search(anyString(), anyString(), any(RelativeRange.class), anyInt(), anyInt(), any(Sorting.class))).thenReturn(searchResult);
    final FieldContentValueAlertCondition condition = getCondition(getParametersMap(0, "message", "something"), "Alert Condition for testing");
    final AlertCondition.CheckResult result = condition.runCheck();
    assertTriggered(condition, result);
}
Also used : SearchHit(org.elasticsearch.search.SearchHit) SearchResult(org.graylog2.indexer.results.SearchResult) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) DateTime(org.joda.time.DateTime) Sorting(org.graylog2.indexer.searches.Sorting) IndexRange(org.graylog2.indexer.ranges.IndexRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) RelativeRange(org.graylog2.plugin.indexer.searches.timeranges.RelativeRange) AbstractAlertCondition(org.graylog2.alerts.AbstractAlertCondition) AlertCondition(org.graylog2.plugin.alarms.AlertCondition) SearchHits(org.elasticsearch.search.SearchHits) TimeValue(org.elasticsearch.common.unit.TimeValue) Test(org.junit.Test) AlertConditionTest(org.graylog2.alerts.AlertConditionTest)

Example 15 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class SearchesTest method determineAffectedIndicesDoesNotIncludesDeflectorTargetIfMissing.

@Test
public void determineAffectedIndicesDoesNotIncludesDeflectorTargetIfMissing() throws Exception {
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final MongoIndexRange indexRange0 = MongoIndexRange.create("graylog_0", now, now.plusDays(1), now, 0);
    final MongoIndexRange indexRange1 = MongoIndexRange.create("graylog_1", now.plusDays(1), now.plusDays(2), now, 0);
    final SortedSet<IndexRange> indices = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(indexRange0).add(indexRange1).build();
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indices);
    final TimeRange absoluteRange = AbsoluteRange.create(now.minusDays(1), now.plusDays(1));
    final TimeRange keywordRange = KeywordRange.create("1 day ago");
    final TimeRange relativeRange = RelativeRange.create(3600);
    assertThat(searches.determineAffectedIndices(absoluteRange, null)).containsOnly(indexRange0.indexName(), indexRange1.indexName());
    assertThat(searches.determineAffectedIndices(keywordRange, null)).containsOnly(indexRange0.indexName(), indexRange1.indexName());
    assertThat(searches.determineAffectedIndices(relativeRange, null)).containsOnly(indexRange0.indexName(), indexRange1.indexName());
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Aggregations

IndexRange (org.graylog2.indexer.ranges.IndexRange)17 DateTime (org.joda.time.DateTime)11 Test (org.junit.Test)10 MongoIndexRange (org.graylog2.indexer.ranges.MongoIndexRange)9 ZonedDateTime (java.time.ZonedDateTime)8 IndexSet (org.graylog2.indexer.IndexSet)5 TimeRange (org.graylog2.plugin.indexer.searches.timeranges.TimeRange)5 NotFoundException (org.graylog2.database.NotFoundException)3 SearchResult (org.graylog2.indexer.results.SearchResult)3 Timed (com.codahale.metrics.annotation.Timed)2 Stopwatch (com.google.common.base.Stopwatch)2 ImmutableSortedSet (com.google.common.collect.ImmutableSortedSet)2 UsingDataSet (com.lordofthejars.nosqlunit.annotation.UsingDataSet)2 ApiOperation (io.swagger.annotations.ApiOperation)2 GET (javax.ws.rs.GET)2 Produces (javax.ws.rs.Produces)2 TimeValue (org.elasticsearch.common.unit.TimeValue)2 SearchHits (org.elasticsearch.search.SearchHits)2 AbstractAlertCondition (org.graylog2.alerts.AbstractAlertCondition)2 AlertConditionTest (org.graylog2.alerts.AlertConditionTest)2