Search in sources :

Example 16 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class SearchesTest method testHistogramWithNonExistingIndex.

@Test
@UsingDataSet(loadStrategy = LoadStrategyEnum.CLEAN_INSERT)
@SuppressWarnings("unchecked")
public void testHistogramWithNonExistingIndex() throws Exception {
    final SortedSet<IndexRange> indexRanges = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(MongoIndexRange.create(INDEX_NAME, new DateTime(0L, UTC), new DateTime(2015, 1, 1, 0, 0, UTC), DateTime.now(UTC), 0, null)).add(MongoIndexRange.create("does-not-exist", new DateTime(0L, UTC), new DateTime(2015, 1, 1, 0, 0, UTC), DateTime.now(UTC), 0, null)).build();
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indexRanges);
    final AbsoluteRange range = AbsoluteRange.create(new DateTime(2015, 1, 1, 0, 0, DateTimeZone.UTC).withZone(UTC), new DateTime(2015, 1, 2, 0, 0, DateTimeZone.UTC).withZone(UTC));
    HistogramResult h = searches.histogram("*", Searches.DateHistogramInterval.HOUR, range);
    assertThat(h.getInterval()).isEqualTo(Searches.DateHistogramInterval.HOUR);
    assertThat(h.getHistogramBoundaries()).isEqualTo(range);
    assertThat(h.getResults()).hasSize(5).containsEntry(new DateTime(2015, 1, 1, 1, 0, UTC).getMillis() / 1000L, 2L).containsEntry(new DateTime(2015, 1, 1, 2, 0, UTC).getMillis() / 1000L, 2L).containsEntry(new DateTime(2015, 1, 1, 3, 0, UTC).getMillis() / 1000L, 2L).containsEntry(new DateTime(2015, 1, 1, 4, 0, UTC).getMillis() / 1000L, 2L).containsEntry(new DateTime(2015, 1, 1, 5, 0, UTC).getMillis() / 1000L, 2L);
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) HistogramResult(org.graylog2.indexer.results.HistogramResult) AbsoluteRange(org.graylog2.plugin.indexer.searches.timeranges.AbsoluteRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) UsingDataSet(com.lordofthejars.nosqlunit.annotation.UsingDataSet) Test(org.junit.Test)

Example 17 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class SearchesTest method determineAffectedIndicesIncludesDeflectorTarget.

@Test
public void determineAffectedIndicesIncludesDeflectorTarget() throws Exception {
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final MongoIndexRange indexRange0 = MongoIndexRange.create("graylog_0", now, now.plusDays(1), now, 0);
    final MongoIndexRange indexRange1 = MongoIndexRange.create("graylog_1", now.plusDays(1), now.plusDays(2), now, 0);
    final MongoIndexRange indexRangeLatest = MongoIndexRange.create("graylog_2", new DateTime(0L, DateTimeZone.UTC), new DateTime(0L, DateTimeZone.UTC), now, 0);
    final SortedSet<IndexRange> indices = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(indexRange0).add(indexRange1).add(indexRangeLatest).build();
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indices);
    final TimeRange absoluteRange = AbsoluteRange.create(now.minusDays(1), now.plusDays(1));
    final TimeRange keywordRange = KeywordRange.create("1 day ago");
    final TimeRange relativeRange = RelativeRange.create(3600);
    assertThat(searches.determineAffectedIndices(absoluteRange, null)).containsExactlyInAnyOrder(indexRangeLatest.indexName(), indexRange0.indexName(), indexRange1.indexName());
    assertThat(searches.determineAffectedIndices(keywordRange, null)).containsExactlyInAnyOrder(indexRangeLatest.indexName(), indexRange0.indexName(), indexRange1.indexName());
    assertThat(searches.determineAffectedIndices(relativeRange, null)).containsExactlyInAnyOrder(indexRangeLatest.indexName(), indexRange0.indexName(), indexRange1.indexName());
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Example 18 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class V20161130141500_DefaultStreamRecalcIndexRanges method upgrade.

@Override
public void upgrade() {
    IndexSet indexSet;
    try {
        indexSet = indexSetRegistry.getDefault();
    } catch (IllegalStateException e) {
        // Try to find the default index set manually if the index set registry cannot find it.
        // This is needed if you run this migration with a 2.2.0-beta.2 state before commit 460cac6af.
        final IndexSetConfig indexSetConfig = indexSetService.findOne(DBQuery.is("default", true)).orElseThrow(() -> new IllegalStateException("No default index set configured! This is a bug!"));
        indexSet = indexSetFactory.create(indexSetConfig);
    }
    final IndexSet defaultIndexSet = indexSet;
    if (!cluster.isConnected()) {
        LOG.info("Cluster not connected yet, delaying migration until it is reachable.");
        while (true) {
            try {
                cluster.waitForConnectedAndDeflectorHealthy();
                break;
            } catch (InterruptedException | TimeoutException e) {
                LOG.warn("Interrupted or timed out waiting for Elasticsearch cluster, checking again.");
            }
        }
    }
    final Set<String> indexRangesWithoutStreams = indexRangeService.findAll().stream().filter(indexRange -> defaultIndexSet.isManagedIndex(indexRange.indexName())).filter(indexRange -> indexRange.streamIds() == null).map(IndexRange::indexName).collect(Collectors.toSet());
    if (indexRangesWithoutStreams.size() == 0) {
        // all ranges have a stream list, even if it is empty, nothing more to do
        return;
    }
    final String currentWriteTarget;
    try {
        currentWriteTarget = defaultIndexSet.getActiveWriteIndex();
    } catch (TooManyAliasesException e) {
        LOG.error("Multiple write targets found for write alias. Cannot continue to assign streams to older indices", e);
        return;
    }
    for (String indexName : defaultIndexSet.getManagedIndices()) {
        if (indexName.equals(currentWriteTarget)) {
            // do not recalculate for current write target
            continue;
        }
        if (!indexRangesWithoutStreams.contains(indexName)) {
            // already computed streams for this index
            continue;
        }
        LOG.info("Recalculating streams in index {}", indexName);
        final CreateNewSingleIndexRangeJob createNewSingleIndexRangeJob = rebuildIndexRangeJobFactory.create(indexSetRegistry.getAll(), indexName);
        createNewSingleIndexRangeJob.execute();
    }
}
Also used : CreateNewSingleIndexRangeJob(org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob) IndexRangeService(org.graylog2.indexer.ranges.IndexRangeService) Logger(org.slf4j.Logger) Cluster(org.graylog2.indexer.cluster.Cluster) IndexSetConfig(org.graylog2.indexer.indexset.IndexSetConfig) ZonedDateTime(java.time.ZonedDateTime) LoggerFactory(org.slf4j.LoggerFactory) Set(java.util.Set) TimeoutException(java.util.concurrent.TimeoutException) DBQuery(org.mongojack.DBQuery) Collectors(java.util.stream.Collectors) IndexRange(org.graylog2.indexer.ranges.IndexRange) Inject(javax.inject.Inject) IndexSetService(org.graylog2.indexer.indexset.IndexSetService) MongoIndexSet(org.graylog2.indexer.MongoIndexSet) IndexSet(org.graylog2.indexer.IndexSet) TooManyAliasesException(org.graylog2.indexer.indices.TooManyAliasesException) IndexSetRegistry(org.graylog2.indexer.IndexSetRegistry) CreateNewSingleIndexRangeJob(org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob) IndexSetConfig(org.graylog2.indexer.indexset.IndexSetConfig) TooManyAliasesException(org.graylog2.indexer.indices.TooManyAliasesException) MongoIndexSet(org.graylog2.indexer.MongoIndexSet) IndexSet(org.graylog2.indexer.IndexSet) TimeoutException(java.util.concurrent.TimeoutException)

Example 19 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class IndexRangesCleanupPeriodical method doRun.

@Override
public void doRun() {
    if (!cluster.isConnected() || !cluster.isHealthy()) {
        LOG.info("Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy");
        return;
    }
    final Set<String> indexNames = ImmutableSet.copyOf(indexSetRegistry.getManagedIndices());
    final SortedSet<IndexRange> indexRanges = indexRangeService.findAll();
    final Set<String> removedIndices = new HashSet<>();
    for (IndexRange indexRange : indexRanges) {
        if (!indexNames.contains(indexRange.indexName())) {
            removedIndices.add(indexRange.indexName());
        }
    }
    if (!removedIndices.isEmpty()) {
        LOG.info("Removing index range information for unavailable indices: {}", removedIndices);
        eventBus.post(IndicesDeletedEvent.create(removedIndices));
    }
}
Also used : IndexRange(org.graylog2.indexer.ranges.IndexRange) HashSet(java.util.HashSet)

Example 20 with IndexRange

use of org.graylog2.indexer.ranges.IndexRange in project graylog2-server by Graylog2.

the class IndexRangesMigrationPeriodical method doRun.

@Override
public void doRun() {
    final MongoIndexRangesMigrationComplete migrationComplete = clusterConfigService.get(MongoIndexRangesMigrationComplete.class);
    if (migrationComplete != null && migrationComplete.complete) {
        LOG.debug("Migration of index ranges (pre Graylog 1.2.2) already complete. Skipping migration process.");
        return;
    }
    while (!cluster.isConnected() || !cluster.isHealthy()) {
        Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
    }
    final Set<String> indexNames = ImmutableSet.copyOf(indexSetRegistry.getManagedIndices());
    // Migrate old MongoDB index ranges
    final SortedSet<IndexRange> mongoIndexRanges = legacyMongoIndexRangeService.findAll();
    for (IndexRange indexRange : mongoIndexRanges) {
        if (indexNames.contains(indexRange.indexName())) {
            LOG.info("Migrating index range from MongoDB: {}", indexRange);
            indexRangeService.save(indexRange);
        } else {
            LOG.info("Removing stale index range from MongoDB: {}", indexRange);
        }
        legacyMongoIndexRangeService.delete(indexRange.indexName());
    }
    // Check whether all index ranges have been migrated
    final int numberOfIndices = indexNames.size();
    final SortedSet<IndexRange> allIndexRanges = indexRangeService.findAll();
    final int numberOfIndexRanges = allIndexRanges.size();
    if (numberOfIndices > numberOfIndexRanges) {
        LOG.info("There are more indices ({}) than there are index ranges ({}). Notifying administrator.", numberOfIndices, numberOfIndexRanges);
        // remove all present index names so we can display the index sets that need manual fixing
        final Set<String> extraIndices = Sets.newHashSet(indexNames);
        allIndexRanges.forEach(indexRange -> extraIndices.remove(indexRange.indexName()));
        final Set<String> affectedIndexSetNames = extraIndices.stream().map(indexSetRegistry::getForIndex).filter(Optional::isPresent).map(Optional::get).map(IndexSet::getConfig).map(IndexSetConfig::title).collect(Collectors.toSet());
        final Notification notification = notificationService.buildNow().addSeverity(Notification.Severity.URGENT).addType(Notification.Type.INDEX_RANGES_RECALCULATION).addDetail("indices", numberOfIndices).addDetail("index_ranges", numberOfIndexRanges).addDetail("index_sets", affectedIndexSetNames.isEmpty() ? null : Joiner.on(", ").join(affectedIndexSetNames));
        notificationService.publishIfFirst(notification);
    }
    clusterConfigService.write(new MongoIndexRangesMigrationComplete(true));
}
Also used : IndexRange(org.graylog2.indexer.ranges.IndexRange) Optional(java.util.Optional) IndexSet(org.graylog2.indexer.IndexSet) Notification(org.graylog2.notifications.Notification)

Aggregations

IndexRange (org.graylog2.indexer.ranges.IndexRange)17 DateTime (org.joda.time.DateTime)11 Test (org.junit.Test)10 MongoIndexRange (org.graylog2.indexer.ranges.MongoIndexRange)9 ZonedDateTime (java.time.ZonedDateTime)8 IndexSet (org.graylog2.indexer.IndexSet)5 TimeRange (org.graylog2.plugin.indexer.searches.timeranges.TimeRange)5 NotFoundException (org.graylog2.database.NotFoundException)3 SearchResult (org.graylog2.indexer.results.SearchResult)3 Timed (com.codahale.metrics.annotation.Timed)2 Stopwatch (com.google.common.base.Stopwatch)2 ImmutableSortedSet (com.google.common.collect.ImmutableSortedSet)2 UsingDataSet (com.lordofthejars.nosqlunit.annotation.UsingDataSet)2 ApiOperation (io.swagger.annotations.ApiOperation)2 GET (javax.ws.rs.GET)2 Produces (javax.ws.rs.Produces)2 TimeValue (org.elasticsearch.common.unit.TimeValue)2 SearchHits (org.elasticsearch.search.SearchHits)2 AbstractAlertCondition (org.graylog2.alerts.AbstractAlertCondition)2 AlertConditionTest (org.graylog2.alerts.AlertConditionTest)2