Search in sources :

Example 1 with Indices

use of org.graylog2.indexer.indices.Indices in project graylog2-server by Graylog2.

the class SearchesTest method determineAffectedIndicesWithRangesDoesNotIncludesDeflectorTargetIfMissing.

@Test
public void determineAffectedIndicesWithRangesDoesNotIncludesDeflectorTargetIfMissing() throws Exception {
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final MongoIndexRange indexRange0 = MongoIndexRange.create("graylog_0", now, now.plusDays(1), now, 0);
    final MongoIndexRange indexRange1 = MongoIndexRange.create("graylog_1", now.plusDays(1), now.plusDays(2), now, 0);
    final SortedSet<IndexRange> indices = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(indexRange0).add(indexRange1).build();
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indices);
    final TimeRange absoluteRange = AbsoluteRange.create(now.minusDays(1), now.plusDays(1));
    final TimeRange keywordRange = KeywordRange.create("1 day ago");
    final TimeRange relativeRange = RelativeRange.create(3600);
    assertThat(searches.determineAffectedIndicesWithRanges(absoluteRange, null)).containsExactly(indexRange0, indexRange1);
    assertThat(searches.determineAffectedIndicesWithRanges(keywordRange, null)).containsExactly(indexRange0, indexRange1);
    assertThat(searches.determineAffectedIndicesWithRanges(relativeRange, null)).containsExactly(indexRange0, indexRange1);
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Example 2 with Indices

use of org.graylog2.indexer.indices.Indices in project graylog2-server by Graylog2.

the class SearchesTest method determineAffectedIndicesFilterIndexPrefix.

@Test
public void determineAffectedIndicesFilterIndexPrefix() throws Exception {
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final MongoIndexRange indexRange0 = MongoIndexRange.create("graylog_0", now, now.plusDays(1), now, 0);
    final MongoIndexRange indexRange1 = MongoIndexRange.create("graylog_1", now.plusDays(1), now.plusDays(2), now, 0);
    final MongoIndexRange b0 = MongoIndexRange.create("b_0", now.plusDays(1), now.plusDays(2), now, 0);
    final MongoIndexRange b1 = MongoIndexRange.create("b_1", now.plusDays(1), now.plusDays(2), now, 0);
    final SortedSet<IndexRange> indices = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(indexRange0).add(indexRange1).add(b0).add(b1).build();
    final Stream bStream = mock(Stream.class);
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indices);
    when(streamService.load(eq("123456789ABCDEF"))).thenReturn(bStream);
    final IndexSet indexSet = mock(IndexSet.class);
    when(indexSet.isManagedIndex(startsWith("b_"))).thenReturn(true);
    when(bStream.getIndexSet()).thenReturn(indexSet);
    final TimeRange absoluteRange = AbsoluteRange.create(now.minusDays(1), now.plusDays(1));
    assertThat(searches.determineAffectedIndices(absoluteRange, "streams:123456789ABCDEF")).containsOnly(b0.indexName(), b1.indexName());
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) Stream(org.graylog2.plugin.streams.Stream) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) IndexSet(org.graylog2.indexer.IndexSet) TestIndexSet(org.graylog2.indexer.TestIndexSet) Test(org.junit.Test)

Example 3 with Indices

use of org.graylog2.indexer.indices.Indices in project graylog2-server by Graylog2.

the class SearchesTest method determineAffectedIndicesWithRangesIncludesDeflectorTarget.

@Test
public void determineAffectedIndicesWithRangesIncludesDeflectorTarget() throws Exception {
    final DateTime now = DateTime.now(DateTimeZone.UTC);
    final MongoIndexRange indexRange0 = MongoIndexRange.create("graylog_0", now, now.plusDays(1), now, 0);
    final MongoIndexRange indexRange1 = MongoIndexRange.create("graylog_1", now.plusDays(1), now.plusDays(2), now, 0);
    final MongoIndexRange indexRangeLatest = MongoIndexRange.create("graylog_2", new DateTime(0L, DateTimeZone.UTC), new DateTime(0L, DateTimeZone.UTC), now, 0);
    final SortedSet<IndexRange> indices = ImmutableSortedSet.orderedBy(IndexRange.COMPARATOR).add(indexRange0).add(indexRange1).add(indexRangeLatest).build();
    when(indexRangeService.find(any(DateTime.class), any(DateTime.class))).thenReturn(indices);
    final TimeRange absoluteRange = AbsoluteRange.create(now.minusDays(1), now.plusDays(1));
    final TimeRange keywordRange = KeywordRange.create("1 day ago");
    final TimeRange relativeRange = RelativeRange.create(3600);
    assertThat(searches.determineAffectedIndicesWithRanges(absoluteRange, null)).containsExactly(indexRangeLatest, indexRange0, indexRange1);
    assertThat(searches.determineAffectedIndicesWithRanges(keywordRange, null)).containsExactly(indexRangeLatest, indexRange0, indexRange1);
    assertThat(searches.determineAffectedIndicesWithRanges(relativeRange, null)).containsExactly(indexRangeLatest, indexRange0, indexRange1);
}
Also used : MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) IndexRange(org.graylog2.indexer.ranges.IndexRange) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) MongoIndexRange(org.graylog2.indexer.ranges.MongoIndexRange) ZonedDateTime(java.time.ZonedDateTime) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Example 4 with Indices

use of org.graylog2.indexer.indices.Indices in project graylog2-server by Graylog2.

the class Indices method areReopened.

public Map<String, Boolean> areReopened(Collection<String> indices) {
    final ClusterStateResponse clusterState = c.admin().cluster().prepareState().all().get();
    final ImmutableOpenMap<String, IndexMetaData> indicesMetaData = clusterState.getState().getMetaData().getIndices();
    return indices.stream().collect(Collectors.toMap(index -> index, index -> checkForReopened(indicesMetaData.get(index))));
}
Also used : ElasticsearchException(org.elasticsearch.ElasticsearchException) GetAliasesResponse(org.elasticsearch.action.admin.indices.alias.get.GetAliasesResponse) SortBuilders(org.elasticsearch.search.sort.SortBuilders) DateTimeZone(org.joda.time.DateTimeZone) MappingMetaData(org.elasticsearch.cluster.metadata.MappingMetaData) LoggerFactory(org.slf4j.LoggerFactory) IndexStats(org.elasticsearch.action.admin.indices.stats.IndexStats) IndexClosedException(org.elasticsearch.indices.IndexClosedException) QueryBuilders(org.elasticsearch.index.query.QueryBuilders) ObjectObjectCursor(com.carrotsearch.hppc.cursors.ObjectObjectCursor) ClusterState(org.elasticsearch.cluster.ClusterState) IndicesStatsRequest(org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest) Settings(org.elasticsearch.common.settings.Settings) WriteConsistencyLevel(org.elasticsearch.action.WriteConsistencyLevel) PutIndexTemplateRequest(org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest) Map(java.util.Map) SearchResponse(org.elasticsearch.action.search.SearchResponse) IndexRangeStats(org.graylog2.indexer.searches.IndexRangeStats) IndexSet(org.graylog2.indexer.IndexSet) IndexMapping(org.graylog2.indexer.IndexMapping) Max(org.elasticsearch.search.aggregations.metrics.max.Max) Collectors.toSet(java.util.stream.Collectors.toSet) SearchHit(org.elasticsearch.search.SearchHit) SearchType(org.elasticsearch.action.search.SearchType) ImmutableSet(com.google.common.collect.ImmutableSet) Min(org.elasticsearch.search.aggregations.metrics.min.Min) ImmutableMap(com.google.common.collect.ImmutableMap) Terms(org.elasticsearch.search.aggregations.bucket.terms.Terms) GetAliasesRequest(org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest) Collection(java.util.Collection) BulkResponse(org.elasticsearch.action.bulk.BulkResponse) AliasMetaData(org.elasticsearch.cluster.metadata.AliasMetaData) Set(java.util.Set) FilterAggregationBuilder(org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder) NotNull(javax.validation.constraints.NotNull) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) DeleteIndexTemplateRequest(org.elasticsearch.action.admin.indices.template.delete.DeleteIndexTemplateRequest) Filter(org.elasticsearch.search.aggregations.bucket.filter.Filter) List(java.util.List) AuditActor(org.graylog2.audit.AuditActor) ClusterHealthRequest(org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest) IndexMetaData(org.elasticsearch.cluster.metadata.IndexMetaData) ClusterStateRequest(org.elasticsearch.action.admin.cluster.state.ClusterStateRequest) Optional(java.util.Optional) BulkRequestBuilder(org.elasticsearch.action.bulk.BulkRequestBuilder) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) ImmutableOpenMap(org.elasticsearch.common.collect.ImmutableOpenMap) HashMap(java.util.HashMap) SearchRequest(org.elasticsearch.action.search.SearchRequest) Singleton(javax.inject.Singleton) ActionFuture(org.elasticsearch.action.ActionFuture) OpenIndexRequest(org.elasticsearch.action.admin.indices.open.OpenIndexRequest) UpdateSettingsRequest(org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest) ForceMergeRequest(org.elasticsearch.action.admin.indices.forcemerge.ForceMergeRequest) XContentHelper(org.elasticsearch.common.xcontent.XContentHelper) Inject(javax.inject.Inject) IndicesStatsResponse(org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse) GetIndexRequestBuilder(org.elasticsearch.action.admin.indices.get.GetIndexRequestBuilder) ImmutableList(com.google.common.collect.ImmutableList) NodeId(org.graylog2.plugin.system.NodeId) ShardStats(org.elasticsearch.action.admin.indices.stats.ShardStats) CreateIndexRequest(org.elasticsearch.action.admin.indices.create.CreateIndexRequest) TimeValue(org.elasticsearch.common.unit.TimeValue) ES_INDEX_CREATE(org.graylog2.audit.AuditEventTypes.ES_INDEX_CREATE) GetIndexResponse(org.elasticsearch.action.admin.indices.get.GetIndexResponse) Duration(com.github.joschi.jadconfig.util.Duration) Messages(org.graylog2.indexer.messages.Messages) AuditEventSender(org.graylog2.audit.AuditEventSender) Nullable(javax.annotation.Nullable) DeleteIndexRequest(org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest) Logger(org.slf4j.Logger) QueryBuilders.matchAllQuery(org.elasticsearch.index.query.QueryBuilders.matchAllQuery) GetIndexRequest(org.elasticsearch.action.admin.indices.get.GetIndexRequest) IndexNotFoundException(org.graylog2.indexer.IndexNotFoundException) Client(org.elasticsearch.client.Client) DateTime(org.joda.time.DateTime) ClusterHealthResponse(org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse) AggregationBuilders(org.elasticsearch.search.aggregations.AggregationBuilders) IOException(java.io.IOException) Maps(com.google.common.collect.Maps) IndicesAdminClient(org.elasticsearch.client.IndicesAdminClient) ClusterStateResponse(org.elasticsearch.action.admin.cluster.state.ClusterStateResponse) TimeUnit(java.util.concurrent.TimeUnit) CloseIndexRequest(org.elasticsearch.action.admin.indices.close.CloseIndexRequest) IndicesExistsResponse(org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse) FlushRequest(org.elasticsearch.action.admin.indices.flush.FlushRequest) DocsStats(org.elasticsearch.index.shard.DocsStats) SearchRequestBuilder(org.elasticsearch.action.search.SearchRequestBuilder) SortParseElement(org.elasticsearch.search.sort.SortParseElement) UnmodifiableIterator(com.google.common.collect.UnmodifiableIterator) ClusterHealthStatus(org.elasticsearch.cluster.health.ClusterHealthStatus) Collections(java.util.Collections) IndicesExistsRequest(org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequest) ClusterStateResponse(org.elasticsearch.action.admin.cluster.state.ClusterStateResponse) IndexMetaData(org.elasticsearch.cluster.metadata.IndexMetaData)

Example 5 with Indices

use of org.graylog2.indexer.indices.Indices in project graylog2-server by Graylog2.

the class Indices method deleteIndexTemplate.

public void deleteIndexTemplate(IndexSet indexSet) {
    final String templateName = indexSet.getConfig().indexTemplateName();
    final DeleteIndexTemplateRequest deleteRequest = c.admin().indices().prepareDeleteTemplate(templateName).request();
    try {
        final boolean acknowledged = c.admin().indices().deleteTemplate(deleteRequest).actionGet().isAcknowledged();
        if (acknowledged) {
            LOG.info("Deleted Graylog index template \"{}\" in Elasticsearch.", templateName);
        }
    } catch (Exception e) {
        LOG.error("Unable to delete the Graylog index template: " + templateName, e);
    }
}
Also used : DeleteIndexTemplateRequest(org.elasticsearch.action.admin.indices.template.delete.DeleteIndexTemplateRequest) ElasticsearchException(org.elasticsearch.ElasticsearchException) IndexClosedException(org.elasticsearch.indices.IndexClosedException) IndexNotFoundException(org.graylog2.indexer.IndexNotFoundException) IOException(java.io.IOException)

Aggregations

IndexSet (org.graylog2.indexer.IndexSet)20 IndexRange (org.graylog2.indexer.ranges.IndexRange)19 Set (java.util.Set)18 Test (org.junit.Test)18 DateTime (org.joda.time.DateTime)17 Map (java.util.Map)16 Collectors (java.util.stream.Collectors)15 List (java.util.List)14 TimeRange (org.graylog2.plugin.indexer.searches.timeranges.TimeRange)14 ZonedDateTime (java.time.ZonedDateTime)13 Inject (javax.inject.Inject)13 JsonNode (com.fasterxml.jackson.databind.JsonNode)11 ImmutableMap (com.google.common.collect.ImmutableMap)11 HashMap (java.util.HashMap)11 Indices (org.graylog2.indexer.indices.Indices)11 MongoIndexRange (org.graylog2.indexer.ranges.MongoIndexRange)11 Logger (org.slf4j.Logger)11 LoggerFactory (org.slf4j.LoggerFactory)11 Optional (java.util.Optional)10 ImmutableSet (com.google.common.collect.ImmutableSet)9