Search in sources :

Example 1 with StreamFilter

use of org.graylog.plugins.views.search.filter.StreamFilter in project graylog2-server by Graylog2.

the class ElasticsearchBackend method generateFilterClause.

// TODO make pluggable
public Optional<QueryBuilder> generateFilterClause(Filter filter, SearchJob job, Query query) {
    if (filter == null) {
        return Optional.empty();
    }
    switch(filter.type()) {
        case AndFilter.NAME:
            final BoolQueryBuilder andBuilder = QueryBuilders.boolQuery();
            filter.filters().stream().map(filter1 -> generateFilterClause(filter1, job, query)).forEach(optQueryBuilder -> optQueryBuilder.ifPresent(andBuilder::must));
            return Optional.of(andBuilder);
        case OrFilter.NAME:
            final BoolQueryBuilder orBuilder = QueryBuilders.boolQuery();
            // TODO for the common case "any of these streams" we can optimize the filter into
            // a single "termsQuery" instead of "termQuery OR termQuery" if all direct children are "StreamFilter"
            filter.filters().stream().map(filter1 -> generateFilterClause(filter1, job, query)).forEach(optQueryBuilder -> optQueryBuilder.ifPresent(orBuilder::should));
            return Optional.of(orBuilder);
        case StreamFilter.NAME:
            // Skipping stream filter, will be extracted elsewhere
            return Optional.empty();
        case QueryStringFilter.NAME:
            return Optional.of(QueryBuilders.queryStringQuery(this.queryStringDecorators.decorate(((QueryStringFilter) filter).query(), job, query)));
    }
    return Optional.empty();
}
Also used : ESSearchTypeHandler(org.graylog.storage.elasticsearch7.views.searchtypes.ESSearchTypeHandler) AndFilter(org.graylog.plugins.views.search.filter.AndFilter) ElasticsearchClient(org.graylog.storage.elasticsearch7.ElasticsearchClient) Arrays(java.util.Arrays) BackendQuery(org.graylog.plugins.views.search.engine.BackendQuery) QueryBackend(org.graylog.plugins.views.search.engine.QueryBackend) Provider(javax.inject.Provider) LoggerFactory(org.slf4j.LoggerFactory) FieldTypeException(org.graylog2.indexer.FieldTypeException) MultiSearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) Map(java.util.Map) IndicesOptions(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.support.IndicesOptions) SearchConfig(org.graylog.plugins.views.search.engine.SearchConfig) Set(java.util.Set) Collectors(java.util.stream.Collectors) BoolQueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.BoolQueryBuilder) Objects(java.util.Objects) List(java.util.List) Filter(org.graylog.plugins.views.search.Filter) Optional(java.util.Optional) SearchResponse(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchResponse) Query(org.graylog.plugins.views.search.Query) SearchTypeErrorParser(org.graylog.plugins.views.search.errors.SearchTypeErrorParser) HashMap(java.util.HashMap) ElasticsearchException(org.graylog2.indexer.ElasticsearchException) ShardOperationFailedException(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.ShardOperationFailedException) ArrayList(java.util.ArrayList) GlobalOverride(org.graylog.plugins.views.search.GlobalOverride) Inject(javax.inject.Inject) HashSet(java.util.HashSet) OrFilter(org.graylog.plugins.views.search.filter.OrFilter) SearchType(org.graylog.plugins.views.search.SearchType) QueryStringFilter(org.graylog.plugins.views.search.filter.QueryStringFilter) SearchTypeError(org.graylog.plugins.views.search.errors.SearchTypeError) QueryResult(org.graylog.plugins.views.search.QueryResult) TimeRangeQueryFactory(org.graylog.storage.elasticsearch7.TimeRangeQueryFactory) SearchJob(org.graylog.plugins.views.search.SearchJob) QueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.QueryBuilder) SearchSourceBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.search.builder.SearchSourceBuilder) Logger(org.slf4j.Logger) SearchRequest(org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.SearchRequest) Maps(com.google.common.collect.Maps) QueryStringDecorators(org.graylog.plugins.views.search.elasticsearch.QueryStringDecorators) Named(com.google.inject.name.Named) IndexLookup(org.graylog.plugins.views.search.elasticsearch.IndexLookup) QueryBuilders(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.QueryBuilders) Message(org.graylog2.plugin.Message) Collections(java.util.Collections) BoolQueryBuilder(org.graylog.shaded.elasticsearch7.org.elasticsearch.index.query.BoolQueryBuilder)

Example 2 with StreamFilter

use of org.graylog.plugins.views.search.filter.StreamFilter in project graylog2-server by Graylog2.

the class Query method shallowMappedFilter.

// TODO: This code assumes that we only use shallow filters for streams.
// If this ever changes, we need to implement a mapper that can handle filter trees.
private Filter shallowMappedFilter(EntityDescriptorIds entityDescriptorIds) {
    return Optional.ofNullable(filter()).map(optFilter -> {
        Set<Filter> newFilters = optFilter.filters().stream().map(filter -> {
            if (filter.type().equals(StreamFilter.NAME)) {
                final StreamFilter streamFilter = (StreamFilter) filter;
                final String streamId = entityDescriptorIds.getOrThrow(streamFilter.streamId(), ModelTypes.STREAM_V1);
                return streamFilter.toBuilder().streamId(streamId).build();
            }
            return filter;
        }).collect(toSet());
        return optFilter.toGenericBuilder().filters(newFilters).build();
    }).orElse(null);
}
Also used : JsonProperty(com.fasterxml.jackson.annotation.JsonProperty) AndFilter(org.graylog.plugins.views.search.filter.AndFilter) EntityDescriptorIds(org.graylog2.contentpacks.EntityDescriptorIds) BackendQuery(org.graylog.plugins.views.search.engine.BackendQuery) InvalidRangeParametersException(org.graylog2.plugin.indexer.searches.timeranges.InvalidRangeParametersException) LoggerFactory(org.slf4j.LoggerFactory) RelativeRange(org.graylog2.plugin.indexer.searches.timeranges.RelativeRange) ElasticsearchQueryString(org.graylog.plugins.views.search.elasticsearch.ElasticsearchQueryString) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) QueryEntity(org.graylog2.contentpacks.model.entities.QueryEntity) JsonIgnore(com.fasterxml.jackson.annotation.JsonIgnore) EmptyTimeRange(org.graylog.plugins.views.search.engine.EmptyTimeRange) StreamSupport(java.util.stream.StreamSupport) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) Collectors.toSet(java.util.stream.Collectors.toSet) ImmutableSet(com.google.common.collect.ImmutableSet) ContentPackable(org.graylog2.contentpacks.ContentPackable) Logger(org.slf4j.Logger) Set(java.util.Set) JsonAutoDetect(com.fasterxml.jackson.annotation.JsonAutoDetect) UUID(java.util.UUID) JsonPOJOBuilder(com.fasterxml.jackson.databind.annotation.JsonPOJOBuilder) Collectors(java.util.stream.Collectors) Objects(java.util.Objects) Traverser(com.google.common.graph.Traverser) ExecutionStateGlobalOverride(org.graylog.plugins.views.search.rest.ExecutionStateGlobalOverride) SearchTypeExecutionState(org.graylog.plugins.views.search.rest.SearchTypeExecutionState) JsonCreator(com.fasterxml.jackson.annotation.JsonCreator) AutoValue(com.google.auto.value.AutoValue) ImmutableSortedSet.of(com.google.common.collect.ImmutableSortedSet.of) JsonInclude(com.fasterxml.jackson.annotation.JsonInclude) Optional(java.util.Optional) MoreObjects.firstNonNull(com.google.common.base.MoreObjects.firstNonNull) JsonDeserialize(com.fasterxml.jackson.databind.annotation.JsonDeserialize) Collections(java.util.Collections) ModelTypes(org.graylog2.contentpacks.model.ModelTypes) Collectors.toSet(java.util.stream.Collectors.toSet) ImmutableSet(com.google.common.collect.ImmutableSet) Set(java.util.Set) ElasticsearchQueryString(org.graylog.plugins.views.search.elasticsearch.ElasticsearchQueryString) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter)

Example 3 with StreamFilter

use of org.graylog.plugins.views.search.filter.StreamFilter in project graylog2-server by Graylog2.

the class QueryEntity method shallowMappedFilter.

// TODO: This code assumes that we only use shallow filters for streams.
// If this ever changes, we need to implement a mapper that can handle filter trees.
private Filter shallowMappedFilter(Map<EntityDescriptor, Object> nativeEntities) {
    return Optional.ofNullable(filter()).map(optFilter -> {
        Set<Filter> newFilters = optFilter.filters().stream().map(filter -> {
            if (filter.type().matches(StreamFilter.NAME)) {
                final StreamFilter streamFilter = (StreamFilter) filter;
                final Stream stream = (Stream) nativeEntities.get(EntityDescriptor.create(streamFilter.streamId(), ModelTypes.STREAM_V1));
                if (Objects.isNull(stream)) {
                    throw new ContentPackException("Could not find matching stream id: " + streamFilter.streamId());
                }
                return streamFilter.toBuilder().streamId(stream.getId()).build();
            }
            return filter;
        }).collect(Collectors.toSet());
        return optFilter.toGenericBuilder().filters(newFilters).build();
    }).orElse(null);
}
Also used : JsonProperty(com.fasterxml.jackson.annotation.JsonProperty) BackendQuery(org.graylog.plugins.views.search.engine.BackendQuery) Query(org.graylog.plugins.views.search.Query) ContentPackException(org.graylog2.contentpacks.exceptions.ContentPackException) GlobalOverride(org.graylog.plugins.views.search.GlobalOverride) ValueReference(org.graylog2.contentpacks.model.entities.references.ValueReference) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) NativeEntityConverter(org.graylog2.contentpacks.NativeEntityConverter) Map(java.util.Map) JsonIgnore(com.fasterxml.jackson.annotation.JsonIgnore) StreamSupport(java.util.stream.StreamSupport) TimeRange(org.graylog2.plugin.indexer.searches.timeranges.TimeRange) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) Collectors.toSet(java.util.stream.Collectors.toSet) ImmutableSet(com.google.common.collect.ImmutableSet) Set(java.util.Set) JsonAutoDetect(com.fasterxml.jackson.annotation.JsonAutoDetect) JsonPOJOBuilder(com.fasterxml.jackson.databind.annotation.JsonPOJOBuilder) Collectors(java.util.stream.Collectors) Objects(java.util.Objects) Traverser(com.google.common.graph.Traverser) Stream(org.graylog2.plugin.streams.Stream) JsonCreator(com.fasterxml.jackson.annotation.JsonCreator) AutoValue(com.google.auto.value.AutoValue) ImmutableSortedSet.of(com.google.common.collect.ImmutableSortedSet.of) JsonInclude(com.fasterxml.jackson.annotation.JsonInclude) Filter(org.graylog.plugins.views.search.Filter) Optional(java.util.Optional) MoreObjects.firstNonNull(com.google.common.base.MoreObjects.firstNonNull) JsonDeserialize(com.fasterxml.jackson.databind.annotation.JsonDeserialize) Collections(java.util.Collections) ModelTypes(org.graylog2.contentpacks.model.ModelTypes) ContentPackException(org.graylog2.contentpacks.exceptions.ContentPackException) Collectors.toSet(java.util.stream.Collectors.toSet) ImmutableSet(com.google.common.collect.ImmutableSet) Set(java.util.Set) Stream(org.graylog2.plugin.streams.Stream) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter)

Example 4 with StreamFilter

use of org.graylog.plugins.views.search.filter.StreamFilter in project graylog2-server by Graylog2.

the class ViewFacadeTest method itShouldCreateADTOFromAnEntity.

@Test
@MongoDBFixtures("ViewFacadeTest.json")
public void itShouldCreateADTOFromAnEntity() throws Exception {
    final StreamImpl stream = new StreamImpl(Collections.emptyMap());
    final Entity viewEntity = createViewEntity();
    final Map<EntityDescriptor, Object> nativeEntities = new HashMap<>(1);
    nativeEntities.put(EntityDescriptor.create(newStreamId, ModelTypes.STREAM_V1), stream);
    final UserImpl fakeUser = new UserImpl(mock(PasswordAlgorithmFactory.class), new Permissions(ImmutableSet.of()), ImmutableMap.of("username", "testuser"));
    when(userService.load("testuser")).thenReturn(fakeUser);
    final NativeEntity<ViewDTO> nativeEntity = facade.createNativeEntity(viewEntity, Collections.emptyMap(), nativeEntities, "testuser");
    assertThat(nativeEntity.descriptor().title()).isEqualTo("title");
    assertThat(nativeEntity.descriptor().type()).isEqualTo(ModelTypes.SEARCH_V1);
    Optional<ViewDTO> resultedView = viewService.get(nativeEntity.descriptor().id().id());
    assertThat(resultedView).isPresent();
    Optional<Search> search = searchDbService.get(resultedView.get().searchId());
    assertThat(search).isPresent();
    final Query query = search.get().queries().iterator().next();
    assertThat(query.filter()).isNotNull();
    assertThat(query.filter().filters()).isNotEmpty();
    final StreamFilter streamFilter = (StreamFilter) query.filter().filters().iterator().next();
    assertThat(streamFilter.streamId()).doesNotMatch(newStreamId);
}
Also used : NativeEntity(org.graylog2.contentpacks.model.entities.NativeEntity) PivotEntity(org.graylog2.contentpacks.model.entities.PivotEntity) QueryEntity(org.graylog2.contentpacks.model.entities.QueryEntity) EventListEntity(org.graylog2.contentpacks.model.entities.EventListEntity) ViewEntity(org.graylog2.contentpacks.model.entities.ViewEntity) Entity(org.graylog2.contentpacks.model.entities.Entity) SearchEntity(org.graylog2.contentpacks.model.entities.SearchEntity) ViewStateEntity(org.graylog2.contentpacks.model.entities.ViewStateEntity) StreamEntity(org.graylog2.contentpacks.model.entities.StreamEntity) MessageListEntity(org.graylog2.contentpacks.model.entities.MessageListEntity) Query(org.graylog.plugins.views.search.Query) HashMap(java.util.HashMap) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) EntityDescriptor(org.graylog2.contentpacks.model.entities.EntityDescriptor) ViewDTO(org.graylog.plugins.views.search.views.ViewDTO) PasswordAlgorithmFactory(org.graylog2.security.PasswordAlgorithmFactory) StreamImpl(org.graylog2.streams.StreamImpl) Search(org.graylog.plugins.views.search.Search) UserImpl(org.graylog2.users.UserImpl) Permissions(org.graylog2.shared.security.Permissions) MongoDBFixtures(org.graylog.testing.mongodb.MongoDBFixtures) Test(org.junit.Test)

Example 5 with StreamFilter

use of org.graylog.plugins.views.search.filter.StreamFilter in project graylog2-server by Graylog2.

the class ViewFacadeTest method itShouldCreateAViewEntity.

@Test
@MongoDBFixtures("ViewFacadeTest.json")
public void itShouldCreateAViewEntity() {
    final ViewDTO viewDTO = viewService.get(viewId).orElseThrow(() -> new NotFoundException("Missing view with id: " + viewId));
    final EntityDescriptor searchDescriptor = EntityDescriptor.create(viewDTO.id(), ModelTypes.SEARCH_V1);
    final EntityDescriptor streamDescriptor = EntityDescriptor.create(streamId, ModelTypes.STREAM_V1);
    final EntityDescriptorIds entityDescriptorIds = EntityDescriptorIds.of(searchDescriptor, streamDescriptor);
    final Optional<Entity> optionalEntity = facade.exportEntity(searchDescriptor, entityDescriptorIds);
    assertThat(optionalEntity).isPresent();
    final Entity entity = optionalEntity.get();
    assertThat(entity).isInstanceOf(EntityV1.class);
    final EntityV1 entityV1 = (EntityV1) entity;
    assertThat(entityV1.id()).isEqualTo(ModelId.of(entityDescriptorIds.get(searchDescriptor).orElse(null)));
    assertThat(entityV1.type()).isEqualTo(ModelTypes.SEARCH_V1);
    final ViewEntity viewEntity = objectMapper.convertValue(entityV1.data(), ViewEntity.class);
    assertThat(viewEntity.title().asString()).isEqualTo(viewDTO.title());
    assertThat(viewEntity.type().toString()).isEqualTo(ViewDTO.Type.SEARCH.toString());
    assertThat(viewEntity.search().queries().size()).isEqualTo(1);
    final QueryEntity queryEntity = viewEntity.search().queries().iterator().next();
    assertThat(queryEntity.filter().filters().size()).isEqualTo(1);
    final StreamFilter filter = (StreamFilter) queryEntity.filter().filters().iterator().next();
    assertThat(filter.streamId()).doesNotMatch(streamId);
}
Also used : EntityV1(org.graylog2.contentpacks.model.entities.EntityV1) ViewDTO(org.graylog.plugins.views.search.views.ViewDTO) EntityDescriptor(org.graylog2.contentpacks.model.entities.EntityDescriptor) NativeEntity(org.graylog2.contentpacks.model.entities.NativeEntity) PivotEntity(org.graylog2.contentpacks.model.entities.PivotEntity) QueryEntity(org.graylog2.contentpacks.model.entities.QueryEntity) EventListEntity(org.graylog2.contentpacks.model.entities.EventListEntity) ViewEntity(org.graylog2.contentpacks.model.entities.ViewEntity) Entity(org.graylog2.contentpacks.model.entities.Entity) SearchEntity(org.graylog2.contentpacks.model.entities.SearchEntity) ViewStateEntity(org.graylog2.contentpacks.model.entities.ViewStateEntity) StreamEntity(org.graylog2.contentpacks.model.entities.StreamEntity) MessageListEntity(org.graylog2.contentpacks.model.entities.MessageListEntity) ViewEntity(org.graylog2.contentpacks.model.entities.ViewEntity) EntityDescriptorIds(org.graylog2.contentpacks.EntityDescriptorIds) NotFoundException(javax.ws.rs.NotFoundException) QueryEntity(org.graylog2.contentpacks.model.entities.QueryEntity) StreamFilter(org.graylog.plugins.views.search.filter.StreamFilter) MongoDBFixtures(org.graylog.testing.mongodb.MongoDBFixtures) Test(org.junit.Test)

Aggregations

StreamFilter (org.graylog.plugins.views.search.filter.StreamFilter)7 BackendQuery (org.graylog.plugins.views.search.engine.BackendQuery)5 Collections (java.util.Collections)4 Objects (java.util.Objects)4 Optional (java.util.Optional)4 Set (java.util.Set)4 Collectors (java.util.stream.Collectors)4 Query (org.graylog.plugins.views.search.Query)4 HashMap (java.util.HashMap)3 Map (java.util.Map)3 Filter (org.graylog.plugins.views.search.Filter)3 GlobalOverride (org.graylog.plugins.views.search.GlobalOverride)3 AndFilter (org.graylog.plugins.views.search.filter.AndFilter)3 QueryEntity (org.graylog2.contentpacks.model.entities.QueryEntity)3 JsonAutoDetect (com.fasterxml.jackson.annotation.JsonAutoDetect)2 JsonCreator (com.fasterxml.jackson.annotation.JsonCreator)2 JsonIgnore (com.fasterxml.jackson.annotation.JsonIgnore)2 JsonInclude (com.fasterxml.jackson.annotation.JsonInclude)2 JsonProperty (com.fasterxml.jackson.annotation.JsonProperty)2 JsonDeserialize (com.fasterxml.jackson.databind.annotation.JsonDeserialize)2