Search in sources :

Example 1 with MapLookupExtractor

use of org.apache.druid.query.extraction.MapLookupExtractor in project druid by druid-io.

the class LongFilteringTest method testLongFilterWithExtractionFn.

@Test
public void testLongFilterWithExtractionFn() {
    final Map<String, String> stringMap = new HashMap<>();
    stringMap.put("1", "Monday");
    stringMap.put("2", "Tuesday");
    stringMap.put("3", "Wednesday");
    stringMap.put("4", "Thursday");
    stringMap.put("5", "Friday");
    stringMap.put("6", "Saturday");
    LookupExtractor mapExtractor = new MapLookupExtractor(stringMap, false);
    LookupExtractionFn exfn = new LookupExtractionFn(mapExtractor, false, "UNKNOWN", false, true);
    assertFilterMatches(new SelectorDimFilter(LONG_COLUMN, "Monday", exfn), ImmutableList.of("1"));
    assertFilterMatches(new SelectorDimFilter(LONG_COLUMN, "Notaday", exfn), ImmutableList.of());
    assertFilterMatches(new BoundDimFilter(LONG_COLUMN, "Fridax", "Fridaz", false, false, null, exfn, StringComparators.ALPHANUMERIC), ImmutableList.of("5"));
    assertFilterMatches(new BoundDimFilter(LONG_COLUMN, "Friday", "Friday", true, true, null, exfn, StringComparators.ALPHANUMERIC), ImmutableList.of());
    assertFilterMatches(new InDimFilter(LONG_COLUMN, Arrays.asList("Caturday", "Saturday", "Tuesday"), exfn), ImmutableList.of("2", "6"));
    // test InFilter HashSet implementation
    List<String> bigList = Arrays.asList("Saturday", "Tuesday", "Caturday", "Xanaday", "Vojuday", "Gribaday", "Kipoday", "Dheferday", "Fakeday", "Qeearaday", "Hello", "World", "1", "2", "3", "4", "5", "6", "7");
    assertFilterMatches(new InDimFilter(LONG_COLUMN, bigList, exfn), ImmutableList.of("2", "6"));
    String jsFn = "function(x) { return(x === 'Wednesday' || x === 'Thursday') }";
    assertFilterMatchesSkipVectorize(new JavaScriptDimFilter(LONG_COLUMN, jsFn, exfn, JavaScriptConfig.getEnabledInstance()), ImmutableList.of("3", "4"));
    assertFilterMatches(new RegexDimFilter(LONG_COLUMN, ".*day", exfn), ImmutableList.of("1", "2", "3", "4", "5", "6"));
    assertFilterMatches(new SearchQueryDimFilter(LONG_COLUMN, new ContainsSearchQuerySpec("s", true), exfn), ImmutableList.of("2", "3", "4"));
}
Also used : RegexDimFilter(org.apache.druid.query.filter.RegexDimFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) HashMap(java.util.HashMap) ContainsSearchQuerySpec(org.apache.druid.query.search.ContainsSearchQuerySpec) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) LookupExtractor(org.apache.druid.query.lookup.LookupExtractor) LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) SearchQueryDimFilter(org.apache.druid.query.filter.SearchQueryDimFilter) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) JavaScriptDimFilter(org.apache.druid.query.filter.JavaScriptDimFilter) Test(org.junit.Test)

Example 2 with MapLookupExtractor

use of org.apache.druid.query.extraction.MapLookupExtractor in project druid by druid-io.

the class FloatAndDoubleFilteringTest method doTestFloatFilterWithExtractionFn.

private void doTestFloatFilterWithExtractionFn(final String columnName) {
    final Map<String, String> stringMap = new HashMap<>();
    stringMap.put("1.0", "Monday");
    stringMap.put("2.0", "Tuesday");
    stringMap.put("3.0", "Wednesday");
    stringMap.put("4.0", "Thursday");
    stringMap.put("5.0", "Friday");
    stringMap.put("6.0", "Saturday");
    LookupExtractor mapExtractor = new MapLookupExtractor(stringMap, false);
    LookupExtractionFn exfn = new LookupExtractionFn(mapExtractor, false, "UNKNOWN", false, true);
    assertFilterMatches(new SelectorDimFilter(columnName, "Monday", exfn), ImmutableList.of("1"));
    assertFilterMatches(new SelectorDimFilter(columnName, "Notaday", exfn), ImmutableList.of());
    assertFilterMatches(new BoundDimFilter(columnName, "Fridax", "Fridaz", false, false, null, exfn, StringComparators.ALPHANUMERIC), ImmutableList.of("5"));
    assertFilterMatches(new BoundDimFilter(columnName, "Friday", "Friday", true, true, null, exfn, StringComparators.ALPHANUMERIC), ImmutableList.of());
    assertFilterMatches(new InDimFilter(columnName, Arrays.asList("Caturday", "Saturday", "Tuesday"), exfn), ImmutableList.of("2", "6"));
    // test InFilter HashSet implementation
    List<String> bigList = Arrays.asList("Saturday", "Tuesday", "Caturday", "Xanaday", "Vojuday", "Gribaday", "Kipoday", "Dheferday", "Fakeday", "Qeearaday", "Hello", "World", "1", "2", "3", "4", "5", "6", "7");
    assertFilterMatches(new InDimFilter(columnName, bigList, exfn), ImmutableList.of("2", "6"));
    String jsFn = "function(x) { return(x === 'Wednesday' || x === 'Thursday') }";
    assertFilterMatchesSkipVectorize(new JavaScriptDimFilter(columnName, jsFn, exfn, JavaScriptConfig.getEnabledInstance()), ImmutableList.of("3", "4"));
    assertFilterMatches(new RegexDimFilter(columnName, ".*day", exfn), ImmutableList.of("1", "2", "3", "4", "5", "6"));
    assertFilterMatches(new SearchQueryDimFilter(columnName, new ContainsSearchQuerySpec("s", true), exfn), ImmutableList.of("2", "3", "4"));
}
Also used : RegexDimFilter(org.apache.druid.query.filter.RegexDimFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) HashMap(java.util.HashMap) ContainsSearchQuerySpec(org.apache.druid.query.search.ContainsSearchQuerySpec) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) LookupExtractor(org.apache.druid.query.lookup.LookupExtractor) LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) SearchQueryDimFilter(org.apache.druid.query.filter.SearchQueryDimFilter) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) JavaScriptDimFilter(org.apache.druid.query.filter.JavaScriptDimFilter)

Example 3 with MapLookupExtractor

use of org.apache.druid.query.extraction.MapLookupExtractor in project druid by druid-io.

the class ColumnComparisonFilterTest method testSelectorWithLookupExtractionFn.

@Test
public void testSelectorWithLookupExtractionFn() {
    final Map<String, String> stringMap = ImmutableMap.of("a", "7");
    LookupExtractor mapExtractor = new MapLookupExtractor(stringMap, false);
    LookupExtractionFn lookupFn = new LookupExtractionFn(mapExtractor, true, null, false, true);
    assertFilterMatchesSkipVectorize(new ColumnComparisonDimFilter(ImmutableList.of(new ExtractionDimensionSpec("dim0", "dim0", lookupFn), new ExtractionDimensionSpec("dim1", "dim1", lookupFn))), ImmutableList.of("2", "5", "7", "8"));
}
Also used : LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) ColumnComparisonDimFilter(org.apache.druid.query.filter.ColumnComparisonDimFilter) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) LookupExtractor(org.apache.druid.query.lookup.LookupExtractor) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) Test(org.junit.Test)

Example 4 with MapLookupExtractor

use of org.apache.druid.query.extraction.MapLookupExtractor in project druid by druid-io.

the class SelectorFilterTest method testSelectorWithLookupExtractionFn.

@Test
public void testSelectorWithLookupExtractionFn() {
    final Map<String, String> stringMap = ImmutableMap.of("1", "HELLO", "a", "HELLO", "abdef", "HELLO", "abc", "UNKNOWN");
    LookupExtractor mapExtractor = new MapLookupExtractor(stringMap, false);
    LookupExtractionFn lookupFn = new LookupExtractionFn(mapExtractor, false, "UNKNOWN", false, true);
    assertFilterMatches(new SelectorDimFilter("dim0", "HELLO", lookupFn), ImmutableList.of("1"));
    assertFilterMatches(new SelectorDimFilter("dim0", "UNKNOWN", lookupFn), ImmutableList.of("0", "2", "3", "4", "5"));
    assertFilterMatches(new SelectorDimFilter("dim1", "HELLO", lookupFn), ImmutableList.of("3", "4"));
    assertFilterMatches(new SelectorDimFilter("dim1", "UNKNOWN", lookupFn), ImmutableList.of("0", "1", "2", "5"));
    assertFilterMatches(new SelectorDimFilter("dim2", "HELLO", lookupFn), ImmutableList.of("0", "3"));
    assertFilterMatches(new SelectorDimFilter("dim2", "UNKNOWN", lookupFn), ImmutableList.of("0", "1", "2", "4", "5"));
    assertFilterMatches(new SelectorDimFilter("dim3", "HELLO", lookupFn), ImmutableList.of());
    assertFilterMatches(new SelectorDimFilter("dim3", "UNKNOWN", lookupFn), ImmutableList.of("0", "1", "2", "3", "4", "5"));
    assertFilterMatches(new SelectorDimFilter("dim4", "HELLO", lookupFn), ImmutableList.of());
    assertFilterMatches(new SelectorDimFilter("dim4", "UNKNOWN", lookupFn), ImmutableList.of("0", "1", "2", "3", "4", "5"));
    final Map<String, String> stringMap2 = ImmutableMap.of("2", "5");
    LookupExtractor mapExtractor2 = new MapLookupExtractor(stringMap2, false);
    LookupExtractionFn lookupFn2 = new LookupExtractionFn(mapExtractor2, true, null, false, true);
    assertFilterMatches(new SelectorDimFilter("dim0", "5", lookupFn2), ImmutableList.of("2", "5"));
    final Map<String, String> stringMap3 = ImmutableMap.of("1", "");
    LookupExtractor mapExtractor3 = new MapLookupExtractor(stringMap3, false);
    LookupExtractionFn lookupFn3 = new LookupExtractionFn(mapExtractor3, false, null, false, true);
    if (NullHandling.replaceWithDefault()) {
        // Nulls and empty strings are considered equivalent
        assertFilterMatches(new SelectorDimFilter("dim0", null, lookupFn3), ImmutableList.of("0", "1", "2", "3", "4", "5"));
    } else {
        assertFilterMatches(new SelectorDimFilter("dim0", null, lookupFn3), ImmutableList.of("0", "2", "3", "4", "5"));
        assertFilterMatches(new SelectorDimFilter("dim0", "", lookupFn3), ImmutableList.of("1"));
    }
    final Map<String, String> stringMap4 = ImmutableMap.of("9", "4");
    LookupExtractor mapExtractor4 = new MapLookupExtractor(stringMap4, false);
    LookupExtractionFn lookupFn4 = new LookupExtractionFn(mapExtractor4, true, null, false, true);
    final Map<String, String> stringMap5 = ImmutableMap.of("5", "44");
    LookupExtractor mapExtractor5 = new MapLookupExtractor(stringMap5, false);
    LookupExtractionFn lookupFn5 = new LookupExtractionFn(mapExtractor5, true, null, false, true);
    final Map<String, String> stringMap6 = ImmutableMap.of("5", "5");
    LookupExtractor mapExtractor6 = new MapLookupExtractor(stringMap6, false);
    LookupExtractionFn lookupFn6 = new LookupExtractionFn(mapExtractor6, true, null, false, true);
    // optimize() tests, check that filter was converted to the proper form
    SelectorDimFilter optFilter1 = new SelectorDimFilter("dim1", "UNKNOWN", lookupFn);
    SelectorDimFilter optFilter2 = new SelectorDimFilter("dim0", "5", lookupFn2);
    SelectorDimFilter optFilter3 = new SelectorDimFilter("dim0", null, lookupFn3);
    SelectorDimFilter optFilter4 = new SelectorDimFilter("dim0", "5", lookupFn4);
    SelectorDimFilter optFilter5 = new SelectorDimFilter("dim0", "5", lookupFn5);
    SelectorDimFilter optFilter6 = new SelectorDimFilter("dim0", "5", lookupFn6);
    InDimFilter optFilter2Optimized = new InDimFilter("dim0", Arrays.asList("2", "5"), null);
    SelectorDimFilter optFilter4Optimized = new SelectorDimFilter("dim0", "5", null);
    SelectorDimFilter optFilter6Optimized = new SelectorDimFilter("dim0", "5", null);
    Assert.assertTrue(optFilter1.equals(optFilter1.optimize()));
    Assert.assertTrue(optFilter2Optimized.equals(optFilter2.optimize()));
    Assert.assertTrue(optFilter3.equals(optFilter3.optimize()));
    Assert.assertTrue(optFilter4Optimized.equals(optFilter4.optimize()));
    Assert.assertTrue(optFilter5.equals(optFilter5.optimize()));
    Assert.assertTrue(optFilter6Optimized.equals(optFilter6.optimize()));
    assertFilterMatches(optFilter1, ImmutableList.of("0", "1", "2", "5"));
    assertFilterMatches(optFilter2, ImmutableList.of("2", "5"));
    if (NullHandling.replaceWithDefault()) {
        // Null and Empty strings are same
        assertFilterMatches(optFilter3, ImmutableList.of("0", "1", "2", "3", "4", "5"));
    } else {
        assertFilterMatches(optFilter3, ImmutableList.of("0", "2", "3", "4", "5"));
    }
    assertFilterMatches(optFilter4, ImmutableList.of("5"));
    assertFilterMatches(optFilter5, ImmutableList.of());
    assertFilterMatches(optFilter6, ImmutableList.of("5"));
    // tests that ExtractionDimFilter (identical to SelectorDimFilter now) optimize() with lookup works
    // remove these when ExtractionDimFilter is removed.
    assertFilterMatches(new ExtractionDimFilter("dim1", "UNKNOWN", lookupFn, null), ImmutableList.of("0", "1", "2", "5"));
    assertFilterMatches(new ExtractionDimFilter("dim0", "5", lookupFn2, null), ImmutableList.of("2", "5"));
    if (NullHandling.replaceWithDefault()) {
        assertFilterMatches(new ExtractionDimFilter("dim0", null, lookupFn3, null), ImmutableList.of("0", "1", "2", "3", "4", "5"));
    } else {
        assertFilterMatches(new ExtractionDimFilter("dim0", null, lookupFn3, null), ImmutableList.of("0", "2", "3", "4", "5"));
        assertFilterMatches(new ExtractionDimFilter("dim0", "", lookupFn3, null), ImmutableList.of("1"));
    }
}
Also used : LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) ExtractionDimFilter(org.apache.druid.query.filter.ExtractionDimFilter) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) LookupExtractor(org.apache.druid.query.lookup.LookupExtractor) Test(org.junit.Test)

Example 5 with MapLookupExtractor

use of org.apache.druid.query.extraction.MapLookupExtractor in project druid by druid-io.

the class GroupByQueryRunnerTest method testGroupByWithExtractionDimFilterKeyisNull.

@Test
public void testGroupByWithExtractionDimFilterKeyisNull() {
    Map<String, String> extractionMap = new HashMap<>();
    MapLookupExtractor mapLookupExtractor = new MapLookupExtractor(extractionMap, false);
    LookupExtractionFn lookupExtractionFn;
    if (NullHandling.replaceWithDefault()) {
        lookupExtractionFn = new LookupExtractionFn(mapLookupExtractor, false, null, true, false);
        extractionMap.put("", "REPLACED_VALUE");
    } else {
        lookupExtractionFn = new LookupExtractionFn(mapLookupExtractor, false, "REPLACED_VALUE", true, false);
        extractionMap.put("", "NOT_USED");
    }
    GroupByQuery query = makeQueryBuilder().setDataSource(QueryRunnerTestHelper.DATA_SOURCE).setQuerySegmentSpec(QueryRunnerTestHelper.FIRST_TO_THIRD).setDimensions(new DefaultDimensionSpec("null_column", "alias")).setAggregatorSpecs(QueryRunnerTestHelper.ROWS_COUNT, new LongSumAggregatorFactory("idx", "index")).setGranularity(QueryRunnerTestHelper.DAY_GRAN).setDimFilter(new ExtractionDimFilter("null_column", "REPLACED_VALUE", lookupExtractionFn, null)).build();
    List<ResultRow> expectedResults = Arrays.asList(makeRow(query, "2011-04-01", "alias", null, "rows", 13L, "idx", 6619L), makeRow(query, "2011-04-02", "alias", null, "rows", 13L, "idx", 5827L));
    Iterable<ResultRow> results = GroupByQueryRunnerTestHelper.runQuery(factory, runner, query);
    TestHelper.assertExpectedObjects(expectedResults, results, "dim-extraction");
}
Also used : LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) HashMap(java.util.HashMap) LongSumAggregatorFactory(org.apache.druid.query.aggregation.LongSumAggregatorFactory) MapLookupExtractor(org.apache.druid.query.extraction.MapLookupExtractor) ExtractionDimFilter(org.apache.druid.query.filter.ExtractionDimFilter) DefaultDimensionSpec(org.apache.druid.query.dimension.DefaultDimensionSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Aggregations

MapLookupExtractor (org.apache.druid.query.extraction.MapLookupExtractor)58 Test (org.junit.Test)56 LookupExtractionFn (org.apache.druid.query.lookup.LookupExtractionFn)40 InitializedNullHandlingTest (org.apache.druid.testing.InitializedNullHandlingTest)30 HashMap (java.util.HashMap)25 ExtractionDimensionSpec (org.apache.druid.query.dimension.ExtractionDimensionSpec)20 LongSumAggregatorFactory (org.apache.druid.query.aggregation.LongSumAggregatorFactory)18 SelectorDimFilter (org.apache.druid.query.filter.SelectorDimFilter)14 Result (org.apache.druid.query.Result)13 ExtractionDimFilter (org.apache.druid.query.filter.ExtractionDimFilter)13 ImmutableList (com.google.common.collect.ImmutableList)11 List (java.util.List)11 NullHandlingTest (org.apache.druid.common.config.NullHandlingTest)11 PreJoinableClause (org.apache.druid.query.planning.PreJoinableClause)11 InDimFilter (org.apache.druid.query.filter.InDimFilter)10 DefaultDimensionSpec (org.apache.druid.query.dimension.DefaultDimensionSpec)8 BoundDimFilter (org.apache.druid.query.filter.BoundDimFilter)8 LookupExtractor (org.apache.druid.query.lookup.LookupExtractor)8 JavaScriptDimFilter (org.apache.druid.query.filter.JavaScriptDimFilter)6 RegexDimFilter (org.apache.druid.query.filter.RegexDimFilter)6