Search in sources :

Example 6 with TimeFormatExtractionFn

use of org.apache.druid.query.extraction.TimeFormatExtractionFn in project druid by druid-io.

the class TimeFilteringTest method testTimeFilterWithTimeFormatExtractionFn.

@Test
public void testTimeFilterWithTimeFormatExtractionFn() {
    ExtractionFn exfn = new TimeFormatExtractionFn("EEEE", DateTimes.inferTzFromString("America/New_York"), "en", null, false);
    assertFilterMatches(new SelectorDimFilter(ColumnHolder.TIME_COLUMN_NAME, "Wednesday", exfn), ImmutableList.of("0", "1", "2", "3", "4", "5"));
}
Also used : ExtractionFn(org.apache.druid.query.extraction.ExtractionFn) JavaScriptExtractionFn(org.apache.druid.query.extraction.JavaScriptExtractionFn) LookupExtractionFn(org.apache.druid.query.lookup.LookupExtractionFn) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) Test(org.junit.Test)

Example 7 with TimeFormatExtractionFn

use of org.apache.druid.query.extraction.TimeFormatExtractionFn in project druid by druid-io.

the class ExpressionSelectorBenchmark method timeFloorUsingExtractionFn.

@Benchmark
public void timeFloorUsingExtractionFn(Blackhole blackhole) {
    final Sequence<Cursor> cursors = new QueryableIndexStorageAdapter(index).makeCursors(null, index.getDataInterval(), VirtualColumns.EMPTY, Granularities.ALL, false, null);
    final List<?> results = cursors.map(cursor -> {
        final DimensionSelector selector = cursor.getColumnSelectorFactory().makeDimensionSelector(new ExtractionDimensionSpec(ColumnHolder.TIME_COLUMN_NAME, "v", new TimeFormatExtractionFn(null, null, null, Granularities.HOUR, true)));
        consumeDimension(cursor, selector, blackhole);
        return null;
    }).toList();
    blackhole.consume(results);
}
Also used : BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Measurement(org.openjdk.jmh.annotations.Measurement) Intervals(org.apache.druid.java.util.common.Intervals) SegmentGenerator(org.apache.druid.segment.generator.SegmentGenerator) Blackhole(org.openjdk.jmh.infra.Blackhole) ColumnValueSelector(org.apache.druid.segment.ColumnValueSelector) Scope(org.openjdk.jmh.annotations.Scope) Warmup(org.openjdk.jmh.annotations.Warmup) DefaultDimensionSpec(org.apache.druid.query.dimension.DefaultDimensionSpec) ColumnHolder(org.apache.druid.segment.column.ColumnHolder) ImmutableList(com.google.common.collect.ImmutableList) OutputTimeUnit(org.openjdk.jmh.annotations.OutputTimeUnit) DimensionSelector(org.apache.druid.segment.DimensionSelector) TearDown(org.openjdk.jmh.annotations.TearDown) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) ExpressionVirtualColumn(org.apache.druid.segment.virtual.ExpressionVirtualColumn) Sequence(org.apache.druid.java.util.common.guava.Sequence) Setup(org.openjdk.jmh.annotations.Setup) QueryableIndexStorageAdapter(org.apache.druid.segment.QueryableIndexStorageAdapter) VirtualColumns(org.apache.druid.segment.VirtualColumns) GeneratorColumnSchema(org.apache.druid.segment.generator.GeneratorColumnSchema) Mode(org.openjdk.jmh.annotations.Mode) Closer(org.apache.druid.java.util.common.io.Closer) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) Param(org.openjdk.jmh.annotations.Param) QueryableIndex(org.apache.druid.segment.QueryableIndex) GeneratorSchemaInfo(org.apache.druid.segment.generator.GeneratorSchemaInfo) ValueType(org.apache.druid.segment.column.ValueType) TestExprMacroTable(org.apache.druid.query.expression.TestExprMacroTable) State(org.openjdk.jmh.annotations.State) StrlenExtractionFn(org.apache.druid.query.extraction.StrlenExtractionFn) Benchmark(org.openjdk.jmh.annotations.Benchmark) Granularities(org.apache.druid.java.util.common.granularity.Granularities) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) Cursor(org.apache.druid.segment.Cursor) NullHandling(org.apache.druid.common.config.NullHandling) LinearShardSpec(org.apache.druid.timeline.partition.LinearShardSpec) DataSegment(org.apache.druid.timeline.DataSegment) Level(org.openjdk.jmh.annotations.Level) ColumnType(org.apache.druid.segment.column.ColumnType) Fork(org.openjdk.jmh.annotations.Fork) BitSet(java.util.BitSet) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) DimensionSelector(org.apache.druid.segment.DimensionSelector) QueryableIndexStorageAdapter(org.apache.druid.segment.QueryableIndexStorageAdapter) Cursor(org.apache.druid.segment.Cursor) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Example 8 with TimeFormatExtractionFn

use of org.apache.druid.query.extraction.TimeFormatExtractionFn in project druid by druid-io.

the class ExpressionSelectorBenchmark method timeFormatUsingExtractionFn.

@Benchmark
public void timeFormatUsingExtractionFn(Blackhole blackhole) {
    final Sequence<Cursor> cursors = new QueryableIndexStorageAdapter(index).makeCursors(null, index.getDataInterval(), VirtualColumns.EMPTY, Granularities.ALL, false, null);
    final List<?> results = cursors.map(cursor -> {
        final DimensionSelector selector = cursor.getColumnSelectorFactory().makeDimensionSelector(new ExtractionDimensionSpec(ColumnHolder.TIME_COLUMN_NAME, "v", new TimeFormatExtractionFn("yyyy-MM-dd", null, null, null, false)));
        consumeDimension(cursor, selector, blackhole);
        return null;
    }).toList();
    blackhole.consume(results);
}
Also used : BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Measurement(org.openjdk.jmh.annotations.Measurement) Intervals(org.apache.druid.java.util.common.Intervals) SegmentGenerator(org.apache.druid.segment.generator.SegmentGenerator) Blackhole(org.openjdk.jmh.infra.Blackhole) ColumnValueSelector(org.apache.druid.segment.ColumnValueSelector) Scope(org.openjdk.jmh.annotations.Scope) Warmup(org.openjdk.jmh.annotations.Warmup) DefaultDimensionSpec(org.apache.druid.query.dimension.DefaultDimensionSpec) ColumnHolder(org.apache.druid.segment.column.ColumnHolder) ImmutableList(com.google.common.collect.ImmutableList) OutputTimeUnit(org.openjdk.jmh.annotations.OutputTimeUnit) DimensionSelector(org.apache.druid.segment.DimensionSelector) TearDown(org.openjdk.jmh.annotations.TearDown) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) ExpressionVirtualColumn(org.apache.druid.segment.virtual.ExpressionVirtualColumn) Sequence(org.apache.druid.java.util.common.guava.Sequence) Setup(org.openjdk.jmh.annotations.Setup) QueryableIndexStorageAdapter(org.apache.druid.segment.QueryableIndexStorageAdapter) VirtualColumns(org.apache.druid.segment.VirtualColumns) GeneratorColumnSchema(org.apache.druid.segment.generator.GeneratorColumnSchema) Mode(org.openjdk.jmh.annotations.Mode) Closer(org.apache.druid.java.util.common.io.Closer) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) Param(org.openjdk.jmh.annotations.Param) QueryableIndex(org.apache.druid.segment.QueryableIndex) GeneratorSchemaInfo(org.apache.druid.segment.generator.GeneratorSchemaInfo) ValueType(org.apache.druid.segment.column.ValueType) TestExprMacroTable(org.apache.druid.query.expression.TestExprMacroTable) State(org.openjdk.jmh.annotations.State) StrlenExtractionFn(org.apache.druid.query.extraction.StrlenExtractionFn) Benchmark(org.openjdk.jmh.annotations.Benchmark) Granularities(org.apache.druid.java.util.common.granularity.Granularities) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) Cursor(org.apache.druid.segment.Cursor) NullHandling(org.apache.druid.common.config.NullHandling) LinearShardSpec(org.apache.druid.timeline.partition.LinearShardSpec) DataSegment(org.apache.druid.timeline.DataSegment) Level(org.openjdk.jmh.annotations.Level) ColumnType(org.apache.druid.segment.column.ColumnType) Fork(org.openjdk.jmh.annotations.Fork) BitSet(java.util.BitSet) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) DimensionSelector(org.apache.druid.segment.DimensionSelector) QueryableIndexStorageAdapter(org.apache.druid.segment.QueryableIndexStorageAdapter) Cursor(org.apache.druid.segment.Cursor) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Example 9 with TimeFormatExtractionFn

use of org.apache.druid.query.extraction.TimeFormatExtractionFn in project druid by druid-io.

the class GroupByQueryRunnerTest method testSubqueryWithOuterTimeFilter.

@Test
public void testSubqueryWithOuterTimeFilter() {
    final GroupByQuery subquery = makeQueryBuilder().setDataSource(QueryRunnerTestHelper.DATA_SOURCE).setQuerySegmentSpec(QueryRunnerTestHelper.FULL_ON_INTERVAL_SPEC).setDimensions(new DefaultDimensionSpec("market", "market"), new DefaultDimensionSpec("quality", "quality")).setAggregatorSpecs(QueryRunnerTestHelper.ROWS_COUNT, new LongSumAggregatorFactory("index", "index")).setGranularity(QueryRunnerTestHelper.DAY_GRAN).build();
    final DimFilter fridayFilter = new SelectorDimFilter(ColumnHolder.TIME_COLUMN_NAME, "Friday", new TimeFormatExtractionFn("EEEE", null, null, null, false));
    final DimFilter firstDaysFilter = new InDimFilter(ColumnHolder.TIME_COLUMN_NAME, ImmutableList.of("1", "2", "3"), new TimeFormatExtractionFn("d", null, null, null, false));
    final GroupByQuery query = makeQueryBuilder().setDataSource(subquery).setQuerySegmentSpec(QueryRunnerTestHelper.FULL_ON_INTERVAL_SPEC).setDimensions(Collections.emptyList()).setDimFilter(firstDaysFilter).setAggregatorSpecs(new FilteredAggregatorFactory(QueryRunnerTestHelper.ROWS_COUNT, fridayFilter)).setGranularity(QueryRunnerTestHelper.DAY_GRAN).build();
    List<ResultRow> expectedResults = Arrays.asList(makeRow(query, "2011-02-01", "rows", 0L), makeRow(query, "2011-02-02", "rows", 0L), makeRow(query, "2011-02-03", "rows", 0L), makeRow(query, "2011-03-01", "rows", 0L), makeRow(query, "2011-03-02", "rows", 0L), makeRow(query, "2011-03-03", "rows", 0L), makeRow(query, "2011-04-01", "rows", 13L), makeRow(query, "2011-04-02", "rows", 0L), makeRow(query, "2011-04-03", "rows", 0L));
    Iterable<ResultRow> results = GroupByQueryRunnerTestHelper.runQuery(factory, runner, query);
    TestHelper.assertExpectedObjects(expectedResults, results, "subquery-time-filter");
}
Also used : FilteredAggregatorFactory(org.apache.druid.query.aggregation.FilteredAggregatorFactory) TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) LongSumAggregatorFactory(org.apache.druid.query.aggregation.LongSumAggregatorFactory) InDimFilter(org.apache.druid.query.filter.InDimFilter) AndDimFilter(org.apache.druid.query.filter.AndDimFilter) RegexDimFilter(org.apache.druid.query.filter.RegexDimFilter) NotDimFilter(org.apache.druid.query.filter.NotDimFilter) DimFilter(org.apache.druid.query.filter.DimFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) JavaScriptDimFilter(org.apache.druid.query.filter.JavaScriptDimFilter) SearchQueryDimFilter(org.apache.druid.query.filter.SearchQueryDimFilter) SelectorDimFilter(org.apache.druid.query.filter.SelectorDimFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) ExtractionDimFilter(org.apache.druid.query.filter.ExtractionDimFilter) OrDimFilter(org.apache.druid.query.filter.OrDimFilter) DefaultDimensionSpec(org.apache.druid.query.dimension.DefaultDimensionSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Example 10 with TimeFormatExtractionFn

use of org.apache.druid.query.extraction.TimeFormatExtractionFn in project druid by druid-io.

the class TopNQueryRunnerTest method testTopNTimeExtraction.

@Test
public void testTopNTimeExtraction() {
    TopNQuery query = new TopNQueryBuilder().dataSource(QueryRunnerTestHelper.DATA_SOURCE).granularity(QueryRunnerTestHelper.ALL_GRAN).dimension(new ExtractionDimensionSpec(ColumnHolder.TIME_COLUMN_NAME, "dayOfWeek", new TimeFormatExtractionFn("EEEE", null, null, null, false))).metric("index").threshold(2).intervals(QueryRunnerTestHelper.FULL_ON_INTERVAL_SPEC).aggregators(QueryRunnerTestHelper.ROWS_COUNT, QueryRunnerTestHelper.INDEX_DOUBLE_SUM).postAggregators(QueryRunnerTestHelper.ADD_ROWS_INDEX_CONSTANT).build();
    List<Result<TopNResultValue>> expectedResults = Collections.singletonList(new Result<>(DateTimes.of("2011-01-12T00:00:00.000Z"), new TopNResultValue(Arrays.<Map<String, Object>>asList(ImmutableMap.of("dayOfWeek", "Wednesday", "rows", 182L, "index", 76010.28100585938, "addRowsIndexConstant", 76193.28100585938), ImmutableMap.of("dayOfWeek", "Thursday", "rows", 182L, "index", 75203.26300811768, "addRowsIndexConstant", 75386.26300811768)))));
    assertExpectedResults(expectedResults, query);
}
Also used : TimeFormatExtractionFn(org.apache.druid.query.extraction.TimeFormatExtractionFn) ExtractionDimensionSpec(org.apache.druid.query.dimension.ExtractionDimensionSpec) Result(org.apache.druid.query.Result) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Aggregations

TimeFormatExtractionFn (org.apache.druid.query.extraction.TimeFormatExtractionFn)12 ExtractionDimensionSpec (org.apache.druid.query.dimension.ExtractionDimensionSpec)9 Test (org.junit.Test)9 DefaultDimensionSpec (org.apache.druid.query.dimension.DefaultDimensionSpec)8 SelectorDimFilter (org.apache.druid.query.filter.SelectorDimFilter)7 InitializedNullHandlingTest (org.apache.druid.testing.InitializedNullHandlingTest)7 OrDimFilter (org.apache.druid.query.filter.OrDimFilter)5 List (java.util.List)3 ExtractionFn (org.apache.druid.query.extraction.ExtractionFn)3 StrlenExtractionFn (org.apache.druid.query.extraction.StrlenExtractionFn)3 ImmutableList (com.google.common.collect.ImmutableList)2 ArrayList (java.util.ArrayList)2 BitSet (java.util.BitSet)2 TimeUnit (java.util.concurrent.TimeUnit)2 NullHandling (org.apache.druid.common.config.NullHandling)2 Intervals (org.apache.druid.java.util.common.Intervals)2 Granularities (org.apache.druid.java.util.common.granularity.Granularities)2 Sequence (org.apache.druid.java.util.common.guava.Sequence)2 Closer (org.apache.druid.java.util.common.io.Closer)2 LongSumAggregatorFactory (org.apache.druid.query.aggregation.LongSumAggregatorFactory)2