Search in sources :

Example 21 with JoinFilterSplit

use of org.apache.druid.segment.join.filter.JoinFilterSplit in project druid by druid-io.

the class JoinFilterAnalyzerTest method test_filterPushDown_factToRegionThreeRHSColumnsAllDirectAndFilterOnRHS.

@Test
public void test_filterPushDown_factToRegionThreeRHSColumnsAllDirectAndFilterOnRHS() {
    JoinableClause factExprToRegon = new JoinableClause(FACT_TO_REGION_PREFIX, new IndexedTableJoinable(regionsTable), JoinType.LEFT, JoinConditionAnalysis.forExpression(StringUtils.format("\"%sregionIsoCode\" == regionIsoCode && \"%scountryIsoCode\" == regionIsoCode && \"%sregionName\" == user", FACT_TO_REGION_PREFIX, FACT_TO_REGION_PREFIX, FACT_TO_REGION_PREFIX), FACT_TO_REGION_PREFIX, ExprMacroTable.nil()));
    List<JoinableClause> joinableClauses = ImmutableList.of(factExprToRegon);
    Filter originalFilter = new OrFilter(ImmutableList.of(new SelectorFilter("r1.regionName", "Fourems Province"), new SelectorFilter("r1.regionIsoCode", "AAAA")));
    JoinFilterPreAnalysis joinFilterPreAnalysis = makeDefaultConfigPreAnalysis(originalFilter, joinableClauses, VirtualColumns.EMPTY);
    HashJoinSegmentStorageAdapter adapter = new HashJoinSegmentStorageAdapter(factSegment.asStorageAdapter(), joinableClauses, joinFilterPreAnalysis);
    // This query doesn't execute because regionName is not a key column, but we can still check the
    // filter rewrites.
    expectedException.expect(IAE.class);
    expectedException.expectMessage("Cannot build hash-join matcher on non-key-based condition: Equality{leftExpr=user, rightColumn='regionName'}");
    JoinTestHelper.verifyCursors(adapter.makeCursors(originalFilter, Intervals.ETERNITY, VirtualColumns.EMPTY, Granularities.ALL, false, null), ImmutableList.of("page", FACT_TO_REGION_PREFIX + "regionName"), ImmutableList.of());
    JoinFilterSplit expectedFilterSplit = new JoinFilterSplit(new OrFilter(ImmutableList.of(new SelectorFilter("user", "Fourems Province"), new SelectorFilter("regionIsoCode", "AAAA"))), null, ImmutableSet.of());
    JoinFilterSplit actualFilterSplit = JoinFilterAnalyzer.splitFilter(joinFilterPreAnalysis);
    Assert.assertEquals(expectedFilterSplit, actualFilterSplit);
}
Also used : SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) JoinFilterPreAnalysis(org.apache.druid.segment.join.filter.JoinFilterPreAnalysis) FalseFilter(org.apache.druid.segment.filter.FalseFilter) OrFilter(org.apache.druid.segment.filter.OrFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) AndFilter(org.apache.druid.segment.filter.AndFilter) BoundFilter(org.apache.druid.segment.filter.BoundFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) ExpressionDimFilter(org.apache.druid.query.filter.ExpressionDimFilter) Filter(org.apache.druid.query.filter.Filter) JoinFilterSplit(org.apache.druid.segment.join.filter.JoinFilterSplit) IndexedTableJoinable(org.apache.druid.segment.join.table.IndexedTableJoinable) OrFilter(org.apache.druid.segment.filter.OrFilter) Test(org.junit.Test)

Example 22 with JoinFilterSplit

use of org.apache.druid.segment.join.filter.JoinFilterSplit in project druid by druid-io.

the class JoinFilterAnalyzerTest method test_filterPushDown_factToCountryFullWithFilterOnChannelAndCountryName.

@Test
public void test_filterPushDown_factToCountryFullWithFilterOnChannelAndCountryName() {
    Filter filter = new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#es.wikipedia"), new SelectorFilter(FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "countryName", "El Salvador")));
    List<JoinableClause> joinableClauses = ImmutableList.of(factToCountryOnIsoCode(JoinType.FULL));
    JoinFilterPreAnalysis joinFilterPreAnalysis = makeDefaultConfigPreAnalysis(filter, joinableClauses, VirtualColumns.EMPTY);
    HashJoinSegmentStorageAdapter adapter = new HashJoinSegmentStorageAdapter(factSegment.asStorageAdapter(), joinableClauses, joinFilterPreAnalysis);
    JoinTestHelper.verifyCursors(adapter.makeCursors(filter, Intervals.ETERNITY, VirtualColumns.EMPTY, Granularities.ALL, false, null), ImmutableList.of("page", "countryIsoCode", "countryNumber", FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "countryIsoCode", FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "countryName", FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "countryNumber"), ImmutableList.of(new Object[] { "Wendigo", "SV", 12L, "SV", "El Salvador", 12L }));
    JoinFilterSplit expectedFilterSplit = new JoinFilterSplit(new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#es.wikipedia"), new InDimFilter("countryIsoCode", ImmutableSet.of("SV"), null, null).toFilter())), new SelectorFilter(FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "countryName", "El Salvador"), ImmutableSet.of());
    JoinFilterSplit actualFilterSplit = JoinFilterAnalyzer.splitFilter(joinFilterPreAnalysis);
    Assert.assertEquals(expectedFilterSplit, actualFilterSplit);
}
Also used : AndFilter(org.apache.druid.segment.filter.AndFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) JoinFilterPreAnalysis(org.apache.druid.segment.join.filter.JoinFilterPreAnalysis) FalseFilter(org.apache.druid.segment.filter.FalseFilter) OrFilter(org.apache.druid.segment.filter.OrFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) AndFilter(org.apache.druid.segment.filter.AndFilter) BoundFilter(org.apache.druid.segment.filter.BoundFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) ExpressionDimFilter(org.apache.druid.query.filter.ExpressionDimFilter) Filter(org.apache.druid.query.filter.Filter) JoinFilterSplit(org.apache.druid.segment.join.filter.JoinFilterSplit) InDimFilter(org.apache.druid.query.filter.InDimFilter) Test(org.junit.Test)

Example 23 with JoinFilterSplit

use of org.apache.druid.segment.join.filter.JoinFilterSplit in project druid by druid-io.

the class JoinFilterAnalyzerTest method test_filterPushDown_factToCountryInnerUsingCountryNumberFilterOnNullsUsingLookup.

@Test
public void test_filterPushDown_factToCountryInnerUsingCountryNumberFilterOnNullsUsingLookup() {
    List<JoinableClause> joinableClauses = ImmutableList.of(factToCountryNameUsingIsoCodeLookup(JoinType.INNER));
    Filter originalFilter = new AndFilter(ImmutableList.of(new SelectorFilter("channel", null), new SelectorFilter(FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v", null)));
    JoinFilterPreAnalysis joinFilterPreAnalysis = makeDefaultConfigPreAnalysis(originalFilter, joinableClauses, VirtualColumns.EMPTY);
    HashJoinSegmentStorageAdapter adapter = new HashJoinSegmentStorageAdapter(factSegment.asStorageAdapter(), joinableClauses, joinFilterPreAnalysis);
    JoinTestHelper.verifyCursors(adapter.makeCursors(originalFilter, Intervals.ETERNITY, VirtualColumns.EMPTY, Granularities.ALL, false, null), ImmutableList.of("page", "countryIsoCode", FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "k", FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v"), ImmutableList.of());
    JoinFilterSplit expectedFilterSplit = new JoinFilterSplit(null, new AndFilter(ImmutableList.of(new SelectorFilter("channel", null), new SelectorFilter(FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v", null))), ImmutableSet.of());
    JoinFilterSplit actualFilterSplit = JoinFilterAnalyzer.splitFilter(joinFilterPreAnalysis);
    Assert.assertEquals(expectedFilterSplit, actualFilterSplit);
}
Also used : AndFilter(org.apache.druid.segment.filter.AndFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) JoinFilterPreAnalysis(org.apache.druid.segment.join.filter.JoinFilterPreAnalysis) FalseFilter(org.apache.druid.segment.filter.FalseFilter) OrFilter(org.apache.druid.segment.filter.OrFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) AndFilter(org.apache.druid.segment.filter.AndFilter) BoundFilter(org.apache.druid.segment.filter.BoundFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) ExpressionDimFilter(org.apache.druid.query.filter.ExpressionDimFilter) Filter(org.apache.druid.query.filter.Filter) JoinFilterSplit(org.apache.druid.segment.join.filter.JoinFilterSplit) Test(org.junit.Test)

Example 24 with JoinFilterSplit

use of org.apache.druid.segment.join.filter.JoinFilterSplit in project druid by druid-io.

the class JoinFilterAnalyzerTest method test_filterPushDown_factToCountryInnerUsingCountryNumberFilterOnChannelAndCountryNameUsingLookup.

@Test
public void test_filterPushDown_factToCountryInnerUsingCountryNumberFilterOnChannelAndCountryNameUsingLookup() {
    List<JoinableClause> joinableClauses = ImmutableList.of(factToCountryNameUsingNumberLookup(JoinType.INNER));
    Filter originalFilter = new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#en.wikipedia"), new SelectorFilter(FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v", "Australia")));
    JoinFilterPreAnalysis joinFilterPreAnalysis = makeDefaultConfigPreAnalysis(originalFilter, joinableClauses, VirtualColumns.EMPTY);
    HashJoinSegmentStorageAdapter adapter = new HashJoinSegmentStorageAdapter(factSegment.asStorageAdapter(), joinableClauses, joinFilterPreAnalysis);
    // In non-SQL-compatible mode, we get an extra row, since the 'null' countryNumber for "Talk:Oswald Tilghman"
    // is interpreted as 0 (a.k.a. Australia).
    JoinTestHelper.verifyCursors(adapter.makeCursors(originalFilter, Intervals.ETERNITY, VirtualColumns.EMPTY, Granularities.ALL, false, null), ImmutableList.of("page", "countryIsoCode", FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "k", FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v"), NullHandling.sqlCompatible() ? ImmutableList.of(new Object[] { "Peremptory norm", "AU", "0", "Australia" }) : ImmutableList.of(new Object[] { "Talk:Oswald Tilghman", null, "0", "Australia" }, new Object[] { "Peremptory norm", "AU", "0", "Australia" }));
    JoinFilterSplit expectedFilterSplit = new JoinFilterSplit(new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#en.wikipedia"), new InDimFilter("countryNumber", ImmutableSet.of("0"), null, null).toFilter())), new SelectorFilter(FACT_TO_COUNTRY_ON_NUMBER_PREFIX + "v", "Australia"), ImmutableSet.of());
    JoinFilterSplit actualFilterSplit = JoinFilterAnalyzer.splitFilter(joinFilterPreAnalysis);
    Assert.assertEquals(expectedFilterSplit, actualFilterSplit);
}
Also used : AndFilter(org.apache.druid.segment.filter.AndFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) JoinFilterPreAnalysis(org.apache.druid.segment.join.filter.JoinFilterPreAnalysis) FalseFilter(org.apache.druid.segment.filter.FalseFilter) OrFilter(org.apache.druid.segment.filter.OrFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) AndFilter(org.apache.druid.segment.filter.AndFilter) BoundFilter(org.apache.druid.segment.filter.BoundFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) ExpressionDimFilter(org.apache.druid.query.filter.ExpressionDimFilter) Filter(org.apache.druid.query.filter.Filter) JoinFilterSplit(org.apache.druid.segment.join.filter.JoinFilterSplit) InDimFilter(org.apache.druid.query.filter.InDimFilter) Test(org.junit.Test)

Example 25 with JoinFilterSplit

use of org.apache.druid.segment.join.filter.JoinFilterSplit in project druid by druid-io.

the class JoinFilterAnalyzerTest method test_filterPushDown_factConcatExpressionToCountryLeftFilterOnChannelAndCountryNameUsingLookup.

@Test
public void test_filterPushDown_factConcatExpressionToCountryLeftFilterOnChannelAndCountryNameUsingLookup() {
    JoinableClause factExprToCountry = new JoinableClause(FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX, LookupJoinable.wrap(countryIsoCodeToNameLookup), JoinType.LEFT, JoinConditionAnalysis.forExpression(StringUtils.format("\"%sk\" == concat(countryIsoCode, regionIsoCode)", FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX), FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX, ExprMacroTable.nil()));
    List<JoinableClause> joinableClauses = ImmutableList.of(factExprToCountry);
    Filter filter = new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#en.wikipedia"), new SelectorFilter("c1.v", "Usca")));
    JoinFilterPreAnalysis joinFilterPreAnalysis = makeDefaultConfigPreAnalysis(filter, joinableClauses, VirtualColumns.EMPTY);
    HashJoinSegmentStorageAdapter adapter = new HashJoinSegmentStorageAdapter(factSegment.asStorageAdapter(), joinableClauses, joinFilterPreAnalysis);
    JoinTestHelper.verifyCursors(adapter.makeCursors(filter, Intervals.ETERNITY, VirtualColumns.EMPTY, Granularities.ALL, false, null), ImmutableList.of("page", FACT_TO_COUNTRY_ON_ISO_CODE_PREFIX + "v"), ImmutableList.of(new Object[] { "President of India", "Usca" }, new Object[] { "Otjiwarongo Airport", "Usca" }, new Object[] { "Carlo Curti", "Usca" }));
    ExpressionVirtualColumn expectedVirtualColumn = new ExpressionVirtualColumn("JOIN-FILTER-PUSHDOWN-VIRTUAL-COLUMN-0", "concat(countryIsoCode, regionIsoCode)", ColumnType.STRING, ExprMacroTable.nil());
    JoinFilterSplit expectedFilterSplit = new JoinFilterSplit(new AndFilter(ImmutableList.of(new SelectorFilter("channel", "#en.wikipedia"), new InDimFilter("JOIN-FILTER-PUSHDOWN-VIRTUAL-COLUMN-0", ImmutableSet.of("USCA"), null, null).toFilter())), new SelectorFilter("c1.v", "Usca"), ImmutableSet.of(expectedVirtualColumn));
    JoinFilterSplit actualFilterSplit = JoinFilterAnalyzer.splitFilter(joinFilterPreAnalysis);
    Assert.assertEquals(expectedFilterSplit.getBaseTableFilter(), actualFilterSplit.getBaseTableFilter());
    Assert.assertEquals(expectedFilterSplit.getJoinTableFilter(), actualFilterSplit.getJoinTableFilter());
    ExpressionVirtualColumn actualVirtualColumn = (ExpressionVirtualColumn) actualFilterSplit.getPushDownVirtualColumns().iterator().next();
    compareExpressionVirtualColumns(expectedVirtualColumn, actualVirtualColumn);
}
Also used : AndFilter(org.apache.druid.segment.filter.AndFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) ExpressionVirtualColumn(org.apache.druid.segment.virtual.ExpressionVirtualColumn) JoinFilterPreAnalysis(org.apache.druid.segment.join.filter.JoinFilterPreAnalysis) FalseFilter(org.apache.druid.segment.filter.FalseFilter) OrFilter(org.apache.druid.segment.filter.OrFilter) BoundDimFilter(org.apache.druid.query.filter.BoundDimFilter) SelectorFilter(org.apache.druid.segment.filter.SelectorFilter) AndFilter(org.apache.druid.segment.filter.AndFilter) BoundFilter(org.apache.druid.segment.filter.BoundFilter) InDimFilter(org.apache.druid.query.filter.InDimFilter) ExpressionDimFilter(org.apache.druid.query.filter.ExpressionDimFilter) Filter(org.apache.druid.query.filter.Filter) JoinFilterSplit(org.apache.druid.segment.join.filter.JoinFilterSplit) InDimFilter(org.apache.druid.query.filter.InDimFilter) Test(org.junit.Test)

Aggregations

Filter (org.apache.druid.query.filter.Filter)35 JoinFilterPreAnalysis (org.apache.druid.segment.join.filter.JoinFilterPreAnalysis)35 JoinFilterSplit (org.apache.druid.segment.join.filter.JoinFilterSplit)35 BoundDimFilter (org.apache.druid.query.filter.BoundDimFilter)34 ExpressionDimFilter (org.apache.druid.query.filter.ExpressionDimFilter)34 InDimFilter (org.apache.druid.query.filter.InDimFilter)34 AndFilter (org.apache.druid.segment.filter.AndFilter)34 BoundFilter (org.apache.druid.segment.filter.BoundFilter)34 FalseFilter (org.apache.druid.segment.filter.FalseFilter)34 OrFilter (org.apache.druid.segment.filter.OrFilter)34 SelectorFilter (org.apache.druid.segment.filter.SelectorFilter)34 Test (org.junit.Test)33 IndexedTableJoinable (org.apache.druid.segment.join.table.IndexedTableJoinable)8 ExpressionVirtualColumn (org.apache.druid.segment.virtual.ExpressionVirtualColumn)6 VirtualColumns (org.apache.druid.segment.VirtualColumns)3 JoinFilterPreAnalysisKey (org.apache.druid.segment.join.filter.JoinFilterPreAnalysisKey)3 JoinableClauses (org.apache.druid.segment.join.filter.JoinableClauses)3 JoinFilterRewriteConfig (org.apache.druid.segment.join.filter.rewrite.JoinFilterRewriteConfig)3 VirtualColumn (org.apache.druid.segment.VirtualColumn)2 Lists (com.google.common.collect.Lists)1