Search in sources :

Example 1 with DrillScanRel

use of org.apache.drill.exec.planner.logical.DrillScanRel in project drill by apache.

the class HivePushPartitionFilterIntoScan method getFilterOnProject.

public static final StoragePluginOptimizerRule getFilterOnProject(OptimizerRulesContext optimizerRulesContext, final String defaultPartitionValue) {
    return new PruneScanRule(RelOptHelper.some(DrillFilterRel.class, RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))), "HivePushPartitionFilterIntoScan:Filter_On_Project_Hive", optimizerRulesContext) {

        @Override
        public PartitionDescriptor getPartitionDescriptor(PlannerSettings settings, TableScan scanRel) {
            return new HivePartitionDescriptor(settings, (DrillScanRel) scanRel, getOptimizerRulesContext().getManagedBuffer(), defaultPartitionValue);
        }

        @Override
        public boolean matches(RelOptRuleCall call) {
            final DrillScanRel scan = (DrillScanRel) call.rel(2);
            GroupScan groupScan = scan.getGroupScan();
            // this rule is applicable only for Hive based partition pruning
            if (PrelUtil.getPlannerSettings(scan.getCluster().getPlanner()).isHepPartitionPruningEnabled()) {
                return groupScan instanceof HiveScan && groupScan.supportsPartitionFilterPushdown() && !scan.partitionFilterPushdown();
            } else {
                return groupScan instanceof HiveScan && groupScan.supportsPartitionFilterPushdown();
            }
        }

        @Override
        public void onMatch(RelOptRuleCall call) {
            final DrillFilterRel filterRel = call.rel(0);
            final DrillProjectRel projectRel = call.rel(1);
            final DrillScanRel scanRel = call.rel(2);
            doOnMatch(call, filterRel, projectRel, scanRel);
        }
    };
}
Also used : GroupScan(org.apache.drill.exec.physical.base.GroupScan) TableScan(org.apache.calcite.rel.core.TableScan) DrillScanRel(org.apache.drill.exec.planner.logical.DrillScanRel) PruneScanRule(org.apache.drill.exec.planner.logical.partition.PruneScanRule) PlannerSettings(org.apache.drill.exec.planner.physical.PlannerSettings) DrillProjectRel(org.apache.drill.exec.planner.logical.DrillProjectRel) HiveScan(org.apache.drill.exec.store.hive.HiveScan) DrillFilterRel(org.apache.drill.exec.planner.logical.DrillFilterRel) RelOptRuleCall(org.apache.calcite.plan.RelOptRuleCall) HivePartitionDescriptor(org.apache.drill.exec.planner.sql.HivePartitionDescriptor)

Example 2 with DrillScanRel

use of org.apache.drill.exec.planner.logical.DrillScanRel in project drill by apache.

the class HivePushPartitionFilterIntoScan method getFilterOnScan.

public static final StoragePluginOptimizerRule getFilterOnScan(OptimizerRulesContext optimizerRulesContext, final String defaultPartitionValue) {
    return new PruneScanRule(RelOptHelper.some(DrillFilterRel.class, RelOptHelper.any(DrillScanRel.class)), "HivePushPartitionFilterIntoScan:Filter_On_Scan_Hive", optimizerRulesContext) {

        @Override
        public PartitionDescriptor getPartitionDescriptor(PlannerSettings settings, TableScan scanRel) {
            return new HivePartitionDescriptor(settings, (DrillScanRel) scanRel, getOptimizerRulesContext().getManagedBuffer(), defaultPartitionValue);
        }

        @Override
        public boolean matches(RelOptRuleCall call) {
            final DrillScanRel scan = (DrillScanRel) call.rel(1);
            GroupScan groupScan = scan.getGroupScan();
            // this rule is applicable only for Hive based partition pruning
            if (PrelUtil.getPlannerSettings(scan.getCluster().getPlanner()).isHepPartitionPruningEnabled()) {
                return groupScan instanceof HiveScan && groupScan.supportsPartitionFilterPushdown() && !scan.partitionFilterPushdown();
            } else {
                return groupScan instanceof HiveScan && groupScan.supportsPartitionFilterPushdown();
            }
        }

        @Override
        public void onMatch(RelOptRuleCall call) {
            final DrillFilterRel filterRel = call.rel(0);
            final DrillScanRel scanRel = call.rel(1);
            doOnMatch(call, filterRel, null, scanRel);
        }
    };
}
Also used : GroupScan(org.apache.drill.exec.physical.base.GroupScan) TableScan(org.apache.calcite.rel.core.TableScan) DrillScanRel(org.apache.drill.exec.planner.logical.DrillScanRel) PruneScanRule(org.apache.drill.exec.planner.logical.partition.PruneScanRule) PlannerSettings(org.apache.drill.exec.planner.physical.PlannerSettings) HiveScan(org.apache.drill.exec.store.hive.HiveScan) DrillFilterRel(org.apache.drill.exec.planner.logical.DrillFilterRel) RelOptRuleCall(org.apache.calcite.plan.RelOptRuleCall) HivePartitionDescriptor(org.apache.drill.exec.planner.sql.HivePartitionDescriptor)

Example 3 with DrillScanRel

use of org.apache.drill.exec.planner.logical.DrillScanRel in project drill by apache.

the class ScanPrule method onMatch.

@Override
public void onMatch(RelOptRuleCall call) {
    final DrillScanRel scan = (DrillScanRel) call.rel(0);
    GroupScan groupScan = scan.getGroupScan();
    DrillDistributionTrait partition = (groupScan.getMaxParallelizationWidth() > 1 || groupScan.getDistributionAffinity() == DistributionAffinity.HARD) ? DrillDistributionTrait.RANDOM_DISTRIBUTED : DrillDistributionTrait.SINGLETON;
    final RelTraitSet traits = scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(partition);
    final DrillScanPrel newScan = ScanPrel.create(scan, traits, groupScan, scan.getRowType());
    call.transformTo(newScan);
}
Also used : GroupScan(org.apache.drill.exec.physical.base.GroupScan) DrillScanRel(org.apache.drill.exec.planner.logical.DrillScanRel) RelTraitSet(org.apache.calcite.plan.RelTraitSet)

Example 4 with DrillScanRel

use of org.apache.drill.exec.planner.logical.DrillScanRel in project drill by apache.

the class FileSystemPartitionDescriptor method createTableScan.

@Override
public TableScan createTableScan(List<PartitionLocation> newPartitionLocation, String cacheFileRoot, boolean wasAllPartitionsPruned, MetadataContext metaContext) throws Exception {
    List<String> newFiles = Lists.newArrayList();
    for (final PartitionLocation location : newPartitionLocation) {
        if (!location.isCompositePartition()) {
            newFiles.add(location.getEntirePartitionLocation());
        } else {
            final Collection<SimplePartitionLocation> subPartitions = location.getPartitionLocationRecursive();
            for (final PartitionLocation subPart : subPartitions) {
                newFiles.add(subPart.getEntirePartitionLocation());
            }
        }
    }
    if (scanRel instanceof DrillScanRel) {
        final FormatSelection formatSelection = (FormatSelection) table.getSelection();
        final FileSelection newFileSelection = new FileSelection(null, newFiles, getBaseTableLocation(), cacheFileRoot, wasAllPartitionsPruned, formatSelection.getSelection().getDirStatus());
        newFileSelection.setMetaContext(metaContext);
        final FileGroupScan newGroupScan = ((FileGroupScan) ((DrillScanRel) scanRel).getGroupScan()).clone(newFileSelection);
        return new DrillScanRel(scanRel.getCluster(), scanRel.getTraitSet().plus(DrillRel.DRILL_LOGICAL), scanRel.getTable(), newGroupScan, scanRel.getRowType(), ((DrillScanRel) scanRel).getColumns(), true);
    } else if (scanRel instanceof EnumerableTableScan) {
        return createNewTableScanFromSelection((EnumerableTableScan) scanRel, newFiles, cacheFileRoot, wasAllPartitionsPruned, metaContext);
    } else {
        throw new UnsupportedOperationException("Only DrillScanRel and EnumerableTableScan is allowed!");
    }
}
Also used : FileSelection(org.apache.drill.exec.store.dfs.FileSelection) DrillScanRel(org.apache.drill.exec.planner.logical.DrillScanRel) FileGroupScan(org.apache.drill.exec.physical.base.FileGroupScan) EnumerableTableScan(org.apache.calcite.adapter.enumerable.EnumerableTableScan) DirPrunedEnumerableTableScan(org.apache.drill.exec.planner.logical.DirPrunedEnumerableTableScan) FormatSelection(org.apache.drill.exec.store.dfs.FormatSelection)

Example 5 with DrillScanRel

use of org.apache.drill.exec.planner.logical.DrillScanRel in project drill by apache.

the class ParquetPruneScanRule method getFilterOnScanParquet.

public static final RelOptRule getFilterOnScanParquet(OptimizerRulesContext optimizerRulesContext) {
    return new PruneScanRule(RelOptHelper.some(DrillFilterRel.class, RelOptHelper.any(DrillScanRel.class)), "PruneScanRule:Filter_On_Scan_Parquet", optimizerRulesContext) {

        @Override
        public PartitionDescriptor getPartitionDescriptor(PlannerSettings settings, TableScan scanRel) {
            return new ParquetPartitionDescriptor(settings, (DrillScanRel) scanRel);
        }

        @Override
        public boolean matches(RelOptRuleCall call) {
            final DrillScanRel scan = call.rel(1);
            GroupScan groupScan = scan.getGroupScan();
            // this rule is applicable only for parquet based partition pruning
            if (PrelUtil.getPlannerSettings(scan.getCluster().getPlanner()).isHepPartitionPruningEnabled()) {
                return groupScan instanceof ParquetGroupScan && groupScan.supportsPartitionFilterPushdown() && !scan.partitionFilterPushdown();
            } else {
                return groupScan instanceof ParquetGroupScan && groupScan.supportsPartitionFilterPushdown();
            }
        }

        @Override
        public void onMatch(RelOptRuleCall call) {
            final DrillFilterRel filterRel = call.rel(0);
            final DrillScanRel scanRel = call.rel(1);
            doOnMatch(call, filterRel, null, scanRel);
        }
    };
}
Also used : FileGroupScan(org.apache.drill.exec.physical.base.FileGroupScan) GroupScan(org.apache.drill.exec.physical.base.GroupScan) ParquetGroupScan(org.apache.drill.exec.store.parquet.ParquetGroupScan) TableScan(org.apache.calcite.rel.core.TableScan) DrillScanRel(org.apache.drill.exec.planner.logical.DrillScanRel) PlannerSettings(org.apache.drill.exec.planner.physical.PlannerSettings) DrillFilterRel(org.apache.drill.exec.planner.logical.DrillFilterRel) ParquetGroupScan(org.apache.drill.exec.store.parquet.ParquetGroupScan) RelOptRuleCall(org.apache.calcite.plan.RelOptRuleCall) ParquetPartitionDescriptor(org.apache.drill.exec.planner.ParquetPartitionDescriptor)

Aggregations

DrillScanRel (org.apache.drill.exec.planner.logical.DrillScanRel)12 GroupScan (org.apache.drill.exec.physical.base.GroupScan)7 PlannerSettings (org.apache.drill.exec.planner.physical.PlannerSettings)5 HiveScan (org.apache.drill.exec.store.hive.HiveScan)5 RelOptRuleCall (org.apache.calcite.plan.RelOptRuleCall)4 TableScan (org.apache.calcite.rel.core.TableScan)4 FileGroupScan (org.apache.drill.exec.physical.base.FileGroupScan)4 DrillFilterRel (org.apache.drill.exec.planner.logical.DrillFilterRel)4 DrillProjectRel (org.apache.drill.exec.planner.logical.DrillProjectRel)4 ParquetGroupScan (org.apache.drill.exec.store.parquet.ParquetGroupScan)3 EnumerableTableScan (org.apache.calcite.adapter.enumerable.EnumerableTableScan)2 RelDataType (org.apache.calcite.rel.type.RelDataType)2 ParquetPartitionDescriptor (org.apache.drill.exec.planner.ParquetPartitionDescriptor)2 DirPrunedEnumerableTableScan (org.apache.drill.exec.planner.logical.DirPrunedEnumerableTableScan)2 PruneScanRule (org.apache.drill.exec.planner.logical.partition.PruneScanRule)2 HivePartitionDescriptor (org.apache.drill.exec.planner.sql.HivePartitionDescriptor)2 FileSelection (org.apache.drill.exec.store.dfs.FileSelection)2 FormatSelection (org.apache.drill.exec.store.dfs.FormatSelection)2 Collection (java.util.Collection)1 RelTraitSet (org.apache.calcite.plan.RelTraitSet)1