Search in sources :

Example 1 with HiveSubScan

use of org.apache.drill.exec.store.hive.HiveSubScan in project drill by apache.

the class ReadersInitializer method init.

/**
 * Selects reader constructor reference as {@link HiveReaderFactory} readerFactory.
 * Then check if input splits are empty creates empty record reader, or one reader per split otherwise.
 *
 * @param ctx    context related to fragment
 * @param config context which holds different Hive configurations
 * @return list containing one or more readers
 */
public static List<RecordReader> init(ExecutorFragmentContext ctx, HiveSubScan config) {
    final HiveReaderFactory readerFactory = getReaderFactory(config);
    final UserGroupInformation proxyUgi = ImpersonationUtil.createProxyUgi(config.getUserName(), ctx.getQueryUserName());
    final List<List<InputSplit>> inputSplits = config.getInputSplits();
    final HiveConf hiveConf = config.getHiveConf();
    if (inputSplits.isEmpty()) {
        return Collections.singletonList(readerFactory.createReader(config.getTable(), null, /*partition*/
        null, /*split*/
        config.getColumns(), ctx, hiveConf, proxyUgi));
    } else {
        IndexedPartitions partitions = getPartitions(config);
        return IntStream.range(0, inputSplits.size()).mapToObj(idx -> readerFactory.createReader(config.getTable(), partitions.get(idx), inputSplits.get(idx), config.getColumns(), ctx, hiveConf, proxyUgi)).collect(Collectors.toList());
    }
}
Also used : IntStream(java.util.stream.IntStream) FragmentContext(org.apache.drill.exec.ops.FragmentContext) HiveSubScan(org.apache.drill.exec.store.hive.HiveSubScan) TextInputFormat(org.apache.hadoop.mapred.TextInputFormat) Collection(java.util.Collection) SchemaPath(org.apache.drill.common.expression.SchemaPath) HiveConf(org.apache.hadoop.hive.conf.HiveConf) Collectors(java.util.stream.Collectors) List(java.util.List) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation) HivePartition(org.apache.drill.exec.store.hive.HivePartition) ImpersonationUtil(org.apache.drill.exec.util.ImpersonationUtil) InputSplit(org.apache.hadoop.mapred.InputSplit) RecordReader(org.apache.drill.exec.store.RecordReader) HiveTableWithColumnCache(org.apache.drill.exec.store.hive.HiveTableWithColumnCache) Collections(java.util.Collections) ExecutorFragmentContext(org.apache.drill.exec.ops.ExecutorFragmentContext) List(java.util.List) HiveConf(org.apache.hadoop.hive.conf.HiveConf) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation)

Aggregations

Collection (java.util.Collection)1 Collections (java.util.Collections)1 List (java.util.List)1 Collectors (java.util.stream.Collectors)1 IntStream (java.util.stream.IntStream)1 SchemaPath (org.apache.drill.common.expression.SchemaPath)1 ExecutorFragmentContext (org.apache.drill.exec.ops.ExecutorFragmentContext)1 FragmentContext (org.apache.drill.exec.ops.FragmentContext)1 RecordReader (org.apache.drill.exec.store.RecordReader)1 HivePartition (org.apache.drill.exec.store.hive.HivePartition)1 HiveSubScan (org.apache.drill.exec.store.hive.HiveSubScan)1 HiveTableWithColumnCache (org.apache.drill.exec.store.hive.HiveTableWithColumnCache)1 ImpersonationUtil (org.apache.drill.exec.util.ImpersonationUtil)1 HiveConf (org.apache.hadoop.hive.conf.HiveConf)1 InputSplit (org.apache.hadoop.mapred.InputSplit)1 TextInputFormat (org.apache.hadoop.mapred.TextInputFormat)1 UserGroupInformation (org.apache.hadoop.security.UserGroupInformation)1