Search in sources :

Example 6 with HiveOutputFormat

use of org.apache.hadoop.hive.ql.io.HiveOutputFormat in project hive by apache.

the class Utilities method createDummyFileForEmptyTable.

@SuppressWarnings("rawtypes")
private static Path createDummyFileForEmptyTable(JobConf job, MapWork work, Path hiveScratchDir, String alias) throws Exception {
    TableDesc tableDesc = work.getAliasToPartnInfo().get(alias).getTableDesc();
    if (tableDesc.isNonNative()) {
        // if it does not need native storage, we can't create an empty file for it.
        return null;
    }
    Properties props = tableDesc.getProperties();
    HiveOutputFormat outFileFormat = HiveFileFormatUtils.getHiveOutputFormat(job, tableDesc);
    Path newPath = createEmptyFile(hiveScratchDir, outFileFormat, job, props, false);
    LOG.info("Changed input file for alias {} to newPath", alias, newPath);
    // update the work
    Map<Path, List<String>> pathToAliases = work.getPathToAliases();
    List<String> newList = new ArrayList<String>(1);
    newList.add(alias);
    pathToAliases.put(newPath, newList);
    work.setPathToAliases(pathToAliases);
    PartitionDesc pDesc = work.getAliasToPartnInfo().get(alias).clone();
    work.addPathToPartitionInfo(newPath, pDesc);
    return newPath;
}
Also used : Path(org.apache.hadoop.fs.Path) ArrayList(java.util.ArrayList) HiveOutputFormat(org.apache.hadoop.hive.ql.io.HiveOutputFormat) ValidWriteIdList(org.apache.hadoop.hive.common.ValidWriteIdList) LinkedList(java.util.LinkedList) ArrayList(java.util.ArrayList) List(java.util.List) PartitionDesc(org.apache.hadoop.hive.ql.plan.PartitionDesc) TableDesc(org.apache.hadoop.hive.ql.plan.TableDesc) Properties(java.util.Properties)

Aggregations

HiveOutputFormat (org.apache.hadoop.hive.ql.io.HiveOutputFormat)6 Properties (java.util.Properties)3 Path (org.apache.hadoop.fs.Path)3 PrestoException (com.facebook.presto.spi.PrestoException)2 IOException (java.io.IOException)2 PartitionDesc (org.apache.hadoop.hive.ql.plan.PartitionDesc)2 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 Method (java.lang.reflect.Method)1 ArrayList (java.util.ArrayList)1 LinkedList (java.util.LinkedList)1 List (java.util.List)1 FlinkHiveException (org.apache.flink.connectors.hive.FlinkHiveException)1 CatalogException (org.apache.flink.table.catalog.exceptions.CatalogException)1 ValidWriteIdList (org.apache.hadoop.hive.common.ValidWriteIdList)1 InvalidOperationException (org.apache.hadoop.hive.metastore.api.InvalidOperationException)1 MetaException (org.apache.hadoop.hive.metastore.api.MetaException)1 UnknownDBException (org.apache.hadoop.hive.metastore.api.UnknownDBException)1 HiveFileFormatUtils (org.apache.hadoop.hive.ql.io.HiveFileFormatUtils)1 RCFileOutputFormat (org.apache.hadoop.hive.ql.io.RCFileOutputFormat)1 MapredParquetOutputFormat (org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat)1