Search in sources :

Example 16 with FileMetaData

use of org.apache.parquet.hadoop.metadata.FileMetaData in project drill by apache.

the class ParquetFileWriter method end.

/**
 * ends a file once all blocks have been written.
 * closes the file.
 * @param extraMetaData the extra meta data to write in the footer
 * @throws IOException if there is an error while writing
 */
public void end(Map<String, String> extraMetaData) throws IOException {
    state = state.end();
    serializeColumnIndexes(columnIndexes, blocks, out, fileEncryptor);
    serializeOffsetIndexes(offsetIndexes, blocks, out, fileEncryptor);
    serializeBloomFilters(bloomFilters, blocks, out, fileEncryptor);
    LOG.debug("{}: end", out.getPos());
    this.footer = new ParquetMetadata(new FileMetaData(schema, extraMetaData, Version.FULL_VERSION), blocks);
    serializeFooter(footer, out, fileEncryptor);
    out.close();
}
Also used : ParquetMetadata(org.apache.parquet.hadoop.metadata.ParquetMetadata) Util.writeFileMetaData(org.apache.parquet.format.Util.writeFileMetaData) FileMetaData(org.apache.parquet.hadoop.metadata.FileMetaData)

Aggregations

FileMetaData (org.apache.parquet.hadoop.metadata.FileMetaData)16 ParquetMetadata (org.apache.parquet.hadoop.metadata.ParquetMetadata)11 Path (org.apache.hadoop.fs.Path)10 ArrayList (java.util.ArrayList)7 BlockMetaData (org.apache.parquet.hadoop.metadata.BlockMetaData)7 Domain (com.facebook.presto.common.predicate.Domain)3 TupleDomain (com.facebook.presto.common.predicate.TupleDomain)3 Type (com.facebook.presto.common.type.Type)3 TypeManager (com.facebook.presto.common.type.TypeManager)3 FileFormatDataSourceStats (com.facebook.presto.hive.FileFormatDataSourceStats)3 HdfsEnvironment (com.facebook.presto.hive.HdfsEnvironment)3 IOException (java.io.IOException)3 ColumnDescriptor (org.apache.parquet.column.ColumnDescriptor)3 RuntimeStats (com.facebook.presto.common.RuntimeStats)2 HdfsContext (com.facebook.presto.hive.HdfsContext)2 HdfsParquetDataSource.buildHdfsParquetDataSource (com.facebook.presto.hive.parquet.HdfsParquetDataSource.buildHdfsParquetDataSource)2 ParquetPageSource (com.facebook.presto.hive.parquet.ParquetPageSource)2 AggregatedMemoryContext (com.facebook.presto.memory.context.AggregatedMemoryContext)2 AggregatedMemoryContext.newSimpleAggregatedMemoryContext (com.facebook.presto.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext)2 Field (com.facebook.presto.parquet.Field)2