Search in sources :

Example 1 with UserMetadataItem

use of org.apache.orc.OrcProto.UserMetadataItem in project hudi by apache.

the class OrcUtils method readFooter.

@Override
public Map<String, String> readFooter(Configuration conf, boolean required, Path orcFilePath, String... footerNames) {
    try (Reader reader = OrcFile.createReader(orcFilePath, OrcFile.readerOptions(conf))) {
        Map<String, String> footerVals = new HashMap<>();
        List<UserMetadataItem> metadataItemList = reader.getFileTail().getFooter().getMetadataList();
        Map<String, String> metadata = metadataItemList.stream().collect(Collectors.toMap(UserMetadataItem::getName, metadataItem -> metadataItem.getValue().toStringUtf8()));
        for (String footerName : footerNames) {
            if (metadata.containsKey(footerName)) {
                footerVals.put(footerName, metadata.get(footerName));
            } else if (required) {
                throw new MetadataNotFoundException("Could not find index in ORC footer. Looked for key " + footerName + " in " + orcFilePath);
            }
        }
        return footerVals;
    } catch (IOException io) {
        throw new HoodieIOException("Unable to read footer for ORC file:" + orcFilePath, io);
    }
}
Also used : HoodieAvroUtils(org.apache.hudi.avro.HoodieAvroUtils) VectorizedRowBatch(org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch) HoodieException(org.apache.hudi.exception.HoodieException) HashMap(java.util.HashMap) ByteBuffer(java.nio.ByteBuffer) OrcFile(org.apache.orc.OrcFile) ArrayList(java.util.ArrayList) BaseKeyGenerator(org.apache.hudi.keygen.BaseKeyGenerator) HashSet(java.util.HashSet) Reader(org.apache.orc.Reader) Options(org.apache.orc.Reader.Options) Map(java.util.Map) Configuration(org.apache.hadoop.conf.Configuration) Path(org.apache.hadoop.fs.Path) HoodieRecord(org.apache.hudi.common.model.HoodieRecord) GenericRecord(org.apache.avro.generic.GenericRecord) Schema(org.apache.avro.Schema) MetadataNotFoundException(org.apache.hudi.exception.MetadataNotFoundException) RecordReader(org.apache.orc.RecordReader) Set(java.util.Set) TypeDescription(org.apache.orc.TypeDescription) IOException(java.io.IOException) BytesColumnVector(org.apache.orc.storage.ql.exec.vector.BytesColumnVector) Collectors(java.util.stream.Collectors) List(java.util.List) HoodieKey(org.apache.hudi.common.model.HoodieKey) HoodieIOException(org.apache.hudi.exception.HoodieIOException) Collections(java.util.Collections) FSUtils(org.apache.hudi.common.fs.FSUtils) UserMetadataItem(org.apache.orc.OrcProto.UserMetadataItem) MetadataNotFoundException(org.apache.hudi.exception.MetadataNotFoundException) HoodieIOException(org.apache.hudi.exception.HoodieIOException) HashMap(java.util.HashMap) UserMetadataItem(org.apache.orc.OrcProto.UserMetadataItem) Reader(org.apache.orc.Reader) RecordReader(org.apache.orc.RecordReader) IOException(java.io.IOException) HoodieIOException(org.apache.hudi.exception.HoodieIOException)

Aggregations

IOException (java.io.IOException)1 ByteBuffer (java.nio.ByteBuffer)1 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 Collectors (java.util.stream.Collectors)1 Schema (org.apache.avro.Schema)1 GenericRecord (org.apache.avro.generic.GenericRecord)1 Configuration (org.apache.hadoop.conf.Configuration)1 Path (org.apache.hadoop.fs.Path)1 HoodieAvroUtils (org.apache.hudi.avro.HoodieAvroUtils)1 FSUtils (org.apache.hudi.common.fs.FSUtils)1 HoodieKey (org.apache.hudi.common.model.HoodieKey)1 HoodieRecord (org.apache.hudi.common.model.HoodieRecord)1 HoodieException (org.apache.hudi.exception.HoodieException)1 HoodieIOException (org.apache.hudi.exception.HoodieIOException)1