Examples with OrcTail org.apache.orc.impl.OrcTail used on opensource projects

Example 11 with OrcTail

use of org.apache.orc.impl.OrcTail in project hive by apache.

the class TestOrcMetadataCache method testGetOrcTailForPathWithFileId.

@Test
public void testGetOrcTailForPathWithFileId() throws Exception {
    DummyMemoryManager mm = new DummyMemoryManager();
    DummyCachePolicy cp = new DummyCachePolicy();
    final int MAX_ALLOC = 64;
    LlapDaemonCacheMetrics metrics = LlapDaemonCacheMetrics.create("", "");
    BuddyAllocator alloc = new BuddyAllocator(false, false, 8, MAX_ALLOC, 1, 4 * 4096, 0, null, mm, metrics, null, true);
    MetadataCache cache = new MetadataCache(alloc, mm, cp, true, metrics);
    Path path = new Path("../data/files/alltypesorc");
    Configuration jobConf = new Configuration();
    Configuration daemonConf = new Configuration();
    CacheTag tag = CacheTag.build("test-table");
    FileSystem fs = FileSystem.get(daemonConf);
    FileStatus fileStatus = fs.getFileStatus(path);
    OrcTail uncached = OrcEncodedDataReader.getOrcTailForPath(fileStatus.getPath(), jobConf, tag, daemonConf, cache, new SyntheticFileId(fileStatus));
    jobConf.set(HiveConf.ConfVars.LLAP_IO_CACHE_ONLY.varname, "true");
    // this should work from the cache, by recalculating the same fileId
    OrcTail cached = OrcEncodedDataReader.getOrcTailForPath(fileStatus.getPath(), jobConf, tag, daemonConf, cache, null);
    assertEquals(uncached.getSerializedTail(), cached.getSerializedTail());
    assertEquals(uncached.getFileTail(), cached.getFileTail());
}

Example 12 with OrcTail

use of org.apache.orc.impl.OrcTail in project hive by apache.

the class OrcNewSplit method readFields.

@Override
public void readFields(DataInput in) throws IOException {
    // deserialize path, offset, length using FileSplit
    super.readFields(in);
    byte flags = in.readByte();
    hasFooter = (OrcSplit.FOOTER_FLAG & flags) != 0;
    isOriginal = (OrcSplit.ORIGINAL_FLAG & flags) != 0;
    hasBase = (OrcSplit.BASE_FLAG & flags) != 0;
    deltas.clear();
    int numDeltas = in.readInt();
    for (int i = 0; i < numDeltas; i++) {
        AcidInputFormat.DeltaMetaData dmd = new AcidInputFormat.DeltaMetaData();
        dmd.readFields(in);
        deltas.add(dmd);
    }
    if (hasFooter) {
        int tailLen = WritableUtils.readVInt(in);
        byte[] tailBuffer = new byte[tailLen];
        in.readFully(tailBuffer);
        OrcProto.FileTail fileTail = OrcProto.FileTail.parseFrom(tailBuffer);
        orcTail = new OrcTail(fileTail, null);
    }
}

Example 13 with OrcTail

use of org.apache.orc.impl.OrcTail in project hive by apache.

the class ExternalCache method createOrcTailFromMs.

private static OrcTail createOrcTailFromMs(HdfsFileStatusWithId file, ByteBuffer bb) throws IOException {
    if (bb == null) {
        return null;
    }
    FileStatus fs = file.getFileStatus();
    ByteBuffer copy = bb.duplicate();
    try {
        OrcTail orcTail = ReaderImpl.extractFileTail(copy, copy.limit(), fs.getModificationTime());
        return orcTail;
    } catch (Exception ex) {
        byte[] data = new byte[bb.remaining()];
        System.arraycopy(bb.array(), bb.arrayOffset() + bb.position(), data, 0, data.length);
        String msg = "Failed to parse the footer stored in cache for file ID " + file.getFileId() + " " + bb + " [ " + Hex.encodeHexString(data) + " ]";
        LOG.error(msg, ex);
        return null;
    }
}