Examples with SchemaEvolution org.apache.orc.impl.SchemaEvolution used on opensource projects

Example 1 with SchemaEvolution

use of org.apache.orc.impl.SchemaEvolution in project hive by apache.

the class OrcInputFormat method pickStripesViaTranslatedSarg.

public static boolean[] pickStripesViaTranslatedSarg(SearchArgument sarg, OrcFile.WriterVersion writerVersion, List<OrcProto.Type> types, List<StripeStatistics> stripeStats, int stripeCount) throws FileFormatException {
    LOG.info("Translated ORC pushdown predicate: " + sarg);
    assert sarg != null;
    if (stripeStats == null || writerVersion == OrcFile.WriterVersion.ORIGINAL) {
        // only do split pruning if HIVE-8732 has been fixed in the writer
        return null;
    }
    // eliminate stripes that doesn't satisfy the predicate condition
    List<PredicateLeaf> sargLeaves = sarg.getLeaves();
    int[] filterColumns = RecordReaderImpl.mapTranslatedSargColumns(types, sargLeaves);
    TypeDescription schema = OrcUtils.convertTypeFromProtobuf(types, 0);
    SchemaEvolution evolution = new SchemaEvolution(schema, null);
    return pickStripesInternal(sarg, filterColumns, stripeStats, stripeCount, null, evolution);
}

Example 2 with SchemaEvolution

use of org.apache.orc.impl.SchemaEvolution in project hive by apache.

the class LlapRecordReader method checkOrcSchemaEvolution.

private boolean checkOrcSchemaEvolution() {
    SchemaEvolution evolution = rp.getSchemaEvolution();
    if (evolution.hasConversion() && !evolution.isOnlyImplicitConversion()) {
        // We do not support data type conversion when reading encoded ORC data.
        return false;
    }
    // TODO: should this just use physical IDs?
    for (int i = 0; i < includes.getReaderLogicalColumnIds().size(); ++i) {
        int projectedColId = includes.getReaderLogicalColumnIds().get(i);
        // Adjust file column index for ORC struct.
        int fileColId = OrcInputFormat.getRootColumn(!isAcidScan) + projectedColId + 1;
        if (!evolution.isPPDSafeConversion(fileColId)) {
            LlapIoImpl.LOG.warn("Unsupported schema evolution! Disabling Llap IO for {}", split);
            return false;
        }
    }
    return true;
}