Search in sources :

Example 1 with AssertionException

use of javolution.testing.AssertionException in project hive by apache.

the class SparkDynamicPartitionPruner method applyFilterToPartitions.

private void applyFilterToPartitions(MapWork work, ObjectInspectorConverters.Converter converter, ExprNodeEvaluator eval, String columnName, Set<Object> values) throws HiveException {
    Object[] row = new Object[1];
    Iterator<Path> it = work.getPathToPartitionInfo().keySet().iterator();
    while (it.hasNext()) {
        Path p = it.next();
        PartitionDesc desc = work.getPathToPartitionInfo().get(p);
        Map<String, String> spec = desc.getPartSpec();
        if (spec == null) {
            throw new AssertionException("No partition spec found in dynamic pruning");
        }
        String partValueString = spec.get(columnName);
        if (partValueString == null) {
            throw new AssertionException("Could not find partition value for column: " + columnName);
        }
        Object partValue = converter.convert(partValueString);
        if (LOG.isDebugEnabled()) {
            LOG.debug("Converted partition value: " + partValue + " original (" + partValueString + ")");
        }
        row[0] = partValue;
        partValue = eval.evaluate(row);
        if (LOG.isDebugEnabled()) {
            LOG.debug("part key expr applied: " + partValue);
        }
        if (!values.contains(partValue)) {
            LOG.info("Pruning path: " + p);
            it.remove();
            work.removePathToAlias(p);
            // HIVE-12244 call currently ineffective
            work.getPartitionDescs().remove(desc);
        }
    }
}
Also used : Path(org.apache.hadoop.fs.Path) AssertionException(javolution.testing.AssertionException) PartitionDesc(org.apache.hadoop.hive.ql.plan.PartitionDesc)

Aggregations

AssertionException (javolution.testing.AssertionException)1 Path (org.apache.hadoop.fs.Path)1 PartitionDesc (org.apache.hadoop.hive.ql.plan.PartitionDesc)1