Search in sources :

Example 1 with Converter

use of org.apache.parquet.io.api.Converter in project parquet-mr by apache.

the class ColumnReadStoreImpl method getPrimitiveConverter.

private PrimitiveConverter getPrimitiveConverter(ColumnDescriptor path) {
    Type currentType = schema;
    Converter currentConverter = recordConverter;
    for (String fieldName : path.getPath()) {
        final GroupType groupType = currentType.asGroupType();
        int fieldIndex = groupType.getFieldIndex(fieldName);
        currentType = groupType.getType(fieldName);
        currentConverter = currentConverter.asGroupConverter().getConverter(fieldIndex);
    }
    PrimitiveConverter converter = currentConverter.asPrimitiveConverter();
    return converter;
}
Also used : GroupType(org.apache.parquet.schema.GroupType) MessageType(org.apache.parquet.schema.MessageType) Type(org.apache.parquet.schema.Type) GroupType(org.apache.parquet.schema.GroupType) PrimitiveConverter(org.apache.parquet.io.api.PrimitiveConverter) Converter(org.apache.parquet.io.api.Converter) GroupConverter(org.apache.parquet.io.api.GroupConverter) PrimitiveConverter(org.apache.parquet.io.api.PrimitiveConverter)

Example 2 with Converter

use of org.apache.parquet.io.api.Converter in project parquet-mr by apache.

the class FilteringGroupConverter method getConverter.

// When a converter is asked for, we get the real one from the delegate, then wrap it
// in a filtering pass-through proxy.
// TODO: making the assumption that getConverter(i) is only called once, is that valid?
@Override
public Converter getConverter(int fieldIndex) {
    // get the real converter from the delegate
    Converter delegateConverter = checkNotNull(delegate.getConverter(fieldIndex), "delegate converter");
    // determine the indexFieldPath for the converter proxy we're about to make, which is
    // this converter's path + the requested fieldIndex
    List<Integer> newIndexFieldPath = new ArrayList<Integer>(indexFieldPath.size() + 1);
    newIndexFieldPath.addAll(indexFieldPath);
    newIndexFieldPath.add(fieldIndex);
    if (delegateConverter.isPrimitive()) {
        PrimitiveColumnIO columnIO = getColumnIO(newIndexFieldPath);
        ColumnPath columnPath = ColumnPath.get(columnIO.getColumnDescriptor().getPath());
        ValueInspector[] valueInspectors = getValueInspectors(columnPath);
        return new FilteringPrimitiveConverter(delegateConverter.asPrimitiveConverter(), valueInspectors);
    } else {
        return new FilteringGroupConverter(delegateConverter.asGroupConverter(), newIndexFieldPath, valueInspectorsByColumn, columnIOsByIndexFieldPath);
    }
}
Also used : ValueInspector(org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate.ValueInspector) ArrayList(java.util.ArrayList) Converter(org.apache.parquet.io.api.Converter) GroupConverter(org.apache.parquet.io.api.GroupConverter) ColumnPath(org.apache.parquet.hadoop.metadata.ColumnPath) PrimitiveColumnIO(org.apache.parquet.io.PrimitiveColumnIO)

Aggregations

Converter (org.apache.parquet.io.api.Converter)2 GroupConverter (org.apache.parquet.io.api.GroupConverter)2 ArrayList (java.util.ArrayList)1 ValueInspector (org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate.ValueInspector)1 ColumnPath (org.apache.parquet.hadoop.metadata.ColumnPath)1 PrimitiveColumnIO (org.apache.parquet.io.PrimitiveColumnIO)1 PrimitiveConverter (org.apache.parquet.io.api.PrimitiveConverter)1 GroupType (org.apache.parquet.schema.GroupType)1 MessageType (org.apache.parquet.schema.MessageType)1 Type (org.apache.parquet.schema.Type)1