Search in sources :

Example 1 with PrimitiveColumnIO

use of org.apache.parquet.io.PrimitiveColumnIO in project parquet-mr by apache.

the class FilteringGroupConverter method getConverter.

// When a converter is asked for, we get the real one from the delegate, then wrap it
// in a filtering pass-through proxy.
// TODO: making the assumption that getConverter(i) is only called once, is that valid?
@Override
public Converter getConverter(int fieldIndex) {
    // get the real converter from the delegate
    Converter delegateConverter = checkNotNull(delegate.getConverter(fieldIndex), "delegate converter");
    // determine the indexFieldPath for the converter proxy we're about to make, which is
    // this converter's path + the requested fieldIndex
    List<Integer> newIndexFieldPath = new ArrayList<Integer>(indexFieldPath.size() + 1);
    newIndexFieldPath.addAll(indexFieldPath);
    newIndexFieldPath.add(fieldIndex);
    if (delegateConverter.isPrimitive()) {
        PrimitiveColumnIO columnIO = getColumnIO(newIndexFieldPath);
        ColumnPath columnPath = ColumnPath.get(columnIO.getColumnDescriptor().getPath());
        ValueInspector[] valueInspectors = getValueInspectors(columnPath);
        return new FilteringPrimitiveConverter(delegateConverter.asPrimitiveConverter(), valueInspectors);
    } else {
        return new FilteringGroupConverter(delegateConverter.asGroupConverter(), newIndexFieldPath, valueInspectorsByColumn, columnIOsByIndexFieldPath);
    }
}
Also used : ValueInspector(org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate.ValueInspector) ArrayList(java.util.ArrayList) Converter(org.apache.parquet.io.api.Converter) GroupConverter(org.apache.parquet.io.api.GroupConverter) ColumnPath(org.apache.parquet.hadoop.metadata.ColumnPath) PrimitiveColumnIO(org.apache.parquet.io.PrimitiveColumnIO)

Example 2 with PrimitiveColumnIO

use of org.apache.parquet.io.PrimitiveColumnIO in project parquet-mr by apache.

the class FilteringGroupConverter method getColumnIO.

private PrimitiveColumnIO getColumnIO(List<Integer> indexFieldPath) {
    PrimitiveColumnIO found = columnIOsByIndexFieldPath.get(indexFieldPath);
    checkArgument(found != null, "Did not find PrimitiveColumnIO for index field path" + indexFieldPath);
    return found;
}
Also used : PrimitiveColumnIO(org.apache.parquet.io.PrimitiveColumnIO)

Example 3 with PrimitiveColumnIO

use of org.apache.parquet.io.PrimitiveColumnIO in project parquet-mr by apache.

the class ParquetWriteProtocol method getProtocol.

private TProtocol getProtocol(ThriftField field, ColumnIO columnIO, Events returnClause) {
    TProtocol p;
    final ThriftType type = field.getType();
    switch(type.getType()) {
        case STOP:
        case VOID:
        default:
            throw new UnsupportedOperationException("can't convert type of " + field);
        case BOOL:
        case BYTE:
        case DOUBLE:
        case I16:
        case I32:
        case I64:
        case STRING:
            p = new PrimitiveWriteProtocol((PrimitiveColumnIO) columnIO, returnClause);
            break;
        case STRUCT:
            p = new StructWriteProtocol((GroupColumnIO) columnIO, (StructType) type, returnClause);
            break;
        case MAP:
            p = new MapWriteProtocol((GroupColumnIO) columnIO, (MapType) type, returnClause);
            break;
        case SET:
            p = new ListWriteProtocol((GroupColumnIO) columnIO, ((SetType) type).getValues(), returnClause);
            break;
        case LIST:
            p = new ListWriteProtocol((GroupColumnIO) columnIO, ((ListType) type).getValues(), returnClause);
            break;
        case ENUM:
            p = new EnumWriteProtocol((PrimitiveColumnIO) columnIO, (EnumType) type, returnClause);
            break;
    }
    return p;
}
Also used : ThriftType(org.apache.parquet.thrift.struct.ThriftType) StructType(org.apache.parquet.thrift.struct.ThriftType.StructType) PrimitiveColumnIO(org.apache.parquet.io.PrimitiveColumnIO) MapType(org.apache.parquet.thrift.struct.ThriftType.MapType) TProtocol(org.apache.thrift.protocol.TProtocol) GroupColumnIO(org.apache.parquet.io.GroupColumnIO) SetType(org.apache.parquet.thrift.struct.ThriftType.SetType) EnumType(org.apache.parquet.thrift.struct.ThriftType.EnumType) ListType(org.apache.parquet.thrift.struct.ThriftType.ListType)

Aggregations

PrimitiveColumnIO (org.apache.parquet.io.PrimitiveColumnIO)3 ArrayList (java.util.ArrayList)1 ValueInspector (org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate.ValueInspector)1 ColumnPath (org.apache.parquet.hadoop.metadata.ColumnPath)1 GroupColumnIO (org.apache.parquet.io.GroupColumnIO)1 Converter (org.apache.parquet.io.api.Converter)1 GroupConverter (org.apache.parquet.io.api.GroupConverter)1 ThriftType (org.apache.parquet.thrift.struct.ThriftType)1 EnumType (org.apache.parquet.thrift.struct.ThriftType.EnumType)1 ListType (org.apache.parquet.thrift.struct.ThriftType.ListType)1 MapType (org.apache.parquet.thrift.struct.ThriftType.MapType)1 SetType (org.apache.parquet.thrift.struct.ThriftType.SetType)1 StructType (org.apache.parquet.thrift.struct.ThriftType.StructType)1 TProtocol (org.apache.thrift.protocol.TProtocol)1