Examples with OrcMap org.apache.orc.mapred.OrcMap used on opensource projects

Example 1 with OrcMap

use of org.apache.orc.mapred.OrcMap in project druid by druid-io.

the class OrcStructConverterTest method testConvertRootFieldWithMapOfNullValuesReturningMapOfNulls.

@Test
public void testConvertRootFieldWithMapOfNullValuesReturningMapOfNulls() {
    final TypeDescription mapType = TypeDescription.createMap(TypeDescription.createInt(), TypeDescription.createFloat());
    final OrcMap<IntWritable, FloatWritable> map = new OrcMap<>(mapType);
    IntStream.range(0, 3).forEach(i -> map.put(new IntWritable(i * 10), null));
    final Map<Integer, Float> expectedResult = new HashMap<>();
    for (Entry<IntWritable, FloatWritable> entry : map.entrySet()) {
        expectedResult.put(entry.getKey().get(), null);
    }
    final OrcStructConverter converter = new OrcStructConverter(false);
    assertConversion(converter, mapType, expectedResult, map);
}

Example 2 with OrcMap

use of org.apache.orc.mapred.OrcMap in project druid by druid-io.

the class OrcStructConverterTest method testConvertRootFieldWithMapOfNonNullPrimitivesReturningValuesAsTheyAre.

@Test
public void testConvertRootFieldWithMapOfNonNullPrimitivesReturningValuesAsTheyAre() {
    final TypeDescription mapType = TypeDescription.createMap(TypeDescription.createInt(), TypeDescription.createFloat());
    final OrcMap<IntWritable, FloatWritable> map = new OrcMap<>(mapType);
    for (int i = 0; i < 3; i++) {
        map.put(new IntWritable(i * 10), new FloatWritable(i / 10.f));
    }
    final Map<Integer, Float> expectedResult = new HashMap<>();
    for (Entry<IntWritable, FloatWritable> entry : map.entrySet()) {
        expectedResult.put(entry.getKey().get(), entry.getValue().get());
    }
    final OrcStructConverter converter = new OrcStructConverter(false);
    assertConversion(converter, mapType, expectedResult, map);
}

Example 3 with OrcMap

use of org.apache.orc.mapred.OrcMap in project druid by druid-io.

the class OrcStructConverter method convertField.

/**
 * Convert a orc struct field as though it were a map, by fieldIndex. Complex types will be transformed
 * into java lists and maps when possible ({@link OrcStructConverter#convertList} and
 * {@link OrcStructConverter#convertMap}), and
 * primitive types will be extracted into an ingestion friendly state (e.g. 'int' and 'long'). Finally,
 * if a field is not present, this method will return null.
 *
 * Note: "Union" types are not currently supported and will be returned as null
 */
@Nullable
Object convertField(OrcStruct struct, int fieldIndex) {
    if (fieldIndex < 0) {
        return null;
    }
    TypeDescription schema = struct.getSchema();
    TypeDescription fieldDescription = schema.getChildren().get(fieldIndex);
    WritableComparable fieldValue = struct.getFieldValue(fieldIndex);
    if (fieldValue == null) {
        return null;
    }
    if (fieldDescription.getCategory().isPrimitive()) {
        return convertPrimitive(fieldDescription, fieldValue, binaryAsString);
    } else {
        /*
          ORC TYPE    WRITABLE TYPE
          array       org.apache.orc.mapred.OrcList
          map         org.apache.orc.mapred.OrcMap
          struct      org.apache.orc.mapred.OrcStruct
          uniontype   org.apache.orc.mapred.OrcUnion
       */
        switch(fieldDescription.getCategory()) {
            case LIST:
                OrcList orcList = (OrcList) fieldValue;
                return convertList(fieldDescription, orcList, binaryAsString);
            case MAP:
                OrcMap map = (OrcMap) fieldValue;
                return convertMap(fieldDescription, map, binaryAsString);
            case STRUCT:
                OrcStruct structMap = (OrcStruct) fieldValue;
                return convertStructToMap(structMap);
            case UNION:
            // sorry union types :(
            default:
                return null;
        }
    }
}