Search in sources :

Example 11 with ArrayType

use of com.facebook.presto.common.type.ArrayType in project presto by prestodb.

the class AbstractTestParquetReader method testSingleLevelArrayOfStructOfStructOfSingleElement.

@Test
public void testSingleLevelArrayOfStructOfStructOfSingleElement() throws Exception {
    Iterable<List> structs = createTestStructs(transform(intsBetween(0, 31_234), Object::toString));
    Iterable<List> structsOfStructs = createTestStructs(structs);
    Iterable<List<List>> values = createTestArrays(structsOfStructs);
    List<String> structFieldNames = singletonList("test");
    List<String> structsOfStructsFieldNames = singletonList("test");
    Type structType = RowType.from(singletonList(field("test", VARCHAR)));
    Type structsOfStructsType = RowType.from(singletonList(field("test", structType)));
    ObjectInspector structObjectInspector = getStandardStructObjectInspector(structFieldNames, singletonList(javaStringObjectInspector));
    tester.testRoundTrip(getStandardListObjectInspector(getStandardStructObjectInspector(structsOfStructsFieldNames, singletonList(structObjectInspector))), values, values, new ArrayType(structsOfStructsType));
    tester.testSingleLevelArraySchemaRoundTrip(getStandardListObjectInspector(getStandardStructObjectInspector(structsOfStructsFieldNames, singletonList(structObjectInspector))), values, values, new ArrayType(structsOfStructsType));
}
Also used : ArrayType(com.facebook.presto.common.type.ArrayType) VarcharType.createUnboundedVarcharType(com.facebook.presto.common.type.VarcharType.createUnboundedVarcharType) StructuralTestUtil.mapType(com.facebook.presto.tests.StructuralTestUtil.mapType) ArrayType(com.facebook.presto.common.type.ArrayType) RowType(com.facebook.presto.common.type.RowType) PrimitiveType(org.apache.parquet.schema.PrimitiveType) MessageType(org.apache.parquet.schema.MessageType) Type(com.facebook.presto.common.type.Type) MessageTypeParser.parseMessageType(org.apache.parquet.schema.MessageTypeParser.parseMessageType) DecimalType.createDecimalType(com.facebook.presto.common.type.DecimalType.createDecimalType) JavaHiveDecimalObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector) PrimitiveObjectInspectorFactory.javaByteObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaByteObjectInspector) PrimitiveObjectInspectorFactory.javaLongObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaLongObjectInspector) PrimitiveObjectInspectorFactory.javaTimestampObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaTimestampObjectInspector) PrimitiveObjectInspectorFactory.javaDateObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaDateObjectInspector) ObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) PrimitiveObjectInspectorFactory.javaByteArrayObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaByteArrayObjectInspector) PrimitiveObjectInspectorFactory.javaFloatObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaFloatObjectInspector) PrimitiveObjectInspectorFactory.javaDoubleObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaDoubleObjectInspector) PrimitiveObjectInspectorFactory.javaIntObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaIntObjectInspector) PrimitiveObjectInspectorFactory.javaShortObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaShortObjectInspector) ObjectInspectorFactory.getStandardStructObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector) PrimitiveObjectInspectorFactory.javaBooleanObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaBooleanObjectInspector) ObjectInspectorFactory.getStandardMapObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardMapObjectInspector) ObjectInspectorFactory.getStandardListObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardListObjectInspector) PrimitiveObjectInspectorFactory.javaStringObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory.javaStringObjectInspector) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Test(org.testng.annotations.Test)

Example 12 with ArrayType

use of com.facebook.presto.common.type.ArrayType in project presto by prestodb.

the class AbstractTestParquetReader method testArrayOfMaps.

@Test
public void testArrayOfMaps() throws Exception {
    Iterable<Map<String, Long>> maps = createNullableTestMaps(transform(intsBetween(0, 10), Object::toString), longsBetween(0, 10));
    List<List<Map<String, Long>>> values = createTestArrays(maps);
    tester.testRoundTrip(getStandardListObjectInspector(getStandardMapObjectInspector(javaStringObjectInspector, javaLongObjectInspector)), values, values, new ArrayType(mapType(VARCHAR, BIGINT)));
}
Also used : ArrayType(com.facebook.presto.common.type.ArrayType) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Map(java.util.Map) HashMap(java.util.HashMap) Test(org.testng.annotations.Test)

Example 13 with ArrayType

use of com.facebook.presto.common.type.ArrayType in project presto by prestodb.

the class AbstractTestParquetReader method testOldAvroArray.

@Test
public void testOldAvroArray() throws Exception {
    MessageType parquetMrAvroSchema = parseMessageType("message avro_schema_old {" + "  optional group my_list (LIST){" + "        repeated int32 array;" + "  }" + "} ");
    Iterable<List<Integer>> nonNullArrayElements = createTestArrays(intsBetween(0, 31_234));
    tester.testSingleLevelArrayRoundTrip(getStandardListObjectInspector(javaIntObjectInspector), nonNullArrayElements, nonNullArrayElements, "my_list", new ArrayType(INTEGER), Optional.of(parquetMrAvroSchema));
}
Also used : ArrayType(com.facebook.presto.common.type.ArrayType) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) MessageType(org.apache.parquet.schema.MessageType) MessageTypeParser.parseMessageType(org.apache.parquet.schema.MessageTypeParser.parseMessageType) Test(org.testng.annotations.Test)

Example 14 with ArrayType

use of com.facebook.presto.common.type.ArrayType in project presto by prestodb.

the class AbstractTestParquetReader method testSingleLevelArrayOfStructOfSingleElement.

@Test
public void testSingleLevelArrayOfStructOfSingleElement() throws Exception {
    Iterable<List> structs = createTestStructs(transform(intsBetween(0, 31_234), Object::toString));
    Iterable<List<List>> values = createTestArrays(structs);
    List<String> structFieldNames = singletonList("test");
    Type structType = RowType.from(singletonList(field("test", VARCHAR)));
    tester.testRoundTrip(getStandardListObjectInspector(getStandardStructObjectInspector(structFieldNames, singletonList(javaStringObjectInspector))), values, values, new ArrayType(structType));
    tester.testSingleLevelArraySchemaRoundTrip(getStandardListObjectInspector(getStandardStructObjectInspector(structFieldNames, singletonList(javaStringObjectInspector))), values, values, new ArrayType(structType));
}
Also used : ArrayType(com.facebook.presto.common.type.ArrayType) VarcharType.createUnboundedVarcharType(com.facebook.presto.common.type.VarcharType.createUnboundedVarcharType) StructuralTestUtil.mapType(com.facebook.presto.tests.StructuralTestUtil.mapType) ArrayType(com.facebook.presto.common.type.ArrayType) RowType(com.facebook.presto.common.type.RowType) PrimitiveType(org.apache.parquet.schema.PrimitiveType) MessageType(org.apache.parquet.schema.MessageType) Type(com.facebook.presto.common.type.Type) MessageTypeParser.parseMessageType(org.apache.parquet.schema.MessageTypeParser.parseMessageType) DecimalType.createDecimalType(com.facebook.presto.common.type.DecimalType.createDecimalType) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Test(org.testng.annotations.Test)

Example 15 with ArrayType

use of com.facebook.presto.common.type.ArrayType in project presto by prestodb.

the class AbstractTestParquetReader method testStructOfArrayAndPrimitive.

@Test
public void testStructOfArrayAndPrimitive() throws Exception {
    Iterable<List<String>> stringArrayField = createNullableTestArrays(transform(intsBetween(0, 31_234), Object::toString));
    List<List> values = createTestStructs(stringArrayField, limit(cycle(ImmutableList.of(1, 3, 5, 7, 11, 13, 17)), 31_234));
    List<String> structFieldNames = asList("stringArrayField", "intField");
    Type structType = RowType.from(asList(field("stringArrayField", new ArrayType(VARCHAR)), field("intField", INTEGER)));
    tester.testRoundTrip(getStandardStructObjectInspector(structFieldNames, asList(getStandardListObjectInspector(javaStringObjectInspector), javaIntObjectInspector)), values, values, structType);
}
Also used : ArrayType(com.facebook.presto.common.type.ArrayType) VarcharType.createUnboundedVarcharType(com.facebook.presto.common.type.VarcharType.createUnboundedVarcharType) StructuralTestUtil.mapType(com.facebook.presto.tests.StructuralTestUtil.mapType) ArrayType(com.facebook.presto.common.type.ArrayType) RowType(com.facebook.presto.common.type.RowType) PrimitiveType(org.apache.parquet.schema.PrimitiveType) MessageType(org.apache.parquet.schema.MessageType) Type(com.facebook.presto.common.type.Type) MessageTypeParser.parseMessageType(org.apache.parquet.schema.MessageTypeParser.parseMessageType) DecimalType.createDecimalType(com.facebook.presto.common.type.DecimalType.createDecimalType) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Test(org.testng.annotations.Test)

Aggregations

ArrayType (com.facebook.presto.common.type.ArrayType)287 Test (org.testng.annotations.Test)219 Type (com.facebook.presto.common.type.Type)99 RowType (com.facebook.presto.common.type.RowType)79 ArrayList (java.util.ArrayList)58 ImmutableList (com.google.common.collect.ImmutableList)54 List (java.util.List)51 MapType (com.facebook.presto.common.type.MapType)39 DecimalType.createDecimalType (com.facebook.presto.common.type.DecimalType.createDecimalType)36 Arrays.asList (java.util.Arrays.asList)34 Collections.singletonList (java.util.Collections.singletonList)33 MessageType (org.apache.parquet.schema.MessageType)30 BlockBuilder (com.facebook.presto.common.block.BlockBuilder)28 VarcharType.createUnboundedVarcharType (com.facebook.presto.common.type.VarcharType.createUnboundedVarcharType)27 MessageTypeParser.parseMessageType (org.apache.parquet.schema.MessageTypeParser.parseMessageType)27 InternalAggregationFunction (com.facebook.presto.operator.aggregation.InternalAggregationFunction)26 PrimitiveType (org.apache.parquet.schema.PrimitiveType)26 StructuralTestUtil.mapType (com.facebook.presto.tests.StructuralTestUtil.mapType)24 Block (com.facebook.presto.common.block.Block)23 DecimalType (com.facebook.presto.common.type.DecimalType)17