Search in sources :

Example 11 with Schema

use of org.apache.arrow.vector.types.pojo.Schema in project parquet-mr by apache.

the class TestSchemaConverter method testArrowTimestampMillisecondToParquet.

@Test
public void testArrowTimestampMillisecondToParquet() {
    MessageType expected = converter.fromArrow(new Schema(asList(field("a", new ArrowType.Timestamp(TimeUnit.MILLISECOND, "UTC"))))).getParquetSchema();
    Assert.assertEquals(expected, Types.buildMessage().addField(Types.optional(INT64).as(TIMESTAMP_MILLIS).named("a")).named("root"));
}
Also used : Schema(org.apache.arrow.vector.types.pojo.Schema) ArrowType(org.apache.arrow.vector.types.pojo.ArrowType) MessageType(org.apache.parquet.schema.MessageType) Test(org.junit.Test)

Example 12 with Schema

use of org.apache.arrow.vector.types.pojo.Schema in project parquet-mr by apache.

the class TestSchemaConverter method testParquetInt64TimeMicrosToArrow.

@Test
public void testParquetInt64TimeMicrosToArrow() {
    MessageType parquet = Types.buildMessage().addField(Types.optional(INT64).as(TIME_MICROS).named("a")).named("root");
    Schema expected = new Schema(asList(field("a", new ArrowType.Time(TimeUnit.MICROSECOND, 64))));
    Assert.assertEquals(expected, converter.fromParquet(parquet).getArrowSchema());
}
Also used : Schema(org.apache.arrow.vector.types.pojo.Schema) ArrowType(org.apache.arrow.vector.types.pojo.ArrowType) MessageType(org.apache.parquet.schema.MessageType) Test(org.junit.Test)

Example 13 with Schema

use of org.apache.arrow.vector.types.pojo.Schema in project parquet-mr by apache.

the class TestSchemaConverter method testArrowTimeMillisecondToParquet.

@Test
public void testArrowTimeMillisecondToParquet() {
    MessageType expected = converter.fromArrow(new Schema(asList(field("a", new ArrowType.Time(TimeUnit.MILLISECOND, 32))))).getParquetSchema();
    Assert.assertEquals(expected, Types.buildMessage().addField(Types.optional(INT32).as(timeType(false, MILLIS)).named("a")).named("root"));
}
Also used : Schema(org.apache.arrow.vector.types.pojo.Schema) ArrowType(org.apache.arrow.vector.types.pojo.ArrowType) MessageType(org.apache.parquet.schema.MessageType) Test(org.junit.Test)

Example 14 with Schema

use of org.apache.arrow.vector.types.pojo.Schema in project parquet-mr by apache.

the class TestSchemaConverter method testParquetInt64TimestampMicrosToArrow.

@Test
public void testParquetInt64TimestampMicrosToArrow() {
    MessageType parquet = Types.buildMessage().addField(Types.optional(INT64).as(TIMESTAMP_MICROS).named("a")).named("root");
    Schema expected = new Schema(asList(field("a", new ArrowType.Timestamp(TimeUnit.MICROSECOND, "UTC"))));
    Assert.assertEquals(expected, converter.fromParquet(parquet).getArrowSchema());
}
Also used : Schema(org.apache.arrow.vector.types.pojo.Schema) ArrowType(org.apache.arrow.vector.types.pojo.ArrowType) MessageType(org.apache.parquet.schema.MessageType) Test(org.junit.Test)

Example 15 with Schema

use of org.apache.arrow.vector.types.pojo.Schema in project parquet-mr by apache.

the class TestSchemaConverter method testParquetFixedBinaryToArrow.

@Test
public void testParquetFixedBinaryToArrow() {
    MessageType parquet = Types.buildMessage().addField(Types.optional(FIXED_LEN_BYTE_ARRAY).length(12).named("a")).named("root");
    Schema expected = new Schema(asList(field("a", new ArrowType.Binary())));
    Assert.assertEquals(expected, converter.fromParquet(parquet).getArrowSchema());
}
Also used : Schema(org.apache.arrow.vector.types.pojo.Schema) ArrowType(org.apache.arrow.vector.types.pojo.ArrowType) MessageType(org.apache.parquet.schema.MessageType) Test(org.junit.Test)

Aggregations

Schema (org.apache.arrow.vector.types.pojo.Schema)20 ArrowType (org.apache.arrow.vector.types.pojo.ArrowType)15 Test (org.junit.Test)15 MessageType (org.apache.parquet.schema.MessageType)13 Field (org.apache.arrow.vector.types.pojo.Field)5 ImmutableList (com.google.common.collect.ImmutableList)2 RootAllocator (org.apache.arrow.memory.RootAllocator)2 ArrowSchema (com.google.cloud.bigquery.storage.v1.ArrowSchema)1 JobConfig (edu.iu.dsc.tws.api.JobConfig)1 Twister2Job (edu.iu.dsc.tws.api.Twister2Job)1 Config (edu.iu.dsc.tws.api.config.Config)1 Twister2Worker (edu.iu.dsc.tws.api.resource.Twister2Worker)1 WorkerEnvironment (edu.iu.dsc.tws.api.resource.WorkerEnvironment)1 ComputeFunc (edu.iu.dsc.tws.api.tset.fn.ComputeFunc)1 MapFunc (edu.iu.dsc.tws.api.tset.fn.MapFunc)1 Table (edu.iu.dsc.tws.common.table.Table)1 ArrowTable (edu.iu.dsc.tws.common.table.arrow.ArrowTable)1 ArrowAllToAll (edu.iu.dsc.tws.comms.table.ArrowAllToAll)1 LogicalPlanBuilder (edu.iu.dsc.tws.comms.utils.LogicalPlanBuilder)1 DataObjectConstants (edu.iu.dsc.tws.data.utils.DataObjectConstants)1