Search in sources :

Example 1 with TupleWriteSupport

use of org.apache.parquet.pig.TupleWriteSupport in project parquet-mr by apache.

the class TestParquetWriteProtocol method validatePig.

private MessageType validatePig(String[] expectations, TBase<?, ?> a) {
    ThriftToPig<TBase<?, ?>> thriftToPig = new ThriftToPig(a.getClass());
    ExpectationValidatingRecordConsumer recordConsumer = new ExpectationValidatingRecordConsumer(new ArrayDeque<String>(Arrays.asList(expectations)));
    Schema pigSchema = thriftToPig.toSchema();
    LOG.info("{}", pigSchema);
    MessageType schema = new PigSchemaConverter().convert(pigSchema);
    LOG.info("{}", schema);
    TupleWriteSupport tupleWriteSupport = new TupleWriteSupport(pigSchema);
    tupleWriteSupport.init(null);
    tupleWriteSupport.prepareForWrite(recordConsumer);
    final Tuple pigTuple = thriftToPig.getPigTuple(a);
    LOG.info("{}", pigTuple);
    tupleWriteSupport.write(pigTuple);
    return schema;
}
Also used : Schema(org.apache.pig.impl.logicalLayer.schema.Schema) PigSchemaConverter(org.apache.parquet.pig.PigSchemaConverter) TBase(org.apache.thrift.TBase) TupleWriteSupport(org.apache.parquet.pig.TupleWriteSupport) ExpectationValidatingRecordConsumer(org.apache.parquet.io.ExpectationValidatingRecordConsumer) ThriftToPig(com.twitter.elephantbird.pig.util.ThriftToPig) MessageType(org.apache.parquet.schema.MessageType) Tuple(org.apache.pig.data.Tuple)

Aggregations

ThriftToPig (com.twitter.elephantbird.pig.util.ThriftToPig)1 ExpectationValidatingRecordConsumer (org.apache.parquet.io.ExpectationValidatingRecordConsumer)1 PigSchemaConverter (org.apache.parquet.pig.PigSchemaConverter)1 TupleWriteSupport (org.apache.parquet.pig.TupleWriteSupport)1 MessageType (org.apache.parquet.schema.MessageType)1 Tuple (org.apache.pig.data.Tuple)1 Schema (org.apache.pig.impl.logicalLayer.schema.Schema)1 TBase (org.apache.thrift.TBase)1