Examples with ThriftBlockWriter com.twitter.elephantbird.mapreduce.io.ThriftBlockWriter used on opensource projects

Example 1 with ThriftBlockWriter

use of com.twitter.elephantbird.mapreduce.io.ThriftBlockWriter in project elephant-bird by twitter.

the class TestThriftMultiFormatLoader method setUp.

@Before
public void setUp() throws Exception {
    Configuration conf = new Configuration();
    Assume.assumeTrue(CoreTestUtil.okToRunLzoTests(conf));
    pigServer = PigTestUtil.makePigServer();
    inputDir.mkdirs();
    // write to block file
    ThriftBlockWriter<TestPerson> blk_writer = new ThriftBlockWriter<TestPerson>(createLzoOut("1-block.lzo", conf), TestPerson.class);
    for (TestPerson rec : records) {
        blk_writer.write(rec);
    }
    blk_writer.close();
    // write tb64 lines
    LzoBinaryB64LineRecordWriter<TestPerson, ThriftWritable<TestPerson>> b64_writer = LzoBinaryB64LineRecordWriter.newThriftWriter(TestPerson.class, createLzoOut("2-b64.lzo", conf));
    for (TestPerson rec : records) {
        thriftWritable.set(rec);
        b64_writer.write(null, thriftWritable);
    }
    b64_writer.close(null);
}