Search in sources :

Example 1 with ColumnWriterOptions

use of com.facebook.presto.orc.ColumnWriterOptions in project presto by prestodb.

the class TestBooleanOutputStream method testWriteBoolean.

@Test
public void testWriteBoolean() {
    List<List<Integer>> testGroups = ImmutableList.of(ImmutableList.of(149, 317, 2), ImmutableList.of(2), ImmutableList.of(1, 2, 4, 0, 8), ImmutableList.of(1, 4, 8, 1024, 10000), ImmutableList.of(14000, 1, 2));
    DataSize compressionSize = new DataSize(1, KILOBYTE);
    for (List<Integer> counts : testGroups) {
        ColumnWriterOptions columnWriterOptions = ColumnWriterOptions.builder().setCompressionKind(NONE).setCompressionMaxBufferSize(compressionSize).build();
        OrcOutputBuffer buffer = new OrcOutputBuffer(columnWriterOptions, Optional.empty());
        BooleanOutputStream output = new BooleanOutputStream(buffer);
        // write multiple booleans together
        for (int count : counts) {
            output.writeBooleans(count, true);
            output.recordCheckpoint();
        }
        output.close();
        List<BooleanStreamCheckpoint> batchWriteCheckpoints = output.getCheckpoints();
        DynamicSliceOutput slice = new DynamicSliceOutput(128);
        buffer.writeDataTo(slice);
        Slice batchWriteBuffer = slice.slice();
        // write one boolean a time
        buffer.reset();
        output.reset();
        for (int count : counts) {
            for (int i = 0; i < count; i++) {
                output.writeBoolean(true);
            }
            output.recordCheckpoint();
        }
        output.close();
        List<BooleanStreamCheckpoint> singleWriteCheckpoints = output.getCheckpoints();
        slice = new DynamicSliceOutput(128);
        buffer.writeDataTo(slice);
        Slice singleWriteBuffer = slice.slice();
        assertEquals(batchWriteCheckpoints.size(), singleWriteCheckpoints.size());
        for (int i = 0; i < batchWriteCheckpoints.size(); i++) {
            assertTrue(checkpointsEqual(batchWriteCheckpoints.get(i), singleWriteCheckpoints.get(i)));
        }
        assertEquals(batchWriteBuffer, singleWriteBuffer);
    }
}
Also used : BooleanStreamCheckpoint(com.facebook.presto.orc.checkpoint.BooleanStreamCheckpoint) BooleanStreamCheckpoint(com.facebook.presto.orc.checkpoint.BooleanStreamCheckpoint) ByteStreamCheckpoint(com.facebook.presto.orc.checkpoint.ByteStreamCheckpoint) OrcOutputBuffer(com.facebook.presto.orc.OrcOutputBuffer) ColumnWriterOptions(com.facebook.presto.orc.ColumnWriterOptions) Slice(io.airlift.slice.Slice) DataSize(io.airlift.units.DataSize) DynamicSliceOutput(io.airlift.slice.DynamicSliceOutput) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Test(org.testng.annotations.Test)

Example 2 with ColumnWriterOptions

use of com.facebook.presto.orc.ColumnWriterOptions in project presto by prestodb.

the class TestSliceDictionaryColumnWriter method getDictionaryColumnWriter.

private DictionaryColumnWriter getDictionaryColumnWriter(OrcEncoding orcEncoding, boolean sortDictionaryKeys) {
    ColumnWriterOptions columnWriterOptions = ColumnWriterOptions.builder().setCompressionKind(SNAPPY).setStringDictionarySortingEnabled(sortDictionaryKeys).build();
    DictionaryColumnWriter writer = new SliceDictionaryColumnWriter(COLUMN_ID, VARCHAR, columnWriterOptions, Optional.empty(), orcEncoding, orcEncoding.createMetadataWriter());
    return writer;
}
Also used : ColumnWriterOptions(com.facebook.presto.orc.ColumnWriterOptions)

Aggregations

ColumnWriterOptions (com.facebook.presto.orc.ColumnWriterOptions)2 OrcOutputBuffer (com.facebook.presto.orc.OrcOutputBuffer)1 BooleanStreamCheckpoint (com.facebook.presto.orc.checkpoint.BooleanStreamCheckpoint)1 ByteStreamCheckpoint (com.facebook.presto.orc.checkpoint.ByteStreamCheckpoint)1 ImmutableList (com.google.common.collect.ImmutableList)1 DynamicSliceOutput (io.airlift.slice.DynamicSliceOutput)1 Slice (io.airlift.slice.Slice)1 DataSize (io.airlift.units.DataSize)1 List (java.util.List)1 Test (org.testng.annotations.Test)1