Search in sources :

Example 1 with SliceDictionaryColumnWriter

use of io.trino.orc.writer.SliceDictionaryColumnWriter in project trino by trinodb.

the class TestSliceDictionaryColumnWriter method testDirectConversion.

@Test
public void testDirectConversion() {
    SliceDictionaryColumnWriter writer = new SliceDictionaryColumnWriter(ROOT_COLUMN, VARCHAR, CompressionKind.NONE, toIntExact(DEFAULT_MAX_COMPRESSION_BUFFER_SIZE.toBytes()), () -> new StringStatisticsBuilder(toIntExact(DEFAULT_MAX_STRING_STATISTICS_LIMIT.toBytes()), new NoOpBloomFilterBuilder()));
    // a single row group exceeds 2G after direct conversion
    byte[] value = new byte[megabytes(1)];
    ThreadLocalRandom.current().nextBytes(value);
    Block data = RunLengthEncodedBlock.create(VARCHAR, Slices.wrappedBuffer(value), 3000);
    writer.beginRowGroup();
    writer.writeBlock(data);
    writer.finishRowGroup();
    assertFalse(writer.tryConvertToDirect(megabytes(64)).isPresent());
}
Also used : StringStatisticsBuilder(io.trino.orc.metadata.statistics.StringStatisticsBuilder) SliceDictionaryColumnWriter(io.trino.orc.writer.SliceDictionaryColumnWriter) RunLengthEncodedBlock(io.trino.spi.block.RunLengthEncodedBlock) Block(io.trino.spi.block.Block) NoOpBloomFilterBuilder(io.trino.orc.metadata.statistics.NoOpBloomFilterBuilder) Test(org.testng.annotations.Test)

Aggregations

NoOpBloomFilterBuilder (io.trino.orc.metadata.statistics.NoOpBloomFilterBuilder)1 StringStatisticsBuilder (io.trino.orc.metadata.statistics.StringStatisticsBuilder)1 SliceDictionaryColumnWriter (io.trino.orc.writer.SliceDictionaryColumnWriter)1 Block (io.trino.spi.block.Block)1 RunLengthEncodedBlock (io.trino.spi.block.RunLengthEncodedBlock)1 Test (org.testng.annotations.Test)1