Search in sources :

Example 6 with DeltaByteArrayWriter

use of org.apache.parquet.column.values.deltastrings.DeltaByteArrayWriter in project parquet-mr by apache.

the class BenchmarkDeltaByteArray method benchmarkSortedStringsWithDeltaLengthByteArrayValuesWriter.

@BenchmarkOptions(benchmarkRounds = 20, warmupRounds = 4)
@Test
public void benchmarkSortedStringsWithDeltaLengthByteArrayValuesWriter() throws IOException {
    DeltaByteArrayWriter writer = new DeltaByteArrayWriter(64 * 1024, 64 * 1024, new DirectByteBufferAllocator());
    DeltaByteArrayReader reader = new DeltaByteArrayReader();
    Utils.writeData(writer, sortedVals);
    ByteBufferInputStream data = writer.getBytes().toInputStream();
    Binary[] bin = Utils.readData(reader, data, values.length);
    System.out.println("size " + data.position());
}
Also used : DirectByteBufferAllocator(org.apache.parquet.bytes.DirectByteBufferAllocator) DeltaByteArrayWriter(org.apache.parquet.column.values.deltastrings.DeltaByteArrayWriter) ByteBufferInputStream(org.apache.parquet.bytes.ByteBufferInputStream) DeltaByteArrayReader(org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader) Binary(org.apache.parquet.io.api.Binary) Test(org.junit.Test) BenchmarkOptions(com.carrotsearch.junitbenchmarks.BenchmarkOptions)

Aggregations

DeltaByteArrayWriter (org.apache.parquet.column.values.deltastrings.DeltaByteArrayWriter)6 Test (org.junit.Test)6 DeltaByteArrayReader (org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader)5 ByteBuffer (java.nio.ByteBuffer)3 Binary (org.apache.parquet.io.api.Binary)3 BenchmarkOptions (com.carrotsearch.junitbenchmarks.BenchmarkOptions)2 ByteBufferInputStream (org.apache.parquet.bytes.ByteBufferInputStream)2 DirectByteBufferAllocator (org.apache.parquet.bytes.DirectByteBufferAllocator)2 ArrayList (java.util.ArrayList)1 ParsedVersion (org.apache.parquet.VersionParser.ParsedVersion)1 BytesInput (org.apache.parquet.bytes.BytesInput)1 ColumnDescriptor (org.apache.parquet.column.ColumnDescriptor)1 ParquetProperties (org.apache.parquet.column.ParquetProperties)1 PageWriter (org.apache.parquet.column.page.PageWriter)1 MemPageStore (org.apache.parquet.column.page.mem.MemPageStore)1 BinaryStatistics (org.apache.parquet.column.statistics.BinaryStatistics)1 ValuesWriter (org.apache.parquet.column.values.ValuesWriter)1 PrimitiveConverter (org.apache.parquet.io.api.PrimitiveConverter)1