Search in sources :

Example 11 with ValuesWriter

use of org.apache.parquet.column.values.ValuesWriter in project parquet-mr by apache.

the class BenchmarkReadingRandomIntegers method prepare.

@BeforeClass
public static void prepare() throws IOException {
    Random random = new Random();
    data = new int[100000 * blockSize];
    for (int i = 0; i < data.length; i++) {
        data[i] = random.nextInt(100) - 200;
    }
    ValuesWriter delta = new DeltaBinaryPackingValuesWriterForInteger(blockSize, miniBlockNum, 100, 20000, new DirectByteBufferAllocator());
    ValuesWriter rle = new RunLengthBitPackingHybridValuesWriter(32, 100, 20000, new DirectByteBufferAllocator());
    for (int i = 0; i < data.length; i++) {
        delta.writeInteger(data[i]);
        rle.writeInteger(data[i]);
    }
    deltaBytes = delta.getBytes().toByteArray();
    rleBytes = rle.getBytes().toByteArray();
}
Also used : DirectByteBufferAllocator(org.apache.parquet.bytes.DirectByteBufferAllocator) DeltaBinaryPackingValuesWriterForInteger(org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriterForInteger) Random(java.util.Random) RunLengthBitPackingHybridValuesWriter(org.apache.parquet.column.values.rle.RunLengthBitPackingHybridValuesWriter) ValuesWriter(org.apache.parquet.column.values.ValuesWriter) RunLengthBitPackingHybridValuesWriter(org.apache.parquet.column.values.rle.RunLengthBitPackingHybridValuesWriter) BeforeClass(org.junit.BeforeClass)

Aggregations

ValuesWriter (org.apache.parquet.column.values.ValuesWriter)11 Test (org.junit.Test)8 FallbackValuesWriter (org.apache.parquet.column.values.fallback.FallbackValuesWriter)6 PlainValuesWriter (org.apache.parquet.column.values.plain.PlainValuesWriter)6 BytesInput (org.apache.parquet.bytes.BytesInput)5 PlainBinaryDictionaryValuesWriter (org.apache.parquet.column.values.dictionary.DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter)5 PlainDoubleDictionaryValuesWriter (org.apache.parquet.column.values.dictionary.DictionaryValuesWriter.PlainDoubleDictionaryValuesWriter)5 PlainFloatDictionaryValuesWriter (org.apache.parquet.column.values.dictionary.DictionaryValuesWriter.PlainFloatDictionaryValuesWriter)5 PlainIntegerDictionaryValuesWriter (org.apache.parquet.column.values.dictionary.DictionaryValuesWriter.PlainIntegerDictionaryValuesWriter)5 PlainLongDictionaryValuesWriter (org.apache.parquet.column.values.dictionary.DictionaryValuesWriter.PlainLongDictionaryValuesWriter)5 BinaryPlainValuesReader (org.apache.parquet.column.values.plain.BinaryPlainValuesReader)5 ValuesReader (org.apache.parquet.column.values.ValuesReader)4 RunLengthBitPackingHybridValuesWriter (org.apache.parquet.column.values.rle.RunLengthBitPackingHybridValuesWriter)4 DirectByteBufferAllocator (org.apache.parquet.bytes.DirectByteBufferAllocator)3 PlainValuesReader (org.apache.parquet.column.values.plain.PlainValuesReader)3 BenchmarkOptions (com.carrotsearch.junitbenchmarks.BenchmarkOptions)2 ColumnDescriptor (org.apache.parquet.column.ColumnDescriptor)2 DeltaBinaryPackingValuesWriter (org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriter)2 Binary (org.apache.parquet.io.api.Binary)2 ArrayList (java.util.ArrayList)1