Search in sources :

Example 1 with ChunkDecompressor

use of com.linkedin.pinot.core.io.compression.ChunkDecompressor in project pinot by linkedin.

the class ColumnIndexContainer method getRawIndexReader.

public static SingleColumnSingleValueReader getRawIndexReader(PinotDataBuffer fwdIndexBuffer, FieldSpec.DataType dataType) throws IOException {
    SingleColumnSingleValueReader reader;
    // TODO: Make compression/decompression configurable.
    ChunkDecompressor decompressor = ChunkCompressorFactory.getDecompressor("snappy");
    switch(dataType) {
        case INT:
        case LONG:
        case FLOAT:
        case DOUBLE:
            reader = new FixedByteChunkSingleValueReader(fwdIndexBuffer, decompressor);
            break;
        case STRING:
            reader = new VarByteChunkSingleValueReader(fwdIndexBuffer, decompressor);
            break;
        default:
            throw new IllegalArgumentException("Illegal data type for raw index reader: " + dataType);
    }
    return reader;
}
Also used : SingleColumnSingleValueReader(com.linkedin.pinot.core.io.reader.SingleColumnSingleValueReader) ChunkDecompressor(com.linkedin.pinot.core.io.compression.ChunkDecompressor) FixedByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader) VarByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.VarByteChunkSingleValueReader)

Example 2 with ChunkDecompressor

use of com.linkedin.pinot.core.io.compression.ChunkDecompressor in project pinot by linkedin.

the class FixedByteChunkSingleValueReaderWriteTest method testInt.

@Test
public void testInt() throws Exception {
    int[] expected = new int[NUM_VALUES];
    for (int i = 0; i < NUM_VALUES; i++) {
        expected[i] = _random.nextInt();
    }
    File outFile = new File(TEST_FILE);
    FileUtils.deleteQuietly(outFile);
    ChunkCompressor compressor = ChunkCompressorFactory.getCompressor("snappy");
    FixedByteChunkSingleValueWriter writer = new FixedByteChunkSingleValueWriter(outFile, compressor, NUM_VALUES, NUM_DOCS_PER_CHUNK, V1Constants.Numbers.INTEGER_SIZE);
    for (int i = 0; i < NUM_VALUES; i++) {
        writer.setInt(i, expected[i]);
    }
    writer.close();
    PinotDataBuffer pinotDataBuffer = PinotDataBuffer.fromFile(outFile, ReadMode.mmap, FileChannel.MapMode.READ_ONLY, getClass().getName());
    ChunkDecompressor uncompressor = ChunkCompressorFactory.getDecompressor("snappy");
    FixedByteChunkSingleValueReader reader = new FixedByteChunkSingleValueReader(pinotDataBuffer, uncompressor);
    ChunkReaderContext context = reader.createContext();
    for (int i = 0; i < NUM_VALUES; i++) {
        int actual = reader.getInt(i, context);
        Assert.assertEquals(actual, expected[i]);
    }
    reader.close();
    FileUtils.deleteQuietly(outFile);
}
Also used : ChunkCompressor(com.linkedin.pinot.core.io.compression.ChunkCompressor) PinotDataBuffer(com.linkedin.pinot.core.segment.memory.PinotDataBuffer) ChunkDecompressor(com.linkedin.pinot.core.io.compression.ChunkDecompressor) FixedByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader) File(java.io.File) FixedByteChunkSingleValueWriter(com.linkedin.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter) ChunkReaderContext(com.linkedin.pinot.core.io.reader.impl.ChunkReaderContext) Test(org.testng.annotations.Test)

Example 3 with ChunkDecompressor

use of com.linkedin.pinot.core.io.compression.ChunkDecompressor in project pinot by linkedin.

the class FixedByteChunkSingleValueReaderWriteTest method testLong.

@Test
public void testLong() throws Exception {
    long[] expected = new long[NUM_VALUES];
    for (int i = 0; i < NUM_VALUES; i++) {
        expected[i] = _random.nextLong();
    }
    File outFile = new File(TEST_FILE);
    FileUtils.deleteQuietly(outFile);
    ChunkCompressor compressor = ChunkCompressorFactory.getCompressor("snappy");
    FixedByteChunkSingleValueWriter writer = new FixedByteChunkSingleValueWriter(outFile, compressor, NUM_VALUES, NUM_DOCS_PER_CHUNK, V1Constants.Numbers.LONG_SIZE);
    for (int i = 0; i < NUM_VALUES; i++) {
        writer.setLong(i, expected[i]);
    }
    writer.close();
    PinotDataBuffer pinotDataBuffer = PinotDataBuffer.fromFile(outFile, ReadMode.mmap, FileChannel.MapMode.READ_ONLY, getClass().getName());
    ChunkDecompressor uncompressor = ChunkCompressorFactory.getDecompressor("snappy");
    FixedByteChunkSingleValueReader reader = new FixedByteChunkSingleValueReader(pinotDataBuffer, uncompressor);
    ChunkReaderContext context = reader.createContext();
    for (int i = 0; i < NUM_VALUES; i++) {
        long actual = reader.getLong(i, context);
        Assert.assertEquals(actual, expected[i]);
    }
    reader.close();
    FileUtils.deleteQuietly(outFile);
}
Also used : ChunkCompressor(com.linkedin.pinot.core.io.compression.ChunkCompressor) PinotDataBuffer(com.linkedin.pinot.core.segment.memory.PinotDataBuffer) ChunkDecompressor(com.linkedin.pinot.core.io.compression.ChunkDecompressor) FixedByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader) File(java.io.File) FixedByteChunkSingleValueWriter(com.linkedin.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter) ChunkReaderContext(com.linkedin.pinot.core.io.reader.impl.ChunkReaderContext) Test(org.testng.annotations.Test)

Example 4 with ChunkDecompressor

use of com.linkedin.pinot.core.io.compression.ChunkDecompressor in project pinot by linkedin.

the class FixedByteChunkSingleValueReaderWriteTest method testFloat.

@Test
public void testFloat() throws Exception {
    float[] expected = new float[NUM_VALUES];
    for (int i = 0; i < NUM_VALUES; i++) {
        expected[i] = _random.nextFloat();
    }
    File outFile = new File(TEST_FILE);
    FileUtils.deleteQuietly(outFile);
    ChunkCompressor compressor = ChunkCompressorFactory.getCompressor("snappy");
    FixedByteChunkSingleValueWriter writer = new FixedByteChunkSingleValueWriter(outFile, compressor, NUM_VALUES, NUM_DOCS_PER_CHUNK, V1Constants.Numbers.FLOAT_SIZE);
    for (int i = 0; i < NUM_VALUES; i++) {
        writer.setFloat(i, expected[i]);
    }
    writer.close();
    PinotDataBuffer pinotDataBuffer = PinotDataBuffer.fromFile(outFile, ReadMode.mmap, FileChannel.MapMode.READ_ONLY, getClass().getName());
    ChunkDecompressor uncompressor = ChunkCompressorFactory.getDecompressor("snappy");
    FixedByteChunkSingleValueReader reader = new FixedByteChunkSingleValueReader(pinotDataBuffer, uncompressor);
    ChunkReaderContext context = reader.createContext();
    for (int i = 0; i < NUM_VALUES; i++) {
        float actual = reader.getFloat(i, context);
        Assert.assertEquals(actual, expected[i]);
    }
    reader.close();
    FileUtils.deleteQuietly(outFile);
}
Also used : ChunkCompressor(com.linkedin.pinot.core.io.compression.ChunkCompressor) PinotDataBuffer(com.linkedin.pinot.core.segment.memory.PinotDataBuffer) ChunkDecompressor(com.linkedin.pinot.core.io.compression.ChunkDecompressor) FixedByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader) File(java.io.File) FixedByteChunkSingleValueWriter(com.linkedin.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter) ChunkReaderContext(com.linkedin.pinot.core.io.reader.impl.ChunkReaderContext) Test(org.testng.annotations.Test)

Example 5 with ChunkDecompressor

use of com.linkedin.pinot.core.io.compression.ChunkDecompressor in project pinot by linkedin.

the class FixedByteChunkSingleValueReaderWriteTest method testDouble.

@Test
public void testDouble() throws Exception {
    double[] expected = new double[NUM_VALUES];
    for (int i = 0; i < NUM_VALUES; i++) {
        expected[i] = _random.nextDouble();
    }
    File outFile = new File(TEST_FILE);
    FileUtils.deleteQuietly(outFile);
    ChunkCompressor compressor = ChunkCompressorFactory.getCompressor("snappy");
    FixedByteChunkSingleValueWriter writer = new FixedByteChunkSingleValueWriter(outFile, compressor, NUM_VALUES, NUM_DOCS_PER_CHUNK, V1Constants.Numbers.DOUBLE_SIZE);
    for (int i = 0; i < NUM_VALUES; i++) {
        writer.setDouble(i, expected[i]);
    }
    writer.close();
    PinotDataBuffer pinotDataBuffer = PinotDataBuffer.fromFile(outFile, ReadMode.mmap, FileChannel.MapMode.READ_ONLY, getClass().getName());
    ChunkDecompressor uncompressor = ChunkCompressorFactory.getDecompressor("snappy");
    FixedByteChunkSingleValueReader reader = new FixedByteChunkSingleValueReader(pinotDataBuffer, uncompressor);
    ChunkReaderContext context = reader.createContext();
    for (int i = 0; i < NUM_VALUES; i++) {
        double actual = reader.getDouble(i, context);
        Assert.assertEquals(actual, expected[i]);
    }
    reader.close();
    FileUtils.deleteQuietly(outFile);
}
Also used : ChunkCompressor(com.linkedin.pinot.core.io.compression.ChunkCompressor) PinotDataBuffer(com.linkedin.pinot.core.segment.memory.PinotDataBuffer) ChunkDecompressor(com.linkedin.pinot.core.io.compression.ChunkDecompressor) FixedByteChunkSingleValueReader(com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader) File(java.io.File) FixedByteChunkSingleValueWriter(com.linkedin.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter) ChunkReaderContext(com.linkedin.pinot.core.io.reader.impl.ChunkReaderContext) Test(org.testng.annotations.Test)

Aggregations

ChunkDecompressor (com.linkedin.pinot.core.io.compression.ChunkDecompressor)7 ChunkReaderContext (com.linkedin.pinot.core.io.reader.impl.ChunkReaderContext)6 PinotDataBuffer (com.linkedin.pinot.core.segment.memory.PinotDataBuffer)6 Test (org.testng.annotations.Test)6 ChunkCompressor (com.linkedin.pinot.core.io.compression.ChunkCompressor)5 FixedByteChunkSingleValueReader (com.linkedin.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader)5 File (java.io.File)5 FixedByteChunkSingleValueWriter (com.linkedin.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter)4 VarByteChunkSingleValueReader (com.linkedin.pinot.core.io.reader.impl.v1.VarByteChunkSingleValueReader)3 GenericRow (com.linkedin.pinot.core.data.GenericRow)1 SingleColumnSingleValueReader (com.linkedin.pinot.core.io.reader.SingleColumnSingleValueReader)1 VarByteChunkSingleValueWriter (com.linkedin.pinot.core.io.writer.impl.v1.VarByteChunkSingleValueWriter)1 Random (java.util.Random)1