Search in sources :

Example 6 with DataChunk

use of org.apache.carbondata.core.metadata.blocklet.datachunk.DataChunk in project carbondata by apache.

the class CompressedDimensionChunkFileBasedReaderV1 method readRawDimensionChunk.

/**
   * Below method will be used to read the raw chunk based on block index
   *
   * @param fileReader file reader to read the blocks from file
   * @param blockletIndex block to be read
   * @return dimension column chunk
   */
@Override
public DimensionRawColumnChunk readRawDimensionChunk(FileHolder fileReader, int blockletIndex) throws IOException {
    DataChunk dataChunk = dimensionColumnChunk.get(blockletIndex);
    ByteBuffer buffer = null;
    synchronized (fileReader) {
        buffer = fileReader.readByteBuffer(filePath, dataChunk.getDataPageOffset(), dataChunk.getDataPageLength());
    }
    DimensionRawColumnChunk rawColumnChunk = new DimensionRawColumnChunk(blockletIndex, buffer, 0, dataChunk.getDataPageLength(), this);
    rawColumnChunk.setFileHolder(fileReader);
    rawColumnChunk.setPagesCount(1);
    rawColumnChunk.setRowCount(new int[] { numberOfRows });
    return rawColumnChunk;
}
Also used : DimensionColumnDataChunk(org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk) ColumnGroupDimensionDataChunk(org.apache.carbondata.core.datastore.chunk.impl.ColumnGroupDimensionDataChunk) VariableLengthDimensionDataChunk(org.apache.carbondata.core.datastore.chunk.impl.VariableLengthDimensionDataChunk) DataChunk(org.apache.carbondata.core.metadata.blocklet.datachunk.DataChunk) FixedLengthDimensionDataChunk(org.apache.carbondata.core.datastore.chunk.impl.FixedLengthDimensionDataChunk) DimensionRawColumnChunk(org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk) ByteBuffer(java.nio.ByteBuffer)

Example 7 with DataChunk

use of org.apache.carbondata.core.metadata.blocklet.datachunk.DataChunk in project carbondata by apache.

the class CarbonUtilTest method testToGetValueCompressionModel.

@Test
public void testToGetValueCompressionModel() {
    List<DataChunk> dataChunkList = new ArrayList<>();
    DataChunk dataChunk = new DataChunk();
    List<Encoding> encodingList = new ArrayList<>();
    encodingList.add(Encoding.DELTA);
    dataChunk.setEncodingList(encodingList);
    List<ValueEncoderMeta> valueEncoderMetas = new ArrayList<>();
    ValueEncoderMeta valueEncoderMeta = new ValueEncoderMeta();
    valueEncoderMeta.setMaxValue(5.0);
    valueEncoderMeta.setMinValue(1.0);
    valueEncoderMeta.setUniqueValue(2.0);
    valueEncoderMeta.setType('n');
    valueEncoderMeta.setDataTypeSelected((byte) 'v');
    valueEncoderMetas.add(valueEncoderMeta);
    dataChunk.setValueEncoderMeta(valueEncoderMetas);
    dataChunkList.add(dataChunk);
    WriterCompressModel writerCompressModel = CarbonUtil.getValueCompressionModel(dataChunkList.get(0).getValueEncoderMeta());
    assertEquals(1, writerCompressModel.getMaxValue().length);
}
Also used : WriterCompressModel(org.apache.carbondata.core.datastore.compression.WriterCompressModel) ArrayList(java.util.ArrayList) DataChunk(org.apache.carbondata.core.metadata.blocklet.datachunk.DataChunk) FixedLengthDimensionDataChunk(org.apache.carbondata.core.datastore.chunk.impl.FixedLengthDimensionDataChunk) Encoding(org.apache.carbondata.core.metadata.encoder.Encoding) ValueEncoderMeta(org.apache.carbondata.core.metadata.ValueEncoderMeta) Test(org.junit.Test)

Aggregations

DataChunk (org.apache.carbondata.core.metadata.blocklet.datachunk.DataChunk)7 ByteBuffer (java.nio.ByteBuffer)5 ArrayList (java.util.ArrayList)3 FixedLengthDimensionDataChunk (org.apache.carbondata.core.datastore.chunk.impl.FixedLengthDimensionDataChunk)3 ValueEncoderMeta (org.apache.carbondata.core.metadata.ValueEncoderMeta)3 DimensionColumnDataChunk (org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk)2 MeasureColumnDataChunk (org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk)2 ColumnGroupDimensionDataChunk (org.apache.carbondata.core.datastore.chunk.impl.ColumnGroupDimensionDataChunk)2 VariableLengthDimensionDataChunk (org.apache.carbondata.core.datastore.chunk.impl.VariableLengthDimensionDataChunk)2 Encoding (org.apache.carbondata.core.metadata.encoder.Encoding)2 FileHolder (org.apache.carbondata.core.datastore.FileHolder)1 DimensionRawColumnChunk (org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk)1 MeasureRawColumnChunk (org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk)1 ReaderCompressModel (org.apache.carbondata.core.datastore.compression.ReaderCompressModel)1 ValueCompressionHolder (org.apache.carbondata.core.datastore.compression.ValueCompressionHolder)1 WriterCompressModel (org.apache.carbondata.core.datastore.compression.WriterCompressModel)1 CarbonReadDataHolder (org.apache.carbondata.core.datastore.dataholder.CarbonReadDataHolder)1 BlockletInfo (org.apache.carbondata.core.metadata.blocklet.BlockletInfo)1 Test (org.junit.Test)1