Search in sources :

Example 1 with DecimalStreamCheckpoint

use of io.prestosql.orc.checkpoint.DecimalStreamCheckpoint in project hetu-core by openlookeng.

the class DecimalColumnWriter method getIndexStreams.

@Override
public List<StreamDataOutput> getIndexStreams(CompressedMetadataWriter metadataWriter) throws IOException {
    checkState(closed);
    ImmutableList.Builder<RowGroupIndex> rowGroupIndexes = ImmutableList.builder();
    List<DecimalStreamCheckpoint> dataCheckpoints = dataStream.getCheckpoints();
    List<LongStreamCheckpoint> scaleCheckpoints = scaleStream.getCheckpoints();
    Optional<List<BooleanStreamCheckpoint>> presentCheckpoints = presentStream.getCheckpoints();
    for (int i = 0; i < rowGroupColumnStatistics.size(); i++) {
        int groupId = i;
        ColumnStatistics columnStatistics = rowGroupColumnStatistics.get(groupId);
        DecimalStreamCheckpoint dataCheckpoint = dataCheckpoints.get(groupId);
        LongStreamCheckpoint scaleCheckpoint = scaleCheckpoints.get(groupId);
        Optional<BooleanStreamCheckpoint> presentCheckpoint = presentCheckpoints.map(checkpoints -> checkpoints.get(groupId));
        List<Integer> positions = createDecimalColumnPositionList(compressed, dataCheckpoint, scaleCheckpoint, presentCheckpoint);
        rowGroupIndexes.add(new RowGroupIndex(positions, columnStatistics));
    }
    Slice slice = metadataWriter.writeRowIndexes(rowGroupIndexes.build());
    Stream stream = new Stream(columnId, StreamKind.ROW_INDEX, slice.length(), false);
    return ImmutableList.of(new StreamDataOutput(slice, stream));
}
Also used : ColumnStatistics(io.prestosql.orc.metadata.statistics.ColumnStatistics) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) StreamDataOutput(io.prestosql.orc.stream.StreamDataOutput) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) DecimalStreamCheckpoint(io.prestosql.orc.checkpoint.DecimalStreamCheckpoint) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) DecimalStreamCheckpoint(io.prestosql.orc.checkpoint.DecimalStreamCheckpoint) RowGroupIndex(io.prestosql.orc.metadata.RowGroupIndex) Slice(io.airlift.slice.Slice) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) LongOutputStream(io.prestosql.orc.stream.LongOutputStream) DecimalOutputStream(io.prestosql.orc.stream.DecimalOutputStream) PresentOutputStream(io.prestosql.orc.stream.PresentOutputStream) Stream(io.prestosql.orc.metadata.Stream)

Example 2 with DecimalStreamCheckpoint

use of io.prestosql.orc.checkpoint.DecimalStreamCheckpoint in project hetu-core by openlookeng.

the class DecimalOutputStream method recordCheckpoint.

@Override
public void recordCheckpoint() {
    checkState(!closed);
    checkpoints.add(new DecimalStreamCheckpoint(buffer.getCheckpoint()));
}
Also used : DecimalStreamCheckpoint(io.prestosql.orc.checkpoint.DecimalStreamCheckpoint)

Example 3 with DecimalStreamCheckpoint

use of io.prestosql.orc.checkpoint.DecimalStreamCheckpoint in project hetu-core by openlookeng.

the class DecimalInputStream method seekToCheckpoint.

@Override
public void seekToCheckpoint(DecimalStreamCheckpoint checkpoint) throws IOException {
    long newCheckpoint = checkpoint.getInputStreamCheckpoint();
    // buffer, so last checkpoint is no longer valid)
    if (block.length() > 0 && decodeCompressedBlockOffset(newCheckpoint) == decodeCompressedBlockOffset(lastCheckpoint)) {
        // and decompressed position is within our block, reposition in the block directly
        int localBlockOffset = decodeDecompressedOffset(newCheckpoint) - decodeDecompressedOffset(lastCheckpoint);
        if (localBlockOffset >= 0 && localBlockOffset < block.length()) {
            this.blockOffset = localBlockOffset;
            // do not change last checkpoint because we have not moved positions
            return;
        }
    }
    chunkLoader.seekToCheckpoint(newCheckpoint);
    lastCheckpoint = newCheckpoint;
    block = Slices.EMPTY_SLICE;
    blockOffset = 0;
}
Also used : DecimalStreamCheckpoint(io.prestosql.orc.checkpoint.DecimalStreamCheckpoint)

Aggregations

DecimalStreamCheckpoint (io.prestosql.orc.checkpoint.DecimalStreamCheckpoint)3 ImmutableList (com.google.common.collect.ImmutableList)1 Slice (io.airlift.slice.Slice)1 BooleanStreamCheckpoint (io.prestosql.orc.checkpoint.BooleanStreamCheckpoint)1 LongStreamCheckpoint (io.prestosql.orc.checkpoint.LongStreamCheckpoint)1 RowGroupIndex (io.prestosql.orc.metadata.RowGroupIndex)1 Stream (io.prestosql.orc.metadata.Stream)1 ColumnStatistics (io.prestosql.orc.metadata.statistics.ColumnStatistics)1 DecimalOutputStream (io.prestosql.orc.stream.DecimalOutputStream)1 LongOutputStream (io.prestosql.orc.stream.LongOutputStream)1 PresentOutputStream (io.prestosql.orc.stream.PresentOutputStream)1 StreamDataOutput (io.prestosql.orc.stream.StreamDataOutput)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1