Search in sources :

Example 1 with ByteStreamCheckpoint

use of io.prestosql.orc.checkpoint.ByteStreamCheckpoint in project hetu-core by openlookeng.

the class TestBooleanOutputStream method checkpointsEqual.

private static boolean checkpointsEqual(BooleanStreamCheckpoint left, BooleanStreamCheckpoint right) {
    assertNotNull(left);
    assertNotNull(right);
    if (left.getOffset() != right.getOffset()) {
        return false;
    }
    ByteStreamCheckpoint leftCheckpoint = left.getByteStreamCheckpoint();
    ByteStreamCheckpoint rightCheckpoint = right.getByteStreamCheckpoint();
    assertNotNull(leftCheckpoint);
    assertNotNull(rightCheckpoint);
    return leftCheckpoint.getInputStreamCheckpoint() == rightCheckpoint.getInputStreamCheckpoint() && leftCheckpoint.getOffset() == rightCheckpoint.getOffset();
}
Also used : ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint)

Example 2 with ByteStreamCheckpoint

use of io.prestosql.orc.checkpoint.ByteStreamCheckpoint in project hetu-core by openlookeng.

the class BooleanOutputStream method getCheckpoints.

@Override
public List<BooleanStreamCheckpoint> getCheckpoints() {
    checkState(closed);
    ImmutableList.Builder<BooleanStreamCheckpoint> booleanStreamCheckpoint = ImmutableList.builder();
    List<ByteStreamCheckpoint> byteStreamCheckpoints = byteOutputStream.getCheckpoints();
    for (int groupId = 0; groupId < checkpointBitOffsets.size(); groupId++) {
        int checkpointBitOffset = checkpointBitOffsets.get(groupId);
        ByteStreamCheckpoint byteStreamCheckpoint = byteStreamCheckpoints.get(groupId);
        booleanStreamCheckpoint.add(new BooleanStreamCheckpoint(checkpointBitOffset, byteStreamCheckpoint));
    }
    return booleanStreamCheckpoint.build();
}
Also used : BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint)

Example 3 with ByteStreamCheckpoint

use of io.prestosql.orc.checkpoint.ByteStreamCheckpoint in project hetu-core by openlookeng.

the class ByteColumnWriter method getIndexStreams.

@Override
public List<StreamDataOutput> getIndexStreams(CompressedMetadataWriter metadataWriter) throws IOException {
    checkState(closed);
    ImmutableList.Builder<RowGroupIndex> rowGroupIndexes = ImmutableList.builder();
    List<ByteStreamCheckpoint> dataCheckpoints = dataStream.getCheckpoints();
    Optional<List<BooleanStreamCheckpoint>> presentCheckpoints = presentStream.getCheckpoints();
    for (int i = 0; i < rowGroupColumnStatistics.size(); i++) {
        int groupId = i;
        ColumnStatistics columnStatistics = rowGroupColumnStatistics.get(groupId);
        ByteStreamCheckpoint dataCheckpoint = dataCheckpoints.get(groupId);
        Optional<BooleanStreamCheckpoint> presentCheckpoint = presentCheckpoints.map(checkpoints -> checkpoints.get(groupId));
        List<Integer> positions = createByteColumnPositionList(compressed, dataCheckpoint, presentCheckpoint);
        rowGroupIndexes.add(new RowGroupIndex(positions, columnStatistics));
    }
    Slice slice = metadataWriter.writeRowIndexes(rowGroupIndexes.build());
    Stream stream = new Stream(columnId, StreamKind.ROW_INDEX, slice.length(), false);
    return ImmutableList.of(new StreamDataOutput(slice, stream));
}
Also used : ColumnStatistics(io.prestosql.orc.metadata.statistics.ColumnStatistics) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint) StreamDataOutput(io.prestosql.orc.stream.StreamDataOutput) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint) RowGroupIndex(io.prestosql.orc.metadata.RowGroupIndex) Slice(io.airlift.slice.Slice) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) ByteOutputStream(io.prestosql.orc.stream.ByteOutputStream) PresentOutputStream(io.prestosql.orc.stream.PresentOutputStream) Stream(io.prestosql.orc.metadata.Stream)

Example 4 with ByteStreamCheckpoint

use of io.prestosql.orc.checkpoint.ByteStreamCheckpoint in project hetu-core by openlookeng.

the class ByteOutputStream method recordCheckpoint.

@Override
public void recordCheckpoint() {
    checkState(!closed);
    checkpoints.add(new ByteStreamCheckpoint(size, buffer.getCheckpoint()));
}
Also used : ByteStreamCheckpoint(io.prestosql.orc.checkpoint.ByteStreamCheckpoint)

Aggregations

ByteStreamCheckpoint (io.prestosql.orc.checkpoint.ByteStreamCheckpoint)4 ImmutableList (com.google.common.collect.ImmutableList)2 BooleanStreamCheckpoint (io.prestosql.orc.checkpoint.BooleanStreamCheckpoint)2 Slice (io.airlift.slice.Slice)1 RowGroupIndex (io.prestosql.orc.metadata.RowGroupIndex)1 Stream (io.prestosql.orc.metadata.Stream)1 ColumnStatistics (io.prestosql.orc.metadata.statistics.ColumnStatistics)1 ByteOutputStream (io.prestosql.orc.stream.ByteOutputStream)1 PresentOutputStream (io.prestosql.orc.stream.PresentOutputStream)1 StreamDataOutput (io.prestosql.orc.stream.StreamDataOutput)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1