Search in sources :

Example 11 with StreamDataOutput

use of io.prestosql.orc.stream.StreamDataOutput in project hetu-core by openlookeng.

the class SliceDirectColumnWriter method getIndexStreams.

@Override
public List<StreamDataOutput> getIndexStreams(CompressedMetadataWriter metadataWriter) throws IOException {
    checkState(closed);
    ImmutableList.Builder<RowGroupIndex> rowGroupIndexes = ImmutableList.builder();
    List<LongStreamCheckpoint> lengthCheckpoints = lengthStream.getCheckpoints();
    List<ByteArrayStreamCheckpoint> dataCheckpoints = dataStream.getCheckpoints();
    Optional<List<BooleanStreamCheckpoint>> presentCheckpoints = presentStream.getCheckpoints();
    for (int i = 0; i < rowGroupColumnStatistics.size(); i++) {
        int groupId = i;
        ColumnStatistics columnStatistics = rowGroupColumnStatistics.get(groupId);
        LongStreamCheckpoint lengthCheckpoint = lengthCheckpoints.get(groupId);
        ByteArrayStreamCheckpoint dataCheckpoint = dataCheckpoints.get(groupId);
        Optional<BooleanStreamCheckpoint> presentCheckpoint = presentCheckpoints.map(checkpoints -> checkpoints.get(groupId));
        List<Integer> positions = createSliceColumnPositionList(compressed, lengthCheckpoint, dataCheckpoint, presentCheckpoint);
        rowGroupIndexes.add(new RowGroupIndex(positions, columnStatistics));
    }
    Slice slice = metadataWriter.writeRowIndexes(rowGroupIndexes.build());
    Stream stream = new Stream(columnId, StreamKind.ROW_INDEX, slice.length(), false);
    return ImmutableList.of(new StreamDataOutput(slice, stream));
}
Also used : ColumnStatistics(io.prestosql.orc.metadata.statistics.ColumnStatistics) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ByteArrayStreamCheckpoint(io.prestosql.orc.checkpoint.ByteArrayStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) StreamDataOutput(io.prestosql.orc.stream.StreamDataOutput) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) ByteArrayStreamCheckpoint(io.prestosql.orc.checkpoint.ByteArrayStreamCheckpoint) RowGroupIndex(io.prestosql.orc.metadata.RowGroupIndex) Slice(io.airlift.slice.Slice) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) LongOutputStream(io.prestosql.orc.stream.LongOutputStream) LongOutputStream.createLengthOutputStream(io.prestosql.orc.stream.LongOutputStream.createLengthOutputStream) PresentOutputStream(io.prestosql.orc.stream.PresentOutputStream) ByteArrayOutputStream(io.prestosql.orc.stream.ByteArrayOutputStream) Stream(io.prestosql.orc.metadata.Stream)

Example 12 with StreamDataOutput

use of io.prestosql.orc.stream.StreamDataOutput in project hetu-core by openlookeng.

the class StructColumnWriter method getIndexStreams.

@Override
public List<StreamDataOutput> getIndexStreams(CompressedMetadataWriter metadataWriter) throws IOException {
    checkState(closed);
    ImmutableList.Builder<RowGroupIndex> rowGroupIndexes = ImmutableList.builder();
    Optional<List<BooleanStreamCheckpoint>> presentCheckpoints = presentStream.getCheckpoints();
    for (int i = 0; i < rowGroupColumnStatistics.size(); i++) {
        int groupId = i;
        ColumnStatistics columnStatistics = rowGroupColumnStatistics.get(groupId);
        Optional<BooleanStreamCheckpoint> presentCheckpoint = presentCheckpoints.map(checkpoints -> checkpoints.get(groupId));
        List<Integer> positions = createStructColumnPositionList(compressed, presentCheckpoint);
        rowGroupIndexes.add(new RowGroupIndex(positions, columnStatistics));
    }
    Slice slice = metadataWriter.writeRowIndexes(rowGroupIndexes.build());
    Stream stream = new Stream(columnId, StreamKind.ROW_INDEX, slice.length(), false);
    ImmutableList.Builder<StreamDataOutput> indexStreams = ImmutableList.builder();
    indexStreams.add(new StreamDataOutput(slice, stream));
    for (ColumnWriter structField : structFields) {
        indexStreams.addAll(structField.getIndexStreams(metadataWriter));
    }
    return indexStreams.build();
}
Also used : ColumnStatistics(io.prestosql.orc.metadata.statistics.ColumnStatistics) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) StreamDataOutput(io.prestosql.orc.stream.StreamDataOutput) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) RowGroupIndex(io.prestosql.orc.metadata.RowGroupIndex) Slice(io.airlift.slice.Slice) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) PresentOutputStream(io.prestosql.orc.stream.PresentOutputStream) Stream(io.prestosql.orc.metadata.Stream)

Example 13 with StreamDataOutput

use of io.prestosql.orc.stream.StreamDataOutput in project hetu-core by openlookeng.

the class TimestampColumnWriter method getIndexStreams.

@Override
public List<StreamDataOutput> getIndexStreams(CompressedMetadataWriter metadataWriter) throws IOException {
    checkState(closed);
    ImmutableList.Builder<RowGroupIndex> rowGroupIndexes = ImmutableList.builder();
    List<LongStreamCheckpoint> secondsCheckpoints = secondsStream.getCheckpoints();
    List<LongStreamCheckpoint> nanosCheckpoints = nanosStream.getCheckpoints();
    Optional<List<BooleanStreamCheckpoint>> presentCheckpoints = presentStream.getCheckpoints();
    for (int i = 0; i < rowGroupColumnStatistics.size(); i++) {
        int groupId = i;
        ColumnStatistics columnStatistics = rowGroupColumnStatistics.get(groupId);
        LongStreamCheckpoint secondsCheckpoint = secondsCheckpoints.get(groupId);
        LongStreamCheckpoint nanosCheckpoint = nanosCheckpoints.get(groupId);
        Optional<BooleanStreamCheckpoint> presentCheckpoint = presentCheckpoints.map(checkpoints -> checkpoints.get(groupId));
        List<Integer> positions = createTimestampColumnPositionList(compressed, secondsCheckpoint, nanosCheckpoint, presentCheckpoint);
        rowGroupIndexes.add(new RowGroupIndex(positions, columnStatistics));
    }
    Slice slice = metadataWriter.writeRowIndexes(rowGroupIndexes.build());
    Stream stream = new Stream(columnId, StreamKind.ROW_INDEX, slice.length(), false);
    return ImmutableList.of(new StreamDataOutput(slice, stream));
}
Also used : ColumnStatistics(io.prestosql.orc.metadata.statistics.ColumnStatistics) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) ImmutableList(com.google.common.collect.ImmutableList) StreamDataOutput(io.prestosql.orc.stream.StreamDataOutput) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) BooleanStreamCheckpoint(io.prestosql.orc.checkpoint.BooleanStreamCheckpoint) LongStreamCheckpoint(io.prestosql.orc.checkpoint.LongStreamCheckpoint) RowGroupIndex(io.prestosql.orc.metadata.RowGroupIndex) Slice(io.airlift.slice.Slice) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) LongOutputStream(io.prestosql.orc.stream.LongOutputStream) PresentOutputStream(io.prestosql.orc.stream.PresentOutputStream) Stream(io.prestosql.orc.metadata.Stream)

Aggregations

Slice (io.airlift.slice.Slice)13 Stream (io.prestosql.orc.metadata.Stream)13 ColumnStatistics (io.prestosql.orc.metadata.statistics.ColumnStatistics)13 StreamDataOutput (io.prestosql.orc.stream.StreamDataOutput)13 ArrayList (java.util.ArrayList)13 ImmutableList (com.google.common.collect.ImmutableList)12 BooleanStreamCheckpoint (io.prestosql.orc.checkpoint.BooleanStreamCheckpoint)12 RowGroupIndex (io.prestosql.orc.metadata.RowGroupIndex)12 PresentOutputStream (io.prestosql.orc.stream.PresentOutputStream)12 List (java.util.List)12 LongStreamCheckpoint (io.prestosql.orc.checkpoint.LongStreamCheckpoint)7 LongOutputStream (io.prestosql.orc.stream.LongOutputStream)7 LongOutputStream.createLengthOutputStream (io.prestosql.orc.stream.LongOutputStream.createLengthOutputStream)4 ByteArrayOutputStream (io.prestosql.orc.stream.ByteArrayOutputStream)2 Slices.utf8Slice (io.airlift.slice.Slices.utf8Slice)1 ByteArrayStreamCheckpoint (io.prestosql.orc.checkpoint.ByteArrayStreamCheckpoint)1 ByteStreamCheckpoint (io.prestosql.orc.checkpoint.ByteStreamCheckpoint)1 DecimalStreamCheckpoint (io.prestosql.orc.checkpoint.DecimalStreamCheckpoint)1 DoubleStreamCheckpoint (io.prestosql.orc.checkpoint.DoubleStreamCheckpoint)1 FloatStreamCheckpoint (io.prestosql.orc.checkpoint.FloatStreamCheckpoint)1