Search in sources :

Example 6 with OrcCorruptionException

use of io.prestosql.orc.OrcCorruptionException in project hetu-core by openlookeng.

the class OrcInputStream method readFully.

public void readFully(Slice buffer, int inputOffset, int inputLength) throws IOException {
    int offset = inputOffset;
    int length = inputLength;
    while (length > 0) {
        if (current != null && current.remaining() == 0) {
            advance();
        }
        if (current == null) {
            throw new OrcCorruptionException(chunkLoader.getOrcDataSourceId(), "Unexpected end of stream");
        }
        int chunkSize = min(length, (int) current.remaining());
        current.readBytes(buffer, offset, chunkSize);
        length -= chunkSize;
        offset += chunkSize;
    }
}
Also used : OrcCorruptionException(io.prestosql.orc.OrcCorruptionException) InputStreamCheckpoint.createInputStreamCheckpoint(io.prestosql.orc.checkpoint.InputStreamCheckpoint.createInputStreamCheckpoint)

Example 7 with OrcCorruptionException

use of io.prestosql.orc.OrcCorruptionException in project hetu-core by openlookeng.

the class UncompressedOrcChunkLoader method seekToCheckpoint.

@Override
public void seekToCheckpoint(long checkpoint) throws OrcCorruptionException {
    int compressedOffset = decodeCompressedBlockOffset(checkpoint);
    if (compressedOffset != 0) {
        throw new OrcCorruptionException(dataReader.getOrcDataSourceId(), "Uncompressed stream does not support seeking to a compressed offset");
    }
    int decompressedOffset = decodeDecompressedOffset(checkpoint);
    nextPosition = decompressedOffset;
    lastCheckpoint = checkpoint;
}
Also used : OrcCorruptionException(io.prestosql.orc.OrcCorruptionException) InputStreamCheckpoint.createInputStreamCheckpoint(io.prestosql.orc.checkpoint.InputStreamCheckpoint.createInputStreamCheckpoint)

Example 8 with OrcCorruptionException

use of io.prestosql.orc.OrcCorruptionException in project hetu-core by openlookeng.

the class DecimalColumnReader method readBlock.

@Override
public Block readBlock() throws IOException {
    if (!rowGroupOpen) {
        openRowGroup();
    }
    seekToOffset();
    Block block;
    if (decimalStream == null && scaleStream == null) {
        if (presentStream == null) {
            throw new OrcCorruptionException(column.getOrcDataSourceId(), "Value is null but present stream is missing");
        }
        presentStream.skip(nextBatchSize);
        block = RunLengthEncodedBlock.create(type, null, nextBatchSize);
    } else if (presentStream == null) {
        checkDataStreamsArePresent();
        block = readNonNullBlock();
    } else {
        checkDataStreamsArePresent();
        boolean[] isNull = new boolean[nextBatchSize];
        int nullCount = presentStream.getUnsetBits(nextBatchSize, isNull);
        if (nullCount == 0) {
            block = readNonNullBlock();
        } else if (nullCount != nextBatchSize) {
            block = readNullBlock(isNull, nextBatchSize - nullCount);
        } else {
            block = RunLengthEncodedBlock.create(DOUBLE, null, nextBatchSize);
        }
    }
    readOffset = 0;
    nextBatchSize = 0;
    return block;
}
Also used : RunLengthEncodedBlock(io.prestosql.spi.block.RunLengthEncodedBlock) LongArrayBlock(io.prestosql.spi.block.LongArrayBlock) Block(io.prestosql.spi.block.Block) Int128ArrayBlock(io.prestosql.spi.block.Int128ArrayBlock) OrcCorruptionException(io.prestosql.orc.OrcCorruptionException)

Example 9 with OrcCorruptionException

use of io.prestosql.orc.OrcCorruptionException in project hetu-core by openlookeng.

the class DoubleColumnReader method readBlock.

@Override
public Block readBlock() throws IOException {
    if (!rowGroupOpen) {
        openRowGroup();
    }
    if (readOffset > 0) {
        if (presentStream != null) {
            // skip ahead the present bit reader, but count the set bits
            // and use this as the skip size for the data reader
            readOffset = presentStream.countBitsSet(readOffset);
        }
        if (readOffset > 0) {
            if (dataStream == null) {
                throw new OrcCorruptionException(column.getOrcDataSourceId(), "Value is not null but data stream is missing");
            }
            dataStream.skip(readOffset);
        }
    }
    Block block;
    if (dataStream == null) {
        if (presentStream == null) {
            throw new OrcCorruptionException(column.getOrcDataSourceId(), "Value is null but present stream is missing");
        }
        presentStream.skip(nextBatchSize);
        block = RunLengthEncodedBlock.create(DOUBLE, null, nextBatchSize);
    } else if (presentStream == null) {
        block = readNonNullBlock();
    } else {
        boolean[] isNull = new boolean[nextBatchSize];
        int nullCount = presentStream.getUnsetBits(nextBatchSize, isNull);
        if (nullCount == 0) {
            block = readNonNullBlock();
        } else if (nullCount != nextBatchSize) {
            block = readNullBlock(isNull, nextBatchSize - nullCount);
        } else {
            block = RunLengthEncodedBlock.create(DOUBLE, null, nextBatchSize);
        }
    }
    readOffset = 0;
    nextBatchSize = 0;
    return block;
}
Also used : RunLengthEncodedBlock(io.prestosql.spi.block.RunLengthEncodedBlock) LongArrayBlock(io.prestosql.spi.block.LongArrayBlock) Block(io.prestosql.spi.block.Block) OrcCorruptionException(io.prestosql.orc.OrcCorruptionException)

Example 10 with OrcCorruptionException

use of io.prestosql.orc.OrcCorruptionException in project hetu-core by openlookeng.

the class FloatColumnReader method readBlock.

@Override
public Block readBlock() throws IOException {
    if (!rowGroupOpen) {
        openRowGroup();
    }
    if (readOffset > 0) {
        if (presentStream != null) {
            // skip ahead the present bit reader, but count the set bits
            // and use this as the skip size for the data reader
            readOffset = presentStream.countBitsSet(readOffset);
        }
        if (readOffset > 0) {
            if (dataStream == null) {
                throw new OrcCorruptionException(column.getOrcDataSourceId(), "Value is not null but data stream is missing");
            }
            dataStream.skip(readOffset);
        }
    }
    Block block;
    if (dataStream == null) {
        if (presentStream == null) {
            throw new OrcCorruptionException(column.getOrcDataSourceId(), "Value is null but present stream is missing");
        }
        presentStream.skip(nextBatchSize);
        block = RunLengthEncodedBlock.create(REAL, null, nextBatchSize);
    } else if (presentStream == null) {
        block = readNonNullBlock();
    } else {
        boolean[] isNull = new boolean[nextBatchSize];
        int nullCount = presentStream.getUnsetBits(nextBatchSize, isNull);
        if (nullCount == 0) {
            block = readNonNullBlock();
        } else if (nullCount != nextBatchSize) {
            block = readNullBlock(isNull, nextBatchSize - nullCount);
        } else {
            block = RunLengthEncodedBlock.create(REAL, null, nextBatchSize);
        }
    }
    readOffset = 0;
    nextBatchSize = 0;
    return block;
}
Also used : RunLengthEncodedBlock(io.prestosql.spi.block.RunLengthEncodedBlock) Block(io.prestosql.spi.block.Block) IntArrayBlock(io.prestosql.spi.block.IntArrayBlock) OrcCorruptionException(io.prestosql.orc.OrcCorruptionException)

Aggregations

OrcCorruptionException (io.prestosql.orc.OrcCorruptionException)37 Block (io.prestosql.spi.block.Block)14 RunLengthEncodedBlock (io.prestosql.spi.block.RunLengthEncodedBlock)12 LongStreamCheckpoint (io.prestosql.orc.checkpoint.LongStreamCheckpoint)7 DecimalStreamCheckpoint (io.prestosql.orc.checkpoint.DecimalStreamCheckpoint)4 InputStreamCheckpoint.createInputStreamCheckpoint (io.prestosql.orc.checkpoint.InputStreamCheckpoint.createInputStreamCheckpoint)4 LongStreamV2Checkpoint (io.prestosql.orc.checkpoint.LongStreamV2Checkpoint)4 LongArrayBlock (io.prestosql.spi.block.LongArrayBlock)4 Slice (io.airlift.slice.Slice)3 LongStreamV1Checkpoint (io.prestosql.orc.checkpoint.LongStreamV1Checkpoint)3 PrestoException (io.prestosql.spi.PrestoException)3 ByteStreamCheckpoint (io.prestosql.orc.checkpoint.ByteStreamCheckpoint)2 ByteArrayInputStream (io.prestosql.orc.stream.ByteArrayInputStream)2 LongInputStream (io.prestosql.orc.stream.LongInputStream)2 WriteIdInfo (io.prestosql.plugin.hive.WriteIdInfo)2 ByteArrayBlock (io.prestosql.spi.block.ByteArrayBlock)2 IntArrayBlock (io.prestosql.spi.block.IntArrayBlock)2 SortOrder (io.prestosql.spi.block.SortOrder)2 VariableWidthBlock (io.prestosql.spi.block.VariableWidthBlock)2 BigintType (io.prestosql.spi.type.BigintType)2