Search in sources :

Example 1 with MemoryOrcDataReader

use of io.prestosql.orc.stream.MemoryOrcDataReader in project hetu-core by openlookeng.

the class AbstractOrcDataSource method readSmallDiskRanges.

private <K> Map<K, OrcDataReader> readSmallDiskRanges(Map<K, DiskRange> diskRanges) throws IOException {
    if (diskRanges.isEmpty()) {
        return ImmutableMap.of();
    }
    Iterable<DiskRange> mergedRanges = mergeAdjacentDiskRanges(diskRanges.values(), maxMergeDistance, maxBufferSize);
    ImmutableMap.Builder<K, OrcDataReader> slices = ImmutableMap.builder();
    if (lazyReadSmallRanges) {
        for (DiskRange mergedRange : mergedRanges) {
            LazyBufferLoader mergedRangeLazyLoader = new LazyBufferLoader(mergedRange);
            for (Entry<K, DiskRange> diskRangeEntry : diskRanges.entrySet()) {
                DiskRange diskRange = diskRangeEntry.getValue();
                if (mergedRange.contains(diskRange)) {
                    slices.put(diskRangeEntry.getKey(), new MergedOrcDataReader(id, diskRange, mergedRangeLazyLoader));
                }
            }
        }
    } else {
        Map<DiskRange, Slice> buffers = new LinkedHashMap<>();
        for (DiskRange mergedRange : mergedRanges) {
            // read full range in one request
            Slice buffer = readFully(mergedRange.getOffset(), mergedRange.getLength());
            buffers.put(mergedRange, buffer);
        }
        for (Entry<K, DiskRange> entry : diskRanges.entrySet()) {
            slices.put(entry.getKey(), new MemoryOrcDataReader(id, getDiskRangeSlice(entry.getValue(), buffers), entry.getValue().getLength()));
        }
    }
    Map<K, OrcDataReader> sliceStreams = slices.build();
    verify(sliceStreams.keySet().equals(diskRanges.keySet()));
    return sliceStreams;
}
Also used : MemoryOrcDataReader(io.prestosql.orc.stream.MemoryOrcDataReader) ImmutableMap(com.google.common.collect.ImmutableMap) LinkedHashMap(java.util.LinkedHashMap) Slice(io.airlift.slice.Slice) OrcDataSourceUtils.getDiskRangeSlice(io.prestosql.orc.OrcDataSourceUtils.getDiskRangeSlice) MemoryOrcDataReader(io.prestosql.orc.stream.MemoryOrcDataReader) AbstractDiskOrcDataReader(io.prestosql.orc.stream.AbstractDiskOrcDataReader) OrcDataReader(io.prestosql.orc.stream.OrcDataReader)

Example 2 with MemoryOrcDataReader

use of io.prestosql.orc.stream.MemoryOrcDataReader in project hetu-core by openlookeng.

the class CachingOrcDataSource method readFully.

@Override
public <K> Map<K, OrcDataReader> readFully(Map<K, DiskRange> diskRanges) throws IOException {
    ImmutableMap.Builder<K, OrcDataReader> builder = ImmutableMap.builder();
    // will not result in eviction of cache that otherwise could have served any of the DiskRanges provided.
    for (Map.Entry<K, DiskRange> entry : diskRanges.entrySet()) {
        DiskRange diskRange = entry.getValue();
        Slice buffer = readFully(diskRange.getOffset(), diskRange.getLength());
        builder.put(entry.getKey(), new MemoryOrcDataReader(dataSource.getId(), buffer, buffer.length()));
    }
    return builder.build();
}
Also used : Slice(io.airlift.slice.Slice) OrcDataReader(io.prestosql.orc.stream.OrcDataReader) MemoryOrcDataReader(io.prestosql.orc.stream.MemoryOrcDataReader) MemoryOrcDataReader(io.prestosql.orc.stream.MemoryOrcDataReader) ImmutableMap(com.google.common.collect.ImmutableMap) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap)

Aggregations

ImmutableMap (com.google.common.collect.ImmutableMap)2 Slice (io.airlift.slice.Slice)2 MemoryOrcDataReader (io.prestosql.orc.stream.MemoryOrcDataReader)2 OrcDataReader (io.prestosql.orc.stream.OrcDataReader)2 OrcDataSourceUtils.getDiskRangeSlice (io.prestosql.orc.OrcDataSourceUtils.getDiskRangeSlice)1 AbstractDiskOrcDataReader (io.prestosql.orc.stream.AbstractDiskOrcDataReader)1 LinkedHashMap (java.util.LinkedHashMap)1 Map (java.util.Map)1