Search in sources :

Example 1 with ListIndexed

use of org.apache.druid.segment.data.ListIndexed in project presto by prestodb.

the class V9SegmentIndexSource method loadIndex.

@Override
public QueryableIndex loadIndex(List<ColumnHandle> columnHandles) throws IOException {
    ByteBuffer indexBuffer = ByteBuffer.wrap(segmentColumnSource.getColumnData(INDEX_METADATA_FILE_NAME));
    GenericIndexed.read(indexBuffer, STRING_STRATEGY);
    GenericIndexed<String> allDimensions = GenericIndexed.read(indexBuffer, STRING_STRATEGY);
    Interval dataInterval = Intervals.utc(indexBuffer.getLong(), indexBuffer.getLong());
    BitmapSerdeFactory segmentBitmapSerdeFactory;
    if (indexBuffer.hasRemaining()) {
        segmentBitmapSerdeFactory = JSON_MAPPER.readValue(SERIALIZER_UTILS.readString(indexBuffer), BitmapSerdeFactory.class);
    } else {
        segmentBitmapSerdeFactory = new BitmapSerde.LegacyBitmapSerdeFactory();
    }
    Metadata metadata = null;
    ByteBuffer metadataBuffer = ByteBuffer.wrap(segmentColumnSource.getColumnData(SEGMENT_METADATA_FILE_NAME));
    try {
        metadata = JSON_MAPPER.readValue(SERIALIZER_UTILS.readBytes(metadataBuffer, metadataBuffer.remaining()), Metadata.class);
    } catch (JsonParseException | JsonMappingException e) {
        // Any jackson deserialization errors are ignored e.g. if metadata contains some aggregator which
        // is no longer supported then it is OK to not use the metadata instead of failing segment loading
        log.warn(e, "Failed to load metadata for segment");
    }
    Map<String, Supplier<ColumnHolder>> columns = new HashMap<>();
    for (ColumnHandle columnHandle : columnHandles) {
        String columnName = ((DruidColumnHandle) columnHandle).getColumnName();
        columns.put(columnName, () -> createColumnHolder(columnName));
    }
    List<String> availableDimensions = Streams.stream(allDimensions.iterator()).filter(columns::containsKey).collect(toImmutableList());
    columns.put(TIME_COLUMN_NAME, () -> createColumnHolder(TIME_COLUMN_NAME));
    Indexed<String> indexed = new ListIndexed<>(availableDimensions);
    // TODO: get rid of the time column by creating Presto's SimpleQueryableIndex impl
    return new SimpleQueryableIndex(dataInterval, indexed, segmentBitmapSerdeFactory.getBitmapFactory(), columns, null, metadata, false);
}
Also used : DruidColumnHandle(com.facebook.presto.druid.DruidColumnHandle) DruidColumnHandle(com.facebook.presto.druid.DruidColumnHandle) ColumnHandle(com.facebook.presto.spi.ColumnHandle) ListIndexed(org.apache.druid.segment.data.ListIndexed) HashMap(java.util.HashMap) BitmapSerde(org.apache.druid.segment.data.BitmapSerde) Metadata(org.apache.druid.segment.Metadata) SimpleQueryableIndex(org.apache.druid.segment.SimpleQueryableIndex) JsonParseException(com.fasterxml.jackson.core.JsonParseException) ByteBuffer(java.nio.ByteBuffer) JsonMappingException(com.fasterxml.jackson.databind.JsonMappingException) Supplier(com.google.common.base.Supplier) BitmapSerdeFactory(org.apache.druid.segment.data.BitmapSerdeFactory) Interval(org.joda.time.Interval)

Aggregations

DruidColumnHandle (com.facebook.presto.druid.DruidColumnHandle)1 ColumnHandle (com.facebook.presto.spi.ColumnHandle)1 JsonParseException (com.fasterxml.jackson.core.JsonParseException)1 JsonMappingException (com.fasterxml.jackson.databind.JsonMappingException)1 Supplier (com.google.common.base.Supplier)1 ByteBuffer (java.nio.ByteBuffer)1 HashMap (java.util.HashMap)1 Metadata (org.apache.druid.segment.Metadata)1 SimpleQueryableIndex (org.apache.druid.segment.SimpleQueryableIndex)1 BitmapSerde (org.apache.druid.segment.data.BitmapSerde)1 BitmapSerdeFactory (org.apache.druid.segment.data.BitmapSerdeFactory)1 ListIndexed (org.apache.druid.segment.data.ListIndexed)1 Interval (org.joda.time.Interval)1