Examples with MemoryColumnHandle io.prestosql.plugin.memory.MemoryColumnHandle used on opensource projects

Example 1 with MemoryColumnHandle

use of io.prestosql.plugin.memory.MemoryColumnHandle in project hetu-core by openlookeng.

the class LogicalPart method partitionPage.

static Map<String, Page> partitionPage(Page page, List<String> partitionedBy, List<MemoryColumnHandle> columns, TypeManager typeManager) {
    // derive the channel numbers that corresponds to the partitionedBy list
    List<MemoryColumnHandle> partitionChannels = new ArrayList<>(partitionedBy.size());
    for (String name : partitionedBy) {
        for (MemoryColumnHandle handle : columns) {
            if (handle.getColumnName().equals(name)) {
                partitionChannels.add(handle);
            }
        }
    }
    // build the partitions
    Map<String, Page> partitions = new HashMap<>();
    MemoryColumnHandle partitionColumnHandle = partitionChannels.get(0);
    Block block = page.getBlock(partitionColumnHandle.getColumnIndex());
    Type type = partitionColumnHandle.getType(typeManager);
    Map<Object, ArrayList<Integer>> uniqueValues = new HashMap<>();
    for (int i = 0; i < page.getPositionCount(); i++) {
        Object value = getNativeValue(type, block, i);
        uniqueValues.putIfAbsent(value, new ArrayList<>());
        uniqueValues.get(value).add(i);
    }
    for (Map.Entry<Object, ArrayList<Integer>> valueAndPosition : uniqueValues.entrySet()) {
        int[] retainedPositions = valueAndPosition.getValue().stream().mapToInt(i -> i).toArray();
        Object valueKey = valueAndPosition.getKey();
        Page subPage = page.getPositions(retainedPositions, 0, retainedPositions.length);
        // NOTE: null partition key is allowed here in the map
        // but when this partition map is sent to coordinator via MemoryDataFragment
        // the JSON parser fails and can't handle null keys in the map
        // the JSON parser will ignore null keys
        // therefore during scheduling if the query predicate is for null
        // we MUST NOT do any partition filtering because the partition map
        // the coordinator has is missing null partitions
        // the coordinator must schedule all splits if the query predicate is null
        // see: MemorySplitManager#getSplits
        // 
        // note: the other option is to use an empty string as the null key
        // then the JSON parser could send the key to the coordinator
        // but then this would cause conflicts with actual empty string values
        partitions.put(valueKey == null ? null : valueKey.toString(), subPage);
    }
    return partitions;
}

Example 2 with MemoryColumnHandle

use of io.prestosql.plugin.memory.MemoryColumnHandle in project hetu-core by openlookeng.

the class StatisticsUtils method fromComputedStatistics.

public static TableStatisticsData fromComputedStatistics(ConnectorSession session, Collection<ComputedStatistics> computedStatistics, long rowCount, Map<String, ColumnHandle> columnHandles, TypeManager typeManager) {
    TableStatisticsData.Builder tableStatBuilder = TableStatisticsData.builder();
    tableStatBuilder.setRowCount(rowCount);
    // organize the column stats into a per-column view
    Map<String, Map<ColumnStatisticType, Block>> perColumnStats = new HashMap<>();
    for (ComputedStatistics stat : computedStatistics) {
        for (Map.Entry<ColumnStatisticMetadata, Block> entry : stat.getColumnStatistics().entrySet()) {
            perColumnStats.putIfAbsent(entry.getKey().getColumnName(), new HashMap<>());
            perColumnStats.get(entry.getKey().getColumnName()).put(entry.getKey().getStatisticType(), entry.getValue());
        }
    }
    // build the per-column statistics
    for (Map.Entry<String, ColumnHandle> entry : columnHandles.entrySet()) {
        Map<ColumnStatisticType, Block> columnStat = perColumnStats.get(entry.getKey());
        if (columnStat == null) {
            continue;
        }
        MemoryColumnHandle handle = (MemoryColumnHandle) entry.getValue();
        Type columnType = handle.getType(typeManager);
        tableStatBuilder.setColumnStatistics(handle.getColumnName(), fromComputedStatistics(session, columnStat, rowCount, columnType));
    }
    return tableStatBuilder.build();
}