Search in sources :

Example 1 with HoodieKeyLookupResult

use of org.apache.hudi.io.HoodieKeyLookupResult in project hudi by apache.

the class ListBasedHoodieBloomIndexHelper method findMatchingFilesForRecordKeys.

@Override
public HoodiePairData<HoodieKey, HoodieRecordLocation> findMatchingFilesForRecordKeys(HoodieWriteConfig config, HoodieEngineContext context, HoodieTable hoodieTable, HoodiePairData<String, String> partitionRecordKeyPairs, HoodieData<Pair<String, HoodieKey>> fileComparisonPairs, Map<String, List<BloomIndexFileInfo>> partitionToFileInfo, Map<String, Long> recordsPerPartition) {
    List<Pair<String, HoodieKey>> fileComparisonPairList = HoodieList.getList(fileComparisonPairs).stream().sorted(Comparator.comparing(Pair::getLeft)).collect(toList());
    List<HoodieKeyLookupResult> keyLookupResults = new ArrayList<>();
    Iterator<List<HoodieKeyLookupResult>> iterator = new HoodieBaseBloomIndexCheckFunction(hoodieTable, config).apply(fileComparisonPairList.iterator());
    while (iterator.hasNext()) {
        keyLookupResults.addAll(iterator.next());
    }
    keyLookupResults = keyLookupResults.stream().filter(lr -> lr.getMatchingRecordKeys().size() > 0).collect(toList());
    return context.parallelize(keyLookupResults).flatMap(lookupResult -> lookupResult.getMatchingRecordKeys().stream().map(recordKey -> new ImmutablePair<>(lookupResult, recordKey)).iterator()).mapToPair(pair -> {
        HoodieKeyLookupResult lookupResult = pair.getLeft();
        String recordKey = pair.getRight();
        return new ImmutablePair<>(new HoodieKey(recordKey, lookupResult.getPartitionPath()), new HoodieRecordLocation(lookupResult.getBaseInstantTime(), lookupResult.getFileId()));
    });
}
Also used : ImmutablePair(org.apache.hudi.common.util.collection.ImmutablePair) HoodieTable(org.apache.hudi.table.HoodieTable) HoodieData(org.apache.hudi.common.data.HoodieData) HoodieWriteConfig(org.apache.hudi.config.HoodieWriteConfig) Iterator(java.util.Iterator) HoodiePairData(org.apache.hudi.common.data.HoodiePairData) HoodieEngineContext(org.apache.hudi.common.engine.HoodieEngineContext) HoodieKeyLookupResult(org.apache.hudi.io.HoodieKeyLookupResult) ArrayList(java.util.ArrayList) HoodieList(org.apache.hudi.common.data.HoodieList) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) HoodieRecordLocation(org.apache.hudi.common.model.HoodieRecordLocation) Map(java.util.Map) HoodieKey(org.apache.hudi.common.model.HoodieKey) Comparator(java.util.Comparator) Pair(org.apache.hudi.common.util.collection.Pair) ArrayList(java.util.ArrayList) HoodieRecordLocation(org.apache.hudi.common.model.HoodieRecordLocation) HoodieKeyLookupResult(org.apache.hudi.io.HoodieKeyLookupResult) ImmutablePair(org.apache.hudi.common.util.collection.ImmutablePair) HoodieKey(org.apache.hudi.common.model.HoodieKey) ArrayList(java.util.ArrayList) HoodieList(org.apache.hudi.common.data.HoodieList) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) ImmutablePair(org.apache.hudi.common.util.collection.ImmutablePair) Pair(org.apache.hudi.common.util.collection.Pair)

Aggregations

ArrayList (java.util.ArrayList)1 Comparator (java.util.Comparator)1 Iterator (java.util.Iterator)1 List (java.util.List)1 Map (java.util.Map)1 Collectors.toList (java.util.stream.Collectors.toList)1 HoodieData (org.apache.hudi.common.data.HoodieData)1 HoodieList (org.apache.hudi.common.data.HoodieList)1 HoodiePairData (org.apache.hudi.common.data.HoodiePairData)1 HoodieEngineContext (org.apache.hudi.common.engine.HoodieEngineContext)1 HoodieKey (org.apache.hudi.common.model.HoodieKey)1 HoodieRecordLocation (org.apache.hudi.common.model.HoodieRecordLocation)1 ImmutablePair (org.apache.hudi.common.util.collection.ImmutablePair)1 Pair (org.apache.hudi.common.util.collection.Pair)1 HoodieWriteConfig (org.apache.hudi.config.HoodieWriteConfig)1 HoodieKeyLookupResult (org.apache.hudi.io.HoodieKeyLookupResult)1 HoodieTable (org.apache.hudi.table.HoodieTable)1