use of com.apple.foundationdb.record.query.plan.temp.Ordering in project fdb-record-layer by FoundationDB.
the class AbstractDataAccessRule method onMatch.
/**
* Method that does the leg work to create the appropriate expression dag for data access using value indexes or
* value index-like scans (primary scans).
*
* Conceptually we do the following work:
*
* <ul>
* <li> This method yields a scan plan for each matching primary candidate ({@link PrimaryScanMatchCandidate}).
* There is only ever going to be exactly one {@link PrimaryScanMatchCandidate} for a primary key. Due to the
* candidate being solely based upon a primary key, the match structure is somewhat limited. In essence, there
* is an implicit guarantee that we can always create a primary scan for a data source.
* </li>
* <li> This method yields an index scan plan for each matching value index candidate
* ({@link ValueIndexScanMatchCandidate}).
* </li>
* <li> This method yields the combinatorial expansion of intersections of distinct-ed index scan plans.
* </li>
* </ul>
*
* The work described above is semantically correct in a sense that it creates a search space that can be explored
* and pruned in suitable ways that will eventually converge into an optimal data access plan.
*
* We can choose to create an index scan for every index that is available regardless what the coverage
* of an index is. The coverage of an index is a measurement that tells us how well an index can answer what a
* filter (or by extension a query) asks for. For instance, a high number of search arguments used in the index scan
* can be associated with high coverage (as in the index scan covers more of the query) and vice versa.
*
* Similarly, we can choose to create the intersection of all possible combinations of suitable scans over indexes
* (that we have matches for). Since we create a logical intersection of these access plans we can leave it up to
* the respective implementation rules (e.g., {@link ImplementIntersectionRule}) to do the right thing and implement
* the physical plan for the intersection if possible (e.g. ensuring compatibly ordered legs, etc.).
*
* In fact, the two before-mentioned approaches are completely valid with respect to correctness of the plan and
* the guaranteed creation of the optimal plan. However, in reality using this approach, although valid and probably
* the conceptually better and more orthogonal approach, will result in a ballooning of the search space very quickly.
* While that may be acceptable for group-by engines and only few index access paths, in an OLTP world where there
* are potentially dozens of indexes, memory footprint and the sheer number of tasks that would be created for
* subsequent exploration and implementation of all these alternatives make the purist approach to planning these
* indexes infeasible.
*
* Thus we would like to eliminate unnecessary exploration by avoiding variations we know can never be successful
* either in creating a successful executable plan (e.g. logical expression may not ever be able to produce a
* compatible ordering) or cannot ever create an optimal plan. In a nutshell, we try to utilize additional
* information that is available in addition to the matching partition in order to make decisions about which
* expression variation to create and which to avoid:
*
* <ul>
* <li> For a matching primary scan candidate ({@link PrimaryScanMatchCandidate})
* we will not create a primary scan if the scan is incompatible with an interesting order that has been
* communicated downwards in the graph.
* </li>
* <li> For a matching index scan candidate ({@link ValueIndexScanMatchCandidate})
* we will not create an index scan if the scan is incompatible with an interesting order that has been
* communicated downwards in the graph.
* </li>
* <li> We will only create a scan if there is no other index scan with a greater coverage (think of coverage
* as the assumed amount of filtering or currently the number of bound predicates) for the search arguments
* which are bound by the query.
* For instance, an index scan {@code INDEX SCAN(i1, a = [5, 5], b = [10, 10])} is still planned along
* {@code INDEX SCAN(i2, x = ["hello", "hello"], y = ["world", "world"], z = [10, inf])} even though
* the latter utilizes three search arguments while the former one only uses two. However, an index scan
* {@code INDEX SCAN(i1, a = [5, 5], b = [10, 10])} is not created (and yielded) if there we also
* have a choice to plan {@code INDEX SCAN(i2, b = [10, 10], a = [5, 5], c = ["Guten", "Morgen"])} as that
* index {@code i2} has a higher coverage compared to {@code i1} <em>and</em> all bound arguments in the scan
* over {@code i2} are also bound in the scan over {@code i1}.
* <li>
* We will only create intersections of scans if we can already establish that the logical intersection
* can be implemented by a {@link com.apple.foundationdb.record.query.plan.plans.RecordQueryIntersectionPlan}.
* That requires that the legs of the intersection are compatibly ordered <em>and</em> that that ordering follows
* a potentially required ordering.
* </li>
* </ul>
*
* @param call the call associated with this planner rule execution
*/
@Override
@SuppressWarnings("java:S135")
public void onMatch(@Nonnull PlannerRuleCall call) {
final PlannerBindings bindings = call.getBindings();
final List<? extends PartialMatch> completeMatches = bindings.getAll(getCompleteMatchMatcher());
final R expression = bindings.get(getExpressionMatcher());
//
if (completeMatches.isEmpty()) {
return;
}
//
// return if there is no pre-determined interesting ordering
//
final Optional<Set<RequestedOrdering>> requestedOrderingsOptional = call.getInterestingProperty(OrderingAttribute.ORDERING);
if (requestedOrderingsOptional.isEmpty()) {
return;
}
final Set<RequestedOrdering> requestedOrderings = requestedOrderingsOptional.get();
//
// group matches by candidates
//
final LinkedHashMap<MatchCandidate, ? extends ImmutableList<? extends PartialMatch>> completeMatchMap = completeMatches.stream().collect(Collectors.groupingBy(PartialMatch::getMatchCandidate, LinkedHashMap::new, ImmutableList.toImmutableList()));
// find the best match for a candidate as there may be more than one due to partial matching
final ImmutableSet<PartialMatch> maximumCoverageMatchPerCandidate = completeMatchMap.entrySet().stream().flatMap(entry -> {
final List<? extends PartialMatch> completeMatchesForCandidate = entry.getValue();
final Optional<? extends PartialMatch> bestMatchForCandidateOptional = completeMatchesForCandidate.stream().max(Comparator.comparing(PartialMatch::getNumBoundParameterPrefix));
return bestMatchForCandidateOptional.map(Stream::of).orElse(Stream.empty());
}).collect(ImmutableSet.toImmutableSet());
final List<PartialMatch> bestMaximumCoverageMatches = maximumCoverageMatches(maximumCoverageMatchPerCandidate, requestedOrderings);
if (bestMaximumCoverageMatches.isEmpty()) {
return;
}
// create scans for all best matches
final Map<PartialMatch, RelationalExpression> bestMatchToExpressionMap = createScansForMatches(bestMaximumCoverageMatches);
final ExpressionRef<RelationalExpression> toBeInjectedReference = GroupExpressionRef.empty();
// create single scan accesses
for (final PartialMatch bestMatch : bestMaximumCoverageMatches) {
final RelationalExpression dataAccessAndCompensationExpression = compensateSingleDataAccess(bestMatch, bestMatchToExpressionMap.get(bestMatch));
toBeInjectedReference.insert(dataAccessAndCompensationExpression);
}
final Map<PartialMatch, RelationalExpression> bestMatchToDistinctExpressionMap = distinctMatchToScanMap(bestMatchToExpressionMap);
@Nullable final KeyExpression commonPrimaryKey = call.getContext().getCommonPrimaryKey();
if (commonPrimaryKey != null) {
final var commonPrimaryKeyParts = commonPrimaryKey.normalizeKeyForPositions();
final var boundPartitions = Lists.<List<PartialMatch>>newArrayList();
// create intersections for all n choose k partitions from k = 2 .. n
IntStream.range(2, bestMaximumCoverageMatches.size() + 1).mapToObj(k -> ChooseK.chooseK(bestMaximumCoverageMatches, k)).flatMap(iterable -> StreamSupport.stream(iterable.spliterator(), false)).forEach(boundPartitions::add);
boundPartitions.stream().flatMap(partition -> createIntersectionAndCompensation(commonPrimaryKeyParts, bestMatchToDistinctExpressionMap, partition, requestedOrderings).stream()).forEach(toBeInjectedReference::insert);
}
call.yield(inject(expression, completeMatches, toBeInjectedReference));
}
use of com.apple.foundationdb.record.query.plan.temp.Ordering in project fdb-record-layer by FoundationDB.
the class AbstractDataAccessRule method createIntersectionAndCompensation.
/**
* Private helper method to plan an intersection and subsequently compensate it using the partial match structures
* kept for all participating data accesses.
* Planning the data access and its compensation for a given match is a two-step approach as we compute
* the compensation for intersections by intersecting the {@link Compensation} for the single data accesses first
* before using the resulting {@link Compensation} to compute the compensating expression for the entire
* intersection.
* @param commonPrimaryKeyParts normalized common primary key
* @param matchToExpressionMap a map from match to single data access expression
* @param partition a partition (i.e. a list of {@link PartialMatch}es that the caller would like to compute
* and intersected data access for
* @param requestedOrderings a set of ordering that have been requested by consuming expressions/plan operators
* @return a optional containing a new {@link RelationalExpression} that represents the data access and its
* compensation, {@code Optional.empty()} if this method was unable to compute the intersection expression
*/
@Nonnull
private static List<RelationalExpression> createIntersectionAndCompensation(@Nonnull final List<KeyExpression> commonPrimaryKeyParts, @Nonnull final Map<PartialMatch, RelationalExpression> matchToExpressionMap, @Nonnull final List<PartialMatch> partition, @Nonnull final Set<RequestedOrdering> requestedOrderings) {
final var expressionsBuilder = ImmutableList.<RelationalExpression>builder();
final var orderingPartialOrder = intersectionOrdering(partition);
final ImmutableSet<BoundKeyPart> equalityBoundKeyParts = partition.stream().map(partialMatch -> partialMatch.getMatchInfo().getBoundKeyParts()).flatMap(boundOrderingKeyParts -> boundOrderingKeyParts.stream().filter(boundOrderingKey -> boundOrderingKey.getComparisonRangeType() == ComparisonRange.Type.EQUALITY)).collect(ImmutableSet.toImmutableSet());
for (final var requestedOrdering : requestedOrderings) {
final var satisfyingOrderingPartsOptional = Ordering.satisfiesKeyPartsOrdering(orderingPartialOrder, requestedOrdering.getOrderingKeyParts(), BoundKeyPart::getKeyPart);
final var comparisonKeyOptional = satisfyingOrderingPartsOptional.map(parts -> parts.stream().filter(part -> !equalityBoundKeyParts.contains(part)).collect(ImmutableList.toImmutableList())).flatMap(parts -> comparisonKey(commonPrimaryKeyParts, equalityBoundKeyParts, parts));
if (comparisonKeyOptional.isEmpty()) {
continue;
}
final KeyExpression comparisonKey = comparisonKeyOptional.get();
final var compensation = partition.stream().map(partialMatch -> partialMatch.compensate(partialMatch.getBoundParameterPrefixMap())).reduce(Compensation.impossibleCompensation(), Compensation::intersect);
final ImmutableList<RelationalExpression> scans = partition.stream().map(partialMatch -> Objects.requireNonNull(matchToExpressionMap.get(partialMatch))).collect(ImmutableList.toImmutableList());
final var logicalIntersectionExpression = LogicalIntersectionExpression.from(scans, comparisonKey);
final var compensatedIntersection = compensation.isNeeded() ? compensation.apply(GroupExpressionRef.of(logicalIntersectionExpression)) : logicalIntersectionExpression;
expressionsBuilder.add(compensatedIntersection);
}
return expressionsBuilder.build();
}
use of com.apple.foundationdb.record.query.plan.temp.Ordering in project fdb-record-layer by FoundationDB.
the class OrderingProperty method deriveForInJoinFromOrderings.
@SuppressWarnings("java:S135")
public static Optional<Ordering> deriveForInJoinFromOrderings(@Nonnull final List<Optional<Ordering>> orderingOptionals, @Nonnull final RecordQueryInJoinPlan inJoinPlan) {
final Optional<Ordering> childOrderingOptional = Iterables.getOnlyElement(orderingOptionals);
if (childOrderingOptional.isPresent()) {
final var childOrdering = childOrderingOptional.get();
final var equalityBoundKeyMap = childOrdering.getEqualityBoundKeyMap();
final var inSource = inJoinPlan.getInSource();
final CorrelationIdentifier inAlias = inJoinPlan.getInAlias();
final SetMultimap<KeyExpression, Comparisons.Comparison> resultEqualityBoundKeyMap = HashMultimap.create(equalityBoundKeyMap);
KeyExpression inKeyExpression = null;
for (final var entry : equalityBoundKeyMap.entries()) {
// TODO we only look for the first entry that matches. That is enough for the in-to-join case,
// however, it is possible that more than one different key expressions are equality-bound
// by this in. That would constitute to more than one concurrent order which we cannot
// express at the moment (we need the PartialOrder approach for that).
final var comparison = entry.getValue();
final var correlatedTo = comparison.getCorrelatedTo();
if (correlatedTo.size() != 1) {
continue;
}
if (inAlias.equals(Iterables.getOnlyElement(correlatedTo))) {
inKeyExpression = entry.getKey();
resultEqualityBoundKeyMap.removeAll(inKeyExpression);
break;
}
}
if (inKeyExpression == null || !inSource.isSorted()) {
//
return Optional.of(new Ordering(resultEqualityBoundKeyMap, ImmutableList.of(), false));
}
//
// Prepend the existing order with the key expression we just found.
//
final var resultOrderingKeyPartsBuilder = ImmutableList.<KeyPart>builder();
resultOrderingKeyPartsBuilder.add(KeyPart.of(inKeyExpression, inSource.isReverse()));
resultOrderingKeyPartsBuilder.addAll(childOrdering.getOrderingKeyParts());
return Optional.of(new Ordering(resultEqualityBoundKeyMap, resultOrderingKeyPartsBuilder.build(), childOrdering.isDistinct()));
} else {
return Optional.empty();
}
}
use of com.apple.foundationdb.record.query.plan.temp.Ordering in project fdb-record-layer by FoundationDB.
the class OrderingProperty method fromKeyAndScanComparisons.
@Nonnull
private static Optional<Ordering> fromKeyAndScanComparisons(@Nullable final KeyExpression keyExpression, @Nonnull final ScanComparisons scanComparisons, final boolean isReverse, final boolean isDistinct) {
if (keyExpression == null) {
return Optional.empty();
}
final ImmutableSetMultimap.Builder<KeyExpression, Comparisons.Comparison> equalityBoundKeyMapBuilder = ImmutableSetMultimap.builder();
final List<KeyExpression> normalizedKeyExpressions = keyExpression.normalizeKeyForPositions();
final List<Comparisons.Comparison> equalityComparisons = scanComparisons.getEqualityComparisons();
for (int i = 0; i < equalityComparisons.size(); i++) {
final KeyExpression normalizedKeyExpression = normalizedKeyExpressions.get(i);
final Comparisons.Comparison comparison = equalityComparisons.get(i);
equalityBoundKeyMapBuilder.put(normalizedKeyExpression, comparison);
}
final ImmutableList.Builder<KeyPart> result = ImmutableList.builder();
for (int i = scanComparisons.getEqualitySize(); i < normalizedKeyExpressions.size(); i++) {
final KeyExpression currentKeyExpression = normalizedKeyExpressions.get(i);
//
// Note that it is not really important here if the keyExpression can be normalized in a lossless way
// or not. A key expression containing repeated fields is sort-compatible with its normalized key
// expression. We used to refuse to compute the sort order in the presence of repeats, however,
// I think that restriction can be relaxed.
//
result.add(KeyPart.of(currentKeyExpression, isReverse));
}
return Optional.of(new Ordering(equalityBoundKeyMapBuilder.build(), result.build(), isDistinct));
}
use of com.apple.foundationdb.record.query.plan.temp.Ordering in project fdb-record-layer by FoundationDB.
the class OrderingProperty method deriveForInUnionFromOrderings.
public static Optional<Ordering> deriveForInUnionFromOrderings(@Nonnull final List<Optional<Ordering>> orderingOptionals, @Nonnull final RecordQueryInUnionPlan inUnionPlan) {
final Optional<Ordering> childOrderingOptional = Iterables.getOnlyElement(orderingOptionals);
if (childOrderingOptional.isPresent()) {
final var childOrdering = childOrderingOptional.get();
final SetMultimap<KeyExpression, Comparisons.Comparison> equalityBoundKeyMap = childOrdering.getEqualityBoundKeyMap();
final KeyExpression comparisonKey = inUnionPlan.getComparisonKey();
final SetMultimap<KeyExpression, Comparisons.Comparison> resultEqualityBoundKeyMap = HashMultimap.create(equalityBoundKeyMap);
final ImmutableList.Builder<KeyPart> resultKeyPartBuilder = ImmutableList.builder();
final List<KeyExpression> normalizedComparisonKeys = comparisonKey.normalizeKeyForPositions();
for (final KeyExpression normalizedKeyExpression : normalizedComparisonKeys) {
resultKeyPartBuilder.add(KeyPart.of(normalizedKeyExpression, inUnionPlan.isReverse()));
}
final var sourceAliases = inUnionPlan.getInSources().stream().map(inSource -> CorrelationIdentifier.of(CORRELATION.identifier(inSource.getBindingName()))).collect(ImmutableSet.toImmutableSet());
for (final var entry : equalityBoundKeyMap.entries()) {
final var correlatedTo = entry.getValue().getCorrelatedTo();
if (correlatedTo.stream().anyMatch(sourceAliases::contains)) {
resultEqualityBoundKeyMap.removeAll(entry.getKey());
}
}
return Optional.of(new Ordering(resultEqualityBoundKeyMap, resultKeyPartBuilder.build(), childOrdering.isDistinct()));
} else {
return Optional.empty();
}
}
Aggregations