Search in sources :

Example 1 with KeyWithValueExpression

use of com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression in project fdb-record-layer by FoundationDB.

the class StandardIndexMaintainer method evaluateIndex.

/**
 * Apply the key and value expressions to a <code>record</code>.
 * @param <M> the message type of the record
 * @param record the record from which the index will extract its key and value
 * @return a list of index keys and values
 */
@Override
public <M extends Message> List<IndexEntry> evaluateIndex(@Nonnull FDBRecord<M> record) {
    final KeyExpression rootExpression = state.index.getRootExpression();
    final List<Key.Evaluated> indexKeys = rootExpression.evaluate(record);
    // so we have to tease them apart.
    if (rootExpression instanceof KeyWithValueExpression) {
        final KeyWithValueExpression keyWithValueExpression = (KeyWithValueExpression) rootExpression;
        return indexKeys.stream().map(key -> new IndexEntry(state.index, keyWithValueExpression.getKey(key), keyWithValueExpression.getValue(key))).collect(Collectors.toList());
    }
    return indexKeys.stream().map(key -> new IndexEntry(state.index, key)).collect(Collectors.toList());
}
Also used : IndexEntry(com.apple.foundationdb.record.IndexEntry) LogMessageKeys(com.apple.foundationdb.record.logging.LogMessageKeys) IndexMaintainerState(com.apple.foundationdb.record.provider.foundationdb.IndexMaintainerState) FDBRecord(com.apple.foundationdb.record.provider.foundationdb.FDBRecord) LoggerFactory(org.slf4j.LoggerFactory) Subspace(com.apple.foundationdb.subspace.Subspace) Transaction(com.apple.foundationdb.Transaction) IndexScanType(com.apple.foundationdb.record.IndexScanType) Tuple(com.apple.foundationdb.tuple.Tuple) Range(com.apple.foundationdb.Range) KeyValueLogMessage(com.apple.foundationdb.record.logging.KeyValueLogMessage) Pair(org.apache.commons.lang3.tuple.Pair) RecordCoreException(com.apple.foundationdb.record.RecordCoreException) PipelineOperation(com.apple.foundationdb.record.PipelineOperation) RecordIndexUniquenessViolation(com.apple.foundationdb.record.RecordIndexUniquenessViolation) GroupingKeyExpression(com.apple.foundationdb.record.metadata.expressions.GroupingKeyExpression) FDBExceptions(com.apple.foundationdb.record.provider.foundationdb.FDBExceptions) FDBRecordStoreBase(com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase) QueryToKeyMatcher(com.apple.foundationdb.record.query.QueryToKeyMatcher) ByteArrayUtil(com.apple.foundationdb.tuple.ByteArrayUtil) FDBIndexableRecord(com.apple.foundationdb.record.provider.foundationdb.FDBIndexableRecord) KeyExpression(com.apple.foundationdb.record.metadata.expressions.KeyExpression) KeyValue(com.apple.foundationdb.KeyValue) FDBStoreTimer(com.apple.foundationdb.record.provider.foundationdb.FDBStoreTimer) Collection(java.util.Collection) Collectors(java.util.stream.Collectors) TupleRange(com.apple.foundationdb.record.TupleRange) Objects(java.util.Objects) KeyValueCursor(com.apple.foundationdb.record.provider.foundationdb.KeyValueCursor) List(java.util.List) EvaluationContext(com.apple.foundationdb.record.EvaluationContext) TupleHelpers(com.apple.foundationdb.tuple.TupleHelpers) API(com.apple.foundationdb.annotation.API) SplitHelper.unpackKey(com.apple.foundationdb.record.provider.foundationdb.SplitHelper.unpackKey) IndexMaintainer(com.apple.foundationdb.record.provider.foundationdb.IndexMaintainer) IndexMaintenanceFilter(com.apple.foundationdb.record.provider.foundationdb.IndexMaintenanceFilter) IndexAggregateFunction(com.apple.foundationdb.record.metadata.IndexAggregateFunction) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression) IndexOperation(com.apple.foundationdb.record.provider.foundationdb.IndexOperation) CompletableFuture(java.util.concurrent.CompletableFuture) AsyncUtil(com.apple.foundationdb.async.AsyncUtil) RangeSet(com.apple.foundationdb.async.RangeSet) Function(java.util.function.Function) CursorStreamingMode(com.apple.foundationdb.record.CursorStreamingMode) ArrayList(java.util.ArrayList) Key(com.apple.foundationdb.record.metadata.Key) ExecuteProperties(com.apple.foundationdb.record.ExecuteProperties) ScanProperties(com.apple.foundationdb.record.ScanProperties) IndexRecordFunction(com.apple.foundationdb.record.metadata.IndexRecordFunction) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) IndexOperationResult(com.apple.foundationdb.record.provider.foundationdb.IndexOperationResult) MoreAsyncUtil(com.apple.foundationdb.async.MoreAsyncUtil) IsolationLevel(com.apple.foundationdb.record.IsolationLevel) Logger(org.slf4j.Logger) Executor(java.util.concurrent.Executor) RecordType(com.apple.foundationdb.record.metadata.RecordType) AsyncIterable(com.apple.foundationdb.async.AsyncIterable) Message(com.google.protobuf.Message) RecordCursor(com.apple.foundationdb.record.RecordCursor) Collections(java.util.Collections) GroupingKeyExpression(com.apple.foundationdb.record.metadata.expressions.GroupingKeyExpression) KeyExpression(com.apple.foundationdb.record.metadata.expressions.KeyExpression) IndexEntry(com.apple.foundationdb.record.IndexEntry) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression)

Example 2 with KeyWithValueExpression

use of com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression in project fdb-record-layer by FoundationDB.

the class FDBOrQueryToUnionTest method deferFetchOnUnionWithInnerFilter.

@DualPlannerTest
void deferFetchOnUnionWithInnerFilter() throws Exception {
    complexQuerySetup(metaData -> {
        // We don't prefer covering indexes over other indexes yet.
        metaData.removeIndex("MySimpleRecord$num_value_3_indexed");
        metaData.addIndex("MySimpleRecord", "coveringIndex", new KeyWithValueExpression(concat(field("num_value_2"), field("num_value_3_indexed")), 1));
    });
    RecordQuery query = RecordQuery.newBuilder().setRecordType("MySimpleRecord").setFilter(Query.or(Query.field("str_value_indexed").startsWith("foo"), Query.and(Query.field("num_value_2").greaterThanOrEquals(2), Query.field("num_value_2").lessThanOrEquals(4)), Query.and(Query.field("num_value_3_indexed").lessThanOrEquals(18), Query.field("num_value_2").greaterThanOrEquals(26)))).build();
    setDeferFetchAfterUnionAndIntersection(true);
    RecordQueryPlan plan = planner.plan(query);
    if (planner instanceof RecordQueryPlanner) {
        final BindingMatcher<? extends RecordQueryPlan> planMatcher = fetchFromPartialRecordPlan(unorderedPrimaryKeyDistinctPlan(unorderedUnionPlan(coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("MySimpleRecord$str_value_indexed")).and(scanComparisons(range("{[foo],[foo]}"))))), coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("coveringIndex")).and(scanComparisons(range("[[2],[4]]"))))), filterPlan(coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("coveringIndex")).and(scanComparisons(range("[[26],>")))))).where(queryComponents(exactly(equalsObject(Query.field("num_value_3_indexed").lessThanOrEquals(18))))))));
        assertMatchesExactly(plan, planMatcher);
        assertEquals(-1829743477, plan.planHash(PlanHashable.PlanHashKind.LEGACY));
        assertEquals(-1168128533, plan.planHash(PlanHashable.PlanHashKind.FOR_CONTINUATION));
        assertEquals(1840217393, plan.planHash(PlanHashable.PlanHashKind.STRUCTURAL_WITHOUT_LITERALS));
    } else {
        final BindingMatcher<? extends RecordQueryPlan> planMatcher = fetchFromPartialRecordPlan(unorderedPrimaryKeyDistinctPlan(unorderedUnionPlan(coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("MySimpleRecord$str_value_indexed")).and(scanComparisons(range("{[foo],[foo]}"))))), coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("coveringIndex")).and(scanComparisons(range("[[2],[4]]"))))), predicatesFilterPlan(coveringIndexPlan().where(indexPlanOf(indexPlan().where(indexName("coveringIndex")).and(scanComparisons(range("[[26],>")))))).where(predicates(only(valuePredicate(fieldValue("num_value_3_indexed"), new Comparisons.SimpleComparison(Comparisons.Type.LESS_THAN_OR_EQUALS, 18))))))));
        assertMatchesExactly(plan, planMatcher);
        assertEquals(331039648, plan.planHash(PlanHashable.PlanHashKind.LEGACY));
        assertEquals(-1539052743, plan.planHash(PlanHashable.PlanHashKind.FOR_CONTINUATION));
        assertEquals(1469293183, plan.planHash(PlanHashable.PlanHashKind.STRUCTURAL_WITHOUT_LITERALS));
    }
}
Also used : RecordQueryPlan(com.apple.foundationdb.record.query.plan.plans.RecordQueryPlan) IndexScanComparisons(com.apple.foundationdb.record.provider.foundationdb.IndexScanComparisons) RecordQueryPlanMatchers.scanComparisons(com.apple.foundationdb.record.query.plan.temp.matchers.RecordQueryPlanMatchers.scanComparisons) Comparisons(com.apple.foundationdb.record.query.expressions.Comparisons) RecordQueryPlanner(com.apple.foundationdb.record.query.plan.RecordQueryPlanner) RecordQuery(com.apple.foundationdb.record.query.RecordQuery) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression)

Example 3 with KeyWithValueExpression

use of com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression in project fdb-record-layer by FoundationDB.

the class GeophileIndexMaintainer method getSpatialFunction.

// If the bottom-right child is GeophileSpatialFunctionKeyExpression, return it. Else error.
@Nonnull
static GeophileSpatialFunctionKeyExpression getSpatialFunction(@Nonnull Index index) {
    KeyExpression rootKey = index.getRootExpression();
    if (rootKey instanceof KeyWithValueExpression) {
        rootKey = ((KeyWithValueExpression) rootKey).getKeyExpression();
    }
    final List<KeyExpression> components = rootKey.normalizeKeyForPositions();
    final KeyExpression rightComponent = components.get(components.size() - 1);
    KeyExpression bottomComponent = rightComponent;
    while (true) {
        if (bottomComponent instanceof GeophileSpatialFunctionKeyExpression) {
            return (GeophileSpatialFunctionKeyExpression) bottomComponent;
        }
        if (bottomComponent instanceof KeyExpressionWithChild) {
            bottomComponent = ((KeyExpressionWithChild) bottomComponent).getChild();
            continue;
        }
        throw new KeyExpression.InvalidExpressionException(String.format("need spatial key expression for %s index", index.getType()), LogMessageKeys.INDEX_NAME, index.getName(), LogMessageKeys.INDEX_KEY, index.getRootExpression());
    }
}
Also used : KeyExpression(com.apple.foundationdb.record.metadata.expressions.KeyExpression) KeyExpressionWithChild(com.apple.foundationdb.record.metadata.expressions.KeyExpressionWithChild) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression) Nonnull(javax.annotation.Nonnull)

Example 4 with KeyWithValueExpression

use of com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression in project fdb-record-layer by FoundationDB.

the class RecordQueryPlanner method indexKeyExpressionForPlan.

// Get the key expression for the index entries of the given index, which includes primary key fields for normal indexes.
private KeyExpression indexKeyExpressionForPlan(@Nullable KeyExpression commonPrimaryKey, @Nonnull Index index) {
    KeyExpression indexKeyExpression = index.getRootExpression();
    if (indexKeyExpression instanceof KeyWithValueExpression) {
        indexKeyExpression = ((KeyWithValueExpression) indexKeyExpression).getKeyExpression();
    }
    if (commonPrimaryKey != null && indexTypes.getValueTypes().contains(index.getType()) && configuration.shouldUseFullKeyForValueIndex()) {
        final List<KeyExpression> keys = new ArrayList<>(commonPrimaryKey.normalizeKeyForPositions());
        index.trimPrimaryKey(keys);
        if (!keys.isEmpty()) {
            keys.add(0, indexKeyExpression);
            indexKeyExpression = Key.Expressions.concat(keys);
        }
    }
    return indexKeyExpression;
}
Also used : RecordTypeKeyExpression(com.apple.foundationdb.record.metadata.expressions.RecordTypeKeyExpression) VersionKeyExpression(com.apple.foundationdb.record.metadata.expressions.VersionKeyExpression) GroupingKeyExpression(com.apple.foundationdb.record.metadata.expressions.GroupingKeyExpression) KeyExpression(com.apple.foundationdb.record.metadata.expressions.KeyExpression) FieldKeyExpression(com.apple.foundationdb.record.metadata.expressions.FieldKeyExpression) NestingKeyExpression(com.apple.foundationdb.record.metadata.expressions.NestingKeyExpression) EmptyKeyExpression(com.apple.foundationdb.record.metadata.expressions.EmptyKeyExpression) ThenKeyExpression(com.apple.foundationdb.record.metadata.expressions.ThenKeyExpression) ArrayList(java.util.ArrayList) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression)

Example 5 with KeyWithValueExpression

use of com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression in project fdb-record-layer by FoundationDB.

the class ValueIndexExpansionVisitor method expand.

@Nonnull
@Override
public MatchCandidate expand(@Nonnull final Quantifier.ForEach baseQuantifier, @Nullable final KeyExpression primaryKey, final boolean isReverse) {
    Debugger.updateIndex(ValueComparisonRangePredicate.Placeholder.class, old -> 0);
    final ImmutableList.Builder<GraphExpansion> allExpansionsBuilder = ImmutableList.builder();
    // add the value for the flow of records
    final QuantifiedColumnValue recordValue = QuantifiedColumnValue.of(baseQuantifier.getAlias(), 0);
    allExpansionsBuilder.add(GraphExpansion.ofResultValueAndQuantifier(recordValue, baseQuantifier));
    KeyExpression rootExpression = index.getRootExpression();
    final int keyValueSplitPoint;
    if (rootExpression instanceof KeyWithValueExpression) {
        final KeyWithValueExpression keyWithValueExpression = (KeyWithValueExpression) rootExpression;
        keyValueSplitPoint = keyWithValueExpression.getSplitPoint();
        rootExpression = keyWithValueExpression.getInnerKey();
    } else {
        keyValueSplitPoint = -1;
    }
    final List<Value> keyValues = Lists.newArrayList();
    final List<Value> valueValues = Lists.newArrayList();
    final VisitorState initialState = VisitorState.of(keyValues, valueValues, baseQuantifier.getAlias(), ImmutableList.of(), keyValueSplitPoint, 0);
    final GraphExpansion keyValueExpansion = pop(rootExpression.expand(push(initialState)));
    allExpansionsBuilder.add(keyValueExpansion);
    final int keySize = keyValues.size();
    if (primaryKey != null) {
        // unfortunately we must copy as the returned list is not guaranteed to be mutable which is needed for the
        // trimPrimaryKey() function as it is causing a side-effect
        final List<KeyExpression> trimmedPrimaryKeys = Lists.newArrayList(primaryKey.normalizeKeyForPositions());
        index.trimPrimaryKey(trimmedPrimaryKeys);
        for (int i = 0; i < trimmedPrimaryKeys.size(); i++) {
            final KeyExpression primaryKeyPart = trimmedPrimaryKeys.get(i);
            final VisitorState initialStateForKeyPart = VisitorState.of(keyValues, Lists.newArrayList(), baseQuantifier.getAlias(), ImmutableList.of(), -1, keySize + i);
            final GraphExpansion primaryKeyPartExpansion = pop(primaryKeyPart.expand(push(initialStateForKeyPart)));
            allExpansionsBuilder.add(primaryKeyPartExpansion);
        }
    }
    final GraphExpansion completeExpansion = GraphExpansion.ofOthers(allExpansionsBuilder.build());
    final List<CorrelationIdentifier> parameters = completeExpansion.getPlaceholderAliases();
    final MatchableSortExpression matchableSortExpression = new MatchableSortExpression(parameters, isReverse, completeExpansion.buildSelect());
    return new ValueIndexScanMatchCandidate(index, recordTypes, ExpressionRefTraversal.withRoot(GroupExpressionRef.of(matchableSortExpression)), parameters, recordValue, keyValues, valueValues, fullKey(index, primaryKey));
}
Also used : ImmutableList(com.google.common.collect.ImmutableList) KeyExpression(com.apple.foundationdb.record.metadata.expressions.KeyExpression) ValueComparisonRangePredicate(com.apple.foundationdb.record.query.predicates.ValueComparisonRangePredicate) MatchableSortExpression(com.apple.foundationdb.record.query.plan.temp.expressions.MatchableSortExpression) QuantifiedColumnValue(com.apple.foundationdb.record.query.predicates.QuantifiedColumnValue) Value(com.apple.foundationdb.record.query.predicates.Value) QuantifiedColumnValue(com.apple.foundationdb.record.query.predicates.QuantifiedColumnValue) KeyWithValueExpression(com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression) Nonnull(javax.annotation.Nonnull)

Aggregations

KeyWithValueExpression (com.apple.foundationdb.record.metadata.expressions.KeyWithValueExpression)5 KeyExpression (com.apple.foundationdb.record.metadata.expressions.KeyExpression)4 Nonnull (javax.annotation.Nonnull)3 GroupingKeyExpression (com.apple.foundationdb.record.metadata.expressions.GroupingKeyExpression)2 KeyValue (com.apple.foundationdb.KeyValue)1 Range (com.apple.foundationdb.Range)1 Transaction (com.apple.foundationdb.Transaction)1 API (com.apple.foundationdb.annotation.API)1 AsyncIterable (com.apple.foundationdb.async.AsyncIterable)1 AsyncUtil (com.apple.foundationdb.async.AsyncUtil)1 MoreAsyncUtil (com.apple.foundationdb.async.MoreAsyncUtil)1 RangeSet (com.apple.foundationdb.async.RangeSet)1 CursorStreamingMode (com.apple.foundationdb.record.CursorStreamingMode)1 EvaluationContext (com.apple.foundationdb.record.EvaluationContext)1 ExecuteProperties (com.apple.foundationdb.record.ExecuteProperties)1 IndexEntry (com.apple.foundationdb.record.IndexEntry)1 IndexScanType (com.apple.foundationdb.record.IndexScanType)1 IsolationLevel (com.apple.foundationdb.record.IsolationLevel)1 PipelineOperation (com.apple.foundationdb.record.PipelineOperation)1 RecordCoreException (com.apple.foundationdb.record.RecordCoreException)1