Search in sources :

Example 1 with TransitivePropertyVisitor

use of org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor in project incubator-rya by apache.

the class QueryRuleset method setRules.

/**
 * Extract the rules from the query string, applying inference rules if configured to.
 * @throws QueryRulesetException if the parsed query can't be parsed and translated into valid rules.
 */
private void setRules() throws QueryRulesetException {
    final ParsedTupleQuery ptq;
    final TupleExpr te;
    try {
        ptq = QueryParserUtil.parseTupleQuery(QueryLanguage.SPARQL, query, null);
    } catch (UnsupportedQueryLanguageException | MalformedQueryException e) {
        throw new QueryRulesetException("Error parsing query:\n" + query, e);
    }
    te = ptq.getTupleExpr();
    // Before converting to rules (and renaming variables), validate that no statement patterns
    // consist of only variables (this would result in a rule  that matches every triple).
    // Needs to be done before inference, since inference rules may create such statement patterns
    // that are OK because they won'd be converted to rules directly.
    te.visit(new QueryModelVisitorBase<QueryRulesetException>() {

        @Override
        public void meet(final StatementPattern node) throws QueryRulesetException {
            if (!(node.getSubjectVar().hasValue() || node.getPredicateVar().hasValue() || node.getObjectVar().hasValue())) {
                throw new QueryRulesetException("Statement pattern with no constants would match every statement:\n" + node + "\nFrom parsed query:\n" + te);
            }
        }
    });
    // Apply inference, if applicable
    if (conf != null && conf.isInfer()) {
        RdfCloudTripleStore store = null;
        try {
            log.info("Applying inference rules");
            store = (RdfCloudTripleStore) RyaSailFactory.getInstance(conf);
            final InferenceEngine inferenceEngine = store.getInferenceEngine();
            // Apply in same order as query evaluation:
            te.visit(new TransitivePropertyVisitor(conf, inferenceEngine));
            te.visit(new SymmetricPropertyVisitor(conf, inferenceEngine));
            te.visit(new InverseOfVisitor(conf, inferenceEngine));
            te.visit(new SubPropertyOfVisitor(conf, inferenceEngine));
            te.visit(new SubClassOfVisitor(conf, inferenceEngine));
            te.visit(new SameAsVisitor(conf, inferenceEngine));
            log.info("Query after inference:\n");
            for (final String line : te.toString().split("\n")) {
                log.info("\t" + line);
            }
        } catch (final Exception e) {
            throw new QueryRulesetException("Error applying inference to parsed query:\n" + te, e);
        } finally {
            if (store != null) {
                try {
                    store.shutDown();
                } catch (final SailException e) {
                    log.error("Error shutting down Sail after applying inference", e);
                }
            }
        }
    }
    // Extract the StatementPatterns and Filters and turn them into rules:
    final RulesetVisitor rv = new RulesetVisitor();
    try {
        te.visit(rv);
        rv.addSchema();
    } catch (final QueryRulesetException e) {
        throw new QueryRulesetException("Error extracting rules from parsed query:\n" + te, e);
    }
    for (final CopyRule candidateRule : rv.rules) {
        boolean unique = true;
        for (final CopyRule otherRule : rv.rules) {
            if (!candidateRule.equals(otherRule) && otherRule.isGeneralizationOf(candidateRule)) {
                unique = false;
                break;
            }
        }
        if (unique) {
            rules.add(candidateRule);
        }
    }
}
Also used : SameAsVisitor(org.apache.rya.rdftriplestore.inference.SameAsVisitor) RdfCloudTripleStore(org.apache.rya.rdftriplestore.RdfCloudTripleStore) SymmetricPropertyVisitor(org.apache.rya.rdftriplestore.inference.SymmetricPropertyVisitor) SubClassOfVisitor(org.apache.rya.rdftriplestore.inference.SubClassOfVisitor) SubPropertyOfVisitor(org.apache.rya.rdftriplestore.inference.SubPropertyOfVisitor) SailException(org.openrdf.sail.SailException) TupleExpr(org.openrdf.query.algebra.TupleExpr) SailException(org.openrdf.sail.SailException) UnsupportedQueryLanguageException(org.openrdf.query.UnsupportedQueryLanguageException) QueryRulesetException(org.apache.rya.accumulo.mr.merge.util.QueryRuleset.QueryRulesetException) MalformedQueryException(org.openrdf.query.MalformedQueryException) IOException(java.io.IOException) FixedStatementPattern(org.apache.rya.rdftriplestore.utils.FixedStatementPattern) StatementPattern(org.openrdf.query.algebra.StatementPattern) InferenceEngine(org.apache.rya.rdftriplestore.inference.InferenceEngine) MalformedQueryException(org.openrdf.query.MalformedQueryException) TransitivePropertyVisitor(org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor) InverseOfVisitor(org.apache.rya.rdftriplestore.inference.InverseOfVisitor) QueryRulesetException(org.apache.rya.accumulo.mr.merge.util.QueryRuleset.QueryRulesetException) UnsupportedQueryLanguageException(org.openrdf.query.UnsupportedQueryLanguageException) ParsedTupleQuery(org.openrdf.query.parser.ParsedTupleQuery)

Example 2 with TransitivePropertyVisitor

use of org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor in project incubator-rya by apache.

the class RdfCloudTripleStoreConnection method evaluateInternal.

@Override
protected CloseableIteration<? extends BindingSet, QueryEvaluationException> evaluateInternal(TupleExpr tupleExpr, final Dataset dataset, BindingSet bindings, final boolean flag) throws SailException {
    verifyIsOpen();
    logger.trace("Incoming query model:\n{}", tupleExpr.toString());
    if (provenanceCollector != null) {
        try {
            provenanceCollector.recordQuery(tupleExpr.toString());
        } catch (final ProvenanceCollectionException e) {
            logger.trace("Provenance failed to record query.", e);
        }
    }
    tupleExpr = tupleExpr.clone();
    final C queryConf = (C) store.getConf().clone();
    if (queryConf == null) {
        // Should not happen, but this is better than a null dereference error.
        throw new SailException("Cloning store.getConf() returned null, aborting.");
    }
    if (bindings != null) {
        final Binding dispPlan = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_QUERYPLAN_FLAG);
        if (dispPlan != null) {
            queryConf.setDisplayQueryPlan(Boolean.parseBoolean(dispPlan.getValue().stringValue()));
        }
        final Binding authBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_QUERY_AUTH);
        if (authBinding != null) {
            queryConf.setAuths(authBinding.getValue().stringValue().split(","));
        }
        final Binding ttlBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_TTL);
        if (ttlBinding != null) {
            queryConf.setTtl(Long.valueOf(ttlBinding.getValue().stringValue()));
        }
        final Binding startTimeBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_STARTTIME);
        if (startTimeBinding != null) {
            queryConf.setStartTime(Long.valueOf(startTimeBinding.getValue().stringValue()));
        }
        final Binding performantBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_PERFORMANT);
        if (performantBinding != null) {
            queryConf.setBoolean(RdfCloudTripleStoreConfiguration.CONF_PERFORMANT, Boolean.parseBoolean(performantBinding.getValue().stringValue()));
        }
        final Binding inferBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_INFER);
        if (inferBinding != null) {
            queryConf.setInfer(Boolean.parseBoolean(inferBinding.getValue().stringValue()));
        }
        final Binding useStatsBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_USE_STATS);
        if (useStatsBinding != null) {
            queryConf.setUseStats(Boolean.parseBoolean(useStatsBinding.getValue().stringValue()));
        }
        final Binding offsetBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_OFFSET);
        if (offsetBinding != null) {
            queryConf.setOffset(Long.parseLong(offsetBinding.getValue().stringValue()));
        }
        final Binding limitBinding = bindings.getBinding(RdfCloudTripleStoreConfiguration.CONF_LIMIT);
        if (limitBinding != null) {
            queryConf.setLimit(Long.parseLong(limitBinding.getValue().stringValue()));
        }
    } else {
        bindings = new QueryBindingSet();
    }
    if (!(tupleExpr instanceof QueryRoot)) {
        tupleExpr = new QueryRoot(tupleExpr);
    }
    try {
        final List<Class<QueryOptimizer>> optimizers = queryConf.getOptimizers();
        final Class<QueryOptimizer> pcjOptimizer = queryConf.getPcjOptimizer();
        if (pcjOptimizer != null) {
            QueryOptimizer opt = null;
            try {
                final Constructor<QueryOptimizer> construct = pcjOptimizer.getDeclaredConstructor(new Class[] {});
                opt = construct.newInstance();
            } catch (final Exception e) {
            }
            if (opt == null) {
                throw new NoSuchMethodException("Could not find valid constructor for " + pcjOptimizer.getName());
            }
            if (opt instanceof Configurable) {
                ((Configurable) opt).setConf(conf);
            }
            opt.optimize(tupleExpr, dataset, bindings);
        }
        final ParallelEvaluationStrategyImpl strategy = new ParallelEvaluationStrategyImpl(new StoreTripleSource<C>(queryConf, ryaDAO), inferenceEngine, dataset, queryConf);
        (new BindingAssigner()).optimize(tupleExpr, dataset, bindings);
        (new ConstantOptimizer(strategy)).optimize(tupleExpr, dataset, bindings);
        (new CompareOptimizer()).optimize(tupleExpr, dataset, bindings);
        (new ConjunctiveConstraintSplitter()).optimize(tupleExpr, dataset, bindings);
        (new DisjunctiveConstraintOptimizer()).optimize(tupleExpr, dataset, bindings);
        (new SameTermFilterOptimizer()).optimize(tupleExpr, dataset, bindings);
        (new QueryModelNormalizer()).optimize(tupleExpr, dataset, bindings);
        (new IterativeEvaluationOptimizer()).optimize(tupleExpr, dataset, bindings);
        if (!optimizers.isEmpty()) {
            for (final Class<QueryOptimizer> optclz : optimizers) {
                QueryOptimizer result = null;
                try {
                    final Constructor<QueryOptimizer> meth = optclz.getDeclaredConstructor(new Class[] {});
                    result = meth.newInstance();
                } catch (final Exception e) {
                }
                try {
                    final Constructor<QueryOptimizer> meth = optclz.getDeclaredConstructor(EvaluationStrategy.class);
                    result = meth.newInstance(strategy);
                } catch (final Exception e) {
                }
                if (result == null) {
                    throw new NoSuchMethodException("Could not find valid constructor for " + optclz.getName());
                }
                if (result instanceof Configurable) {
                    ((Configurable) result).setConf(conf);
                }
                result.optimize(tupleExpr, dataset, bindings);
            }
        }
        (new FilterOptimizer()).optimize(tupleExpr, dataset, bindings);
        (new OrderLimitOptimizer()).optimize(tupleExpr, dataset, bindings);
        logger.trace("Optimized query model:\n{}", tupleExpr.toString());
        if (queryConf.isInfer() && this.inferenceEngine != null) {
            try {
                tupleExpr.visit(new DomainRangeVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new SomeValuesFromVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new AllValuesFromVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new HasValueVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new IntersectionOfVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new ReflexivePropertyVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new PropertyChainVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new TransitivePropertyVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new SymmetricPropertyVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new InverseOfVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new SubPropertyOfVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new SubClassOfVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new SameAsVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new OneOfVisitor(queryConf, inferenceEngine));
                tupleExpr.visit(new HasSelfVisitor(queryConf, inferenceEngine));
            } catch (final Exception e) {
                logger.error("Error encountered while visiting query node.", e);
            }
        }
        if (queryConf.isPerformant()) {
            tupleExpr.visit(new SeparateFilterJoinsVisitor());
        // tupleExpr.visit(new FilterTimeIndexVisitor(queryConf));
        // tupleExpr.visit(new PartitionFilterTimeIndexVisitor(queryConf));
        }
        final FilterRangeVisitor rangeVisitor = new FilterRangeVisitor(queryConf);
        tupleExpr.visit(rangeVisitor);
        // this has to be done twice to get replace the statementpatterns with the right ranges
        tupleExpr.visit(rangeVisitor);
        EvaluationStatistics stats = null;
        if (!queryConf.isUseStats() && queryConf.isPerformant() || rdfEvalStatsDAO == null) {
            stats = new DefaultStatistics();
        } else if (queryConf.isUseStats()) {
            if (queryConf.isUseSelectivity()) {
                stats = new RdfCloudTripleStoreSelectivityEvaluationStatistics<C>(queryConf, rdfEvalStatsDAO, selectEvalDAO);
            } else {
                stats = new RdfCloudTripleStoreEvaluationStatistics<C>(queryConf, rdfEvalStatsDAO);
            }
        }
        if (stats != null) {
            if (stats instanceof RdfCloudTripleStoreSelectivityEvaluationStatistics) {
                final QueryJoinSelectOptimizer qjso = new QueryJoinSelectOptimizer(stats, selectEvalDAO);
                qjso.optimize(tupleExpr, dataset, bindings);
            } else {
                final QueryJoinOptimizer qjo = new QueryJoinOptimizer(stats);
                // TODO: Make pluggable
                qjo.optimize(tupleExpr, dataset, bindings);
            }
        }
        final CloseableIteration<BindingSet, QueryEvaluationException> iter = strategy.evaluate(tupleExpr, EmptyBindingSet.getInstance());
        final CloseableIteration<BindingSet, QueryEvaluationException> iterWrap = new CloseableIteration<BindingSet, QueryEvaluationException>() {

            @Override
            public void remove() throws QueryEvaluationException {
                iter.remove();
            }

            @Override
            public BindingSet next() throws QueryEvaluationException {
                return iter.next();
            }

            @Override
            public boolean hasNext() throws QueryEvaluationException {
                return iter.hasNext();
            }

            @Override
            public void close() throws QueryEvaluationException {
                iter.close();
                strategy.shutdown();
            }
        };
        return iterWrap;
    } catch (final QueryEvaluationException e) {
        throw new SailException(e);
    } catch (final Exception e) {
        throw new SailException(e);
    }
}
Also used : OneOfVisitor(org.apache.rya.rdftriplestore.inference.OneOfVisitor) FilterOptimizer(org.openrdf.query.algebra.evaluation.impl.FilterOptimizer) SameTermFilterOptimizer(org.openrdf.query.algebra.evaluation.impl.SameTermFilterOptimizer) EvaluationStatistics(org.openrdf.query.algebra.evaluation.impl.EvaluationStatistics) RdfCloudTripleStoreSelectivityEvaluationStatistics(org.apache.rya.rdftriplestore.evaluation.RdfCloudTripleStoreSelectivityEvaluationStatistics) RdfCloudTripleStoreEvaluationStatistics(org.apache.rya.rdftriplestore.evaluation.RdfCloudTripleStoreEvaluationStatistics) DisjunctiveConstraintOptimizer(org.openrdf.query.algebra.evaluation.impl.DisjunctiveConstraintOptimizer) DomainRangeVisitor(org.apache.rya.rdftriplestore.inference.DomainRangeVisitor) IterativeEvaluationOptimizer(org.openrdf.query.algebra.evaluation.impl.IterativeEvaluationOptimizer) BindingAssigner(org.openrdf.query.algebra.evaluation.impl.BindingAssigner) QueryRoot(org.openrdf.query.algebra.QueryRoot) QueryJoinOptimizer(org.apache.rya.rdftriplestore.evaluation.QueryJoinOptimizer) InverseOfVisitor(org.apache.rya.rdftriplestore.inference.InverseOfVisitor) Binding(org.openrdf.query.Binding) QueryBindingSet(org.openrdf.query.algebra.evaluation.QueryBindingSet) EmptyBindingSet(org.openrdf.query.impl.EmptyBindingSet) BindingSet(org.openrdf.query.BindingSet) ProvenanceCollectionException(org.apache.rya.rdftriplestore.provenance.ProvenanceCollectionException) ReflexivePropertyVisitor(org.apache.rya.rdftriplestore.inference.ReflexivePropertyVisitor) SymmetricPropertyVisitor(org.apache.rya.rdftriplestore.inference.SymmetricPropertyVisitor) QueryJoinSelectOptimizer(org.apache.rya.rdftriplestore.evaluation.QueryJoinSelectOptimizer) QueryBindingSet(org.openrdf.query.algebra.evaluation.QueryBindingSet) IntersectionOfVisitor(org.apache.rya.rdftriplestore.inference.IntersectionOfVisitor) QueryEvaluationException(org.openrdf.query.QueryEvaluationException) RdfCloudTripleStoreEvaluationStatistics(org.apache.rya.rdftriplestore.evaluation.RdfCloudTripleStoreEvaluationStatistics) HasSelfVisitor(org.apache.rya.rdftriplestore.inference.HasSelfVisitor) OrderLimitOptimizer(org.openrdf.query.algebra.evaluation.impl.OrderLimitOptimizer) SameAsVisitor(org.apache.rya.rdftriplestore.inference.SameAsVisitor) HasValueVisitor(org.apache.rya.rdftriplestore.inference.HasValueVisitor) SameTermFilterOptimizer(org.openrdf.query.algebra.evaluation.impl.SameTermFilterOptimizer) SubClassOfVisitor(org.apache.rya.rdftriplestore.inference.SubClassOfVisitor) SeparateFilterJoinsVisitor(org.apache.rya.rdftriplestore.evaluation.SeparateFilterJoinsVisitor) Configurable(org.apache.hadoop.conf.Configurable) PropertyChainVisitor(org.apache.rya.rdftriplestore.inference.PropertyChainVisitor) FilterRangeVisitor(org.apache.rya.rdftriplestore.evaluation.FilterRangeVisitor) ParallelEvaluationStrategyImpl(org.apache.rya.rdftriplestore.evaluation.ParallelEvaluationStrategyImpl) TransitivePropertyVisitor(org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor) SomeValuesFromVisitor(org.apache.rya.rdftriplestore.inference.SomeValuesFromVisitor) DefaultStatistics(org.apache.rya.rdftriplestore.utils.DefaultStatistics) QueryModelNormalizer(org.openrdf.query.algebra.evaluation.impl.QueryModelNormalizer) ConjunctiveConstraintSplitter(org.openrdf.query.algebra.evaluation.impl.ConjunctiveConstraintSplitter) SubPropertyOfVisitor(org.apache.rya.rdftriplestore.inference.SubPropertyOfVisitor) RdfCloudTripleStoreSelectivityEvaluationStatistics(org.apache.rya.rdftriplestore.evaluation.RdfCloudTripleStoreSelectivityEvaluationStatistics) SailException(org.openrdf.sail.SailException) QueryOptimizer(org.openrdf.query.algebra.evaluation.QueryOptimizer) SailException(org.openrdf.sail.SailException) ProvenanceCollectionException(org.apache.rya.rdftriplestore.provenance.ProvenanceCollectionException) QueryEvaluationException(org.openrdf.query.QueryEvaluationException) NoSuchElementException(java.util.NoSuchElementException) RyaDAOException(org.apache.rya.api.persist.RyaDAOException) CloseableIteration(info.aduna.iteration.CloseableIteration) CompareOptimizer(org.openrdf.query.algebra.evaluation.impl.CompareOptimizer) ConstantOptimizer(org.openrdf.query.algebra.evaluation.impl.ConstantOptimizer) AllValuesFromVisitor(org.apache.rya.rdftriplestore.inference.AllValuesFromVisitor)

Example 3 with TransitivePropertyVisitor

use of org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor in project incubator-rya by apache.

the class SparqlQueryPigEngine method generatePigScript.

public String generatePigScript(final String sparql) throws Exception {
    Preconditions.checkNotNull(sparql, "Sparql query cannot be null");
    final QueryParser parser = new SPARQLParser();
    final ParsedQuery parsedQuery = parser.parseQuery(sparql, null);
    final QueryRoot tupleExpr = new QueryRoot(parsedQuery.getTupleExpr());
    if (inference || stats) {
        if (inference) {
            tupleExpr.visit(new TransitivePropertyVisitor(conf, inferenceEngine));
            tupleExpr.visit(new SymmetricPropertyVisitor(conf, inferenceEngine));
            tupleExpr.visit(new InverseOfVisitor(conf, inferenceEngine));
        }
        if (stats) {
            (new QueryJoinOptimizer(rdfCloudTripleStoreEvaluationStatistics)).optimize(tupleExpr, null, null);
        }
    }
    sparqlToPigTransformVisitor.meet(tupleExpr);
    return sparqlToPigTransformVisitor.getPigScript();
}
Also used : QueryParser(org.openrdf.query.parser.QueryParser) SPARQLParser(org.openrdf.query.parser.sparql.SPARQLParser) QueryRoot(org.openrdf.query.algebra.QueryRoot) ParsedQuery(org.openrdf.query.parser.ParsedQuery) QueryJoinOptimizer(org.apache.rya.rdftriplestore.evaluation.QueryJoinOptimizer) SymmetricPropertyVisitor(org.apache.rya.rdftriplestore.inference.SymmetricPropertyVisitor) TransitivePropertyVisitor(org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor) InverseOfVisitor(org.apache.rya.rdftriplestore.inference.InverseOfVisitor)

Aggregations

InverseOfVisitor (org.apache.rya.rdftriplestore.inference.InverseOfVisitor)3 SymmetricPropertyVisitor (org.apache.rya.rdftriplestore.inference.SymmetricPropertyVisitor)3 TransitivePropertyVisitor (org.apache.rya.rdftriplestore.inference.TransitivePropertyVisitor)3 QueryJoinOptimizer (org.apache.rya.rdftriplestore.evaluation.QueryJoinOptimizer)2 SameAsVisitor (org.apache.rya.rdftriplestore.inference.SameAsVisitor)2 SubClassOfVisitor (org.apache.rya.rdftriplestore.inference.SubClassOfVisitor)2 SubPropertyOfVisitor (org.apache.rya.rdftriplestore.inference.SubPropertyOfVisitor)2 QueryRoot (org.openrdf.query.algebra.QueryRoot)2 SailException (org.openrdf.sail.SailException)2 CloseableIteration (info.aduna.iteration.CloseableIteration)1 IOException (java.io.IOException)1 NoSuchElementException (java.util.NoSuchElementException)1 Configurable (org.apache.hadoop.conf.Configurable)1 QueryRulesetException (org.apache.rya.accumulo.mr.merge.util.QueryRuleset.QueryRulesetException)1 RyaDAOException (org.apache.rya.api.persist.RyaDAOException)1 RdfCloudTripleStore (org.apache.rya.rdftriplestore.RdfCloudTripleStore)1 FilterRangeVisitor (org.apache.rya.rdftriplestore.evaluation.FilterRangeVisitor)1 ParallelEvaluationStrategyImpl (org.apache.rya.rdftriplestore.evaluation.ParallelEvaluationStrategyImpl)1 QueryJoinSelectOptimizer (org.apache.rya.rdftriplestore.evaluation.QueryJoinSelectOptimizer)1 RdfCloudTripleStoreEvaluationStatistics (org.apache.rya.rdftriplestore.evaluation.RdfCloudTripleStoreEvaluationStatistics)1