Search in sources :

Example 46 with VisibilityBindingSet

use of org.apache.rya.api.model.VisibilityBindingSet in project incubator-rya by apache.

the class PcjDocumentsIntegrationTest method purge.

@Test
public void purge() throws Exception {
    final String sparql = "SELECT ?name ?age " + "{" + "FILTER(?age < 30) ." + "?name <http://hasAge> ?age." + "?name <http://playsSport> \"Soccer\" " + "}";
    final String pcjTableName = "testPcj";
    final MongoPcjDocuments pcjs = new MongoPcjDocuments(getMongoClient(), conf.getRyaInstanceName());
    pcjs.createPcj(pcjTableName, sparql);
    // Add a few results to the PCJ table.
    final MapBindingSet alice = new MapBindingSet();
    alice.addBinding("name", new URIImpl("http://Alice"));
    alice.addBinding("age", new NumericLiteralImpl(14, XMLSchema.INTEGER));
    final MapBindingSet bob = new MapBindingSet();
    bob.addBinding("name", new URIImpl("http://Bob"));
    bob.addBinding("age", new NumericLiteralImpl(16, XMLSchema.INTEGER));
    final MapBindingSet charlie = new MapBindingSet();
    charlie.addBinding("name", new URIImpl("http://Charlie"));
    charlie.addBinding("age", new NumericLiteralImpl(12, XMLSchema.INTEGER));
    pcjs.addResults(pcjTableName, Sets.<VisibilityBindingSet>newHashSet(new VisibilityBindingSet(alice), new VisibilityBindingSet(bob), new VisibilityBindingSet(charlie)));
    // Make sure the cardinality was updated.
    PcjMetadata metadata = pcjs.getPcjMetadata(pcjTableName);
    assertEquals(3, metadata.getCardinality());
    // Purge the data.
    pcjs.purgePcjs(pcjTableName);
    // Make sure the cardinality was updated to 0.
    metadata = pcjs.getPcjMetadata(pcjTableName);
    assertEquals(0, metadata.getCardinality());
}
Also used : VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) NumericLiteralImpl(org.openrdf.model.impl.NumericLiteralImpl) URIImpl(org.openrdf.model.impl.URIImpl) PcjMetadata(org.apache.rya.indexing.pcj.storage.PcjMetadata) MapBindingSet(org.openrdf.query.impl.MapBindingSet) Test(org.junit.Test)

Example 47 with VisibilityBindingSet

use of org.apache.rya.api.model.VisibilityBindingSet in project incubator-rya by apache.

the class PcjDocumentsIntegrationTest method addResults.

/**
 * Ensure when results have been written to the PCJ table that they are in Accumulo.
 * <p>
 * The method being tested is {@link PcjTables#addResults(Connector, String, java.util.Collection)}
 */
@Test
public void addResults() throws Exception {
    final String sparql = "SELECT ?name ?age " + "{" + "FILTER(?age < 30) ." + "?name <http://hasAge> ?age." + "?name <http://playsSport> \"Soccer\" " + "}";
    final String pcjTableName = "testPcj";
    final MongoPcjDocuments pcjs = new MongoPcjDocuments(getMongoClient(), conf.getRyaInstanceName());
    pcjs.createPcj(pcjTableName, sparql);
    // Add a few results to the PCJ table.
    final MapBindingSet alice = new MapBindingSet();
    alice.addBinding("name", new URIImpl("http://Alice"));
    alice.addBinding("age", new NumericLiteralImpl(14, XMLSchema.INTEGER));
    final MapBindingSet bob = new MapBindingSet();
    bob.addBinding("name", new URIImpl("http://Bob"));
    bob.addBinding("age", new NumericLiteralImpl(16, XMLSchema.INTEGER));
    final MapBindingSet charlie = new MapBindingSet();
    charlie.addBinding("name", new URIImpl("http://Charlie"));
    charlie.addBinding("age", new NumericLiteralImpl(12, XMLSchema.INTEGER));
    final Set<BindingSet> expected = Sets.<BindingSet>newHashSet(alice, bob, charlie);
    pcjs.addResults(pcjTableName, Sets.<VisibilityBindingSet>newHashSet(new VisibilityBindingSet(alice), new VisibilityBindingSet(bob), new VisibilityBindingSet(charlie)));
    // Make sure the cardinality was updated.
    final PcjMetadata metadata = pcjs.getPcjMetadata(pcjTableName);
    assertEquals(3, metadata.getCardinality());
    // Scan Accumulo for the stored results.
    final Collection<BindingSet> fetchedResults = loadPcjResults(pcjTableName);
    assertEquals(expected, fetchedResults);
}
Also used : MapBindingSet(org.openrdf.query.impl.MapBindingSet) VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) BindingSet(org.openrdf.query.BindingSet) VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) NumericLiteralImpl(org.openrdf.model.impl.NumericLiteralImpl) URIImpl(org.openrdf.model.impl.URIImpl) PcjMetadata(org.apache.rya.indexing.pcj.storage.PcjMetadata) MapBindingSet(org.openrdf.query.impl.MapBindingSet) Test(org.junit.Test)

Example 48 with VisibilityBindingSet

use of org.apache.rya.api.model.VisibilityBindingSet in project incubator-rya by apache.

the class VisibilityBindingSetStringConverter method convert.

@Override
public VisibilityBindingSet convert(final String bindingSetString, final VariableOrder varOrder) {
    // Try to split the binding set string over the visibility delimiter.
    final String[] strings = bindingSetString.split("" + VISIBILITY_DELIM);
    // Convert the binding set string into a BindingSet.
    final BindingSet bindingSet = super.convert(strings[BINDING_SET_STRING_INDEX], varOrder);
    // If a visibility expression is present, then also include it.
    return (strings.length > 1) ? new VisibilityBindingSet(bindingSet, strings[VISIBILITY_EXPRESSION_INDEX]) : new VisibilityBindingSet(bindingSet);
}
Also used : VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) BindingSet(org.openrdf.query.BindingSet) VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet)

Example 49 with VisibilityBindingSet

use of org.apache.rya.api.model.VisibilityBindingSet in project incubator-rya by apache.

the class MongoPcjDocuments method populatePcj.

/**
 * Scan Rya for results that solve the PCJ's query and store them in the PCJ
 * document.
 * <p>
 * This method assumes the PCJ document has already been created.
 *
 * @param pcjId - The Id of the PCJ that will receive the results. (not null)
 * @param ryaConn - A connection to the Rya store that will be queried to find results. (not null)
 * @throws PCJStorageException If results could not be written to the PCJ results document,
 *  the PCJ results document does not exist, or the query that is being execute was malformed.
 */
public void populatePcj(final String pcjId, final RepositoryConnection ryaConn) throws PCJStorageException {
    checkNotNull(pcjId);
    checkNotNull(ryaConn);
    try {
        // Fetch the query that needs to be executed from the PCJ metadata document.
        final PcjMetadata pcjMetadata = getPcjMetadata(pcjId);
        final String sparql = pcjMetadata.getSparql();
        // Query Rya for results to the SPARQL query.
        final TupleQuery query = ryaConn.prepareTupleQuery(QueryLanguage.SPARQL, sparql);
        final TupleQueryResult results = query.evaluate();
        // Load batches of 1000 of them at a time into the PCJ results document.
        final Set<VisibilityBindingSet> batch = new HashSet<>(1000);
        while (results.hasNext()) {
            final VisibilityBindingSet bs = new VisibilityBindingSet(results.next());
            batch.add(bs);
            if (batch.size() == 1000) {
                addResults(pcjId, batch);
                batch.clear();
            }
        }
        if (!batch.isEmpty()) {
            addResults(pcjId, batch);
        }
    } catch (RepositoryException | MalformedQueryException | QueryEvaluationException e) {
        throw new PCJStorageException("Could not populate a PCJ document with Rya results for the pcj with Id: " + pcjId, e);
    }
}
Also used : VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) QueryEvaluationException(org.openrdf.query.QueryEvaluationException) MalformedQueryException(org.openrdf.query.MalformedQueryException) PcjMetadata(org.apache.rya.indexing.pcj.storage.PcjMetadata) TupleQuery(org.openrdf.query.TupleQuery) RepositoryException(org.openrdf.repository.RepositoryException) PCJStorageException(org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException) TupleQueryResult(org.openrdf.query.TupleQueryResult) HashSet(java.util.HashSet)

Example 50 with VisibilityBindingSet

use of org.apache.rya.api.model.VisibilityBindingSet in project incubator-rya by apache.

the class PcjTables method writeResults.

/**
 * Add a collection of results to a specific PCJ table.
 *
 * @param accumuloConn - A connection to the Accumulo that hosts the PCJ table. (not null)
 * @param pcjTableName - The name of the PCJ table that will receive the results. (not null)
 * @param results - Binding sets that will be written to the PCJ table. (not null)
 * @throws PCJStorageException The provided PCJ table doesn't exist, is missing the
 *   PCJ metadata, or the result could not be written to it.
 */
private void writeResults(final Connector accumuloConn, final String pcjTableName, final Collection<VisibilityBindingSet> results) throws PCJStorageException {
    checkNotNull(accumuloConn);
    checkNotNull(pcjTableName);
    checkNotNull(results);
    // Fetch the variable orders from the PCJ table.
    final PcjMetadata metadata = getPcjMetadata(accumuloConn, pcjTableName);
    // Write each result formatted using each of the variable orders.
    BatchWriter writer = null;
    try {
        writer = accumuloConn.createBatchWriter(pcjTableName, new BatchWriterConfig());
        for (final VisibilityBindingSet result : results) {
            final Set<Mutation> addResultMutations = makeWriteResultMutations(metadata.getVarOrders(), result);
            writer.addMutations(addResultMutations);
        }
    } catch (TableNotFoundException | MutationsRejectedException e) {
        throw new PCJStorageException("Could not add results to the PCJ table named: " + pcjTableName, e);
    } finally {
        if (writer != null) {
            try {
                writer.close();
            } catch (final MutationsRejectedException e) {
                throw new PCJStorageException("Could not add results to a PCJ table because some of the mutations were rejected.", e);
            }
        }
    }
}
Also used : TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) VisibilityBindingSet(org.apache.rya.api.model.VisibilityBindingSet) BatchWriterConfig(org.apache.accumulo.core.client.BatchWriterConfig) PcjMetadata(org.apache.rya.indexing.pcj.storage.PcjMetadata) BatchWriter(org.apache.accumulo.core.client.BatchWriter) Mutation(org.apache.accumulo.core.data.Mutation) ConditionalMutation(org.apache.accumulo.core.data.ConditionalMutation) PCJStorageException(org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException) MutationsRejectedException(org.apache.accumulo.core.client.MutationsRejectedException)

Aggregations

VisibilityBindingSet (org.apache.rya.api.model.VisibilityBindingSet)140 Test (org.junit.Test)105 MapBindingSet (org.openrdf.query.impl.MapBindingSet)93 ValueFactory (org.openrdf.model.ValueFactory)66 ValueFactoryImpl (org.openrdf.model.impl.ValueFactoryImpl)66 HashSet (java.util.HashSet)52 ArrayList (java.util.ArrayList)31 UUID (java.util.UUID)28 VisibilityStatement (org.apache.rya.api.model.VisibilityStatement)28 TopologyFactory (org.apache.rya.streams.kafka.topology.TopologyFactory)24 BindingSet (org.openrdf.query.BindingSet)24 TopologyBuilder (org.apache.kafka.streams.processor.TopologyBuilder)23 RandomUUIDFactory (org.apache.rya.api.function.projection.RandomUUIDFactory)23 URIImpl (org.openrdf.model.impl.URIImpl)19 QueryBindingSet (org.openrdf.query.algebra.evaluation.QueryBindingSet)19 VariableOrder (org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder)18 Bytes (org.apache.fluo.api.data.Bytes)16 PcjMetadata (org.apache.rya.indexing.pcj.storage.PcjMetadata)15 ProcessorContext (org.apache.kafka.streams.processor.ProcessorContext)12 Statement (org.openrdf.model.Statement)12