Examples with TableNotFoundException org.apache.accumulo.core.client.TableNotFoundException used on opensource projects

Search in sources :

Example 71 with TableNotFoundException

use of org.apache.accumulo.core.client.TableNotFoundException in project Gaffer by gchq.

the class GetRDDOfAllElementsHandlerIT method getGraphForDirectRDDForIngestAggregation.

private Graph getGraphForDirectRDDForIngestAggregation(final KeyPackage keyPackage, final String tableName) throws InterruptedException, AccumuloException, AccumuloSecurityException, IOException, OperationException, StoreException, TableNotFoundException {
    Schema schema = getSchemaForIngestAggregationChecking();
    final Graph graph = _getGraphForDirectRDD(keyPackage, tableName, schema, null);
    AccumuloStore accumuloStore = new AccumuloStore();
    accumuloStore.initialise(tableName, schema, PROPERTIES_B);
    // using the RFileReaderRDD
    for (int i = 0; i < 2; i++) {
        final String dir = tempFolder.getAbsolutePath() + File.separator + "files";
        File filesDir = new File(dir);
        final String file = dir + File.separator + "file" + i + ".rf";
        final String failure = tempFolder.getAbsolutePath() + File.separator + "failures";
        File failuresDir = new File(failure);
        try {
            filesDir.mkdir();
            failuresDir.mkdir();
        } catch (Exception e) {
            LOGGER.error("Failed to create directory: " + e.getMessage());
        }
        writeFile(keyPackage, graph.getSchema(), file);
        accumuloStore.getConnection().tableOperations().importDirectory(tableName, dir, failure, false);
    }
    return graph;
}
Also used : Graph(uk.gov.gchq.gaffer.graph.Graph) Schema(uk.gov.gchq.gaffer.store.schema.Schema) AccumuloStore(uk.gov.gchq.gaffer.accumulostore.AccumuloStore) CachableBlockFile(org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile) File(java.io.File) RFile(org.apache.accumulo.core.file.rfile.RFile) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) OperationException(uk.gov.gchq.gaffer.operation.OperationException) StoreException(uk.gov.gchq.gaffer.store.StoreException) AccumuloSecurityException(org.apache.accumulo.core.client.AccumuloSecurityException) IOException(java.io.IOException) AccumuloException(org.apache.accumulo.core.client.AccumuloException)

Example 72 with TableNotFoundException

use of org.apache.accumulo.core.client.TableNotFoundException in project Gaffer by gchq.

the class CoreKeyGroupByAggregatorIteratorTest method shouldAggregateEverythingWhenGroupByIsSetToBlank.

public void shouldAggregateEverythingWhenGroupByIsSetToBlank(final AccumuloStore store, final AccumuloElementConverter elementConverter) throws StoreException {
    final String visibilityString = "public";
    try {
        // Create edge
        final Edge edge = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        // THIS EDGE WILL BE REDUCED MEANING ITS CQ (columnQualifier) will only occur once because its key is equal.
        final Edge edge4 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 4).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 2).build();
        final Edge edge5 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 10).build();
        final Edge edge6 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 5).build();
        // Accumulo key
        final Key key = elementConverter.getKeysFromEdge(edge).getFirst();
        final Key key2 = elementConverter.getKeysFromEdge(edge2).getFirst();
        final Key key3 = elementConverter.getKeysFromEdge(edge3).getFirst();
        final Key key4 = elementConverter.getKeysFromEdge(edge4).getFirst();
        final Key key5 = elementConverter.getKeysFromEdge(edge5).getFirst();
        final Key key6 = elementConverter.getKeysFromEdge(edge6).getFirst();
        // Accumulo values
        final Value value1 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge.getProperties());
        final Value value2 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge2.getProperties());
        final Value value3 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge3.getProperties());
        final Value value4 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge4.getProperties());
        final Value value5 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge5.getProperties());
        final Value value6 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge6.getProperties());
        // Create mutation
        final Mutation m1 = new Mutation(key.getRow());
        m1.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value1);
        final Mutation m2 = new Mutation(key2.getRow());
        m2.put(key2.getColumnFamily(), key2.getColumnQualifier(), new ColumnVisibility(key2.getColumnVisibility()), key2.getTimestamp(), value2);
        final Mutation m3 = new Mutation(key.getRow());
        m3.put(key3.getColumnFamily(), key3.getColumnQualifier(), new ColumnVisibility(key3.getColumnVisibility()), key3.getTimestamp(), value3);
        final Mutation m4 = new Mutation(key.getRow());
        m4.put(key4.getColumnFamily(), key4.getColumnQualifier(), new ColumnVisibility(key4.getColumnVisibility()), key4.getTimestamp(), value4);
        final Mutation m5 = new Mutation(key.getRow());
        m5.put(key5.getColumnFamily(), key5.getColumnQualifier(), new ColumnVisibility(key5.getColumnVisibility()), key5.getTimestamp(), value5);
        final Mutation m6 = new Mutation(key.getRow());
        m6.put(key6.getColumnFamily(), key6.getColumnQualifier(), new ColumnVisibility(key6.getColumnVisibility()), key6.getTimestamp(), value6);
        // Write mutation
        final BatchWriterConfig writerConfig = new BatchWriterConfig();
        writerConfig.setMaxMemory(1000000L);
        writerConfig.setMaxLatency(1000L, TimeUnit.MILLISECONDS);
        writerConfig.setMaxWriteThreads(1);
        final BatchWriter writer = store.getConnection().createBatchWriter(store.getTableName(), writerConfig);
        writer.addMutation(m1);
        writer.addMutation(m2);
        writer.addMutation(m3);
        writer.addMutation(m4);
        writer.addMutation(m5);
        writer.addMutation(m6);
        writer.close();
        Edge expectedEdge1 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 5).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 10).property(AccumuloPropertyNames.COUNT, 20).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        // Read data back and check we get one merged element
        final Authorizations authorizations = new Authorizations(visibilityString);
        final Scanner scanner = store.getConnection().createScanner(store.getTableName(), authorizations);
        final IteratorSetting iteratorSetting = new IteratorSettingBuilder(AccumuloStoreConstants.COLUMN_QUALIFIER_AGGREGATOR_ITERATOR_PRIORITY, "KeyCombiner", CoreKeyGroupByAggregatorIterator.class).all().view(new View.Builder().edge(TestGroups.EDGE, new ViewElementDefinition.Builder().groupBy().build()).build()).schema(store.getSchema()).keyConverter(store.getKeyPackage().getKeyConverter()).build();
        scanner.addScanIterator(iteratorSetting);
        final Iterator<Entry<Key, Value>> it = scanner.iterator();
        Entry<Key, Value> entry = it.next();
        Element readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge1, readEdge);
        assertEquals(5, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(10, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(20, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        if (it.hasNext()) {
            fail("Additional row found.");
        }
    } catch (final AccumuloException | TableNotFoundException e) {
        fail(this.getClass().getSimpleName() + " failed with exception: " + e);
    }
}
Also used : Scanner(org.apache.accumulo.core.client.Scanner) AccumuloException(org.apache.accumulo.core.client.AccumuloException) Authorizations(org.apache.accumulo.core.security.Authorizations) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) Element(uk.gov.gchq.gaffer.data.element.Element) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) Entry(java.util.Map.Entry) IteratorSetting(org.apache.accumulo.core.client.IteratorSetting) Value(org.apache.accumulo.core.data.Value) BatchWriterConfig(org.apache.accumulo.core.client.BatchWriterConfig) Mutation(org.apache.accumulo.core.data.Mutation) ColumnVisibility(org.apache.accumulo.core.security.ColumnVisibility) BatchWriter(org.apache.accumulo.core.client.BatchWriter) Edge(uk.gov.gchq.gaffer.data.element.Edge) Key(org.apache.accumulo.core.data.Key)

Example 73 with TableNotFoundException

use of org.apache.accumulo.core.client.TableNotFoundException in project Gaffer by gchq.

the class CoreKeyGroupByAggregatorIteratorTest method testAggregatingMultiplePropertySets.

private void testAggregatingMultiplePropertySets(final AccumuloStore store, final AccumuloElementConverter elementConverter) throws StoreException {
    String visibilityString = "public";
    try {
        // Create edge
        final Edge edge = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 8).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        // THIS EDGE WILL BE REDUCED MEANING ITS CQ (columnQualifier) will only occur once because its key is equal.
        final Edge edge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 2).build();
        final Edge edge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 10).build();
        // Accumulo key
        final Key key = elementConverter.getKeysFromEdge(edge).getFirst();
        final Key key2 = elementConverter.getKeysFromEdge(edge2).getFirst();
        final Key key3 = elementConverter.getKeysFromEdge(edge3).getFirst();
        // Accumulo values
        final Value value1 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge.getProperties());
        final Value value2 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge2.getProperties());
        final Value value3 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge3.getProperties());
        // Create mutation
        final Mutation m1 = new Mutation(key.getRow());
        m1.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value1);
        final Mutation m2 = new Mutation(key.getRow());
        m2.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value2);
        final Mutation m3 = new Mutation(key.getRow());
        m3.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value3);
        final Mutation m4 = new Mutation(key2.getRow());
        m4.put(key2.getColumnFamily(), key2.getColumnQualifier(), new ColumnVisibility(key2.getColumnVisibility()), key2.getTimestamp(), value1);
        final Mutation m5 = new Mutation(key.getRow());
        m5.put(key3.getColumnFamily(), key3.getColumnQualifier(), new ColumnVisibility(key3.getColumnVisibility()), key3.getTimestamp(), value1);
        // Write mutation
        final BatchWriterConfig writerConfig = new BatchWriterConfig();
        writerConfig.setMaxMemory(1000000L);
        writerConfig.setMaxLatency(1000L, TimeUnit.MILLISECONDS);
        writerConfig.setMaxWriteThreads(1);
        final BatchWriter writer = store.getConnection().createBatchWriter(store.getTableName(), writerConfig);
        writer.addMutation(m1);
        writer.addMutation(m2);
        writer.addMutation(m3);
        writer.addMutation(m4);
        writer.addMutation(m5);
        writer.close();
        // Read data back and check we get one merged element
        final Authorizations authorizations = new Authorizations(visibilityString);
        final Scanner scanner = store.getConnection().createScanner(store.getTableName(), authorizations);
        final IteratorSetting iteratorSetting = new IteratorSettingBuilder(AccumuloStoreConstants.COLUMN_QUALIFIER_AGGREGATOR_ITERATOR_PRIORITY, "KeyCombiner", CoreKeyGroupByAggregatorIterator.class).all().schema(store.getSchema()).view(new View.Builder().edge(TestGroups.EDGE, new ViewElementDefinition.Builder().groupBy().build()).build()).keyConverter(store.getKeyPackage().getKeyConverter()).build();
        scanner.addScanIterator(iteratorSetting);
        final Iterator<Entry<Key, Value>> it = scanner.iterator();
        final Entry<Key, Value> entry = it.next();
        final Element readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        final Edge expectedEdge = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 9).property(AccumuloPropertyNames.COUNT, 15).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        assertEquals(expectedEdge, readEdge);
        assertEquals(9, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(15, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        // Check no more entries
        if (it.hasNext()) {
            fail("Additional row found.");
        }
    } catch (final AccumuloException | TableNotFoundException e) {
        fail(this.getClass().getSimpleName() + " failed with exception: " + e);
    }
}
Also used : Scanner(org.apache.accumulo.core.client.Scanner) Element(uk.gov.gchq.gaffer.data.element.Element) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) Entry(java.util.Map.Entry) BatchWriterConfig(org.apache.accumulo.core.client.BatchWriterConfig) ColumnVisibility(org.apache.accumulo.core.security.ColumnVisibility) AccumuloException(org.apache.accumulo.core.client.AccumuloException) Authorizations(org.apache.accumulo.core.security.Authorizations) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) IteratorSetting(org.apache.accumulo.core.client.IteratorSetting) Value(org.apache.accumulo.core.data.Value) Mutation(org.apache.accumulo.core.data.Mutation) BatchWriter(org.apache.accumulo.core.client.BatchWriter) Edge(uk.gov.gchq.gaffer.data.element.Edge) Key(org.apache.accumulo.core.data.Key)

Example 74 with TableNotFoundException

use of org.apache.accumulo.core.client.TableNotFoundException in project Gaffer by gchq.

the class CoreKeyGroupByAggregatorIteratorTest method shouldPartiallyAggregateColumnQualifierOverCQ1GroupBy.

public void shouldPartiallyAggregateColumnQualifierOverCQ1GroupBy(final AccumuloStore store, final AccumuloElementConverter elementConverter) throws StoreException {
    final String visibilityString = "public";
    try {
        // Create edge
        final Edge edge = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 2).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 3).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        // THIS EDGE WILL BE REDUCED MEANING ITS CQ (columnQualifier) will only occur once because its key is equal.
        final Edge edge4 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 2).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 4).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 2).build();
        final Edge edge5 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 10).build();
        final Edge edge6 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 4).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 6).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 5).build();
        // Accumulo key
        final Key key = elementConverter.getKeysFromEdge(edge).getFirst();
        final Key key2 = elementConverter.getKeysFromEdge(edge2).getFirst();
        final Key key3 = elementConverter.getKeysFromEdge(edge3).getFirst();
        final Key key4 = elementConverter.getKeysFromEdge(edge4).getFirst();
        final Key key5 = elementConverter.getKeysFromEdge(edge5).getFirst();
        final Key key6 = elementConverter.getKeysFromEdge(edge6).getFirst();
        // Accumulo values
        final Value value1 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge.getProperties());
        final Value value2 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge2.getProperties());
        final Value value3 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge3.getProperties());
        final Value value4 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge4.getProperties());
        final Value value5 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge5.getProperties());
        final Value value6 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge6.getProperties());
        // Create mutation
        final Mutation m1 = new Mutation(key.getRow());
        m1.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value1);
        final Mutation m2 = new Mutation(key2.getRow());
        m2.put(key2.getColumnFamily(), key2.getColumnQualifier(), new ColumnVisibility(key2.getColumnVisibility()), key2.getTimestamp(), value2);
        final Mutation m3 = new Mutation(key.getRow());
        m3.put(key3.getColumnFamily(), key3.getColumnQualifier(), new ColumnVisibility(key3.getColumnVisibility()), key3.getTimestamp(), value3);
        final Mutation m4 = new Mutation(key.getRow());
        m4.put(key4.getColumnFamily(), key4.getColumnQualifier(), new ColumnVisibility(key4.getColumnVisibility()), key4.getTimestamp(), value4);
        final Mutation m5 = new Mutation(key.getRow());
        m5.put(key5.getColumnFamily(), key5.getColumnQualifier(), new ColumnVisibility(key5.getColumnVisibility()), key5.getTimestamp(), value5);
        final Mutation m6 = new Mutation(key.getRow());
        m6.put(key6.getColumnFamily(), key6.getColumnQualifier(), new ColumnVisibility(key6.getColumnVisibility()), key6.getTimestamp(), value6);
        // Write mutation
        final BatchWriterConfig writerConfig = new BatchWriterConfig();
        writerConfig.setMaxMemory(1000000L);
        writerConfig.setMaxLatency(1000L, TimeUnit.MILLISECONDS);
        writerConfig.setMaxWriteThreads(1);
        final BatchWriter writer = store.getConnection().createBatchWriter(store.getTableName(), writerConfig);
        writer.addMutation(m1);
        writer.addMutation(m2);
        writer.addMutation(m3);
        writer.addMutation(m4);
        writer.addMutation(m5);
        writer.addMutation(m6);
        writer.close();
        Edge expectedEdge1 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 6).property(AccumuloPropertyNames.COUNT, 3).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        Edge expectedEdge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 2).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 4).property(AccumuloPropertyNames.COUNT, 2).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        Edge expectedEdge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.COUNT, 10).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        Edge expectedEdge4 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 4).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 6).property(AccumuloPropertyNames.COUNT, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        // Read data back and check we get one merged element
        final Authorizations authorizations = new Authorizations(visibilityString);
        final Scanner scanner = store.getConnection().createScanner(store.getTableName(), authorizations);
        final IteratorSetting iteratorSetting = new IteratorSettingBuilder(AccumuloStoreConstants.COLUMN_QUALIFIER_AGGREGATOR_ITERATOR_PRIORITY, "KeyCombiner", CoreKeyGroupByAggregatorIterator.class).all().view(new View.Builder().edge(TestGroups.EDGE, new ViewElementDefinition.Builder().groupBy(AccumuloPropertyNames.COLUMN_QUALIFIER).build()).build()).schema(store.getSchema()).keyConverter(store.getKeyPackage().getKeyConverter()).build();
        scanner.addScanIterator(iteratorSetting);
        final Iterator<Entry<Key, Value>> it = scanner.iterator();
        Entry<Key, Value> entry = it.next();
        Element readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge1, readEdge);
        assertEquals(1, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(6, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(3, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        entry = it.next();
        readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge2, readEdge);
        assertEquals(2, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(4, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(2, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        entry = it.next();
        readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge3, readEdge);
        assertEquals(3, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(5, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(10, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        entry = it.next();
        readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge4, readEdge);
        assertEquals(4, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(6, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(5, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        if (it.hasNext()) {
            fail("Additional row found.");
        }
    } catch (final AccumuloException | TableNotFoundException e) {
        fail(this.getClass().getSimpleName() + " failed with exception: " + e);
    }
}
Also used : Scanner(org.apache.accumulo.core.client.Scanner) AccumuloException(org.apache.accumulo.core.client.AccumuloException) Authorizations(org.apache.accumulo.core.security.Authorizations) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) Element(uk.gov.gchq.gaffer.data.element.Element) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) Entry(java.util.Map.Entry) IteratorSetting(org.apache.accumulo.core.client.IteratorSetting) Value(org.apache.accumulo.core.data.Value) BatchWriterConfig(org.apache.accumulo.core.client.BatchWriterConfig) Mutation(org.apache.accumulo.core.data.Mutation) ColumnVisibility(org.apache.accumulo.core.security.ColumnVisibility) BatchWriter(org.apache.accumulo.core.client.BatchWriter) Edge(uk.gov.gchq.gaffer.data.element.Edge) Key(org.apache.accumulo.core.data.Key)

Example 75 with TableNotFoundException

use of org.apache.accumulo.core.client.TableNotFoundException in project Gaffer by gchq.

the class CoreKeyGroupByAggregatorIteratorTest method shouldAggregatePropertiesOnlyWhenGroupByIsSetToCQ1CQ2.

public void shouldAggregatePropertiesOnlyWhenGroupByIsSetToCQ1CQ2(final AccumuloStore store, final AccumuloElementConverter elementConverter) throws StoreException {
    final String visibilityString = "public";
    try {
        // Create edge
        final Edge edge = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        final Edge edge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 1).build();
        // THIS EDGE WILL BE REDUCED MEANING ITS CQ (columnQualifier) will only occur once because its key is equal.
        final Edge edge4 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 4).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 2).build();
        final Edge edge5 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 10).build();
        final Edge edge6 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).property(AccumuloPropertyNames.COUNT, 5).build();
        // Accumulo key
        final Key key = elementConverter.getKeysFromEdge(edge).getFirst();
        final Key key2 = elementConverter.getKeysFromEdge(edge2).getFirst();
        final Key key3 = elementConverter.getKeysFromEdge(edge3).getFirst();
        final Key key4 = elementConverter.getKeysFromEdge(edge4).getFirst();
        final Key key5 = elementConverter.getKeysFromEdge(edge5).getFirst();
        final Key key6 = elementConverter.getKeysFromEdge(edge6).getFirst();
        // Accumulo values
        final Value value1 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge.getProperties());
        final Value value2 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge2.getProperties());
        final Value value3 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge3.getProperties());
        final Value value4 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge4.getProperties());
        final Value value5 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge5.getProperties());
        final Value value6 = elementConverter.getValueFromProperties(TestGroups.EDGE, edge6.getProperties());
        // Create mutation
        final Mutation m1 = new Mutation(key.getRow());
        m1.put(key.getColumnFamily(), key.getColumnQualifier(), new ColumnVisibility(key.getColumnVisibility()), key.getTimestamp(), value1);
        final Mutation m2 = new Mutation(key2.getRow());
        m2.put(key2.getColumnFamily(), key2.getColumnQualifier(), new ColumnVisibility(key2.getColumnVisibility()), key2.getTimestamp(), value2);
        final Mutation m3 = new Mutation(key.getRow());
        m3.put(key3.getColumnFamily(), key3.getColumnQualifier(), new ColumnVisibility(key3.getColumnVisibility()), key3.getTimestamp(), value3);
        final Mutation m4 = new Mutation(key.getRow());
        m4.put(key4.getColumnFamily(), key4.getColumnQualifier(), new ColumnVisibility(key4.getColumnVisibility()), key4.getTimestamp(), value4);
        final Mutation m5 = new Mutation(key.getRow());
        m5.put(key5.getColumnFamily(), key5.getColumnQualifier(), new ColumnVisibility(key5.getColumnVisibility()), key5.getTimestamp(), value5);
        final Mutation m6 = new Mutation(key.getRow());
        m6.put(key6.getColumnFamily(), key6.getColumnQualifier(), new ColumnVisibility(key6.getColumnVisibility()), key6.getTimestamp(), value6);
        // Write mutation
        final BatchWriterConfig writerConfig = new BatchWriterConfig();
        writerConfig.setMaxMemory(1000000L);
        writerConfig.setMaxLatency(1000L, TimeUnit.MILLISECONDS);
        writerConfig.setMaxWriteThreads(1);
        final BatchWriter writer = store.getConnection().createBatchWriter(store.getTableName(), writerConfig);
        writer.addMutation(m1);
        writer.addMutation(m2);
        writer.addMutation(m3);
        writer.addMutation(m4);
        writer.addMutation(m5);
        writer.addMutation(m6);
        writer.close();
        Edge expectedEdge1 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 1).property(AccumuloPropertyNames.COUNT, 3).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        Edge expectedEdge2 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 1).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 4).property(AccumuloPropertyNames.COUNT, 2).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        Edge expectedEdge3 = new Edge.Builder().group(TestGroups.EDGE).source("1").dest("2").directed(true).property(AccumuloPropertyNames.COLUMN_QUALIFIER, 3).property(AccumuloPropertyNames.COLUMN_QUALIFIER_2, 5).property(AccumuloPropertyNames.COUNT, 15).property(AccumuloPropertyNames.PROP_1, 0).property(AccumuloPropertyNames.PROP_2, 0).property(AccumuloPropertyNames.PROP_3, 0).property(AccumuloPropertyNames.PROP_4, 0).build();
        // Read data back and check we get one merged element
        final Authorizations authorizations = new Authorizations(visibilityString);
        final Scanner scanner = store.getConnection().createScanner(store.getTableName(), authorizations);
        final IteratorSetting iteratorSetting = new IteratorSettingBuilder(AccumuloStoreConstants.COLUMN_QUALIFIER_AGGREGATOR_ITERATOR_PRIORITY, "KeyCombiner", CoreKeyGroupByAggregatorIterator.class).all().view(new View.Builder().edge(TestGroups.EDGE, new ViewElementDefinition.Builder().groupBy(AccumuloPropertyNames.COLUMN_QUALIFIER, AccumuloPropertyNames.COLUMN_QUALIFIER_2).build()).build()).schema(store.getSchema()).keyConverter(store.getKeyPackage().getKeyConverter()).build();
        scanner.addScanIterator(iteratorSetting);
        final Iterator<Entry<Key, Value>> it = scanner.iterator();
        Entry<Key, Value> entry = it.next();
        Element readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge1, readEdge);
        assertEquals(1, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(1, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(3, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        entry = it.next();
        readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge2, readEdge);
        assertEquals(1, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(4, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(2, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        entry = it.next();
        readEdge = elementConverter.getFullElement(entry.getKey(), entry.getValue(), false);
        assertEquals(expectedEdge3, readEdge);
        assertEquals(3, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER));
        assertEquals(5, readEdge.getProperty(AccumuloPropertyNames.COLUMN_QUALIFIER_2));
        assertEquals(15, readEdge.getProperty(AccumuloPropertyNames.COUNT));
        if (it.hasNext()) {
            fail("Additional row found.");
        }
    } catch (final AccumuloException | TableNotFoundException e) {
        fail(this.getClass().getSimpleName() + " failed with exception: " + e);
    }
}
Also used : Scanner(org.apache.accumulo.core.client.Scanner) AccumuloException(org.apache.accumulo.core.client.AccumuloException) Authorizations(org.apache.accumulo.core.security.Authorizations) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) Element(uk.gov.gchq.gaffer.data.element.Element) IteratorSettingBuilder(uk.gov.gchq.gaffer.accumulostore.utils.IteratorSettingBuilder) TableNotFoundException(org.apache.accumulo.core.client.TableNotFoundException) Entry(java.util.Map.Entry) IteratorSetting(org.apache.accumulo.core.client.IteratorSetting) Value(org.apache.accumulo.core.data.Value) BatchWriterConfig(org.apache.accumulo.core.client.BatchWriterConfig) Mutation(org.apache.accumulo.core.data.Mutation) ColumnVisibility(org.apache.accumulo.core.security.ColumnVisibility) BatchWriter(org.apache.accumulo.core.client.BatchWriter) Edge(uk.gov.gchq.gaffer.data.element.Edge) Key(org.apache.accumulo.core.data.Key)

Aggregations

TableNotFoundException (org.apache.accumulo.core.client.TableNotFoundException)160 AccumuloException (org.apache.accumulo.core.client.AccumuloException)100 AccumuloSecurityException (org.apache.accumulo.core.client.AccumuloSecurityException)89 Text (org.apache.hadoop.io.Text)51 Value (org.apache.accumulo.core.data.Value)46 Key (org.apache.accumulo.core.data.Key)42 Scanner (org.apache.accumulo.core.client.Scanner)36 IOException (java.io.IOException)34 BatchWriter (org.apache.accumulo.core.client.BatchWriter)31 BatchWriterConfig (org.apache.accumulo.core.client.BatchWriterConfig)31 Connector (org.apache.accumulo.core.client.Connector)30 Mutation (org.apache.accumulo.core.data.Mutation)29 Range (org.apache.accumulo.core.data.Range)28 Authorizations (org.apache.accumulo.core.security.Authorizations)26 ArrayList (java.util.ArrayList)25 Entry (java.util.Map.Entry)25 TableExistsException (org.apache.accumulo.core.client.TableExistsException)25 IteratorSetting (org.apache.accumulo.core.client.IteratorSetting)23 MutationsRejectedException (org.apache.accumulo.core.client.MutationsRejectedException)23 HashMap (java.util.HashMap)19
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial