Search in sources :

Example 6 with AccumuloStore

use of uk.gov.gchq.gaffer.accumulostore.AccumuloStore in project Gaffer by gchq.

the class InputFormatTest method shouldReturnCorrectDataToMapReduceJob.

private void shouldReturnCorrectDataToMapReduceJob(final Schema schema, final KeyPackage kp, final List<Element> data, final View view, final User user, final String instanceName, final Set<String> expectedResults) throws Exception {
    final AccumuloStore store = new MockAccumuloStore();
    final AccumuloProperties properties = AccumuloProperties.loadStoreProperties(StreamUtil.storeProps(getClass()));
    switch(kp) {
        case BYTE_ENTITY_KEY_PACKAGE:
            properties.setKeyPackageClass(ByteEntityKeyPackage.class.getName());
            properties.setInstance(instanceName + "_BYTE_ENTITY");
            break;
        case CLASSIC_KEY_PACKAGE:
            properties.setKeyPackageClass(ClassicKeyPackage.class.getName());
            properties.setInstance(instanceName + "_CLASSIC");
    }
    try {
        store.initialise(schema, properties);
    } catch (StoreException e) {
        fail("StoreException thrown: " + e);
    }
    setupGraph(store, data);
    // Set up local conf
    final JobConf conf = new JobConf();
    conf.set("fs.default.name", "file:///");
    conf.set("mapred.job.tracker", "local");
    final FileSystem fs = FileSystem.getLocal(conf);
    // Update configuration with instance, table name, etc.
    store.updateConfiguration(conf, view, user);
    // Run Driver
    final File outputFolder = testFolder.newFolder();
    FileUtils.deleteDirectory(outputFolder);
    final Driver driver = new Driver(outputFolder.getAbsolutePath());
    driver.setConf(conf);
    driver.run(new String[] {});
    // Read results and check correct
    final SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path(outputFolder + "/part-m-00000"), conf);
    final Text text = new Text();
    final Set<String> results = new HashSet<>();
    while (reader.next(text)) {
        results.add(text.toString());
    }
    reader.close();
    assertEquals(expectedResults, results);
    FileUtils.deleteDirectory(outputFolder);
}
Also used : Path(org.apache.hadoop.fs.Path) ClassicKeyPackage(uk.gov.gchq.gaffer.accumulostore.key.core.impl.classic.ClassicKeyPackage) MockAccumuloStore(uk.gov.gchq.gaffer.accumulostore.MockAccumuloStore) AccumuloProperties(uk.gov.gchq.gaffer.accumulostore.AccumuloProperties) Text(org.apache.hadoop.io.Text) ByteEntityKeyPackage(uk.gov.gchq.gaffer.accumulostore.key.core.impl.byteEntity.ByteEntityKeyPackage) StoreException(uk.gov.gchq.gaffer.store.StoreException) SequenceFile(org.apache.hadoop.io.SequenceFile) FileSystem(org.apache.hadoop.fs.FileSystem) AccumuloStore(uk.gov.gchq.gaffer.accumulostore.AccumuloStore) MockAccumuloStore(uk.gov.gchq.gaffer.accumulostore.MockAccumuloStore) JobConf(org.apache.hadoop.mapred.JobConf) SequenceFile(org.apache.hadoop.io.SequenceFile) File(java.io.File) HashSet(java.util.HashSet)

Example 7 with AccumuloStore

use of uk.gov.gchq.gaffer.accumulostore.AccumuloStore in project Gaffer by gchq.

the class TableUtilsTest method shouldThrowExceptionIfTableNameIsNotSpecified.

@Test(expected = AccumuloRuntimeException.class)
public void shouldThrowExceptionIfTableNameIsNotSpecified() throws StoreException {
    // Given
    final Schema schema = new Schema.Builder().type("int", Integer.class).type("string", String.class).type("boolean", Boolean.class).edge("EDGE", new SchemaEdgeDefinition.Builder().source("string").destination("string").directed("boolean").build()).build();
    final AccumuloProperties properties = new AccumuloProperties();
    properties.setStoreClass(SingleUseMockAccumuloStore.class.getName());
    final AccumuloStore store = new AccumuloStore();
    store.initialise(schema, properties);
    // When
    TableUtils.ensureTableExists(store);
    fail("The expected exception was not thrown.");
}
Also used : SingleUseMockAccumuloStore(uk.gov.gchq.gaffer.accumulostore.SingleUseMockAccumuloStore) AccumuloProperties(uk.gov.gchq.gaffer.accumulostore.AccumuloProperties) Schema(uk.gov.gchq.gaffer.store.schema.Schema) AccumuloStore(uk.gov.gchq.gaffer.accumulostore.AccumuloStore) MockAccumuloStore(uk.gov.gchq.gaffer.accumulostore.MockAccumuloStore) SingleUseMockAccumuloStore(uk.gov.gchq.gaffer.accumulostore.SingleUseMockAccumuloStore) Test(org.junit.Test)

Aggregations

AccumuloStore (uk.gov.gchq.gaffer.accumulostore.AccumuloStore)7 MockAccumuloStore (uk.gov.gchq.gaffer.accumulostore.MockAccumuloStore)6 AccumuloProperties (uk.gov.gchq.gaffer.accumulostore.AccumuloProperties)5 FileSystem (org.apache.hadoop.fs.FileSystem)3 Path (org.apache.hadoop.fs.Path)3 JobConf (org.apache.hadoop.mapred.JobConf)3 Test (org.junit.Test)3 Schema (uk.gov.gchq.gaffer.store.schema.Schema)3 BufferedWriter (java.io.BufferedWriter)2 OutputStreamWriter (java.io.OutputStreamWriter)2 KeyRangePartitioner (org.apache.accumulo.core.client.mapreduce.lib.partition.KeyRangePartitioner)2 RangePartitioner (org.apache.accumulo.core.client.mapreduce.lib.partition.RangePartitioner)2 Job (org.apache.hadoop.mapreduce.Job)2 SingleUseMockAccumuloStore (uk.gov.gchq.gaffer.accumulostore.SingleUseMockAccumuloStore)2 AddElementsFromHdfs (uk.gov.gchq.gaffer.hdfs.operation.AddElementsFromHdfs)2 File (java.io.File)1 HashSet (java.util.HashSet)1 SequenceFile (org.apache.hadoop.io.SequenceFile)1 Text (org.apache.hadoop.io.Text)1 ByteEntityKeyPackage (uk.gov.gchq.gaffer.accumulostore.key.core.impl.byteEntity.ByteEntityKeyPackage)1