Search in sources :

Example 1 with CassandraHadoopScanRunner

use of com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner in project titan by thinkaurelius.

the class CassandraScanJobIT method testSimpleScan.

@Test
public void testSimpleScan() throws InterruptedException, ExecutionException, IOException, BackendException {
    int keys = 1000;
    int cols = 40;
    String[][] values = KeyValueStoreUtil.generateData(keys, cols);
    //Make it only half the number of columns for every 2nd key
    for (int i = 0; i < values.length; i++) {
        if (i % 2 == 0)
            values[i] = Arrays.copyOf(values[i], cols / 2);
    }
    log.debug("Loading values: " + keys + "x" + cols);
    KeyColumnValueStoreManager mgr = new CassandraThriftStoreManager(GraphDatabaseConfiguration.buildGraphConfiguration());
    KeyColumnValueStore store = mgr.openDatabase("edgestore");
    StoreTransaction tx = mgr.beginTransaction(StandardBaseTransactionConfig.of(TimestampProviders.MICRO));
    KeyColumnValueStoreUtil.loadValues(store, tx, values);
    // noop on Cassandra, but harmless
    tx.commit();
    SimpleScanJobRunner runner = (ScanJob job, Configuration jobConf, String rootNSName) -> {
        try {
            return new CassandraHadoopScanRunner(job).scanJobConf(jobConf).scanJobConfRoot(rootNSName).partitionerOverride("org.apache.cassandra.dht.Murmur3Partitioner").run();
        } catch (ClassNotFoundException e) {
            throw new RuntimeException(e);
        }
    };
    SimpleScanJob.runBasicTests(keys, cols, runner);
}
Also used : KeyColumnValueStore(com.thinkaurelius.titan.diskstorage.keycolumnvalue.KeyColumnValueStore) GraphDatabaseConfiguration(com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration) TitanHadoopConfiguration(com.thinkaurelius.titan.hadoop.config.TitanHadoopConfiguration) CassandraThriftStoreManager(com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager) StoreTransaction(com.thinkaurelius.titan.diskstorage.keycolumnvalue.StoreTransaction) KeyColumnValueStoreManager(com.thinkaurelius.titan.diskstorage.keycolumnvalue.KeyColumnValueStoreManager) CassandraHadoopScanRunner(com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner) ScanJob(com.thinkaurelius.titan.diskstorage.keycolumnvalue.scan.ScanJob) Test(org.junit.Test) TitanGraphBaseTest(com.thinkaurelius.titan.graphdb.TitanGraphBaseTest)

Example 2 with CassandraHadoopScanRunner

use of com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner in project titan by thinkaurelius.

the class MapReduceIndexJobs method cassandraRepair.

public static ScanMetrics cassandraRepair(Properties titanProperties, String indexName, String relationType, String partitionerName, Configuration hadoopBaseConf) throws InterruptedException, IOException, ClassNotFoundException {
    IndexRepairJob job = new IndexRepairJob();
    CassandraHadoopScanRunner cr = new CassandraHadoopScanRunner(job);
    ModifiableConfiguration mc = getIndexJobConf(indexName, relationType);
    copyPropertiesToInputAndOutputConf(hadoopBaseConf, titanProperties);
    cr.partitionerOverride(partitionerName);
    cr.scanJobConf(mc);
    cr.scanJobConfRoot(GraphDatabaseConfiguration.class.getName() + "#JOB_NS");
    cr.baseHadoopConf(hadoopBaseConf);
    return cr.run();
}
Also used : IndexRepairJob(com.thinkaurelius.titan.graphdb.olap.job.IndexRepairJob) CassandraHadoopScanRunner(com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner) ModifiableConfiguration(com.thinkaurelius.titan.diskstorage.configuration.ModifiableConfiguration)

Example 3 with CassandraHadoopScanRunner

use of com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner in project titan by thinkaurelius.

the class MapReduceIndexJobs method cassandraRemove.

public static ScanMetrics cassandraRemove(Properties titanProperties, String indexName, String relationType, String partitionerName, Configuration hadoopBaseConf) throws InterruptedException, IOException, ClassNotFoundException {
    IndexRemoveJob job = new IndexRemoveJob();
    CassandraHadoopScanRunner cr = new CassandraHadoopScanRunner(job);
    ModifiableConfiguration mc = getIndexJobConf(indexName, relationType);
    copyPropertiesToInputAndOutputConf(hadoopBaseConf, titanProperties);
    cr.partitionerOverride(partitionerName);
    cr.scanJobConf(mc);
    cr.scanJobConfRoot(GraphDatabaseConfiguration.class.getName() + "#JOB_NS");
    cr.baseHadoopConf(hadoopBaseConf);
    return cr.run();
}
Also used : CassandraHadoopScanRunner(com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner) ModifiableConfiguration(com.thinkaurelius.titan.diskstorage.configuration.ModifiableConfiguration) IndexRemoveJob(com.thinkaurelius.titan.graphdb.olap.job.IndexRemoveJob)

Aggregations

CassandraHadoopScanRunner (com.thinkaurelius.titan.hadoop.scan.CassandraHadoopScanRunner)3 ModifiableConfiguration (com.thinkaurelius.titan.diskstorage.configuration.ModifiableConfiguration)2 CassandraThriftStoreManager (com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager)1 KeyColumnValueStore (com.thinkaurelius.titan.diskstorage.keycolumnvalue.KeyColumnValueStore)1 KeyColumnValueStoreManager (com.thinkaurelius.titan.diskstorage.keycolumnvalue.KeyColumnValueStoreManager)1 StoreTransaction (com.thinkaurelius.titan.diskstorage.keycolumnvalue.StoreTransaction)1 ScanJob (com.thinkaurelius.titan.diskstorage.keycolumnvalue.scan.ScanJob)1 TitanGraphBaseTest (com.thinkaurelius.titan.graphdb.TitanGraphBaseTest)1 GraphDatabaseConfiguration (com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration)1 IndexRemoveJob (com.thinkaurelius.titan.graphdb.olap.job.IndexRemoveJob)1 IndexRepairJob (com.thinkaurelius.titan.graphdb.olap.job.IndexRepairJob)1 TitanHadoopConfiguration (com.thinkaurelius.titan.hadoop.config.TitanHadoopConfiguration)1 Test (org.junit.Test)1