Search in sources :

Example 1 with PairRDDFunctions

use of org.apache.spark.rdd.PairRDDFunctions in project Gaffer by gchq.

the class ImportKeyValuePairRDDToAccumuloHandler method prepareKeyValues.

@Override
protected void prepareKeyValues(final ImportKeyValuePairRDDToAccumulo operation, final AccumuloKeyRangePartitioner partitioner) throws OperationException {
    final OrderedRDDFunctions orderedRDDFunctions = new OrderedRDDFunctions(operation.getInput(), ORDERING_CLASS_TAG, KEY_CLASS_TAG, VALUE_CLASS_TAG, scala.reflect.ClassTag$.MODULE$.apply(Tuple2.class));
    final PairRDDFunctions pairRDDFunctions = new PairRDDFunctions(orderedRDDFunctions.repartitionAndSortWithinPartitions(partitioner), KEY_CLASS_TAG, VALUE_CLASS_TAG, ORDERING_CLASS_TAG);
    pairRDDFunctions.saveAsNewAPIHadoopFile(operation.getOutputPath(), Key.class, Value.class, AccumuloFileOutputFormat.class, getConfiguration(operation));
}
Also used : PairRDDFunctions(org.apache.spark.rdd.PairRDDFunctions) OrderedRDDFunctions(org.apache.spark.rdd.OrderedRDDFunctions) Tuple2(scala.Tuple2)

Aggregations

OrderedRDDFunctions (org.apache.spark.rdd.OrderedRDDFunctions)1 PairRDDFunctions (org.apache.spark.rdd.PairRDDFunctions)1 Tuple2 (scala.Tuple2)1