Search in sources :

Example 1 with JobFactory

use of uk.gov.gchq.gaffer.hdfs.operation.handler.job.factory.JobFactory in project Gaffer by gchq.

the class HBaseAddElementsFromHdfsJobFactoryTest method shouldSetupJob.

@Test
public void shouldSetupJob() throws IOException, StoreException {
    // Given
    final JobConf localConf = createLocalConf();
    final FileSystem fs = FileSystem.getLocal(localConf);
    fs.mkdirs(new Path(outputDir));
    final JobFactory factory = getJobFactory();
    final Job job = mock(Job.class);
    final MapReduce operation = getMapReduceOperation();
    final HBaseStore store = new SingleUseMiniHBaseStore();
    final Schema schema = Schema.fromJson(StreamUtil.schemas(getClass()));
    final HBaseProperties properties = HBaseProperties.loadStoreProperties(StreamUtil.storeProps(getClass()));
    store.initialise("graphId", schema, properties);
    given(job.getConfiguration()).willReturn(localConf);
    // When
    factory.setupJob(job, operation, TextMapperGeneratorImpl.class.getName(), store);
    // Then
    verify(job).setJarByClass(factory.getClass());
    verify(job).setJobName("Ingest HDFS data: Generator=" + TextMapperGeneratorImpl.class.getName() + ", output=" + outputDir);
    verify(job).setMapperClass(AddElementsFromHdfsMapper.class);
    verify(job).setMapOutputKeyClass(ImmutableBytesWritable.class);
    verify(job).setMapOutputValueClass(KeyValue.class);
    verify(job).setReducerClass(AddElementsFromHdfsReducer.class);
    verify(job).setOutputKeyClass(ImmutableBytesWritable.class);
    verify(job).setOutputValueClass(KeyValue.class);
    verify(job).setOutputFormatClass(HFileOutputFormat2.class);
    assertEquals(fs.makeQualified(new Path(outputDir)).toString(), job.getConfiguration().get("mapreduce.output.fileoutputformat.outputdir"));
    verify(job).setNumReduceTasks(1);
}
Also used : Path(org.apache.hadoop.fs.Path) JobFactory(uk.gov.gchq.gaffer.hdfs.operation.handler.job.factory.JobFactory) SingleUseMiniHBaseStore(uk.gov.gchq.gaffer.hbasestore.SingleUseMiniHBaseStore) HBaseStore(uk.gov.gchq.gaffer.hbasestore.HBaseStore) FileSystem(org.apache.hadoop.fs.FileSystem) Schema(uk.gov.gchq.gaffer.store.schema.Schema) SingleUseMiniHBaseStore(uk.gov.gchq.gaffer.hbasestore.SingleUseMiniHBaseStore) Job(org.apache.hadoop.mapreduce.Job) HBaseProperties(uk.gov.gchq.gaffer.hbasestore.HBaseProperties) JobConf(org.apache.hadoop.mapred.JobConf) MapReduce(uk.gov.gchq.gaffer.hdfs.operation.MapReduce) Test(org.junit.jupiter.api.Test) AbstractJobFactoryTest(uk.gov.gchq.gaffer.hdfs.operation.hander.job.factory.AbstractJobFactoryTest)

Aggregations

FileSystem (org.apache.hadoop.fs.FileSystem)1 Path (org.apache.hadoop.fs.Path)1 JobConf (org.apache.hadoop.mapred.JobConf)1 Job (org.apache.hadoop.mapreduce.Job)1 Test (org.junit.jupiter.api.Test)1 HBaseProperties (uk.gov.gchq.gaffer.hbasestore.HBaseProperties)1 HBaseStore (uk.gov.gchq.gaffer.hbasestore.HBaseStore)1 SingleUseMiniHBaseStore (uk.gov.gchq.gaffer.hbasestore.SingleUseMiniHBaseStore)1 MapReduce (uk.gov.gchq.gaffer.hdfs.operation.MapReduce)1 AbstractJobFactoryTest (uk.gov.gchq.gaffer.hdfs.operation.hander.job.factory.AbstractJobFactoryTest)1 JobFactory (uk.gov.gchq.gaffer.hdfs.operation.handler.job.factory.JobFactory)1 Schema (uk.gov.gchq.gaffer.store.schema.Schema)1