Search in sources :

Example 1 with FlowProcess

use of cascading.flow.FlowProcess in project elephant-bird by twitter.

the class TestCombinedSequenceFile method testHadoopConf.

@Test
public void testHadoopConf() {
    CombinedSequenceFile csfScheme = new CombinedSequenceFile(Fields.ALL);
    JobConf conf = new JobConf();
    FlowProcess fp = new HadoopFlowProcess();
    Tap<JobConf, RecordReader, OutputCollector> tap = new TempHfs(conf, "test", CombinedSequenceFile.class, false);
    csfScheme.sourceConfInit(fp, tap, conf);
    assertEquals("MapReduceInputFormatWrapper shold wrap mapred.SequenceFileinputFormat", "org.apache.hadoop.mapred.SequenceFileInputFormat", conf.get(MapReduceInputFormatWrapper.CLASS_CONF_KEY));
    assertEquals("Delegate combiner should wrap MapReduceInputFormatWrapper", "com.twitter.elephantbird.mapreduce.input.MapReduceInputFormatWrapper", conf.get(DelegateCombineFileInputFormat.COMBINED_INPUT_FORMAT_DELEGATE));
    assertEquals("DeprecatedInputFormatWrapper should wrap Delegate combiner", "com.twitter.elephantbird.mapreduce.input.combine.DelegateCombineFileInputFormat", conf.get(DeprecatedInputFormatWrapper.CLASS_CONF_KEY));
}
Also used : OutputCollector(org.apache.hadoop.mapred.OutputCollector) HadoopFlowProcess(cascading.flow.hadoop.HadoopFlowProcess) FlowProcess(cascading.flow.FlowProcess) RecordReader(org.apache.hadoop.mapred.RecordReader) HadoopFlowProcess(cascading.flow.hadoop.HadoopFlowProcess) TempHfs(cascading.tap.hadoop.util.TempHfs) JobConf(org.apache.hadoop.mapred.JobConf) Test(org.junit.Test)

Example 2 with FlowProcess

use of cascading.flow.FlowProcess in project elephant-bird by twitter.

the class TestCombinedSequenceFile method testHadoopConf.

@Test
public void testHadoopConf() {
    CombinedSequenceFile csfScheme = new CombinedSequenceFile(Fields.ALL);
    JobConf conf = new JobConf();
    FlowProcess fp = new HadoopFlowProcess();
    Tap<Configuration, RecordReader, OutputCollector> tap = new TempHfs(conf, "test", CombinedSequenceFile.class, false);
    csfScheme.sourceConfInit(fp, tap, conf);
    assertEquals("MapReduceInputFormatWrapper shold wrap mapred.SequenceFileinputFormat", "org.apache.hadoop.mapred.SequenceFileInputFormat", conf.get(MapReduceInputFormatWrapper.CLASS_CONF_KEY));
    assertEquals("Delegate combiner should wrap MapReduceInputFormatWrapper", "com.twitter.elephantbird.mapreduce.input.MapReduceInputFormatWrapper", conf.get(DelegateCombineFileInputFormat.COMBINED_INPUT_FORMAT_DELEGATE));
    assertEquals("DeprecatedInputFormatWrapper should wrap Delegate combiner", "com.twitter.elephantbird.mapreduce.input.combine.DelegateCombineFileInputFormat", conf.get(DeprecatedInputFormatWrapper.CLASS_CONF_KEY));
}
Also used : OutputCollector(org.apache.hadoop.mapred.OutputCollector) HadoopFlowProcess(cascading.flow.hadoop.HadoopFlowProcess) FlowProcess(cascading.flow.FlowProcess) Configuration(org.apache.hadoop.conf.Configuration) RecordReader(org.apache.hadoop.mapred.RecordReader) HadoopFlowProcess(cascading.flow.hadoop.HadoopFlowProcess) TempHfs(cascading.tap.hadoop.util.TempHfs) JobConf(org.apache.hadoop.mapred.JobConf) Test(org.junit.Test)

Aggregations

FlowProcess (cascading.flow.FlowProcess)2 HadoopFlowProcess (cascading.flow.hadoop.HadoopFlowProcess)2 TempHfs (cascading.tap.hadoop.util.TempHfs)2 JobConf (org.apache.hadoop.mapred.JobConf)2 OutputCollector (org.apache.hadoop.mapred.OutputCollector)2 RecordReader (org.apache.hadoop.mapred.RecordReader)2 Test (org.junit.Test)2 Configuration (org.apache.hadoop.conf.Configuration)1