Examples with MRPipeline org.apache.crunch.impl.mr.MRPipeline used on opensource projects

Example 26 with MRPipeline

use of org.apache.crunch.impl.mr.MRPipeline in project crunch by cloudera.

the class AvroFileSourceTargetTest method testSpecific.

@Test
public void testSpecific() throws IOException {
    GenericRecord savedRecord = new GenericData.Record(Person.SCHEMA$);
    savedRecord.put("name", "John Doe");
    savedRecord.put("age", 42);
    savedRecord.put("siblingnames", Lists.newArrayList("Jimmy", "Jane"));
    populateGenericFile(Lists.newArrayList(savedRecord), Person.SCHEMA$);
    Pipeline pipeline = new MRPipeline(AvroFileSourceTargetTest.class);
    PCollection<Person> genericCollection = pipeline.read(At.avroFile(avroFile.getAbsolutePath(), Avros.records(Person.class)));
    List<Person> personList = Lists.newArrayList(genericCollection.materialize());
    Person expectedPerson = new Person();
    expectedPerson.setName("John Doe");
    expectedPerson.setAge(42);
    List<CharSequence> siblingNames = Lists.newArrayList();
    siblingNames.add("Jimmy");
    siblingNames.add("Jane");
    expectedPerson.setSiblingnames(siblingNames);
    assertEquals(Lists.newArrayList(expectedPerson), Lists.newArrayList(personList));
}

Example 27 with MRPipeline

use of org.apache.crunch.impl.mr.MRPipeline in project crunch by cloudera.

the class WordCountTest method testWritablesWithSecondUseToOutput.

@Test
public void testWritablesWithSecondUseToOutput() throws IOException {
    runSecond = true;
    useToOutput = true;
    run(new MRPipeline(WordCountTest.class), WritableTypeFamily.getInstance());
}

Example 28 with MRPipeline

use of org.apache.crunch.impl.mr.MRPipeline in project crunch by cloudera.

the class WordCountTest method testAvroWithSecond.

@Test
public void testAvroWithSecond() throws IOException {
    runSecond = true;
    run(new MRPipeline(WordCountTest.class), AvroTypeFamily.getInstance());
}

Example 29 with MRPipeline

use of org.apache.crunch.impl.mr.MRPipeline in project crunch by cloudera.

the class MaterializeToMapTest method testMRMaterializeToMap.

@Test
public void testMRMaterializeToMap() throws IOException {
    Pipeline p = new MRPipeline(MaterializeToMapTest.class);
    String inputFile = FileHelper.createTempCopyOf("set1.txt");
    PCollection<String> c = p.readTextFile(inputFile);
    PTypeFamily tf = c.getTypeFamily();
    PTable<Integer, String> t = c.parallelDo(new Set1Mapper(), tf.tableOf(tf.ints(), tf.strings()));
    Map<Integer, String> m = t.materializeToMap();
    assertMatches(m);
}

Example 30 with MRPipeline

use of org.apache.crunch.impl.mr.MRPipeline in project crunch by cloudera.

the class PCollectionGetSizeTest method testGetSizeOfEmptyIntermediatePCollection_MRPipeline.

@Test
public void testGetSizeOfEmptyIntermediatePCollection_MRPipeline() throws IOException {
    PCollection<String> emptyIntermediate = createPesistentEmptyIntermediate(new MRPipeline(this.getClass()));
    assertThat(emptyIntermediate.getSize(), is(0L));
}