Examples with VespaCounters com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters used on opensource projects

Example 1 with VespaCounters

use of com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters in project vespa by vespa-engine.

the class VespaStorageTest method assertAllDocumentsOk.

private void assertAllDocumentsOk(String script, Configuration conf) throws Exception {
    PigServer ps = setup(script, conf);
    List<ExecJob> jobs = ps.executeBatch();
    PigStats stats = jobs.get(0).getStatistics();
    for (JobStats js : stats.getJobGraph()) {
        Counters hadoopCounters = ((MRJobStats) js).getHadoopCounters();
        assertNotNull(hadoopCounters);
        VespaCounters counters = VespaCounters.get(hadoopCounters);
        assertEquals(10, counters.getDocumentsSent());
        assertEquals(0, counters.getDocumentsFailed());
        assertEquals(10, counters.getDocumentsOk());
    }
}

Example 2 with VespaCounters

use of com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters in project vespa by vespa-engine.

the class MapReduceTest method requireThatMapOnlyJobSucceeds.

@Test
public void requireThatMapOnlyJobSucceeds() throws Exception {
    Job job = Job.getInstance(conf);
    job.setJarByClass(MapReduceTest.class);
    job.setMapperClass(FeedMapper.class);
    job.setOutputFormatClass(VespaOutputFormat.class);
    job.setMapOutputValueClass(Text.class);
    FileInputFormat.setInputPaths(job, metricsJsonPath);
    boolean success = job.waitForCompletion(true);
    assertTrue("Job Failed", success);
    VespaCounters counters = VespaCounters.get(job);
    assertEquals(10, counters.getDocumentsSent());
    assertEquals(0, counters.getDocumentsFailed());
    assertEquals(10, counters.getDocumentsOk());
}

Example 3 with VespaCounters

use of com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters in project vespa by vespa-engine.

the class MapReduceTest method requireThatMapReduceJobSucceeds.

@Test
public void requireThatMapReduceJobSucceeds() throws Exception {
    Job job = Job.getInstance(conf);
    job.setJarByClass(MapReduceTest.class);
    job.setMapperClass(FeedMapper.class);
    job.setOutputFormatClass(VespaOutputFormat.class);
    job.setMapOutputValueClass(Text.class);
    job.setReducerClass(FeedReducer.class);
    job.setNumReduceTasks(1);
    FileInputFormat.setInputPaths(job, metricsJsonPath);
    boolean success = job.waitForCompletion(true);
    assertTrue("Job Failed", success);
    VespaCounters counters = VespaCounters.get(job);
    assertEquals(10, counters.getDocumentsSent());
    assertEquals(0, counters.getDocumentsFailed());
    assertEquals(10, counters.getDocumentsOk());
}

Example 4 with VespaCounters

use of com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters in project vespa by vespa-engine.

the class MapReduceTest method requireThatTransformMapJobSucceeds.

@Test
public void requireThatTransformMapJobSucceeds() throws Exception {
    Job job = Job.getInstance(conf);
    job.setJarByClass(MapReduceTest.class);
    job.setMapperClass(ParsingMapper.class);
    job.setOutputFormatClass(VespaOutputFormat.class);
    job.setMapOutputValueClass(Text.class);
    job.setReducerClass(FeedReducer.class);
    job.setNumReduceTasks(1);
    FileInputFormat.setInputPaths(job, metricsCsvPath);
    boolean success = job.waitForCompletion(true);
    assertTrue("Job Failed", success);
    VespaCounters counters = VespaCounters.get(job);
    assertEquals(10, counters.getDocumentsSent());
    assertEquals(0, counters.getDocumentsFailed());
    assertEquals(10, counters.getDocumentsOk());
    assertEquals(0, counters.getDocumentsSkipped());
}

Example 5 with VespaCounters

use of com.yahoo.vespa.hadoop.mapreduce.util.VespaCounters in project vespa by vespa-engine.

the class VespaOutputFormat method getRecordWriter.

@Override
public RecordWriter getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
    VespaCounters counters = VespaCounters.get(context);
    VespaConfiguration configuration = VespaConfiguration.get(context.getConfiguration(), configOverride);
    return new VespaRecordWriter(configuration, counters);
}