Search in sources :

Example 1 with JobCommitCompletedEvent

use of org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitCompletedEvent in project hadoop by apache.

the class TestCommitterEventHandler method testBasic.

@Test
public void testBasic() throws Exception {
    AppContext mockContext = mock(AppContext.class);
    OutputCommitter mockCommitter = mock(OutputCommitter.class);
    Clock mockClock = mock(Clock.class);
    CommitterEventHandler handler = new CommitterEventHandler(mockContext, mockCommitter, new TestingRMHeartbeatHandler());
    YarnConfiguration conf = new YarnConfiguration();
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    JobContext mockJobContext = mock(JobContext.class);
    ApplicationAttemptId attemptid = ApplicationAttemptId.fromString("appattempt_1234567890000_0001_0");
    JobId jobId = TypeConverter.toYarn(TypeConverter.fromYarn(attemptid.getApplicationId()));
    WaitForItHandler waitForItHandler = new WaitForItHandler();
    when(mockContext.getApplicationID()).thenReturn(attemptid.getApplicationId());
    when(mockContext.getApplicationAttemptId()).thenReturn(attemptid);
    when(mockContext.getEventHandler()).thenReturn(waitForItHandler);
    when(mockContext.getClock()).thenReturn(mockClock);
    handler.init(conf);
    handler.start();
    try {
        handler.handle(new CommitterJobCommitEvent(jobId, mockJobContext));
        String user = UserGroupInformation.getCurrentUser().getShortUserName();
        Path startCommitFile = MRApps.getStartJobCommitFile(conf, user, jobId);
        Path endCommitSuccessFile = MRApps.getEndJobCommitSuccessFile(conf, user, jobId);
        Path endCommitFailureFile = MRApps.getEndJobCommitFailureFile(conf, user, jobId);
        Event e = waitForItHandler.getAndClearEvent();
        assertNotNull(e);
        assertTrue(e instanceof JobCommitCompletedEvent);
        FileSystem fs = FileSystem.get(conf);
        assertTrue(startCommitFile.toString(), fs.exists(startCommitFile));
        assertTrue(endCommitSuccessFile.toString(), fs.exists(endCommitSuccessFile));
        assertFalse(endCommitFailureFile.toString(), fs.exists(endCommitFailureFile));
        verify(mockCommitter).commitJob(any(JobContext.class));
    } finally {
        handler.stop();
    }
}
Also used : Path(org.apache.hadoop.fs.Path) OutputCommitter(org.apache.hadoop.mapreduce.OutputCommitter) AppContext(org.apache.hadoop.mapreduce.v2.app.AppContext) ApplicationAttemptId(org.apache.hadoop.yarn.api.records.ApplicationAttemptId) Clock(org.apache.hadoop.yarn.util.Clock) SystemClock(org.apache.hadoop.yarn.util.SystemClock) YarnConfiguration(org.apache.hadoop.yarn.conf.YarnConfiguration) FileSystem(org.apache.hadoop.fs.FileSystem) JobCommitCompletedEvent(org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitCompletedEvent) JobEvent(org.apache.hadoop.mapreduce.v2.app.job.event.JobEvent) Event(org.apache.hadoop.yarn.event.Event) JobCommitFailedEvent(org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitFailedEvent) JobCommitCompletedEvent(org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitCompletedEvent) JobContext(org.apache.hadoop.mapreduce.JobContext) JobId(org.apache.hadoop.mapreduce.v2.api.records.JobId) Test(org.junit.Test)

Aggregations

FileSystem (org.apache.hadoop.fs.FileSystem)1 Path (org.apache.hadoop.fs.Path)1 JobContext (org.apache.hadoop.mapreduce.JobContext)1 OutputCommitter (org.apache.hadoop.mapreduce.OutputCommitter)1 JobId (org.apache.hadoop.mapreduce.v2.api.records.JobId)1 AppContext (org.apache.hadoop.mapreduce.v2.app.AppContext)1 JobCommitCompletedEvent (org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitCompletedEvent)1 JobCommitFailedEvent (org.apache.hadoop.mapreduce.v2.app.job.event.JobCommitFailedEvent)1 JobEvent (org.apache.hadoop.mapreduce.v2.app.job.event.JobEvent)1 ApplicationAttemptId (org.apache.hadoop.yarn.api.records.ApplicationAttemptId)1 YarnConfiguration (org.apache.hadoop.yarn.conf.YarnConfiguration)1 Event (org.apache.hadoop.yarn.event.Event)1 Clock (org.apache.hadoop.yarn.util.Clock)1 SystemClock (org.apache.hadoop.yarn.util.SystemClock)1 Test (org.junit.Test)1