Examples with DagInfo org.apache.tez.history.parser.datamodel.DagInfo used on opensource projects

Example 6 with DagInfo

use of org.apache.tez.history.parser.datamodel.DagInfo in project tez by apache.

the class TestHistoryParser method testParserWithSuccessfulJob.

/**
 * Run a word count example in mini cluster and check if it is possible to download
 * data from ATS and parse it. Also, run with SimpleHistoryLogging option and verify
 * if it matches with ATS data.
 *
 * @throws Exception
 */
@Test
public void testParserWithSuccessfulJob() throws Exception {
    // Run basic word count example.
    String dagId = runWordCount(WordCount.TokenProcessor.class.getName(), WordCount.SumProcessor.class.getName(), "WordCount", true);
    // Export the data from ATS
    String[] args = { "--dagId=" + dagId, "--downloadDir=" + DOWNLOAD_DIR, "--yarnTimelineAddress=" + yarnTimelineAddress };
    int result = ATSImportTool.process(args);
    assertTrue(result == 0);
    // Parse ATS data and verify results
    DagInfo dagInfoFromATS = getDagInfo(dagId);
    verifyDagInfo(dagInfoFromATS, true);
    verifyJobSpecificInfo(dagInfoFromATS);
    checkConfig(dagInfoFromATS);
    // Now run with SimpleHistoryLogging
    dagId = runWordCount(WordCount.TokenProcessor.class.getName(), WordCount.SumProcessor.class.getName(), "WordCount", false);
    // For all flushes to happen and to avoid half-cooked download.
    Thread.sleep(10000);
    DagInfo shDagInfo = getDagInfoFromSimpleHistory(dagId);
    verifyDagInfo(shDagInfo, false);
    verifyJobSpecificInfo(shDagInfo);
    // Compare dagInfo by parsing ATS data with DagInfo obtained by parsing SimpleHistoryLog
    isDAGEqual(dagInfoFromATS, shDagInfo);
}

Example 7 with DagInfo

use of org.apache.tez.history.parser.datamodel.DagInfo in project tez by apache.

the class TestHistoryParser method getDagInfoFromSimpleHistory.

private DagInfo getDagInfoFromSimpleHistory(String dagId) throws TezException, IOException {
    TezDAGID tezDAGID = TezDAGID.fromString(dagId);
    ApplicationAttemptId applicationAttemptId = ApplicationAttemptId.newInstance(tezDAGID.getApplicationId(), 1);
    Path historyPath = new Path(conf.get("fs.defaultFS") + SIMPLE_HISTORY_DIR + HISTORY_TXT + "." + applicationAttemptId);
    FileSystem fs = historyPath.getFileSystem(conf);
    Path localPath = new Path(DOWNLOAD_DIR, HISTORY_TXT);
    fs.copyToLocalFile(historyPath, localPath);
    File localFile = new File(DOWNLOAD_DIR, HISTORY_TXT);
    // Now parse via SimpleHistory
    SimpleHistoryParser parser = new SimpleHistoryParser(localFile);
    DagInfo dagInfo = parser.getDAGData(dagId);
    assertTrue(dagInfo.getDagId().equals(dagId));
    return dagInfo;
}

Example 8 with DagInfo

use of org.apache.tez.history.parser.datamodel.DagInfo in project tez by apache.

the class TestHistoryParser method getDagInfo.

private DagInfo getDagInfo(String dagId) throws TezException {
    // Parse downloaded contents
    File downloadedFile = new File(DOWNLOAD_DIR + Path.SEPARATOR + dagId + ".zip");
    ATSFileParser parser = new ATSFileParser(downloadedFile);
    DagInfo dagInfo = parser.getDAGData(dagId);
    assertTrue(dagInfo.getDagId().equals(dagId));
    return dagInfo;
}