Examples with DAGClient org.apache.tez.dag.api.client.DAGClient used on opensource projects

Example 1 with DAGClient

use of org.apache.tez.dag.api.client.DAGClient in project hive by apache.

the class TezTask method submit.

DAGClient submit(JobConf conf, DAG dag, Path scratchDir, LocalResource appJarLr, TezSessionState sessionState, List<LocalResource> additionalLr, String[] inputOutputJars, Map<String, LocalResource> inputOutputLocalResources) throws Exception {
    perfLogger.PerfLogBegin(CLASS_NAME, PerfLogger.TEZ_SUBMIT_DAG);
    DAGClient dagClient = null;
    Map<String, LocalResource> resourceMap = new HashMap<String, LocalResource>();
    if (additionalLr != null) {
        for (LocalResource lr : additionalLr) {
            if (lr.getType() == LocalResourceType.FILE) {
                // TEZ AM will only localize FILE (no script operators in the AM)
                resourceMap.put(utils.getBaseName(lr), lr);
            }
        }
    }
    try {
        try {
            // ready to start execution on the cluster
            sessionState.getSession().addAppMasterLocalFiles(resourceMap);
            dagClient = sessionState.getSession().submitDAG(dag);
        } catch (SessionNotRunning nr) {
            console.printInfo("Tez session was closed. Reopening...");
            // close the old one, but keep the tmp files around
            // TODO Why is the session being create using a conf instance belonging to TezTask
            //      - instead of the session conf instance.
            TezSessionPoolManager.getInstance().reopenSession(sessionState, this.conf, inputOutputJars, true);
            console.printInfo("Session re-established.");
            dagClient = sessionState.getSession().submitDAG(dag);
        }
    } catch (Exception e) {
        // In case of any other exception, retry. If this also fails, report original error and exit.
        try {
            console.printInfo("Dag submit failed due to " + e.getMessage() + " stack trace: " + Arrays.toString(e.getStackTrace()) + " retrying...");
            TezSessionPoolManager.getInstance().reopenSession(sessionState, this.conf, inputOutputJars, true);
            dagClient = sessionState.getSession().submitDAG(dag);
        } catch (Exception retryException) {
            // we failed to submit after retrying. Destroy session and bail.
            TezSessionPoolManager.getInstance().destroySession(sessionState);
            throw retryException;
        }
    }
    perfLogger.PerfLogEnd(CLASS_NAME, PerfLogger.TEZ_SUBMIT_DAG);
    return new SyncDagClient(dagClient);
}

Example 2 with DAGClient

use of org.apache.tez.dag.api.client.DAGClient in project hive by apache.

the class TezJobMonitor method killRunningJobs.

/**
   * killRunningJobs tries to terminate execution of all
   * currently running tez queries. No guarantees, best effort only.
   *
   * {@link org.apache.hadoop.hive.ql.exec.tez.TezJobExecHelper#killRunningJobs()} makes use of
   * this method via reflection.
   */
public static void killRunningJobs() {
    synchronized (shutdownList) {
        for (DAGClient c : shutdownList) {
            try {
                System.err.println("Trying to shutdown DAG");
                c.tryKillDAG();
            } catch (Exception e) {
            // ignore
            }
        }
    }
}