Search in sources :

Example 61 with DAGPlan

use of org.apache.tez.dag.api.records.DAGProtos.DAGPlan in project tez by apache.

the class DAGClientAMProtocolBlockingPBServerImpl method submitDAG.

@Override
public SubmitDAGResponseProto submitDAG(RpcController controller, SubmitDAGRequestProto request) throws ServiceException {
    UserGroupInformation user = getRPCUser();
    if (!real.getACLManager().checkAMModifyAccess(user)) {
        throw new AccessControlException("User " + user + " cannot perform AM modify operation");
    }
    real.updateLastHeartbeatTime();
    try {
        if (request.hasSerializedRequestPath()) {
            // need to deserialize large request from hdfs
            Path requestPath = new Path(request.getSerializedRequestPath());
            try (FSDataInputStream fsDataInputStream = stagingFs.open(requestPath)) {
                CodedInputStream in = CodedInputStream.newInstance(fsDataInputStream);
                in.setSizeLimit(Integer.MAX_VALUE);
                request = SubmitDAGRequestProto.parseFrom(in);
            } catch (IOException e) {
                throw wrapException(e);
            }
        }
        DAGPlan dagPlan = request.getDAGPlan();
        Map<String, LocalResource> additionalResources = null;
        if (request.hasAdditionalAmResources()) {
            additionalResources = DagTypeConverters.convertFromPlanLocalResources(request.getAdditionalAmResources());
        }
        String dagId = real.submitDAG(dagPlan, additionalResources);
        return SubmitDAGResponseProto.newBuilder().setDagId(dagId).build();
    } catch (TezException e) {
        throw wrapException(e);
    }
}
Also used : Path(org.apache.hadoop.fs.Path) TezException(org.apache.tez.dag.api.TezException) DAGPlan(org.apache.tez.dag.api.records.DAGProtos.DAGPlan) CodedInputStream(com.google.protobuf.CodedInputStream) AccessControlException(java.security.AccessControlException) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream) IOException(java.io.IOException) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation) LocalResource(org.apache.hadoop.yarn.api.records.LocalResource)

Aggregations

DAGPlan (org.apache.tez.dag.api.records.DAGProtos.DAGPlan)61 TezDAGID (org.apache.tez.dag.records.TezDAGID)20 Path (org.apache.hadoop.fs.Path)19 DAGHistoryEvent (org.apache.tez.dag.history.DAGHistoryEvent)18 DAGSubmittedEvent (org.apache.tez.dag.history.events.DAGSubmittedEvent)18 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)17 Configuration (org.apache.hadoop.conf.Configuration)16 Test (org.junit.Test)16 SystemClock (org.apache.hadoop.yarn.util.SystemClock)15 LocalResource (org.apache.hadoop.yarn.api.records.LocalResource)14 DAGRecoveryData (org.apache.tez.dag.app.RecoveryParser.DAGRecoveryData)12 RecoveryService (org.apache.tez.dag.history.recovery.RecoveryService)11 HashMap (java.util.HashMap)10 TezConfiguration (org.apache.tez.dag.api.TezConfiguration)7 VertexPlan (org.apache.tez.dag.api.records.DAGProtos.VertexPlan)7 ApplicationAttemptId (org.apache.hadoop.yarn.api.records.ApplicationAttemptId)6 TezVertexID (org.apache.tez.dag.records.TezVertexID)6 DefaultHadoopShim (org.apache.tez.hadoop.shim.DefaultHadoopShim)5 Credentials (org.apache.hadoop.security.Credentials)4 DAGInitializedEvent (org.apache.tez.dag.history.events.DAGInitializedEvent)4