Search in sources :

Example 1 with ContainerStopRequest

use of org.apache.tez.serviceplugins.api.ContainerStopRequest in project tez by apache.

the class TestMockDAGAppMaster method testConcurrencyLimit.

@Test(timeout = 100000)
public void testConcurrencyLimit() throws Exception {
    // the test relies on local mode behavior of launching a new container per task.
    // so task concurrency == container concurrency
    TezConfiguration tezconf = new TezConfiguration(defaultConf);
    final int concurrencyLimit = 5;
    MockTezClient tezClient = new MockTezClient("testMockAM", tezconf, true, null, null, null, null, false, false, concurrencyLimit * 4, 1000);
    tezClient.start();
    MockDAGAppMaster mockApp = tezClient.getLocalClient().getMockApp();
    MockContainerLauncher mockLauncher = mockApp.getContainerLauncher();
    mockLauncher.startScheduling(false);
    final AtomicInteger concurrency = new AtomicInteger(0);
    final AtomicBoolean exceededConcurrency = new AtomicBoolean(false);
    mockApp.containerDelegate = new ContainerDelegate() {

        @Override
        public void stop(ContainerStopRequest event) {
            concurrency.decrementAndGet();
        }

        @Override
        public void launch(ContainerLaunchRequest event) {
            int maxConc = concurrency.incrementAndGet();
            if (maxConc > concurrencyLimit) {
                exceededConcurrency.set(true);
            }
            System.out.println("Launched: " + maxConc);
        }
    };
    DAG dag = DAG.create("testConcurrencyLimit");
    Vertex vA = Vertex.create("A", ProcessorDescriptor.create("Proc.class"), 20).setConf(TezConfiguration.TEZ_AM_VERTEX_MAX_TASK_CONCURRENCY, String.valueOf(concurrencyLimit));
    dag.addVertex(vA);
    mockLauncher.startScheduling(true);
    DAGClient dagClient = tezClient.submitDAG(dag);
    dagClient.waitForCompletion();
    Assert.assertEquals(DAGStatus.State.SUCCEEDED, dagClient.getDAGStatus(null).getState());
    Assert.assertFalse(exceededConcurrency.get());
    tezClient.stop();
}
Also used : Vertex(org.apache.tez.dag.api.Vertex) ContainerDelegate(org.apache.tez.dag.app.MockDAGAppMaster.ContainerDelegate) ContainerStopRequest(org.apache.tez.serviceplugins.api.ContainerStopRequest) DAG(org.apache.tez.dag.api.DAG) MockContainerLauncher(org.apache.tez.dag.app.MockDAGAppMaster.MockContainerLauncher) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) ContainerLaunchRequest(org.apache.tez.serviceplugins.api.ContainerLaunchRequest) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) DAGClient(org.apache.tez.dag.api.client.DAGClient) TezConfiguration(org.apache.tez.dag.api.TezConfiguration) Test(org.junit.Test)

Example 2 with ContainerStopRequest

use of org.apache.tez.serviceplugins.api.ContainerStopRequest in project tez by apache.

the class ContainerLauncherManager method handle.

@Override
public void handle(ContainerLauncherEvent event) {
    int launcherId = event.getLauncherId();
    String schedulerName = appContext.getTaskSchedulerName(event.getSchedulerId());
    String taskCommName = appContext.getTaskCommunicatorName(event.getTaskCommId());
    switch(event.getType()) {
        case CONTAINER_LAUNCH_REQUEST:
            ContainerLauncherLaunchRequestEvent launchEvent = (ContainerLauncherLaunchRequestEvent) event;
            ContainerLaunchRequest launchRequest = new ContainerLaunchRequest(launchEvent.getNodeId(), launchEvent.getContainerId(), launchEvent.getContainerToken(), launchEvent.getContainerLaunchContext(), launchEvent.getContainer(), schedulerName, taskCommName);
            try {
                containerLaunchers[launcherId].launchContainer(launchRequest);
            } catch (Exception e) {
                String msg = "Error when launching container" + ", containerLauncher=" + Utils.getContainerLauncherIdentifierString(launcherId, appContext) + ", scheduler=" + Utils.getTaskSchedulerIdentifierString(event.getSchedulerId(), appContext) + ", taskCommunicator=" + Utils.getTaskCommIdentifierString(event.getTaskCommId(), appContext);
                LOG.error(msg, e);
                sendEvent(new DAGAppMasterEventUserServiceFatalError(DAGAppMasterEventType.CONTAINER_LAUNCHER_SERVICE_FATAL_ERROR, msg, e));
            }
            break;
        case CONTAINER_STOP_REQUEST:
            ContainerStopRequest stopRequest = new ContainerStopRequest(event.getNodeId(), event.getContainerId(), event.getContainerToken(), schedulerName, taskCommName);
            try {
                containerLaunchers[launcherId].stopContainer(stopRequest);
            } catch (Exception e) {
                String msg = "Error when stopping container" + ", containerLauncher=" + Utils.getContainerLauncherIdentifierString(launcherId, appContext) + ", scheduler=" + Utils.getTaskSchedulerIdentifierString(event.getSchedulerId(), appContext) + ", taskCommunicator=" + Utils.getTaskCommIdentifierString(event.getTaskCommId(), appContext);
                LOG.error(msg, e);
                sendEvent(new DAGAppMasterEventUserServiceFatalError(DAGAppMasterEventType.CONTAINER_LAUNCHER_SERVICE_FATAL_ERROR, msg, e));
            }
            break;
    }
}
Also used : DAGAppMasterEventUserServiceFatalError(org.apache.tez.dag.app.dag.event.DAGAppMasterEventUserServiceFatalError) ContainerLaunchRequest(org.apache.tez.serviceplugins.api.ContainerLaunchRequest) ContainerStopRequest(org.apache.tez.serviceplugins.api.ContainerStopRequest) ContainerLauncherLaunchRequestEvent(org.apache.tez.dag.app.rm.ContainerLauncherLaunchRequestEvent) TezUncheckedException(org.apache.tez.dag.api.TezUncheckedException) UnknownHostException(java.net.UnknownHostException) TezException(org.apache.tez.dag.api.TezException)

Aggregations

ContainerLaunchRequest (org.apache.tez.serviceplugins.api.ContainerLaunchRequest)2 ContainerStopRequest (org.apache.tez.serviceplugins.api.ContainerStopRequest)2 UnknownHostException (java.net.UnknownHostException)1 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)1 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)1 DAG (org.apache.tez.dag.api.DAG)1 TezConfiguration (org.apache.tez.dag.api.TezConfiguration)1 TezException (org.apache.tez.dag.api.TezException)1 TezUncheckedException (org.apache.tez.dag.api.TezUncheckedException)1 Vertex (org.apache.tez.dag.api.Vertex)1 DAGClient (org.apache.tez.dag.api.client.DAGClient)1 ContainerDelegate (org.apache.tez.dag.app.MockDAGAppMaster.ContainerDelegate)1 MockContainerLauncher (org.apache.tez.dag.app.MockDAGAppMaster.MockContainerLauncher)1 DAGAppMasterEventUserServiceFatalError (org.apache.tez.dag.app.dag.event.DAGAppMasterEventUserServiceFatalError)1 ContainerLauncherLaunchRequestEvent (org.apache.tez.dag.app.rm.ContainerLauncherLaunchRequestEvent)1 Test (org.junit.Test)1