Search in sources :

Example 1 with AllocatedEvaluator

use of org.apache.reef.driver.evaluator.AllocatedEvaluator in project heron by twitter.

the class HeronMasterDriver method killWorkers.

/**
   * Terminates any yarn containers associated with the given containers.
   */
public void killWorkers(Set<ContainerPlan> containers) {
    for (ContainerPlan container : containers) {
        LOG.log(Level.INFO, "Find and kill container for worker {0}", container.getId());
        Optional<HeronWorker> worker = multiKeyWorkerMap.lookupByWorkerId(container.getId());
        if (worker.isPresent()) {
            LOG.log(Level.INFO, "Killing container {0} for worker {1}", new Object[] { worker.get().evaluator.getId(), worker.get().workerId });
            AllocatedEvaluator evaluator = multiKeyWorkerMap.detachEvaluatorAndRemove(worker.get());
            evaluator.close();
        } else {
            LOG.log(Level.WARNING, "Did not find worker for {0}", container.getId());
        }
        containerPlans.remove(container.getId());
    }
}
Also used : ContainerPlan(com.twitter.heron.spi.packing.PackingPlan.ContainerPlan) AllocatedEvaluator(org.apache.reef.driver.evaluator.AllocatedEvaluator)

Example 2 with AllocatedEvaluator

use of org.apache.reef.driver.evaluator.AllocatedEvaluator in project heron by twitter.

the class HeronMasterDriver method killTopology.

public void killTopology() {
    LOG.log(Level.INFO, "Kill topology: {0}", topologyName);
    isTopologyKilled.set(true);
    tMaster.killTMaster();
    for (HeronWorker worker : multiKeyWorkerMap.getHeronWorkers()) {
        AllocatedEvaluator evaluator = multiKeyWorkerMap.detachEvaluatorAndRemove(worker);
        LOG.log(Level.INFO, "Killing container {0} for worker {1}", new Object[] { evaluator.getId(), worker.workerId });
        evaluator.close();
    }
}
Also used : AllocatedEvaluator(org.apache.reef.driver.evaluator.AllocatedEvaluator)

Example 3 with AllocatedEvaluator

use of org.apache.reef.driver.evaluator.AllocatedEvaluator in project heron by twitter.

the class HeronMasterDriverTest method onNextAllocatedEvaluatorStartsWorker.

@Test
public void onNextAllocatedEvaluatorStartsWorker() throws Exception {
    PackingPlan packingPlan = PackingTestUtils.testPackingPlan("test", new RoundRobinPacking());
    spyDriver.scheduleHeronWorkers(packingPlan);
    assertTrue(spyDriver.lookupByContainerPlan(1).isPresent());
    PackingPlan.ContainerPlan containerPlan = spyDriver.lookupByContainerPlan(1).get();
    AllocatedEvaluator mockEvaluator = createMockEvaluator("test", getCpu(containerPlan), getRam(containerPlan));
    assertFalse(spyDriver.lookupByEvaluatorId("test").isPresent());
    spyDriver.new ContainerAllocationHandler().onNext(mockEvaluator);
    assertTrue(spyDriver.lookupByEvaluatorId("test").isPresent());
    assertEquals(Integer.valueOf(1), spyDriver.lookupByEvaluatorId("test").get());
    verify(mockEvaluator, times(1)).submitContext(any(Configuration.class));
}
Also used : RoundRobinPacking(com.twitter.heron.packing.roundrobin.RoundRobinPacking) Configuration(org.apache.reef.tang.Configuration) PackingPlan(com.twitter.heron.spi.packing.PackingPlan) AllocatedEvaluator(org.apache.reef.driver.evaluator.AllocatedEvaluator) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) Test(org.junit.Test)

Example 4 with AllocatedEvaluator

use of org.apache.reef.driver.evaluator.AllocatedEvaluator in project heron by twitter.

the class HeronMasterDriverTest method createMockEvaluator.

private AllocatedEvaluator createMockEvaluator(String evaluatorId, int cores, ByteAmount mem) {
    EvaluatorDescriptor descriptor = mock(EvaluatorDescriptor.class);
    when(descriptor.getMemory()).thenReturn(((Long) mem.asMegabytes()).intValue());
    when(descriptor.getNumberOfCores()).thenReturn(cores);
    AllocatedEvaluator mockEvaluator = mock(AllocatedEvaluator.class);
    when(mockEvaluator.getEvaluatorDescriptor()).thenReturn(descriptor);
    when(mockEvaluator.getId()).thenReturn(evaluatorId);
    return mockEvaluator;
}
Also used : EvaluatorDescriptor(org.apache.reef.driver.evaluator.EvaluatorDescriptor) AllocatedEvaluator(org.apache.reef.driver.evaluator.AllocatedEvaluator)

Example 5 with AllocatedEvaluator

use of org.apache.reef.driver.evaluator.AllocatedEvaluator in project heron by twitter.

the class HeronMasterDriver method restartWorker.

public void restartWorker(int id) throws ContainerAllocationException {
    LOG.log(Level.INFO, "Find & restart container for id={0}", id);
    Optional<HeronWorker> worker = multiKeyWorkerMap.lookupByWorkerId(id);
    if (!worker.isPresent()) {
        LOG.log(Level.WARNING, "Requesting a new container for: {0}", id);
        ContainerPlan containerPlan = containerPlans.get(id);
        if (containerPlan == null) {
            throw new IllegalArgumentException(String.format("There is no container for %s in packing plan.", id));
        }
        worker = Optional.of(new HeronWorker(id, containerPlan.getRequiredResource()));
    } else {
        AllocatedEvaluator evaluator = multiKeyWorkerMap.detachEvaluatorAndRemove(worker.get());
        LOG.log(Level.INFO, "Shutting down container {0}", evaluator.getId());
        evaluator.close();
    }
    requestContainerForWorker(worker.get().workerId, worker.get());
}
Also used : ContainerPlan(com.twitter.heron.spi.packing.PackingPlan.ContainerPlan) AllocatedEvaluator(org.apache.reef.driver.evaluator.AllocatedEvaluator)

Aggregations

AllocatedEvaluator (org.apache.reef.driver.evaluator.AllocatedEvaluator)11 Test (org.junit.Test)5 PrepareForTest (org.powermock.core.classloader.annotations.PrepareForTest)5 HashSet (java.util.HashSet)3 PackingPlan (com.twitter.heron.spi.packing.PackingPlan)2 ContainerPlan (com.twitter.heron.spi.packing.PackingPlan.ContainerPlan)2 EvaluatorDescriptor (org.apache.reef.driver.evaluator.EvaluatorDescriptor)2 Configuration (org.apache.reef.tang.Configuration)2 RoundRobinPacking (com.twitter.heron.packing.roundrobin.RoundRobinPacking)1 EvaluatorRequest (org.apache.reef.driver.evaluator.EvaluatorRequest)1 FailedEvaluator (org.apache.reef.driver.evaluator.FailedEvaluator)1