Search in sources :

Example 71 with PackingPlan

use of org.apache.heron.spi.packing.PackingPlan in project heron by twitter.

the class UpdateJsonDryRunRenderer method render.

public String render() {
    StringBuilder builder = new StringBuilder();
    JsonFormatterUtils newFormatter = new JsonFormatterUtils();
    JsonFormatterUtils oldFormatter = new JsonFormatterUtils();
    String topologyName = response.getTopology().getName();
    String packingClassName = Context.packingClass(response.getConfig());
    PackingPlan newPackingPlan = response.getPackingPlan();
    PackingPlan oldPackingPlan = response.getOldPackingPlan();
    try {
        String newPackingPlanJson = newFormatter.renderPackingPlan(topologyName, packingClassName, newPackingPlan);
        String oldPackingPlanJson = oldFormatter.renderPackingPlan(topologyName, packingClassName, oldPackingPlan);
        builder.append("New packing plan:\n");
        builder.append(newPackingPlanJson).append("\n");
        builder.append("Old packing plan:\n");
        builder.append(oldPackingPlanJson).append("\n");
        return builder.toString();
    } catch (JsonProcessingException e) {
        return "ERROR: " + e.getMessage();
    }
}
Also used : PackingPlan(org.apache.heron.spi.packing.PackingPlan) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException)

Example 72 with PackingPlan

use of org.apache.heron.spi.packing.PackingPlan in project heron by twitter.

the class RuntimeManagerRunner method buildNewPackingPlan.

@VisibleForTesting
PackingPlans.PackingPlan buildNewPackingPlan(PackingPlans.PackingPlan currentProtoPlan, Map<String, Integer> changeRequests, Integer containerNum, TopologyAPI.Topology topology) throws PackingException {
    PackingPlanProtoDeserializer deserializer = new PackingPlanProtoDeserializer();
    PackingPlanProtoSerializer serializer = new PackingPlanProtoSerializer();
    PackingPlan currentPackingPlan = deserializer.fromProto(currentProtoPlan);
    Map<String, Integer> componentCounts = currentPackingPlan.getComponentCounts();
    Map<String, Integer> componentChanges = parallelismDelta(componentCounts, changeRequests);
    // Create an instance of the packing class
    String repackingClass = Context.repackingClass(config);
    IRepacking packing;
    try {
        // create an instance of the packing class
        packing = ReflectionUtils.newInstance(repackingClass);
    } catch (IllegalAccessException | InstantiationException | ClassNotFoundException e) {
        throw new IllegalArgumentException("Failed to instantiate packing instance: " + repackingClass, e);
    }
    LOG.info("Updating packing plan using " + repackingClass);
    try {
        packing.initialize(config, topology);
        PackingPlan packedPlan = null;
        if (containerNum == null) {
            packedPlan = packing.repack(currentPackingPlan, componentChanges);
        } else {
            packedPlan = packing.repack(currentPackingPlan, containerNum, componentChanges);
        }
        return serializer.toProto(packedPlan);
    } finally {
        SysUtils.closeIgnoringExceptions(packing);
    }
}
Also used : PackingPlanProtoDeserializer(org.apache.heron.spi.packing.PackingPlanProtoDeserializer) PackingPlanProtoSerializer(org.apache.heron.spi.packing.PackingPlanProtoSerializer) IRepacking(org.apache.heron.spi.packing.IRepacking) PackingPlan(org.apache.heron.spi.packing.PackingPlan) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Example 73 with PackingPlan

use of org.apache.heron.spi.packing.PackingPlan in project heron by twitter.

the class RuntimeManagerRunner method getCurrentContainerNumber.

private int getCurrentContainerNumber(String topologyName) {
    SchedulerStateManagerAdaptor manager = Runtime.schedulerStateManagerAdaptor(runtime);
    PackingPlans.PackingPlan currentPlan = manager.getPackingPlan(topologyName);
    PackingPlanProtoDeserializer deserializer = new PackingPlanProtoDeserializer();
    PackingPlan cPlan = deserializer.fromProto(currentPlan);
    return cPlan.getContainers().size();
}
Also used : PackingPlans(org.apache.heron.proto.system.PackingPlans) PackingPlanProtoDeserializer(org.apache.heron.spi.packing.PackingPlanProtoDeserializer) PackingPlan(org.apache.heron.spi.packing.PackingPlan) SchedulerStateManagerAdaptor(org.apache.heron.spi.statemgr.SchedulerStateManagerAdaptor)

Example 74 with PackingPlan

use of org.apache.heron.spi.packing.PackingPlan in project heron by twitter.

the class RuntimeManagerRunner method containersNumChangeDetected.

private static boolean containersNumChangeDetected(PackingPlans.PackingPlan currentProtoPlan, int numContainers) {
    PackingPlanProtoDeserializer deserializer = new PackingPlanProtoDeserializer();
    PackingPlan currentPlan = deserializer.fromProto(currentProtoPlan);
    return currentPlan.getContainers().size() != numContainers;
}
Also used : PackingPlanProtoDeserializer(org.apache.heron.spi.packing.PackingPlanProtoDeserializer) PackingPlan(org.apache.heron.spi.packing.PackingPlan)

Example 75 with PackingPlan

use of org.apache.heron.spi.packing.PackingPlan in project heron by twitter.

the class UpdateDryRunRenderTest method setUp.

@Before
public void setUp() throws Exception {
    // set up original packing plan
    final String COMPONENT_A = "exclaim1";
    final String COMPONENT_B = "word";
    ContainerPlan containerPlanA = PackingTestUtils.testContainerPlan(1, new Pair<>(COMPONENT_A, 1), new Pair<>(COMPONENT_A, 3), new Pair<>(COMPONENT_B, 5));
    ContainerPlan containerPlanB = PackingTestUtils.testContainerPlan(2, new Pair<>(COMPONENT_A, 2), new Pair<>(COMPONENT_A, 4), new Pair<>(COMPONENT_B, 6));
    Set<ContainerPlan> containerPlans = new HashSet<>();
    containerPlans.add(containerPlanA);
    containerPlans.add(containerPlanB);
    originalPlan = new PackingPlan("ORIG", containerPlans);
    // setup new packing plan A: word:1, exclaim1:9
    Set<ContainerPlan> containerPlansA = new HashSet<>();
    containerPlansA.add(containerPlanA);
    containerPlansA.add(PackingTestUtils.testContainerPlan(2, new Pair<>(COMPONENT_A, 4), new Pair<>(COMPONENT_A, 2), new Pair<>(COMPONENT_A, 6)));
    containerPlansA.add(PackingTestUtils.testContainerPlan(3, new Pair<>(COMPONENT_A, 7), new Pair<>(COMPONENT_A, 8), new Pair<>(COMPONENT_A, 9)));
    containerPlansA.add(PackingTestUtils.testContainerPlan(4, new Pair<>(COMPONENT_A, 10)));
    newPlanA = new PackingPlan("A", containerPlansA);
    // setup new packing plan B: word:1, exclaim1:1
    Set<ContainerPlan> containerPlansB = new HashSet<>();
    containerPlansB.add(PackingTestUtils.testContainerPlan(1, new Pair<>(COMPONENT_A, 3), new Pair<>(COMPONENT_B, 5)));
    newPlanB = new PackingPlan("B", containerPlansB);
}
Also used : ContainerPlan(org.apache.heron.spi.packing.PackingPlan.ContainerPlan) PackingPlan(org.apache.heron.spi.packing.PackingPlan) HashSet(java.util.HashSet) Pair(org.apache.heron.common.basics.Pair) Before(org.junit.Before)

Aggregations

PackingPlan (org.apache.heron.spi.packing.PackingPlan)83 Test (org.junit.Test)43 HashSet (java.util.HashSet)32 Resource (org.apache.heron.spi.packing.Resource)18 Config (org.apache.heron.spi.common.Config)15 PrepareForTest (org.powermock.core.classloader.annotations.PrepareForTest)14 ByteAmount (org.apache.heron.common.basics.ByteAmount)13 HashMap (java.util.HashMap)10 SchedulerStateManagerAdaptor (org.apache.heron.spi.statemgr.SchedulerStateManagerAdaptor)9 TopologyAPI (org.apache.heron.api.generated.TopologyAPI)8 RoundRobinPacking (org.apache.heron.packing.roundrobin.RoundRobinPacking)7 InstanceId (org.apache.heron.spi.packing.InstanceId)7 Pair (org.apache.heron.common.basics.Pair)6 PackingPlans (org.apache.heron.proto.system.PackingPlans)6 PackingPlanProtoDeserializer (org.apache.heron.spi.packing.PackingPlanProtoDeserializer)6 ContainerPlan (org.apache.heron.spi.packing.PackingPlan.ContainerPlan)5 ArrayList (java.util.ArrayList)4 PackingPlanProtoSerializer (org.apache.heron.spi.packing.PackingPlanProtoSerializer)4 EvaluatorRequest (org.apache.reef.driver.evaluator.EvaluatorRequest)4 Before (org.junit.Before)3