Examples with Topology org.apache.heron.api.generated.TopologyAPI.Topology used on opensource projects

Example 1 with Topology

use of org.apache.heron.api.generated.TopologyAPI.Topology in project heron by twitter.

the class HeronExecutorTaskTest method providesConfigsNeededForExecutorCmd.

@Test
public void providesConfigsNeededForExecutorCmd() throws Exception {
    Topology testTopology = createTestTopology("testTopology");
    HeronExecutorTask spyTask = getSpyOnHeronExecutorTask(null);
    Mockito.doReturn("file").when(spyTask).getTopologyDefnFile();
    Mockito.doReturn(testTopology).when(spyTask).getTopology("file");
    String[] command = spyTask.getExecutorCommand();
    // only two configs; state manager root and url should be null.
    int nullCounter = 2;
    for (String subCommand : command) {
        String[] flagArg = SchedulerUtils.splitCommandArg(subCommand);
        if (flagArg.length > 1 && flagArg[1].equals("null")) {
            nullCounter--;
        }
    }
    Assert.assertEquals(0, nullCounter);
}

Example 2 with Topology

use of org.apache.heron.api.generated.TopologyAPI.Topology in project heron by twitter.

the class ScaleUpResolver method buildNewPackingPlan.

@VisibleForTesting
PackingPlan buildNewPackingPlan(Map<String, Integer> changeRequests, PackingPlan currentPackingPlan) {
    Map<String, Integer> componentDeltas = new HashMap<>();
    Map<String, Integer> componentCounts = currentPackingPlan.getComponentCounts();
    for (String compName : changeRequests.keySet()) {
        if (!componentCounts.containsKey(compName)) {
            throw new IllegalArgumentException(String.format("Invalid component name in scale up diagnosis: %s. Valid components include: %s", compName, Arrays.toString(componentCounts.keySet().toArray(new String[componentCounts.keySet().size()]))));
        }
        Integer newValue = changeRequests.get(compName);
        int delta = newValue - componentCounts.get(compName);
        if (delta == 0) {
            LOG.info(String.format("New parallelism for %s is unchanged: %d", compName, newValue));
            continue;
        }
        componentDeltas.put(compName, delta);
    }
    // Create an instance of the packing class
    IRepacking packing = getRepackingClass(Context.repackingClass(config));
    Topology topology = physicalPlanProvider.get().getTopology();
    try {
        packing.initialize(config, topology);
        return packing.repack(currentPackingPlan, componentDeltas);
    } finally {
        SysUtils.closeIgnoringExceptions(packing);
    }
}

Example 3 with Topology

use of org.apache.heron.api.generated.TopologyAPI.Topology in project heron by twitter.

the class HeronExecutorTask method getExecutorCommand.

String[] getExecutorCommand() throws InvalidTopologyException {
    String topologyDefFile = getTopologyDefnFile();
    Topology topology = getTopology(topologyDefFile);
    Config config = SchedulerConfigUtils.loadConfig(cluster, role, env, topologyJar, topologyDefFile, verboseMode, topology);
    Config runtime = Config.newBuilder().put(Key.COMPONENT_RAMMAP, componentRamMap).put(Key.TOPOLOGY_DEFINITION, topology).build();
    Map<ExecutorPort, String> ports = new HashMap<>();
    for (ExecutorPort executorPort : ExecutorPort.getRequiredPorts()) {
        int port = SysUtils.getFreePort();
        if (port == -1) {
            throw new RuntimeException("Failed to find available ports for executor");
        }
        ports.put(executorPort, String.valueOf(port));
    }
    String[] executorCmd = SchedulerUtils.getExecutorCommand(config, runtime, heronExecutorId, ports);
    LOG.info("Executor command line: " + Arrays.toString(executorCmd));
    return executorCmd;
}