Examples with ByteAmount org.apache.heron.common.basics.ByteAmount used on opensource projects

Example 11 with ByteAmount

use of org.apache.heron.common.basics.ByteAmount in project heron by twitter.

the class FirstFitDecreasingPackingTest method testInvalidRamInstance.

/**
 * Test invalid RAM for instance
 */
@Test(expected = PackingException.class)
public void testInvalidRamInstance() throws Exception {
    ByteAmount maxContainerRam = ByteAmount.fromGigabytes(10);
    ByteAmount boltRam = ByteAmount.ZERO;
    topologyConfig.setContainerRamRequested(maxContainerRam);
    topologyConfig.setComponentRam(BOLT_NAME, boltRam);
    topology = getTopology(spoutParallelism, boltParallelism, topologyConfig);
    doPackingTest(topology, instanceDefaultResources.cloneWithRam(boltRam), boltParallelism, instanceDefaultResources, spoutParallelism, 0, getDefaultMaxContainerResource().cloneWithRam(maxContainerRam));
}

Example 12 with ByteAmount

use of org.apache.heron.common.basics.ByteAmount in project heron by twitter.

the class FirstFitDecreasingPackingTest method scaleDownAndUpNoPadding.

/**
 * Test the scenario where scaling down and up is simultaneously requested and padding is
 * configured
 */
@Test
public void scaleDownAndUpNoPadding() throws Exception {
    int paddingPercentage = 0;
    topologyConfig.setContainerPaddingPercentage(paddingPercentage);
    ByteAmount spoutRam = ByteAmount.fromGigabytes(4);
    ByteAmount maxContainerRam = ByteAmount.fromGigabytes(12);
    topologyConfig.setContainerRamRequested(maxContainerRam);
    topologyConfig.setComponentRam(SPOUT_NAME, spoutRam);
    boltParallelism = 3;
    spoutParallelism = 1;
    topology = getTopology(spoutParallelism, boltParallelism, topologyConfig);
    int spoutScalingUp = 1;
    int boltScalingDown = -1;
    Map<String, Integer> componentChanges = new HashMap<>();
    // 2 spouts
    componentChanges.put(SPOUT_NAME, spoutScalingUp);
    // 2 bolts
    componentChanges.put(BOLT_NAME, boltScalingDown);
    int numContainersBeforeRepack = 2;
    int numContainersAfterRepack = 2;
    doPackingAndScalingTest(topology, componentChanges, instanceDefaultResources, boltParallelism, instanceDefaultResources.cloneWithRam(spoutRam), spoutParallelism, numContainersBeforeRepack, numContainersAfterRepack, getDefaultMaxContainerResource().cloneWithRam(maxContainerRam));
}

Example 13 with ByteAmount

use of org.apache.heron.common.basics.ByteAmount in project heron by twitter.

the class FirstFitDecreasingPackingTest method testPartialRamMap.

/**
 * Test the scenario RAM map config is partially set
 */
@Test
public void testPartialRamMap() throws Exception {
    // Explicit set resources for container
    ByteAmount maxContainerRam = ByteAmount.fromGigabytes(10);
    // Explicit set component RAM map
    ByteAmount boltRam = ByteAmount.fromGigabytes(4);
    topologyConfig.setContainerRamRequested(maxContainerRam);
    topologyConfig.setComponentRam(BOLT_NAME, boltRam);
    topology = getTopology(spoutParallelism, boltParallelism, topologyConfig);
    doPackingTest(topology, instanceDefaultResources.cloneWithRam(boltRam), boltParallelism, instanceDefaultResources, spoutParallelism, 3, getDefaultMaxContainerResource().cloneWithRam(maxContainerRam));
}

Example 14 with ByteAmount

use of org.apache.heron.common.basics.ByteAmount in project heron by twitter.

the class RoundRobinPacking method getContainerRamPadding.

private ByteAmount getContainerRamPadding(List<TopologyAPI.Config.KeyValue> topologyConfig) {
    ByteAmount stmgrRam = TopologyUtils.getConfigWithDefault(topologyConfig, org.apache.heron.api.Config.TOPOLOGY_STMGR_RAM, DEFAULT_DAEMON_PROCESS_RAM_PADDING);
    ByteAmount metricsmgrRam = TopologyUtils.getConfigWithDefault(topologyConfig, org.apache.heron.api.Config.TOPOLOGY_METRICSMGR_RAM, DEFAULT_DAEMON_PROCESS_RAM_PADDING);
    String reliabilityMode = TopologyUtils.getConfigWithDefault(topologyConfig, org.apache.heron.api.Config.TOPOLOGY_RELIABILITY_MODE, org.apache.heron.api.Config.TopologyReliabilityMode.ATMOST_ONCE.name());
    boolean isStateful = org.apache.heron.api.Config.TopologyReliabilityMode.EFFECTIVELY_ONCE.name().equals(reliabilityMode);
    ByteAmount ckptmgrRam = TopologyUtils.getConfigWithDefault(topologyConfig, org.apache.heron.api.Config.TOPOLOGY_STATEFUL_CKPTMGR_RAM, isStateful ? DEFAULT_DAEMON_PROCESS_RAM_PADDING : ByteAmount.ZERO);
    ByteAmount daemonProcessPadding = stmgrRam.plus(metricsmgrRam).plus(ckptmgrRam);
    // return the container padding if it's set, otherwise return the total daemon request ram
    return TopologyUtils.getConfigWithDefault(topologyConfig, org.apache.heron.api.Config.TOPOLOGY_CONTAINER_RAM_PADDING, daemonProcessPadding);
}

Example 15 with ByteAmount

use of org.apache.heron.common.basics.ByteAmount in project heron by twitter.

the class HeronMasterDriver method requestContainerForWorker.

@VisibleForTesting
void requestContainerForWorker(int id, final HeronWorker worker) {
    int cpu = worker.cores;
    ByteAmount mem = worker.mem;
    EvaluatorRequest evaluatorRequest = createEvaluatorRequest(cpu, mem);
    LOG.info(String.format("Requesting container for worker: %d, RAM: %s, CPU: %d", id, mem, cpu));
    requestor.submit(evaluatorRequest);
}