Examples with NodeCapacity org.apache.hyracks.api.job.resource.NodeCapacity used on opensource projects

Example 1 with NodeCapacity

use of org.apache.hyracks.api.job.resource.NodeCapacity in project asterixdb by apache.

the class NCApplication method getCapacity.

@Override
public NodeCapacity getCapacity() {
    StorageProperties storageProperties = runtimeContext.getStorageProperties();
    // Deducts the reserved buffer cache size and memory component size from the maxium heap size,
    // and deducts one core for processing heartbeats.
    long memorySize = Runtime.getRuntime().maxMemory() - storageProperties.getBufferCacheSize() - storageProperties.getMemoryComponentGlobalBudget();
    int allCores = Runtime.getRuntime().availableProcessors();
    int maximumCoresForComputation = allCores > 1 ? allCores - 1 : allCores;
    return new NodeCapacity(memorySize, maximumCoresForComputation);
}

Example 2 with NodeCapacity

use of org.apache.hyracks.api.job.resource.NodeCapacity in project asterixdb by apache.

the class NodeManagerTest method mockNodeControllerState.

private NodeControllerState mockNodeControllerState(String nodeId, boolean invalidIpAddr) {
    NodeControllerState ncState = mock(NodeControllerState.class);
    String ipAddr = invalidIpAddr ? "255.255.255:255" : "127.0.0.2";
    NetworkAddress dataAddr = new NetworkAddress(ipAddr, 1001);
    NetworkAddress resultAddr = new NetworkAddress(ipAddr, 1002);
    NetworkAddress msgAddr = new NetworkAddress(ipAddr, 1003);
    when(ncState.getCapacity()).thenReturn(new NodeCapacity(NODE_MEMORY_SIZE, NODE_CORES));
    when(ncState.getDataPort()).thenReturn(dataAddr);
    when(ncState.getDatasetPort()).thenReturn(resultAddr);
    when(ncState.getMessagingPort()).thenReturn(msgAddr);
    NCConfig ncConfig = new NCConfig(nodeId);
    ncConfig.setDataPublicAddress(ipAddr);
    when(ncState.getNCConfig()).thenReturn(ncConfig);
    return ncState;
}

Example 3 with NodeCapacity

use of org.apache.hyracks.api.job.resource.NodeCapacity in project asterixdb by apache.

the class NodeManager method removeDeadNodes.

@Override
public Pair<Collection<String>, Collection<JobId>> removeDeadNodes() throws HyracksException {
    Set<String> deadNodes = new HashSet<>();
    Set<JobId> affectedJobIds = new HashSet<>();
    Iterator<Map.Entry<String, NodeControllerState>> nodeIterator = nodeRegistry.entrySet().iterator();
    while (nodeIterator.hasNext()) {
        Map.Entry<String, NodeControllerState> entry = nodeIterator.next();
        String nodeId = entry.getKey();
        NodeControllerState state = entry.getValue();
        if (state.incrementLastHeartbeatDuration() >= ccConfig.getHeartbeatMaxMisses()) {
            deadNodes.add(nodeId);
            affectedJobIds.addAll(state.getActiveJobIds());
            // Removes the node from node map.
            nodeIterator.remove();
            // Removes the node from IP map.
            removeNodeFromIpAddressMap(nodeId, state);
            // Updates the cluster capacity.
            resourceManager.update(nodeId, new NodeCapacity(0L, 0));
            LOGGER.info(entry.getKey() + " considered dead");
        }
    }
    return Pair.of(deadNodes, affectedJobIds);
}

Example 4 with NodeCapacity

use of org.apache.hyracks.api.job.resource.NodeCapacity in project asterixdb by apache.

the class JobCapacityControllerTest method makeResourceManagerWithCapacity.

private IResourceManager makeResourceManagerWithCapacity(long memorySize, int cores) throws HyracksException {
    IResourceManager resourceManager = new ResourceManager();
    resourceManager.update("node1", new NodeCapacity(memorySize, cores));
    return resourceManager;
}

Example 5 with NodeCapacity

use of org.apache.hyracks.api.job.resource.NodeCapacity in project asterixdb by apache.

the class NodeManager method removeNode.

@Override
public void removeNode(String nodeId) throws HyracksException {
    NodeControllerState ncState = nodeRegistry.remove(nodeId);
    removeNodeFromIpAddressMap(nodeId, ncState);
    // Updates the cluster capacity.
    resourceManager.update(nodeId, new NodeCapacity(0L, 0));
}