Search in sources :

Example 11 with MutablePair

use of org.apache.commons.lang3.tuple.MutablePair in project apex-core by apache.

the class BlacklistBasedResourceRequestHandler method reissueContainerRequests.

@Override
public void reissueContainerRequests(AMRMClient<ContainerRequest> amRmClient, Map<StreamingContainerAgent.ContainerStartRequest, MutablePair<Integer, ContainerRequest>> requestedResources, int loopCounter, ResourceRequestHandler resourceRequestor, List<ContainerRequest> containerRequests, List<ContainerRequest> removedContainerRequests) {
    if (!requestedResources.isEmpty()) {
        // Check if any requests timed out, create new requests in that case
        recreateContainerRequest(requestedResources, loopCounter, resourceRequestor, removedContainerRequests);
    }
    // Issue all host specific requests first
    if (!hostSpecificRequestsMap.isEmpty()) {
        LOG.info("Issue Host specific requests first");
        // Blacklist all the nodes and issue request for host specific
        Entry<String, List<ContainerRequest>> set = hostSpecificRequestsMap.entrySet().iterator().next();
        List<ContainerRequest> requests = set.getValue();
        List<String> blacklistNodes = resourceRequestor.getNodesExceptHost(requests.get(0).getNodes());
        amRmClient.updateBlacklist(blacklistNodes, requests.get(0).getNodes());
        blacklistedNodesForHostSpecificRequests = blacklistNodes;
        LOG.info("Sending {} request(s) after blacklisting all nodes other than {}", requests.size(), requests.get(0).getNodes());
        for (ContainerRequest cr : requests) {
            ContainerStartRequest csr = hostSpecificRequests.get(cr);
            ContainerRequest newCr = new ContainerRequest(cr.getCapability(), null, null, cr.getPriority());
            MutablePair<Integer, ContainerRequest> pair = new MutablePair<>(loopCounter, newCr);
            requestedResources.put(csr, pair);
            containerRequests.add(newCr);
            hostSpecificRequests.remove(cr);
        }
        hostSpecificRequestsMap.remove(set.getKey());
    } else {
        if (blacklistedNodesForHostSpecificRequests != null) {
            // Remove the blacklisted nodes during host specific requests
            LOG.debug("All requests done.. Removing nodes from blacklist {}", blacklistedNodesForHostSpecificRequests);
            amRmClient.updateBlacklist(null, blacklistedNodesForHostSpecificRequests);
            blacklistedNodesForHostSpecificRequests = null;
        }
        // Proceed with other requests after host specific requests are done
        if (!otherContainerRequests.isEmpty()) {
            for (Entry<ContainerRequest, ContainerStartRequest> entry : otherContainerRequests.entrySet()) {
                ContainerRequest cr = entry.getKey();
                ContainerStartRequest csr = entry.getValue();
                MutablePair<Integer, ContainerRequest> pair = new MutablePair<>(loopCounter, cr);
                requestedResources.put(csr, pair);
                containerRequests.add(cr);
            }
            otherContainerRequests.clear();
        }
    }
}
Also used : MutablePair(org.apache.commons.lang3.tuple.MutablePair) ContainerStartRequest(com.datatorrent.stram.StreamingContainerAgent.ContainerStartRequest) List(java.util.List) ArrayList(java.util.ArrayList) ContainerRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest)

Example 12 with MutablePair

use of org.apache.commons.lang3.tuple.MutablePair in project gatk by broadinstitute.

the class BAQ method calculateQueryRange.

/**
     * Determine the appropriate start and stop offsets in the reads for the bases given the cigar string
     * @param read
     * @return
     */
private final Pair<Integer, Integer> calculateQueryRange(final GATKRead read) {
    int queryStart = -1, queryStop = -1;
    int readI = 0;
    // iterate over the cigar elements to determine the start and stop of the read bases for the BAQ calculation
    for (CigarElement elt : read.getCigarElements()) {
        switch(elt.getOperator()) {
            // cannot handle these
            case N:
                return null;
            // ignore pads, hard clips, and deletions
            case H:
            // ignore pads, hard clips, and deletions
            case P:
            // ignore pads, hard clips, and deletions
            case D:
                break;
            case I:
            case S:
            case M:
            case EQ:
            case X:
                int prev = readI;
                readI += elt.getLength();
                if (elt.getOperator() != CigarOperator.S) {
                    if (queryStart == -1) {
                        queryStart = prev;
                    }
                    queryStop = readI;
                }
                // queryStart or queryStop
                break;
            default:
                throw new GATKException("BUG: Unexpected CIGAR element " + elt + " in read " + read.getName());
        }
    }
    if (queryStop == queryStart) {
        //System.err.printf("WARNING -- read is completely clipped away: " + read.format());
        return null;
    }
    return new MutablePair<>(queryStart, queryStop);
}
Also used : MutablePair(org.apache.commons.lang3.tuple.MutablePair) GATKException(org.broadinstitute.hellbender.exceptions.GATKException) CigarElement(htsjdk.samtools.CigarElement)

Aggregations

MutablePair (org.apache.commons.lang3.tuple.MutablePair)12 ArrayList (java.util.ArrayList)5 ContainerStartRequest (com.datatorrent.stram.StreamingContainerAgent.ContainerStartRequest)4 Map (java.util.Map)4 ContainerRequest (org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest)4 IOException (java.io.IOException)3 HashMap (java.util.HashMap)3 List (java.util.List)2 Pair (org.apache.commons.lang3.tuple.Pair)2 Column (com.alibaba.datax.common.element.Column)1 DataXException (com.alibaba.datax.common.exception.DataXException)1 ContainerResource (com.datatorrent.stram.StreamingContainerManager.ContainerResource)1 StramEvent (com.datatorrent.stram.api.StramEvent)1 StreamingContainer (com.datatorrent.stram.engine.StreamingContainer)1 PTContainer (com.datatorrent.stram.plan.physical.PTContainer)1 StramDelegationTokenIdentifier (com.datatorrent.stram.security.StramDelegationTokenIdentifier)1 PulsarServerException (com.yahoo.pulsar.broker.PulsarServerException)1 MessageIdData (com.yahoo.pulsar.common.api.proto.PulsarApi.MessageIdData)1 CigarElement (htsjdk.samtools.CigarElement)1 DataWriter (info.ata4.io.DataWriter)1