Search in sources :

Example 1 with PeekableIterator

use of com.baidu.hugegraph.computer.core.sort.flusher.PeekableIterator in project hugegraph-computer by hugegraph.

the class ComputeManager method input.

public WorkerStat input() {
    WorkerStat workerStat = new WorkerStat();
    this.recvManager.waitReceivedAllMessages();
    Map<Integer, PeekableIterator<KvEntry>> vertices = this.recvManager.vertexPartitions();
    Map<Integer, PeekableIterator<KvEntry>> edges = this.recvManager.edgePartitions();
    // TODO: parallel input process
    for (Map.Entry<Integer, PeekableIterator<KvEntry>> entry : vertices.entrySet()) {
        int partition = entry.getKey();
        PeekableIterator<KvEntry> vertexIter = entry.getValue();
        PeekableIterator<KvEntry> edgesIter = edges.getOrDefault(partition, PeekableIterator.emptyIterator());
        FileGraphPartition part = new FileGraphPartition(this.context, this.managers, partition);
        PartitionStat partitionStat = null;
        ComputerException inputException = null;
        try {
            partitionStat = part.input(vertexIter, edgesIter);
        } catch (ComputerException e) {
            inputException = e;
        } finally {
            try {
                vertexIter.close();
                edgesIter.close();
            } catch (Exception e) {
                String message = "Failed to close vertex or edge file " + "iterator";
                ComputerException closeException = new ComputerException(message, e);
                if (inputException != null) {
                    inputException.addSuppressed(closeException);
                } else {
                    throw closeException;
                }
            }
            if (inputException != null) {
                throw inputException;
            }
        }
        workerStat.add(partitionStat);
        this.partitions.put(partition, part);
    }
    return workerStat;
}
Also used : KvEntry(com.baidu.hugegraph.computer.core.store.entry.KvEntry) ComputerException(com.baidu.hugegraph.computer.core.common.exception.ComputerException) ComputerException(com.baidu.hugegraph.computer.core.common.exception.ComputerException) PartitionStat(com.baidu.hugegraph.computer.core.graph.partition.PartitionStat) WorkerStat(com.baidu.hugegraph.computer.core.worker.WorkerStat) PeekableIterator(com.baidu.hugegraph.computer.core.sort.flusher.PeekableIterator) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) HashMap(java.util.HashMap) Map(java.util.Map)

Aggregations

ComputerException (com.baidu.hugegraph.computer.core.common.exception.ComputerException)1 PartitionStat (com.baidu.hugegraph.computer.core.graph.partition.PartitionStat)1 PeekableIterator (com.baidu.hugegraph.computer.core.sort.flusher.PeekableIterator)1 KvEntry (com.baidu.hugegraph.computer.core.store.entry.KvEntry)1 WorkerStat (com.baidu.hugegraph.computer.core.worker.WorkerStat)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)1