Examples with ComputerException com.baidu.hugegraph.computer.core.common.exception.ComputerException used on opensource projects

Example 16 with ComputerException

use of com.baidu.hugegraph.computer.core.common.exception.ComputerException in project hugegraph-computer by hugegraph.

the class HdfsOutputMerger method init.

protected void init(Config config) {
    try {
        Configuration configuration = new Configuration();
        this.fs = HdfsOutput.openHDFS(config, configuration);
        String dir = config.get(ComputerOptions.OUTPUT_HDFS_DIR);
        String jobId = config.get(ComputerOptions.JOB_ID);
        int partitions = config.get(ComputerOptions.JOB_PARTITIONS_COUNT);
        this.sourcePaths = this.paths(dir, jobId, partitions);
        this.mergedPath = new Path(new Path(dir, jobId), MERGED_FILE_NAME);
    } catch (Exception e) {
        throw new ComputerException("Failed to init hdfs output merger", e);
    }
}

Example 17 with ComputerException

use of com.baidu.hugegraph.computer.core.common.exception.ComputerException in project hugegraph-computer by hugegraph.

the class HdfsOutputMerger method merge.

protected void merge() {
    try {
        this.fs.create(this.mergedPath, true).close();
        this.fs.concat(this.mergedPath, this.sourcePaths);
    } catch (IOException e) {
        throw new ComputerException("Failed to merge hdfs output files", e);
    }
}

Example 18 with ComputerException

use of com.baidu.hugegraph.computer.core.common.exception.ComputerException in project hugegraph-computer by hugegraph.

the class WriteBuffers method prepareSorting.

/**
 * Can remove synchronized if MessageSendManager.finish() only called by
 * single thread
 */
public synchronized void prepareSorting() {
    // Ensure last sorting task finished
    while (!this.sortingBuffer.isEmpty()) {
        try {
            this.wait();
        } catch (InterruptedException e) {
            throw new ComputerException("Interrupted when waiting " + "sorting buffer empty");
        }
    }
    // Record total message bytes
    this.totalCount += this.writingBuffer.writeCount();
    this.totalBytes += this.writingBuffer.numBytes();
    // Swap the writing buffer and sorting buffer pointer
    WriteBuffer temp = this.writingBuffer;
    this.writingBuffer = this.sortingBuffer;
    this.sortingBuffer = temp;
}

Example 19 with ComputerException

use of com.baidu.hugegraph.computer.core.common.exception.ComputerException in project hugegraph-computer by hugegraph.

the class SortManager method sort.

public CompletableFuture<ByteBuffer> sort(MessageType type, WriteBuffers buffer) {
    return CompletableFuture.supplyAsync(() -> {
        RandomAccessInput bufferForRead = buffer.wrapForRead();
        // TODO：This ByteBuffer should be allocated from the off-heap
        BytesOutput output = IOFactory.createBytesOutput(this.capacity);
        InnerSortFlusher flusher = this.createSortFlusher(type, output, this.flushThreshold);
        try {
            this.sorter.sortBuffer(bufferForRead, flusher, type == MessageType.EDGE);
        } catch (Exception e) {
            throw new ComputerException("Failed to sort buffers of %s " + "message", e, type.name());
        }
        return ByteBuffer.wrap(output.buffer(), 0, (int) output.position());
    }, this.sortExecutor);
}

Example 20 with ComputerException

use of com.baidu.hugegraph.computer.core.common.exception.ComputerException in project hugegraph-computer by hugegraph.

the class SortManager method createSortFlusher.

private InnerSortFlusher createSortFlusher(MessageType type, RandomAccessOutput output, int flushThreshold) {
    PointerCombiner combiner;
    boolean needSortSubKv;
    switch(type) {
        case VERTEX:
            combiner = new VertexValueCombiner(this.context);
            needSortSubKv = false;
            break;
        case EDGE:
            combiner = new EdgeValueCombiner(this.context);
            needSortSubKv = true;
            break;
        case MSG:
            combiner = this.createMessageCombiner();
            needSortSubKv = false;
            break;
        default:
            throw new ComputerException("Unsupported combine message " + "type for %s", type);
    }
    InnerSortFlusher flusher;
    if (combiner == null) {
        flusher = new KvInnerSortFlusher(output);
    } else {
        if (needSortSubKv) {
            flusher = new CombineSubKvInnerSortFlusher(output, combiner, flushThreshold);
        } else {
            flusher = new CombineKvInnerSortFlusher(output, combiner);
        }
    }
    return flusher;
}