Examples with InnerSortFlusher com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher used on opensource projects

Example 1 with InnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher in project hugegraph-computer by hugegraph.

the class SortManager method sort.

public CompletableFuture<ByteBuffer> sort(MessageType type, WriteBuffers buffer) {
    return CompletableFuture.supplyAsync(() -> {
        RandomAccessInput bufferForRead = buffer.wrapForRead();
        // TODO：This ByteBuffer should be allocated from the off-heap
        BytesOutput output = IOFactory.createBytesOutput(this.capacity);
        InnerSortFlusher flusher = this.createSortFlusher(type, output, this.flushThreshold);
        try {
            this.sorter.sortBuffer(bufferForRead, flusher, type == MessageType.EDGE);
        } catch (Exception e) {
            throw new ComputerException("Failed to sort buffers of %s " + "message", e, type.name());
        }
        return ByteBuffer.wrap(output.buffer(), 0, (int) output.position());
    }, this.sortExecutor);
}

Example 2 with InnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher in project hugegraph-computer by hugegraph.

the class SortManager method createSortFlusher.

private InnerSortFlusher createSortFlusher(MessageType type, RandomAccessOutput output, int flushThreshold) {
    PointerCombiner combiner;
    boolean needSortSubKv;
    switch(type) {
        case VERTEX:
            combiner = new VertexValueCombiner(this.context);
            needSortSubKv = false;
            break;
        case EDGE:
            combiner = new EdgeValueCombiner(this.context);
            needSortSubKv = true;
            break;
        case MSG:
            combiner = this.createMessageCombiner();
            needSortSubKv = false;
            break;
        default:
            throw new ComputerException("Unsupported combine message " + "type for %s", type);
    }
    InnerSortFlusher flusher;
    if (combiner == null) {
        flusher = new KvInnerSortFlusher(output);
    } else {
        if (needSortSubKv) {
            flusher = new CombineSubKvInnerSortFlusher(output, combiner, flushThreshold);
        } else {
            flusher = new CombineKvInnerSortFlusher(output, combiner);
        }
    }
    return flusher;
}

Example 3 with InnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher in project hugegraph-computer by hugegraph.

the class SortLargeDataTest method sortBuffer.

private static RandomAccessInput sortBuffer(Sorter sorter, RandomAccessInput input) throws Exception {
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    PointerCombiner combiner = SorterTestUtil.createPointerCombiner(IntValue::new, new IntValueSumCombiner());
    InnerSortFlusher flusher = new CombineKvInnerSortFlusher(output, combiner);
    sorter.sortBuffer(input, flusher, false);
    return EntriesUtil.inputFromOutput(output);
}

Example 4 with InnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher in project hugegraph-computer by hugegraph.

the class FlusherTest method testExceptionCaseForFlusher.

@Test
public void testExceptionCaseForFlusher() {
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    InnerSortFlusher flusher = new KvInnerSortFlusher(output);
    List<KvEntry> entries = new ArrayList<>();
    Assert.assertThrows(IllegalArgumentException.class, () -> {
        flusher.flush(entries.iterator());
    }, e -> {
        String errorMsg = "Parameter entries can't be empty";
        Assert.assertContains(errorMsg, e.getMessage());
    });
}

Example 5 with InnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher in project hugegraph-computer by hugegraph.

the class FlusherTest method testOverwriteCombiner.

@Test
public void testOverwriteCombiner() throws Exception {
    List<Integer> data = ImmutableList.of(1, 2, 3, 5, 1, 3, 1, 1, 3, 4);
    BytesInput input = SorterTestUtil.inputFromKvMap(data);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    PointerCombiner combiner = SorterTestUtil.createPointerCombiner(IntValue::new, new OverwriteCombiner<>());
    InnerSortFlusher flusher = new CombineKvInnerSortFlusher(output, combiner);
    Sorter sorter = SorterTestUtil.createSorter(CONFIG);
    sorter.sortBuffer(input, flusher, false);
    BytesInput result = EntriesUtil.inputFromOutput(output);
    // Assert result
    KvEntriesInput iter = new KvEntriesInput(result);
    SorterTestUtil.assertKvEntry(iter.next(), 1, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 3, 4);
    iter.close();
}