Search in sources :

Example 1 with KvInnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher in project hugegraph-computer by hugegraph.

the class SortManager method createSortFlusher.

private InnerSortFlusher createSortFlusher(MessageType type, RandomAccessOutput output, int flushThreshold) {
    PointerCombiner combiner;
    boolean needSortSubKv;
    switch(type) {
        case VERTEX:
            combiner = new VertexValueCombiner(this.context);
            needSortSubKv = false;
            break;
        case EDGE:
            combiner = new EdgeValueCombiner(this.context);
            needSortSubKv = true;
            break;
        case MSG:
            combiner = this.createMessageCombiner();
            needSortSubKv = false;
            break;
        default:
            throw new ComputerException("Unsupported combine message " + "type for %s", type);
    }
    InnerSortFlusher flusher;
    if (combiner == null) {
        flusher = new KvInnerSortFlusher(output);
    } else {
        if (needSortSubKv) {
            flusher = new CombineSubKvInnerSortFlusher(output, combiner, flushThreshold);
        } else {
            flusher = new CombineKvInnerSortFlusher(output, combiner);
        }
    }
    return flusher;
}
Also used : CombineSubKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineSubKvInnerSortFlusher) KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) CombineSubKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineSubKvInnerSortFlusher) CombineSubKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineSubKvInnerSortFlusher) KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) InnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher) EdgeValueCombiner(com.baidu.hugegraph.computer.core.combiner.EdgeValueCombiner) VertexValueCombiner(com.baidu.hugegraph.computer.core.combiner.VertexValueCombiner) PointerCombiner(com.baidu.hugegraph.computer.core.combiner.PointerCombiner) ComputerException(com.baidu.hugegraph.computer.core.common.exception.ComputerException) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher)

Example 2 with KvInnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher in project hugegraph-computer by hugegraph.

the class FlusherTest method testKvInnerSortFlusher.

@Test
public void testKvInnerSortFlusher() throws Exception {
    List<Integer> map = ImmutableList.of(2, 1, 3, 1, 2, 1, 4, 1);
    BytesInput input = SorterTestUtil.inputFromKvMap(map);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    Sorter sorter = SorterTestUtil.createSorter(CONFIG);
    sorter.sortBuffer(input, new KvInnerSortFlusher(output), false);
    BytesInput result = EntriesUtil.inputFromOutput(output);
    EntryIterator iter = new KvEntriesInput(result);
    SorterTestUtil.assertKvEntry(iter.next(), 2, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 2, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 3, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 4, 1);
    iter.close();
}
Also used : KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) Sorter(com.baidu.hugegraph.computer.core.sort.Sorter) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) Test(org.junit.Test)

Example 3 with KvInnerSortFlusher

use of com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher in project hugegraph-computer by hugegraph.

the class FlusherTest method testExceptionCaseForFlusher.

@Test
public void testExceptionCaseForFlusher() {
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    InnerSortFlusher flusher = new KvInnerSortFlusher(output);
    List<KvEntry> entries = new ArrayList<>();
    Assert.assertThrows(IllegalArgumentException.class, () -> {
        flusher.flush(entries.iterator());
    }, e -> {
        String errorMsg = "Parameter entries can't be empty";
        Assert.assertContains(errorMsg, e.getMessage());
    });
}
Also used : KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) InnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) KvEntry(com.baidu.hugegraph.computer.core.store.entry.KvEntry) ArrayList(java.util.ArrayList) Test(org.junit.Test)

Aggregations

CombineKvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher)3 KvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher)3 BytesOutput (com.baidu.hugegraph.computer.core.io.BytesOutput)2 InnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher)2 Test (org.junit.Test)2 EdgeValueCombiner (com.baidu.hugegraph.computer.core.combiner.EdgeValueCombiner)1 PointerCombiner (com.baidu.hugegraph.computer.core.combiner.PointerCombiner)1 VertexValueCombiner (com.baidu.hugegraph.computer.core.combiner.VertexValueCombiner)1 ComputerException (com.baidu.hugegraph.computer.core.common.exception.ComputerException)1 BytesInput (com.baidu.hugegraph.computer.core.io.BytesInput)1 Sorter (com.baidu.hugegraph.computer.core.sort.Sorter)1 CombineSubKvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.CombineSubKvInnerSortFlusher)1 EntryIterator (com.baidu.hugegraph.computer.core.store.EntryIterator)1 KvEntriesInput (com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput)1 KvEntry (com.baidu.hugegraph.computer.core.store.entry.KvEntry)1 ArrayList (java.util.ArrayList)1