Search in sources :

Example 1 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class StoreTestUtil method kvEntriesFromMap.

public static List<KvEntry> kvEntriesFromMap(List<Integer> map) throws IOException {
    BytesOutput data = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    Iterator<Integer> iterator = map.iterator();
    while (iterator.hasNext()) {
        // Write key length
        writeData(data, iterator.next());
        // Write value length
        writeData(data, iterator.next());
    }
    BytesInput input = IOFactory.createBytesInput(data.buffer(), (int) data.position());
    KvEntriesInput iter = new KvEntriesInput(input);
    List<KvEntry> entries = new ArrayList<>();
    while (iter.hasNext()) {
        entries.add(iter.next());
    }
    iter.close();
    return entries;
}
Also used : BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) KvEntry(com.baidu.hugegraph.computer.core.store.entry.KvEntry) ArrayList(java.util.ArrayList) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput)

Example 2 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class SorterTest method testSortSubKvBuffer.

@Test
public void testSortSubKvBuffer() throws Exception {
    Config config = UnitTestBase.updateWithRequiredOptions(ComputerOptions.INPUT_MAX_EDGES_IN_ONE_VERTEX, "2");
    /*
         * Assert result
         * key 1 subKv 3 1, 5 1
         * key 2 subKv 5 1, 8 2
         * key 2 subKv 9 1
         * key 3 subKv 2 2, 3 1
         * key 3 subKv 4 1
         */
    BytesInput input = this.sortedSubKvBuffer(config);
    EntryIterator iter = new KvEntriesInput(input, true);
    SorterTestUtil.assertSubKvByKv(iter.next(), 1, 3, 1, 5, 1);
    SorterTestUtil.assertSubKvByKv(iter.next(), 2, 5, 1, 8, 2);
    SorterTestUtil.assertSubKvByKv(iter.next(), 2, 9, 1);
    SorterTestUtil.assertSubKvByKv(iter.next(), 3, 2, 2, 3, 1);
    SorterTestUtil.assertSubKvByKv(iter.next(), 3, 4, 1);
    iter.close();
}
Also used : BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) Config(com.baidu.hugegraph.computer.core.config.Config) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) Test(org.junit.Test)

Example 3 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class SorterTest method testSortKvBuffer.

@Test
public void testSortKvBuffer() throws Exception {
    Config config = UnitTestBase.updateWithRequiredOptions(ComputerOptions.HGKV_MAX_FILE_SIZE, "32", ComputerOptions.HGKV_DATABLOCK_SIZE, "16", ComputerOptions.HGKV_MERGE_FILES_NUM, "3");
    List<Integer> map = ImmutableList.of(2, 3, 1, 23, 6, 2, 5, 9, 2, 2, 6, 1, 1, 20);
    BytesInput input = SorterTestUtil.inputFromKvMap(map);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    Sorter sorter = SorterTestUtil.createSorter(config);
    PointerCombiner combiner = SorterTestUtil.createPointerCombiner(IntValue::new, new IntValueSumCombiner());
    sorter.sortBuffer(input, new CombineKvInnerSortFlusher(output, combiner), false);
    BytesInput resultInput = EntriesUtil.inputFromOutput(output);
    KvEntriesInput iter = new KvEntriesInput(resultInput);
    SorterTestUtil.assertKvEntry(iter.next(), 1, 43);
    SorterTestUtil.assertKvEntry(iter.next(), 2, 5);
    SorterTestUtil.assertKvEntry(iter.next(), 5, 9);
    SorterTestUtil.assertKvEntry(iter.next(), 6, 3);
    iter.close();
}
Also used : IntValueSumCombiner(com.baidu.hugegraph.computer.core.combiner.IntValueSumCombiner) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) Config(com.baidu.hugegraph.computer.core.config.Config) Sorter(com.baidu.hugegraph.computer.core.sort.Sorter) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) PointerCombiner(com.baidu.hugegraph.computer.core.combiner.PointerCombiner) IntValue(com.baidu.hugegraph.computer.core.graph.value.IntValue) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) Test(org.junit.Test)

Example 4 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class FlusherTest method testKvInnerSortFlusher.

@Test
public void testKvInnerSortFlusher() throws Exception {
    List<Integer> map = ImmutableList.of(2, 1, 3, 1, 2, 1, 4, 1);
    BytesInput input = SorterTestUtil.inputFromKvMap(map);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    Sorter sorter = SorterTestUtil.createSorter(CONFIG);
    sorter.sortBuffer(input, new KvInnerSortFlusher(output), false);
    BytesInput result = EntriesUtil.inputFromOutput(output);
    EntryIterator iter = new KvEntriesInput(result);
    SorterTestUtil.assertKvEntry(iter.next(), 2, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 2, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 3, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 4, 1);
    iter.close();
}
Also used : KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) Sorter(com.baidu.hugegraph.computer.core.sort.Sorter) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) Test(org.junit.Test)

Example 5 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class FlusherTest method testOverwriteCombiner.

@Test
public void testOverwriteCombiner() throws Exception {
    List<Integer> data = ImmutableList.of(1, 2, 3, 5, 1, 3, 1, 1, 3, 4);
    BytesInput input = SorterTestUtil.inputFromKvMap(data);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    PointerCombiner combiner = SorterTestUtil.createPointerCombiner(IntValue::new, new OverwriteCombiner<>());
    InnerSortFlusher flusher = new CombineKvInnerSortFlusher(output, combiner);
    Sorter sorter = SorterTestUtil.createSorter(CONFIG);
    sorter.sortBuffer(input, flusher, false);
    BytesInput result = EntriesUtil.inputFromOutput(output);
    // Assert result
    KvEntriesInput iter = new KvEntriesInput(result);
    SorterTestUtil.assertKvEntry(iter.next(), 1, 1);
    SorterTestUtil.assertKvEntry(iter.next(), 3, 4);
    iter.close();
}
Also used : KvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) InnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) Sorter(com.baidu.hugegraph.computer.core.sort.Sorter) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) PointerCombiner(com.baidu.hugegraph.computer.core.combiner.PointerCombiner) IntValue(com.baidu.hugegraph.computer.core.graph.value.IntValue) CombineKvInnerSortFlusher(com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher) Test(org.junit.Test)

Aggregations

BytesInput (com.baidu.hugegraph.computer.core.io.BytesInput)10 KvEntriesInput (com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput)10 Test (org.junit.Test)7 BytesOutput (com.baidu.hugegraph.computer.core.io.BytesOutput)5 BytesId (com.baidu.hugegraph.computer.core.graph.id.BytesId)3 Id (com.baidu.hugegraph.computer.core.graph.id.Id)3 Sorter (com.baidu.hugegraph.computer.core.sort.Sorter)3 CombineKvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher)3 KvEntry (com.baidu.hugegraph.computer.core.store.entry.KvEntry)3 PointerCombiner (com.baidu.hugegraph.computer.core.combiner.PointerCombiner)2 Config (com.baidu.hugegraph.computer.core.config.Config)2 IntValue (com.baidu.hugegraph.computer.core.graph.value.IntValue)2 KvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher)2 EntryIterator (com.baidu.hugegraph.computer.core.store.EntryIterator)2 IntValueSumCombiner (com.baidu.hugegraph.computer.core.combiner.IntValueSumCombiner)1 Writable (com.baidu.hugegraph.computer.core.io.Writable)1 InnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher)1 InputSorter (com.baidu.hugegraph.computer.core.sort.sorter.InputSorter)1 JavaInputSorter (com.baidu.hugegraph.computer.core.sort.sorter.JavaInputSorter)1 EntryOutput (com.baidu.hugegraph.computer.core.store.entry.EntryOutput)1