Search in sources :

Example 6 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class EntryOutputTest method testWriteKvEntry.

@Test
public void testWriteKvEntry() throws Exception {
    List<Integer> entries = ImmutableList.of(1, 5, 6, 6, 2, 1, 4, 8);
    List<Id> data = intListToLongIds(entries);
    BytesOutput output = IOFactory.createBytesOutput(Constants.SMALL_BUF_SIZE);
    EntryOutput entryOutput = new EntryOutputImpl(output);
    for (int i = 0; i < data.size(); ) {
        Id id = data.get(i++);
        Writable value = data.get(i++);
        entryOutput.writeEntry(id, value);
    }
    // Assert result
    BytesInput input = EntriesUtil.inputFromOutput(output);
    EntryIterator iter = new KvEntriesInput(input);
    SorterTestUtil.assertKvEntry(iter.next(), BytesId.of(1), BytesId.of(5));
    SorterTestUtil.assertKvEntry(iter.next(), BytesId.of(6), BytesId.of(6));
    SorterTestUtil.assertKvEntry(iter.next(), BytesId.of(2), BytesId.of(1));
    SorterTestUtil.assertKvEntry(iter.next(), BytesId.of(4), BytesId.of(8));
    iter.close();
}
Also used : EntryOutput(com.baidu.hugegraph.computer.core.store.entry.EntryOutput) EntryOutputImpl(com.baidu.hugegraph.computer.core.store.entry.EntryOutputImpl) BytesOutput(com.baidu.hugegraph.computer.core.io.BytesOutput) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) Writable(com.baidu.hugegraph.computer.core.io.Writable) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) Id(com.baidu.hugegraph.computer.core.graph.id.Id) BytesId(com.baidu.hugegraph.computer.core.graph.id.BytesId) Test(org.junit.Test)

Example 7 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class EntryOutputTest method testSubKvNotNeedSort.

@Test
public void testSubKvNotNeedSort() throws Exception {
    List<Integer> entries = ImmutableList.of(5, 6, 6, 2, 1, 4, 8, 1, 2, 2, 6, 1);
    BytesInput input = inputFromEntries(entries, false);
    EntryIterator iter = new KvEntriesInput(input, true);
    // Assert entry1
    KvEntry kvEntry1 = iter.next();
    Assert.assertEquals(3, kvEntry1.numSubEntries());
    Id key1 = StoreTestUtil.idFromPointer(kvEntry1.key());
    Assert.assertEquals(BytesId.of(5), key1);
    EntryIterator kvEntry1SubKvs = EntriesUtil.subKvIterFromEntry(kvEntry1);
    KvEntry subKv1 = kvEntry1SubKvs.next();
    Assert.assertEquals(0, subKv1.numSubEntries());
    SorterTestUtil.assertKvEntry(subKv1, BytesId.of(6), BytesId.of(6));
    SorterTestUtil.assertKvEntry(kvEntry1SubKvs.next(), BytesId.of(2), BytesId.of(1));
    SorterTestUtil.assertKvEntry(kvEntry1SubKvs.next(), BytesId.of(4), BytesId.of(8));
    // Assert entry2
    KvEntry kvEntry2 = iter.next();
    Id key2 = StoreTestUtil.idFromPointer(kvEntry2.key());
    Assert.assertEquals(BytesId.of(1), key2);
    EntryIterator kvEntry2SubKvs = EntriesUtil.subKvIterFromEntry(kvEntry2);
    SorterTestUtil.assertKvEntry(kvEntry2SubKvs.next(), BytesId.of(2), BytesId.of(2));
    SorterTestUtil.assertKvEntry(kvEntry2SubKvs.next(), BytesId.of(6), BytesId.of(1));
    iter.close();
}
Also used : BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) KvEntry(com.baidu.hugegraph.computer.core.store.entry.KvEntry) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) Id(com.baidu.hugegraph.computer.core.graph.id.Id) BytesId(com.baidu.hugegraph.computer.core.graph.id.BytesId) Test(org.junit.Test)

Example 8 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class StoreTestUtil method hgkvDirFromSubKvMap.

public static void hgkvDirFromSubKvMap(Config config, List<List<Integer>> map, String path) throws IOException {
    BytesInput input = SorterTestUtil.inputFromSubKvMap(map);
    KvEntriesInput iter = new KvEntriesInput(input, true);
    try (KvEntryFileWriter builder = new HgkvDirBuilderImpl(config, path)) {
        while (iter.hasNext()) {
            builder.write(iter.next());
        }
    }
    iter.close();
}
Also used : BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) HgkvDirBuilderImpl(com.baidu.hugegraph.computer.core.store.file.hgkvfile.builder.HgkvDirBuilderImpl)

Example 9 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class KvEntryWriterImpl method sortAndWriteSubKvs.

private void sortAndWriteSubKvs() throws IOException {
    BytesInput input = EntriesUtil.inputFromOutput(this.subKvBuffer);
    InputSorter sorter = new JavaInputSorter();
    Iterator<KvEntry> subKvs = sorter.sort(new KvEntriesInput(input));
    while (subKvs.hasNext()) {
        KvEntry subKv = subKvs.next();
        subKv.key().write(this.output);
        subKv.value().write(this.output);
    }
}
Also used : JavaInputSorter(com.baidu.hugegraph.computer.core.sort.sorter.JavaInputSorter) InputSorter(com.baidu.hugegraph.computer.core.sort.sorter.InputSorter) JavaInputSorter(com.baidu.hugegraph.computer.core.sort.sorter.JavaInputSorter) BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput)

Example 10 with KvEntriesInput

use of com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput in project hugegraph-computer by hugegraph.

the class EntryOutputTest method testSubKvNeedSort.

@Test
public void testSubKvNeedSort() throws Exception {
    List<Integer> entries = ImmutableList.of(5, 6, 6, 2, 1, 4, 8, 1, 2, 2, 6, 1);
    BytesInput input = inputFromEntries(entries, true);
    EntryIterator iter = new KvEntriesInput(input, true);
    // Assert entry1
    KvEntry kvEntry1 = iter.next();
    Id key1 = StoreTestUtil.idFromPointer(kvEntry1.key());
    Assert.assertEquals(BytesId.of(5), key1);
    EntryIterator kvEntry1SubKvs = EntriesUtil.subKvIterFromEntry(kvEntry1);
    KvEntry subKv1 = kvEntry1SubKvs.next();
    Assert.assertEquals(0, subKv1.numSubEntries());
    SorterTestUtil.assertKvEntry(subKv1, BytesId.of(2), BytesId.of(1));
    KvEntry subKv2 = kvEntry1SubKvs.next();
    Assert.assertEquals(0, subKv2.numSubEntries());
    SorterTestUtil.assertKvEntry(subKv2, BytesId.of(4), BytesId.of(8));
    KvEntry subKv3 = kvEntry1SubKvs.next();
    Assert.assertEquals(0, subKv3.numSubEntries());
    SorterTestUtil.assertKvEntry(subKv3, BytesId.of(6), BytesId.of(6));
    // Assert entry2
    KvEntry kvEntry2 = iter.next();
    Id key2 = StoreTestUtil.idFromPointer(kvEntry2.key());
    Assert.assertEquals(BytesId.of(1), key2);
    EntryIterator kvEntry2SubKvs = EntriesUtil.subKvIterFromEntry(kvEntry2);
    SorterTestUtil.assertKvEntry(kvEntry2SubKvs.next(), BytesId.of(2), BytesId.of(2));
    SorterTestUtil.assertKvEntry(kvEntry2SubKvs.next(), BytesId.of(6), BytesId.of(1));
    iter.close();
}
Also used : BytesInput(com.baidu.hugegraph.computer.core.io.BytesInput) KvEntry(com.baidu.hugegraph.computer.core.store.entry.KvEntry) KvEntriesInput(com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput) Id(com.baidu.hugegraph.computer.core.graph.id.Id) BytesId(com.baidu.hugegraph.computer.core.graph.id.BytesId) Test(org.junit.Test)

Aggregations

BytesInput (com.baidu.hugegraph.computer.core.io.BytesInput)10 KvEntriesInput (com.baidu.hugegraph.computer.core.store.buffer.KvEntriesInput)10 Test (org.junit.Test)7 BytesOutput (com.baidu.hugegraph.computer.core.io.BytesOutput)5 BytesId (com.baidu.hugegraph.computer.core.graph.id.BytesId)3 Id (com.baidu.hugegraph.computer.core.graph.id.Id)3 Sorter (com.baidu.hugegraph.computer.core.sort.Sorter)3 CombineKvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.CombineKvInnerSortFlusher)3 KvEntry (com.baidu.hugegraph.computer.core.store.entry.KvEntry)3 PointerCombiner (com.baidu.hugegraph.computer.core.combiner.PointerCombiner)2 Config (com.baidu.hugegraph.computer.core.config.Config)2 IntValue (com.baidu.hugegraph.computer.core.graph.value.IntValue)2 KvInnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.KvInnerSortFlusher)2 EntryIterator (com.baidu.hugegraph.computer.core.store.EntryIterator)2 IntValueSumCombiner (com.baidu.hugegraph.computer.core.combiner.IntValueSumCombiner)1 Writable (com.baidu.hugegraph.computer.core.io.Writable)1 InnerSortFlusher (com.baidu.hugegraph.computer.core.sort.flusher.InnerSortFlusher)1 InputSorter (com.baidu.hugegraph.computer.core.sort.sorter.InputSorter)1 JavaInputSorter (com.baidu.hugegraph.computer.core.sort.sorter.JavaInputSorter)1 EntryOutput (com.baidu.hugegraph.computer.core.store.entry.EntryOutput)1