Search in sources :

Example 1 with DisperseEvenlySelector

use of com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector in project hugegraph-computer by hugegraph.

the class HgkvFileSorter method mergeInputs.

@Override
public void mergeInputs(List<String> inputs, OuterSortFlusher flusher, List<String> outputs, boolean withSubKv) throws Exception {
    Function<String, EntryIterator> fileToInput;
    Function<String, KvEntryFileWriter> fileToWriter;
    if (withSubKv) {
        fileToInput = o -> new HgkvDir4SubKvReaderImpl(o).iterator();
    } else {
        fileToInput = o -> new HgkvDirReaderImpl(o).iterator();
    }
    fileToWriter = path -> new HgkvDirBuilderImpl(this.config, path);
    InputFilesSelector selector = new DisperseEvenlySelector();
    List<SelectedFiles> selectResult = selector.selectedByHgkvFile(inputs, outputs);
    this.sorter.mergeFile(selectResult, fileToInput, fileToWriter, flusher);
}
Also used : DisperseEvenlySelector(com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector) HgkvDir4SubKvReaderImpl(com.baidu.hugegraph.computer.core.store.file.hgkvfile.reader.HgkvDir4SubKvReaderImpl) HgkvDirReaderImpl(com.baidu.hugegraph.computer.core.store.file.hgkvfile.reader.HgkvDirReaderImpl) SelectedFiles(com.baidu.hugegraph.computer.core.store.file.select.SelectedFiles) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) KvEntryFileWriter(com.baidu.hugegraph.computer.core.store.KvEntryFileWriter) HgkvDirBuilderImpl(com.baidu.hugegraph.computer.core.store.file.hgkvfile.builder.HgkvDirBuilderImpl) InputFilesSelector(com.baidu.hugegraph.computer.core.store.file.select.InputFilesSelector)

Example 2 with DisperseEvenlySelector

use of com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector in project hugegraph-computer by hugegraph.

the class BufferFileSorter method mergeInputs.

@Override
public void mergeInputs(List<String> inputs, OuterSortFlusher flusher, List<String> outputs, boolean withSubKv) throws Exception {
    Function<String, EntryIterator> fileToInput;
    Function<String, KvEntryFileWriter> fileToWriter;
    if (withSubKv) {
        fileToInput = o -> new BufferFileSubEntryReader(o).iterator();
    } else {
        fileToInput = o -> new BufferFileEntryReader(o).iterator();
    }
    fileToWriter = BufferFileEntryBuilder::new;
    InputFilesSelector selector = new DisperseEvenlySelector();
    List<SelectedFiles> selectResult = selector.selectedByBufferFile(inputs, outputs);
    this.sorter.mergeFile(selectResult, fileToInput, fileToWriter, flusher);
}
Also used : DisperseEvenlySelector(com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector) BufferFileSubEntryReader(com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileSubEntryReader) BufferFileEntryReader(com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileEntryReader) SelectedFiles(com.baidu.hugegraph.computer.core.store.file.select.SelectedFiles) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) BufferFileEntryBuilder(com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileEntryBuilder) KvEntryFileWriter(com.baidu.hugegraph.computer.core.store.KvEntryFileWriter) InputFilesSelector(com.baidu.hugegraph.computer.core.store.file.select.InputFilesSelector)

Example 3 with DisperseEvenlySelector

use of com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector in project hugegraph-computer by hugegraph.

the class SorterTest method testExceptionCaseForSelector.

@Test
public void testExceptionCaseForSelector() {
    // Parameter inputs size < outputs size
    String input1 = StoreTestUtil.availablePathById("1");
    String input2 = StoreTestUtil.availablePathById("2");
    List<String> inputs = ImmutableList.of(input1, input2);
    String output1 = StoreTestUtil.availablePathById("3");
    String output2 = StoreTestUtil.availablePathById("4");
    String output3 = StoreTestUtil.availablePathById("5");
    List<String> outputs = ImmutableList.of(output1, output2, output3);
    InputFilesSelector selector = new DisperseEvenlySelector();
    Assert.assertThrows(IllegalArgumentException.class, () -> {
        selector.selectedByHgkvFile(inputs, outputs);
    }, e -> {
        String errorMsg = "inputs size of InputFilesSelector must be >= " + "outputs size";
        Assert.assertContains(errorMsg, e.getMessage());
    });
}
Also used : DisperseEvenlySelector(com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector) InputFilesSelector(com.baidu.hugegraph.computer.core.store.file.select.InputFilesSelector) Test(org.junit.Test)

Aggregations

DisperseEvenlySelector (com.baidu.hugegraph.computer.core.store.file.select.DisperseEvenlySelector)3 InputFilesSelector (com.baidu.hugegraph.computer.core.store.file.select.InputFilesSelector)3 EntryIterator (com.baidu.hugegraph.computer.core.store.EntryIterator)2 KvEntryFileWriter (com.baidu.hugegraph.computer.core.store.KvEntryFileWriter)2 SelectedFiles (com.baidu.hugegraph.computer.core.store.file.select.SelectedFiles)2 BufferFileEntryBuilder (com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileEntryBuilder)1 BufferFileEntryReader (com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileEntryReader)1 BufferFileSubEntryReader (com.baidu.hugegraph.computer.core.store.file.bufferfile.BufferFileSubEntryReader)1 HgkvDirBuilderImpl (com.baidu.hugegraph.computer.core.store.file.hgkvfile.builder.HgkvDirBuilderImpl)1 HgkvDir4SubKvReaderImpl (com.baidu.hugegraph.computer.core.store.file.hgkvfile.reader.HgkvDir4SubKvReaderImpl)1 HgkvDirReaderImpl (com.baidu.hugegraph.computer.core.store.file.hgkvfile.reader.HgkvDirReaderImpl)1 Test (org.junit.Test)1