Search in sources :

Example 1 with InputsSorterImpl

use of com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl in project hugegraph-computer by hugegraph.

the class SorterImpl method sortBuffers.

private void sortBuffers(List<EntryIterator> entries, OuterSortFlusher flusher, String output) throws IOException {
    InputsSorter sorter = new InputsSorterImpl();
    try (HgkvDirBuilder builder = new HgkvDirBuilderImpl(this.config, output)) {
        EntryIterator result = sorter.sort(entries);
        flusher.flush(result, builder);
    }
}
Also used : HgkvDirBuilder(com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilder) InputsSorterImpl(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl) InputsSorter(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorter) EntryIterator(com.baidu.hugegraph.computer.core.store.hgkvfile.buffer.EntryIterator) HgkvDirBuilderImpl(com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilderImpl)

Example 2 with InputsSorterImpl

use of com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl in project hugegraph-computer by hugegraph.

the class FileMergerImpl method mergeInputs.

private void mergeInputs(List<String> inputs, Function<String, EntryIterator> inputToIter, OuterSortFlusher flusher, String output, Function<String, KvEntryFileWriter> fileToWriter) throws Exception {
    /*
         * File value format is different, upper layer is required to
         * provide the file reading mode
         */
    List<EntryIterator> entries = inputs.stream().map(inputToIter).collect(Collectors.toList());
    InputsSorter sorter = new InputsSorterImpl();
    // Merge inputs and write to output
    try (EntryIterator sortedKv = sorter.sort(entries);
        KvEntryFileWriter builder = fileToWriter.apply(output)) {
        flusher.flush(sortedKv, builder);
    }
}
Also used : InputsSorterImpl(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator) InputsSorter(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorter) KvEntryFileWriter(com.baidu.hugegraph.computer.core.store.KvEntryFileWriter)

Example 3 with InputsSorterImpl

use of com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl in project hugegraph-computer by hugegraph.

the class DefaultSorter method iterator.

public PeekableIterator<KvEntry> iterator(List<String> inputs, Function<String, EntryIterator> fileToEntries) throws IOException {
    List<EntryIterator> entries = inputs.stream().map(fileToEntries).collect(Collectors.toList());
    InputsSorterImpl sorter = new InputsSorterImpl();
    return PeekableIteratorAdaptor.of(sorter.sort(entries));
}
Also used : InputsSorterImpl(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator)

Example 4 with InputsSorterImpl

use of com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl in project hugegraph-computer by hugegraph.

the class DefaultSorter method mergeBuffers.

public void mergeBuffers(List<EntryIterator> entries, KvEntryFileWriter writer, OuterSortFlusher flusher) throws IOException {
    InputsSorter sorter = new InputsSorterImpl();
    EntryIterator result = sorter.sort(entries);
    flusher.flush(result, writer);
}
Also used : InputsSorterImpl(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl) InputsSorter(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorter) EntryIterator(com.baidu.hugegraph.computer.core.store.EntryIterator)

Example 5 with InputsSorterImpl

use of com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl in project hugegraph-computer by hugegraph.

the class HgkvDirMergerImpl method mergeInputs.

private void mergeInputs(List<String> inputs, InputToEntries inputToEntries, OuterSortFlusher flusher, String output) throws Exception {
    /*
         * File value format is different, upper layer is required to
         * provide the file reading mode
         */
    List<EntryIterator> entries = inputs.stream().map(inputToEntries::inputToEntries).collect(Collectors.toList());
    InputsSorter sorter = new InputsSorterImpl();
    // Merge inputs and write to output
    try (EntryIterator sortedKv = sorter.sort(entries);
        HgkvDirBuilder builder = new HgkvDirBuilderImpl(this.config, output)) {
        flusher.flush(sortedKv, builder);
    }
}
Also used : HgkvDirBuilder(com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilder) InputsSorterImpl(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl) EntryIterator(com.baidu.hugegraph.computer.core.store.hgkvfile.buffer.EntryIterator) InputsSorter(com.baidu.hugegraph.computer.core.sort.sorter.InputsSorter) HgkvDirBuilderImpl(com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilderImpl)

Aggregations

InputsSorterImpl (com.baidu.hugegraph.computer.core.sort.sorter.InputsSorterImpl)6 InputsSorter (com.baidu.hugegraph.computer.core.sort.sorter.InputsSorter)4 EntryIterator (com.baidu.hugegraph.computer.core.store.EntryIterator)3 EntryIterator (com.baidu.hugegraph.computer.core.store.hgkvfile.buffer.EntryIterator)3 HgkvDirBuilder (com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilder)2 HgkvDirBuilderImpl (com.baidu.hugegraph.computer.core.store.hgkvfile.file.builder.HgkvDirBuilderImpl)2 KvEntryFileWriter (com.baidu.hugegraph.computer.core.store.KvEntryFileWriter)1 HgkvDirReader (com.baidu.hugegraph.computer.core.store.hgkvfile.file.reader.HgkvDirReader)1 HgkvDirReaderImpl (com.baidu.hugegraph.computer.core.store.hgkvfile.file.reader.HgkvDirReaderImpl)1 ArrayList (java.util.ArrayList)1