Search in sources :

Example 1 with WorkProcessor

use of io.trino.operator.WorkProcessor in project trino by trinodb.

the class MergeSortedPages method pageWithPositions.

private static WorkProcessor<PageWithPosition> pageWithPositions(WorkProcessor<Page> pages, AggregatedMemoryContext aggregatedMemoryContext) {
    return pages.flatMap(page -> {
        LocalMemoryContext memoryContext = aggregatedMemoryContext.newLocalMemoryContext(MergeSortedPages.class.getSimpleName());
        memoryContext.setBytes(page.getRetainedSizeInBytes());
        return WorkProcessor.create(new WorkProcessor.Process<PageWithPosition>() {

            int position;

            @Override
            public ProcessState<PageWithPosition> process() {
                if (position >= page.getPositionCount()) {
                    memoryContext.close();
                    return ProcessState.finished();
                }
                return ProcessState.ofResult(new PageWithPosition(page, position++));
            }
        });
    });
}
Also used : ProcessState(io.trino.operator.WorkProcessor.ProcessState) LocalMemoryContext(io.trino.memory.context.LocalMemoryContext) WorkProcessor(io.trino.operator.WorkProcessor)

Example 2 with WorkProcessor

use of io.trino.operator.WorkProcessor in project trino by trinodb.

the class BenchmarkPagesSort method runPagesMergeSortBenchmark.

@Benchmark
public List<Page> runPagesMergeSortBenchmark(MergeSortedBenchmarkData data) {
    WorkProcessor<Page> sortedPagesWork = mergeSortedPages(data.getSplitPages().stream().map(WorkProcessor::fromIterable).collect(toImmutableList()), ORDERING_COMPILER.compilePageWithPositionComparator(data.getSortTypes(), data.getSortChannels(), data.getSortOrders()), data.getOutputChannels(), data.getTypes(), (pageBuilder, pageWithPosition) -> pageBuilder.isFull(), false, newSimpleAggregatedMemoryContext(), new DriverYieldSignal());
    ImmutableList.Builder<Page> sortedPages = ImmutableList.builder();
    while (true) {
        sortedPagesWork.process();
        if (sortedPagesWork.isFinished()) {
            return sortedPages.build();
        }
        sortedPages.add(sortedPagesWork.getResult());
    }
}
Also used : ImmutableList(com.google.common.collect.ImmutableList) ImmutableList.toImmutableList(com.google.common.collect.ImmutableList.toImmutableList) WorkProcessor(io.trino.operator.WorkProcessor) DriverYieldSignal(io.trino.operator.DriverYieldSignal) Page(io.trino.spi.Page) SequencePageBuilder.createSequencePage(io.trino.SequencePageBuilder.createSequencePage) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Example 3 with WorkProcessor

use of io.trino.operator.WorkProcessor in project trino by trinodb.

the class TestMergeSortedPages method mergeSortedPages.

private static MaterializedResult mergeSortedPages(List<Type> types, List<Integer> sortChannels, List<SortOrder> sortOrder, List<List<Page>> sortedPages) {
    List<WorkProcessor<Page>> pageProducers = sortedPages.stream().map(WorkProcessor::fromIterable).collect(toImmutableList());
    PageWithPositionComparator comparator = new SimplePageWithPositionComparator(types, sortChannels, sortOrder, TYPE_OPERATORS_CACHE);
    AggregatedMemoryContext memoryContext = newSimpleAggregatedMemoryContext().newAggregatedMemoryContext();
    WorkProcessor<Page> mergedPages = MergeSortedPages.mergeSortedPages(pageProducers, comparator, types, memoryContext, new DriverYieldSignal());
    assertTrue(mergedPages.process());
    if (mergedPages.isFinished()) {
        return toMaterializedResult(TEST_SESSION, types, ImmutableList.of());
    }
    Page page = mergedPages.getResult();
    assertTrue(mergedPages.process());
    assertTrue(mergedPages.isFinished());
    assertEquals(memoryContext.getBytes(), 0L);
    return toMaterializedResult(TEST_SESSION, types, ImmutableList.of(page));
}
Also used : WorkProcessor(io.trino.operator.WorkProcessor) DriverYieldSignal(io.trino.operator.DriverYieldSignal) SimplePageWithPositionComparator(io.trino.operator.SimplePageWithPositionComparator) Page(io.trino.spi.Page) PageWithPositionComparator(io.trino.operator.PageWithPositionComparator) SimplePageWithPositionComparator(io.trino.operator.SimplePageWithPositionComparator) AggregatedMemoryContext.newSimpleAggregatedMemoryContext(io.trino.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext) AggregatedMemoryContext(io.trino.memory.context.AggregatedMemoryContext)

Aggregations

WorkProcessor (io.trino.operator.WorkProcessor)3 DriverYieldSignal (io.trino.operator.DriverYieldSignal)2 Page (io.trino.spi.Page)2 ImmutableList (com.google.common.collect.ImmutableList)1 ImmutableList.toImmutableList (com.google.common.collect.ImmutableList.toImmutableList)1 SequencePageBuilder.createSequencePage (io.trino.SequencePageBuilder.createSequencePage)1 AggregatedMemoryContext (io.trino.memory.context.AggregatedMemoryContext)1 AggregatedMemoryContext.newSimpleAggregatedMemoryContext (io.trino.memory.context.AggregatedMemoryContext.newSimpleAggregatedMemoryContext)1 LocalMemoryContext (io.trino.memory.context.LocalMemoryContext)1 PageWithPositionComparator (io.trino.operator.PageWithPositionComparator)1 SimplePageWithPositionComparator (io.trino.operator.SimplePageWithPositionComparator)1 ProcessState (io.trino.operator.WorkProcessor.ProcessState)1 Benchmark (org.openjdk.jmh.annotations.Benchmark)1