Search in sources :

Example 1 with UpdatablePageSource

use of io.trino.spi.connector.UpdatablePageSource in project trino by trinodb.

the class MergeOperator method addSplit.

@Override
public Supplier<Optional<UpdatablePageSource>> addSplit(Split split) {
    requireNonNull(split, "split is null");
    checkArgument(split.getConnectorSplit() instanceof RemoteSplit, "split is not a remote split");
    checkState(!blockedOnSplits.isDone(), "noMoreSplits has been called already");
    TaskContext taskContext = operatorContext.getDriverContext().getPipelineContext().getTaskContext();
    DirectExchangeClient client = closer.register(directExchangeClientSupplier.get(taskContext.getTaskId().getQueryId(), new ExchangeId(format("direct-exchange-merge-%s-%s", taskContext.getTaskId().getStageId().getId(), sourceId)), operatorContext.localUserMemoryContext(), taskContext::sourceTaskFailed, RetryPolicy.NONE));
    RemoteSplit remoteSplit = (RemoteSplit) split.getConnectorSplit();
    // Only fault tolerant execution mode is expected to execute external exchanges.
    // MergeOperator is used for distributed sort only and it is not compatible (and disabled) with fault tolerant execution mode.
    DirectExchangeInput exchangeInput = (DirectExchangeInput) remoteSplit.getExchangeInput();
    client.addLocation(exchangeInput.getTaskId(), URI.create(exchangeInput.getLocation()));
    client.noMoreLocations();
    pageProducers.add(client.pages().map(serializedPage -> {
        Page page = pagesSerde.deserialize(serializedPage);
        operatorContext.recordNetworkInput(serializedPage.length(), page.getPositionCount());
        return page;
    }));
    return Optional::empty;
}
Also used : MergeSortedPages.mergeSortedPages(io.trino.util.MergeSortedPages.mergeSortedPages) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) OrderingCompiler(io.trino.sql.gen.OrderingCompiler) Type(io.trino.spi.type.Type) Page(io.trino.spi.Page) SettableFuture(com.google.common.util.concurrent.SettableFuture) Supplier(java.util.function.Supplier) DirectExchangeInput(io.trino.split.RemoteSplit.DirectExchangeInput) ArrayList(java.util.ArrayList) RemoteSplit(io.trino.split.RemoteSplit) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) PagesSerdeFactory(io.trino.execution.buffer.PagesSerdeFactory) Closer(com.google.common.io.Closer) PlanNodeId(io.trino.sql.planner.plan.PlanNodeId) Objects.requireNonNull(java.util.Objects.requireNonNull) URI(java.net.URI) IOException(java.io.IOException) MoreLists.mappedCopy(io.trino.util.MoreLists.mappedCopy) PagesSerde(io.trino.execution.buffer.PagesSerde) String.format(java.lang.String.format) SortOrder(io.trino.spi.connector.SortOrder) Preconditions.checkState(com.google.common.base.Preconditions.checkState) UncheckedIOException(java.io.UncheckedIOException) List(java.util.List) Split(io.trino.metadata.Split) UpdatablePageSource(io.trino.spi.connector.UpdatablePageSource) Optional(java.util.Optional) ExchangeId(io.trino.spi.exchange.ExchangeId) RemoteSplit(io.trino.split.RemoteSplit) Page(io.trino.spi.Page) DirectExchangeInput(io.trino.split.RemoteSplit.DirectExchangeInput) ExchangeId(io.trino.spi.exchange.ExchangeId)

Example 2 with UpdatablePageSource

use of io.trino.spi.connector.UpdatablePageSource in project trino by trinodb.

the class TableScanOperator method addSplit.

@Override
public Supplier<Optional<UpdatablePageSource>> addSplit(Split split) {
    requireNonNull(split, "split is null");
    checkState(this.split == null, "Table scan split already set");
    if (finished) {
        return Optional::empty;
    }
    this.split = split;
    Object splitInfo = split.getInfo();
    if (splitInfo != null) {
        operatorContext.setInfoSupplier(Suppliers.ofInstance(new SplitOperatorInfo(split.getCatalogName(), splitInfo)));
    }
    blocked.set(null);
    if (split.getConnectorSplit() instanceof EmptySplit) {
        source = new EmptyPageSource();
    }
    return () -> {
        if (source instanceof UpdatablePageSource) {
            return Optional.of((UpdatablePageSource) source);
        }
        return Optional.empty();
    };
}
Also used : UpdatablePageSource(io.trino.spi.connector.UpdatablePageSource) EmptyPageSource(io.trino.spi.connector.EmptyPageSource) EmptySplit(io.trino.split.EmptySplit)

Aggregations

UpdatablePageSource (io.trino.spi.connector.UpdatablePageSource)2 Preconditions.checkArgument (com.google.common.base.Preconditions.checkArgument)1 Preconditions.checkState (com.google.common.base.Preconditions.checkState)1 Closer (com.google.common.io.Closer)1 ListenableFuture (com.google.common.util.concurrent.ListenableFuture)1 SettableFuture (com.google.common.util.concurrent.SettableFuture)1 PagesSerde (io.trino.execution.buffer.PagesSerde)1 PagesSerdeFactory (io.trino.execution.buffer.PagesSerdeFactory)1 Split (io.trino.metadata.Split)1 Page (io.trino.spi.Page)1 EmptyPageSource (io.trino.spi.connector.EmptyPageSource)1 SortOrder (io.trino.spi.connector.SortOrder)1 ExchangeId (io.trino.spi.exchange.ExchangeId)1 Type (io.trino.spi.type.Type)1 EmptySplit (io.trino.split.EmptySplit)1 RemoteSplit (io.trino.split.RemoteSplit)1 DirectExchangeInput (io.trino.split.RemoteSplit.DirectExchangeInput)1 OrderingCompiler (io.trino.sql.gen.OrderingCompiler)1 PlanNodeId (io.trino.sql.planner.plan.PlanNodeId)1 MergeSortedPages.mergeSortedPages (io.trino.util.MergeSortedPages.mergeSortedPages)1