Search in sources :

Example 1 with AnalyzedStatement

use of io.crate.analyze.AnalyzedStatement in project crate by crate.

the class BatchPortalTest method testEachStatementReceivesCorrectParams.

@Test
public void testEachStatementReceivesCorrectParams() throws Throwable {
    SQLExecutor sqlExecutor = SQLExecutor.builder(clusterService).addTable("create table t1 (x int)").build();
    Plan insertPlan = new Plan() {

        @Override
        public StatementType type() {
            return StatementType.INSERT;
        }

        @Override
        public void executeOrFail(DependencyCarrier executor, PlannerContext plannerContext, RowConsumer consumer, Row params, SubQueryResults subQueryResults) {
            consumer.accept(InMemoryBatchIterator.of(params, null), null);
        }
    };
    Planner planner = new Planner(Settings.EMPTY, clusterService, sqlExecutor.nodeCtx, new TableStats(), null, null, sqlExecutor.schemas(), new StubUserManager(), mock(SessionSettingRegistry.class)) {

        @Override
        public Plan plan(AnalyzedStatement analyzedStatement, PlannerContext plannerContext) {
            return insertPlan;
        }
    };
    DependencyCarrier executor = mock(DependencyCarrier.class, Answers.RETURNS_MOCKS);
    Session session = new Session(sqlExecutor.nodeCtx, sqlExecutor.analyzer, planner, new JobsLogs(() -> false), false, executor, AccessControl.DISABLED, SessionContext.systemSessionContext());
    session.parse("S_1", "insert into t1(x) values(1)", Collections.emptyList());
    session.bind("Portal", "S_1", Collections.emptyList(), null);
    final ArrayList<Object[]> s1Rows = new ArrayList<>();
    session.execute("Portal", 0, new BaseResultReceiver() {

        @Override
        public void setNextRow(Row row) {
            s1Rows.add(row.materialize());
        }
    });
    session.parse("S_2", "insert into t1(x) values(?)", Collections.emptyList());
    session.bind("Portal", "S_2", Collections.singletonList(2), null);
    final ArrayList<Object[]> s2Rows = new ArrayList<>();
    session.execute("Portal", 0, new BaseResultReceiver() {

        @Override
        public void setNextRow(Row row) {
            s2Rows.add(row.materialize());
        }
    });
    session.sync().get(5, TimeUnit.SECONDS);
    assertThat(s1Rows, contains(emptyArray()));
    assertThat(s2Rows, contains(arrayContaining(is(2))));
}
Also used : DependencyCarrier(io.crate.planner.DependencyCarrier) SubQueryResults(io.crate.planner.operators.SubQueryResults) ArrayList(java.util.ArrayList) Plan(io.crate.planner.Plan) TableStats(io.crate.statistics.TableStats) StubUserManager(io.crate.user.StubUserManager) SessionSettingRegistry(io.crate.metadata.settings.session.SessionSettingRegistry) PlannerContext(io.crate.planner.PlannerContext) SQLExecutor(io.crate.testing.SQLExecutor) BaseResultReceiver(io.crate.action.sql.BaseResultReceiver) Planner(io.crate.planner.Planner) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) RowConsumer(io.crate.data.RowConsumer) Row(io.crate.data.Row) JobsLogs(io.crate.execution.engine.collect.stats.JobsLogs) Session(io.crate.action.sql.Session) CrateDummyClusterServiceUnitTest(io.crate.test.integration.CrateDummyClusterServiceUnitTest) Test(org.junit.Test)

Example 2 with AnalyzedStatement

use of io.crate.analyze.AnalyzedStatement in project crate by crate.

the class Session method quickExec.

/**
 * Execute a query in one step, avoiding the parse/bind/execute/sync procedure.
 * Opposed to using parse/bind/execute/sync this method is thread-safe.
 *
 * @param parse A function to parse the statement; This can be used to cache the parsed statement.
 *              Use {@link #quickExec(String, ResultReceiver, Row)} to use the regular parser
 */
public void quickExec(String statement, Function<String, Statement> parse, ResultReceiver<?> resultReceiver, Row params) {
    CoordinatorTxnCtx txnCtx = new CoordinatorTxnCtx(sessionContext);
    Statement parsedStmt = parse.apply(statement);
    AnalyzedStatement analyzedStatement = analyzer.analyze(parsedStmt, sessionContext, ParamTypeHints.EMPTY);
    RoutingProvider routingProvider = new RoutingProvider(Randomness.get().nextInt(), planner.getAwarenessAttributes());
    UUID jobId = UUIDs.dirtyUUID();
    ClusterState clusterState = planner.currentClusterState();
    PlannerContext plannerContext = new PlannerContext(clusterState, routingProvider, jobId, txnCtx, nodeCtx, 0, params);
    Plan plan;
    try {
        plan = planner.plan(analyzedStatement, plannerContext);
    } catch (Throwable t) {
        jobsLogs.logPreExecutionFailure(jobId, statement, SQLExceptions.messageOf(t), sessionContext.sessionUser());
        throw t;
    }
    StatementClassifier.Classification classification = StatementClassifier.classify(plan);
    jobsLogs.logExecutionStart(jobId, statement, sessionContext.sessionUser(), classification);
    JobsLogsUpdateListener jobsLogsUpdateListener = new JobsLogsUpdateListener(jobId, jobsLogs);
    if (!analyzedStatement.isWriteOperation()) {
        resultReceiver = new RetryOnFailureResultReceiver(executor.clusterService(), clusterState, // clusterState at the time of the index check is used
        indexName -> clusterState.metadata().hasIndex(indexName), resultReceiver, jobId, (newJobId, retryResultReceiver) -> retryQuery(newJobId, analyzedStatement, routingProvider, new RowConsumerToResultReceiver(retryResultReceiver, 0, jobsLogsUpdateListener), params, txnCtx, nodeCtx));
    }
    RowConsumerToResultReceiver consumer = new RowConsumerToResultReceiver(resultReceiver, 0, jobsLogsUpdateListener);
    plan.execute(executor, plannerContext, consumer, params, SubQueryResults.EMPTY);
}
Also used : ParamTypeHints(io.crate.analyze.ParamTypeHints) RetryOnFailureResultReceiver(io.crate.protocols.postgres.RetryOnFailureResultReceiver) Analyzer(io.crate.analyze.Analyzer) DependencyCarrier(io.crate.planner.DependencyCarrier) ClusterState(org.elasticsearch.cluster.ClusterState) Relations(io.crate.analyze.Relations) RowN(io.crate.data.RowN) TransactionState(io.crate.protocols.postgres.TransactionState) Map(java.util.Map) JobsLogsUpdateListener(io.crate.protocols.postgres.JobsLogsUpdateListener) TableInfo(io.crate.metadata.table.TableInfo) NodeContext(io.crate.metadata.NodeContext) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) UUIDs(org.elasticsearch.common.UUIDs) UUID(java.util.UUID) Lists2(io.crate.common.collections.Lists2) List(java.util.List) Logger(org.apache.logging.log4j.Logger) AnalyzedDiscard(io.crate.analyze.AnalyzedDiscard) Row(io.crate.data.Row) Symbol(io.crate.expression.symbol.Symbol) AnalyzedBegin(io.crate.analyze.AnalyzedBegin) SubQueryResults(io.crate.planner.operators.SubQueryResults) Statement(io.crate.sql.tree.Statement) VisibleForTesting(io.crate.common.annotations.VisibleForTesting) Row1(io.crate.data.Row1) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) AnalyzedCommit(io.crate.analyze.AnalyzedCommit) AccessControl(io.crate.auth.AccessControl) QueriedSelectRelation(io.crate.analyze.QueriedSelectRelation) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JobsLogs(io.crate.execution.engine.collect.stats.JobsLogs) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) Portal(io.crate.protocols.postgres.Portal) Symbols(io.crate.expression.symbol.Symbols) FormatCodes(io.crate.protocols.postgres.FormatCodes) SqlParser(io.crate.sql.parser.SqlParser) Nullable(javax.annotation.Nullable) Iterator(java.util.Iterator) RelationInfo(io.crate.metadata.RelationInfo) Target(io.crate.sql.tree.DiscardStatement.Target) DataType(io.crate.types.DataType) Planner(io.crate.planner.Planner) RoutingProvider(io.crate.metadata.RoutingProvider) RowConsumer(io.crate.data.RowConsumer) StatementClassifier(io.crate.planner.operators.StatementClassifier) AbstractTableRelation(io.crate.analyze.relations.AbstractTableRelation) PlannerContext(io.crate.planner.PlannerContext) Plan(io.crate.planner.Plan) AnalyzedRelation(io.crate.analyze.relations.AnalyzedRelation) ReadOnlyException(io.crate.exceptions.ReadOnlyException) SQLExceptions(io.crate.exceptions.SQLExceptions) LogManager(org.apache.logging.log4j.LogManager) Randomness(org.elasticsearch.common.Randomness) AnalyzedDeallocate(io.crate.analyze.AnalyzedDeallocate) RoutingProvider(io.crate.metadata.RoutingProvider) ClusterState(org.elasticsearch.cluster.ClusterState) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) Statement(io.crate.sql.tree.Statement) Plan(io.crate.planner.Plan) StatementClassifier(io.crate.planner.operators.StatementClassifier) PlannerContext(io.crate.planner.PlannerContext) JobsLogsUpdateListener(io.crate.protocols.postgres.JobsLogsUpdateListener) RetryOnFailureResultReceiver(io.crate.protocols.postgres.RetryOnFailureResultReceiver) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) UUID(java.util.UUID)

Example 3 with AnalyzedStatement

use of io.crate.analyze.AnalyzedStatement in project crate by crate.

the class Session method describe.

public DescribeResult describe(char type, String portalOrStatement) {
    if (LOGGER.isDebugEnabled()) {
        LOGGER.debug("method=describe type={} portalOrStatement={}", type, portalOrStatement);
    }
    switch(type) {
        case 'P':
            Portal portal = getSafePortal(portalOrStatement);
            var analyzedStmt = portal.analyzedStatement();
            return new DescribeResult(portal.preparedStmt().parameterTypes(), analyzedStmt.outputs(), resolveTableFromSelect(analyzedStmt));
        case 'S':
            /*
                 * describe might be called without prior bind call.
                 *
                 * If the client uses server-side prepared statements this is usually the case.
                 *
                 * E.g. the statement is first prepared:
                 *
                 *      parse stmtName=S_1 query=insert into t (x) values ($1) paramTypes=[integer]
                 *      describe type=S portalOrStatement=S_1
                 *      sync
                 *
                 * and then used with different bind calls:
                 *
                 *      bind portalName= statementName=S_1 params=[0]
                 *      describe type=P portalOrStatement=
                 *      execute
                 *
                 *      bind portalName= statementName=S_1 params=[1]
                 *      describe type=P portalOrStatement=
                 *      execute
                 */
            PreparedStmt preparedStmt = preparedStatements.get(portalOrStatement);
            AnalyzedStatement analyzedStatement = preparedStmt.analyzedStatement();
            return new DescribeResult(preparedStmt.parameterTypes(), analyzedStatement.outputs(), resolveTableFromSelect(analyzedStatement));
        default:
            throw new AssertionError("Unsupported type: " + type);
    }
}
Also used : Portal(io.crate.protocols.postgres.Portal) AnalyzedStatement(io.crate.analyze.AnalyzedStatement)

Example 4 with AnalyzedStatement

use of io.crate.analyze.AnalyzedStatement in project crate by crate.

the class Session method bulkExec.

private CompletableFuture<?> bulkExec(Statement statement, List<DeferredExecution> toExec) {
    assert toExec.size() >= 1 : "Must have at least 1 deferred execution for bulk exec";
    var jobId = UUIDs.dirtyUUID();
    var routingProvider = new RoutingProvider(Randomness.get().nextInt(), planner.getAwarenessAttributes());
    var clusterState = executor.clusterService().state();
    var txnCtx = new CoordinatorTxnCtx(sessionContext);
    var plannerContext = new PlannerContext(clusterState, routingProvider, jobId, txnCtx, nodeCtx, 0, null);
    PreparedStmt firstPreparedStatement = toExec.get(0).portal().preparedStmt();
    AnalyzedStatement analyzedStatement = firstPreparedStatement.analyzedStatement();
    Plan plan;
    try {
        plan = planner.plan(analyzedStatement, plannerContext);
    } catch (Throwable t) {
        jobsLogs.logPreExecutionFailure(jobId, firstPreparedStatement.rawStatement(), SQLExceptions.messageOf(t), sessionContext.sessionUser());
        throw t;
    }
    jobsLogs.logExecutionStart(jobId, firstPreparedStatement.rawStatement(), sessionContext.sessionUser(), StatementClassifier.classify(plan));
    var bulkArgs = Lists2.map(toExec, x -> (Row) new RowN(x.portal().params().toArray()));
    List<CompletableFuture<Long>> rowCounts = plan.executeBulk(executor, plannerContext, bulkArgs, SubQueryResults.EMPTY);
    CompletableFuture<Void> allRowCounts = CompletableFuture.allOf(rowCounts.toArray(new CompletableFuture[0]));
    List<CompletableFuture<?>> resultReceiverFutures = Lists2.map(toExec, x -> x.resultReceiver().completionFuture());
    CompletableFuture<Void> allResultReceivers = CompletableFuture.allOf(resultReceiverFutures.toArray(new CompletableFuture[0]));
    return allRowCounts.exceptionally(// swallow exception - failures are set per item in emitResults
    t -> null).thenAccept(ignored -> emitRowCountsToResultReceivers(jobId, jobsLogs, toExec, rowCounts)).runAfterBoth(allResultReceivers, () -> {
    });
}
Also used : ParamTypeHints(io.crate.analyze.ParamTypeHints) RetryOnFailureResultReceiver(io.crate.protocols.postgres.RetryOnFailureResultReceiver) Analyzer(io.crate.analyze.Analyzer) DependencyCarrier(io.crate.planner.DependencyCarrier) ClusterState(org.elasticsearch.cluster.ClusterState) Relations(io.crate.analyze.Relations) RowN(io.crate.data.RowN) TransactionState(io.crate.protocols.postgres.TransactionState) Map(java.util.Map) JobsLogsUpdateListener(io.crate.protocols.postgres.JobsLogsUpdateListener) TableInfo(io.crate.metadata.table.TableInfo) NodeContext(io.crate.metadata.NodeContext) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) UUIDs(org.elasticsearch.common.UUIDs) UUID(java.util.UUID) Lists2(io.crate.common.collections.Lists2) List(java.util.List) Logger(org.apache.logging.log4j.Logger) AnalyzedDiscard(io.crate.analyze.AnalyzedDiscard) Row(io.crate.data.Row) Symbol(io.crate.expression.symbol.Symbol) AnalyzedBegin(io.crate.analyze.AnalyzedBegin) SubQueryResults(io.crate.planner.operators.SubQueryResults) Statement(io.crate.sql.tree.Statement) VisibleForTesting(io.crate.common.annotations.VisibleForTesting) Row1(io.crate.data.Row1) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) AnalyzedCommit(io.crate.analyze.AnalyzedCommit) AccessControl(io.crate.auth.AccessControl) QueriedSelectRelation(io.crate.analyze.QueriedSelectRelation) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JobsLogs(io.crate.execution.engine.collect.stats.JobsLogs) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) Portal(io.crate.protocols.postgres.Portal) Symbols(io.crate.expression.symbol.Symbols) FormatCodes(io.crate.protocols.postgres.FormatCodes) SqlParser(io.crate.sql.parser.SqlParser) Nullable(javax.annotation.Nullable) Iterator(java.util.Iterator) RelationInfo(io.crate.metadata.RelationInfo) Target(io.crate.sql.tree.DiscardStatement.Target) DataType(io.crate.types.DataType) Planner(io.crate.planner.Planner) RoutingProvider(io.crate.metadata.RoutingProvider) RowConsumer(io.crate.data.RowConsumer) StatementClassifier(io.crate.planner.operators.StatementClassifier) AbstractTableRelation(io.crate.analyze.relations.AbstractTableRelation) PlannerContext(io.crate.planner.PlannerContext) Plan(io.crate.planner.Plan) AnalyzedRelation(io.crate.analyze.relations.AnalyzedRelation) ReadOnlyException(io.crate.exceptions.ReadOnlyException) SQLExceptions(io.crate.exceptions.SQLExceptions) LogManager(org.apache.logging.log4j.LogManager) Randomness(org.elasticsearch.common.Randomness) AnalyzedDeallocate(io.crate.analyze.AnalyzedDeallocate) RoutingProvider(io.crate.metadata.RoutingProvider) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) Plan(io.crate.planner.Plan) RowN(io.crate.data.RowN) CompletableFuture(java.util.concurrent.CompletableFuture) PlannerContext(io.crate.planner.PlannerContext) AnalyzedStatement(io.crate.analyze.AnalyzedStatement)

Example 5 with AnalyzedStatement

use of io.crate.analyze.AnalyzedStatement in project crate by crate.

the class Session method analyze.

public void analyze(String statementName, Statement statement, List<DataType> paramTypes, @Nullable String query) {
    AnalyzedStatement analyzedStatement;
    DataType[] parameterTypes;
    try {
        analyzedStatement = analyzer.analyze(statement, sessionContext, new ParamTypeHints(paramTypes));
        parameterTypes = parameterTypeExtractor.getParameterTypes(x -> Relations.traverseDeepSymbols(analyzedStatement, x));
    } catch (Throwable t) {
        jobsLogs.logPreExecutionFailure(UUIDs.dirtyUUID(), query == null ? statementName : query, SQLExceptions.messageOf(t), sessionContext.sessionUser());
        throw t;
    }
    preparedStatements.put(statementName, new PreparedStmt(statement, analyzedStatement, query, parameterTypes));
}
Also used : ParamTypeHints(io.crate.analyze.ParamTypeHints) RetryOnFailureResultReceiver(io.crate.protocols.postgres.RetryOnFailureResultReceiver) Analyzer(io.crate.analyze.Analyzer) DependencyCarrier(io.crate.planner.DependencyCarrier) ClusterState(org.elasticsearch.cluster.ClusterState) Relations(io.crate.analyze.Relations) RowN(io.crate.data.RowN) TransactionState(io.crate.protocols.postgres.TransactionState) Map(java.util.Map) JobsLogsUpdateListener(io.crate.protocols.postgres.JobsLogsUpdateListener) TableInfo(io.crate.metadata.table.TableInfo) NodeContext(io.crate.metadata.NodeContext) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) UUIDs(org.elasticsearch.common.UUIDs) UUID(java.util.UUID) Lists2(io.crate.common.collections.Lists2) List(java.util.List) Logger(org.apache.logging.log4j.Logger) AnalyzedDiscard(io.crate.analyze.AnalyzedDiscard) Row(io.crate.data.Row) Symbol(io.crate.expression.symbol.Symbol) AnalyzedBegin(io.crate.analyze.AnalyzedBegin) SubQueryResults(io.crate.planner.operators.SubQueryResults) Statement(io.crate.sql.tree.Statement) VisibleForTesting(io.crate.common.annotations.VisibleForTesting) Row1(io.crate.data.Row1) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) AnalyzedCommit(io.crate.analyze.AnalyzedCommit) AccessControl(io.crate.auth.AccessControl) QueriedSelectRelation(io.crate.analyze.QueriedSelectRelation) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JobsLogs(io.crate.execution.engine.collect.stats.JobsLogs) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) Portal(io.crate.protocols.postgres.Portal) Symbols(io.crate.expression.symbol.Symbols) FormatCodes(io.crate.protocols.postgres.FormatCodes) SqlParser(io.crate.sql.parser.SqlParser) Nullable(javax.annotation.Nullable) Iterator(java.util.Iterator) RelationInfo(io.crate.metadata.RelationInfo) Target(io.crate.sql.tree.DiscardStatement.Target) DataType(io.crate.types.DataType) Planner(io.crate.planner.Planner) RoutingProvider(io.crate.metadata.RoutingProvider) RowConsumer(io.crate.data.RowConsumer) StatementClassifier(io.crate.planner.operators.StatementClassifier) AbstractTableRelation(io.crate.analyze.relations.AbstractTableRelation) PlannerContext(io.crate.planner.PlannerContext) Plan(io.crate.planner.Plan) AnalyzedRelation(io.crate.analyze.relations.AnalyzedRelation) ReadOnlyException(io.crate.exceptions.ReadOnlyException) SQLExceptions(io.crate.exceptions.SQLExceptions) LogManager(org.apache.logging.log4j.LogManager) Randomness(org.elasticsearch.common.Randomness) AnalyzedDeallocate(io.crate.analyze.AnalyzedDeallocate) DataType(io.crate.types.DataType) AnalyzedStatement(io.crate.analyze.AnalyzedStatement) ParamTypeHints(io.crate.analyze.ParamTypeHints)

Aggregations

AnalyzedStatement (io.crate.analyze.AnalyzedStatement)5 Row (io.crate.data.Row)4 RowConsumer (io.crate.data.RowConsumer)4 JobsLogs (io.crate.execution.engine.collect.stats.JobsLogs)4 DependencyCarrier (io.crate.planner.DependencyCarrier)4 Plan (io.crate.planner.Plan)4 Planner (io.crate.planner.Planner)4 PlannerContext (io.crate.planner.PlannerContext)4 SubQueryResults (io.crate.planner.operators.SubQueryResults)4 Portal (io.crate.protocols.postgres.Portal)4 AnalyzedBegin (io.crate.analyze.AnalyzedBegin)3 AnalyzedCommit (io.crate.analyze.AnalyzedCommit)3 AnalyzedDeallocate (io.crate.analyze.AnalyzedDeallocate)3 AnalyzedDiscard (io.crate.analyze.AnalyzedDiscard)3 Analyzer (io.crate.analyze.Analyzer)3 ParamTypeHints (io.crate.analyze.ParamTypeHints)3 QueriedSelectRelation (io.crate.analyze.QueriedSelectRelation)3 Relations (io.crate.analyze.Relations)3 AbstractTableRelation (io.crate.analyze.relations.AbstractTableRelation)3 AnalyzedRelation (io.crate.analyze.relations.AnalyzedRelation)3