Search in sources :

Example 1 with ExecuteResult

use of io.confluent.ksql.KsqlExecutionContext.ExecuteResult in project ksql by confluentinc.

the class EngineExecutor method execute.

ExecuteResult execute(final KsqlPlan plan, final boolean restoreInProgress) {
    if (!plan.getQueryPlan().isPresent()) {
        final String ddlResult = plan.getDdlCommand().map(ddl -> executeDdl(ddl, plan.getStatementText(), false, Collections.emptySet(), restoreInProgress)).orElseThrow(() -> new IllegalStateException("DdlResult should be present if there is no physical plan."));
        return ExecuteResult.of(ddlResult);
    }
    final QueryPlan queryPlan = plan.getQueryPlan().get();
    final KsqlConstants.PersistentQueryType persistentQueryType = plan.getPersistentQueryType().get();
    // that attempt to write to a sink (i.e. INSERT or CREATE_AS).
    if (persistentQueryType != KsqlConstants.PersistentQueryType.CREATE_SOURCE) {
        final DataSource sinkSource = engineContext.getMetaStore().getSource(queryPlan.getSink().get());
        if (sinkSource != null && sinkSource.isSource()) {
            throw new KsqlException(String.format("Cannot insert into read-only %s: %s", sinkSource.getDataSourceType().getKsqlType().toLowerCase(), sinkSource.getName().text()));
        }
    }
    final Optional<String> ddlResult = plan.getDdlCommand().map(ddl -> executeDdl(ddl, plan.getStatementText(), true, queryPlan.getSources(), restoreInProgress));
    // Return if the source to create already exists.
    if (ddlResult.isPresent() && ddlResult.get().contains("already exists")) {
        return ExecuteResult.of(ddlResult.get());
    }
    // must be executed.
    if (persistentQueryType == KsqlConstants.PersistentQueryType.CREATE_SOURCE && !isSourceTableMaterializationEnabled()) {
        LOG.info(String.format("Source table query '%s' won't be materialized because '%s' is disabled.", plan.getStatementText(), KsqlConfig.KSQL_SOURCE_TABLE_MATERIALIZATION_ENABLED));
        return ExecuteResult.of(ddlResult.get());
    }
    return ExecuteResult.of(executePersistentQuery(queryPlan, plan.getStatementText(), persistentQueryType));
}
Also used : DataSource(io.confluent.ksql.metastore.model.DataSource) PushPhysicalPlanCreator(io.confluent.ksql.physical.scalablepush.PushPhysicalPlanCreator) CreateTableAsSelect(io.confluent.ksql.parser.tree.CreateTableAsSelect) Arrays(java.util.Arrays) InternalFunctionRegistry(io.confluent.ksql.function.InternalFunctionRegistry) SourceName(io.confluent.ksql.name.SourceName) RoutingOptions(io.confluent.ksql.execution.streams.RoutingOptions) PushPhysicalPlanManager(io.confluent.ksql.physical.scalablepush.PushPhysicalPlanManager) PushPhysicalPlanBuilder(io.confluent.ksql.physical.scalablepush.PushPhysicalPlanBuilder) RoutingNodeType(io.confluent.ksql.util.KsqlConstants.RoutingNodeType) TransientQueryMetadata(io.confluent.ksql.util.TransientQueryMetadata) ExecuteResult(io.confluent.ksql.KsqlExecutionContext.ExecuteResult) Map(java.util.Map) KsqlBareOutputNode(io.confluent.ksql.planner.plan.KsqlBareOutputNode) QueryId(io.confluent.ksql.query.QueryId) ExecutionStep(io.confluent.ksql.execution.plan.ExecutionStep) RefinementInfo(io.confluent.ksql.serde.RefinementInfo) ImmutableAnalysis(io.confluent.ksql.analyzer.ImmutableAnalysis) Sink(io.confluent.ksql.parser.tree.Sink) Set(java.util.Set) Relation(io.confluent.ksql.parser.tree.Relation) ConfiguredStatement(io.confluent.ksql.statement.ConfiguredStatement) MetaStoreImpl(io.confluent.ksql.metastore.MetaStoreImpl) KsqlException(io.confluent.ksql.util.KsqlException) TransientQueryQueue(io.confluent.ksql.query.TransientQueryQueue) PullQueryResult(io.confluent.ksql.physical.pull.PullQueryResult) Iterables(com.google.common.collect.Iterables) FormatOptions(io.confluent.ksql.schema.utils.FormatOptions) PushRouting(io.confluent.ksql.physical.scalablepush.PushRouting) UnqualifiedColumnReferenceExp(io.confluent.ksql.execution.expression.tree.UnqualifiedColumnReferenceExp) CreateStreamAsSelect(io.confluent.ksql.parser.tree.CreateStreamAsSelect) SessionConfig(io.confluent.ksql.config.SessionConfig) CreateStream(io.confluent.ksql.parser.tree.CreateStream) SingleColumn(io.confluent.ksql.parser.tree.SingleColumn) MetaStore(io.confluent.ksql.metastore.MetaStore) KsqlStructuredDataOutputNode(io.confluent.ksql.planner.plan.KsqlStructuredDataOutputNode) PushRoutingOptions(io.confluent.ksql.physical.scalablepush.PushRoutingOptions) PlanInfoExtractor(io.confluent.ksql.execution.plan.PlanInfoExtractor) DataSourceNode(io.confluent.ksql.planner.plan.DataSourceNode) QueryContainer(io.confluent.ksql.parser.tree.QueryContainer) OutputNode(io.confluent.ksql.planner.plan.OutputNode) Throwables(com.google.common.base.Throwables) PushQueryMetadata(io.confluent.ksql.util.PushQueryMetadata) PushQueryQueuePopulator(io.confluent.ksql.physical.scalablepush.PushQueryQueuePopulator) ValueFormat(io.confluent.ksql.serde.ValueFormat) Table(io.confluent.ksql.parser.tree.Table) KsqlStatementException(io.confluent.ksql.util.KsqlStatementException) CreateAsSelect(io.confluent.ksql.parser.tree.CreateAsSelect) KsqlTopic(io.confluent.ksql.execution.ddl.commands.KsqlTopic) OutputRefinement(io.confluent.ksql.parser.OutputRefinement) LogicalPlanNode(io.confluent.ksql.planner.LogicalPlanNode) Query(io.confluent.ksql.parser.tree.Query) ServiceContext(io.confluent.ksql.services.ServiceContext) LoggerFactory(org.slf4j.LoggerFactory) AliasedRelation(io.confluent.ksql.parser.tree.AliasedRelation) Formats(io.confluent.ksql.execution.plan.Formats) MutableMetaStore(io.confluent.ksql.metastore.MutableMetaStore) Context(io.vertx.core.Context) CreateTable(io.confluent.ksql.parser.tree.CreateTable) Locale(java.util.Locale) PersistentQueryMetadata(io.confluent.ksql.util.PersistentQueryMetadata) KsqlTable(io.confluent.ksql.metastore.model.KsqlTable) TopicPartition(org.apache.kafka.common.TopicPartition) ImmutableSet(com.google.common.collect.ImmutableSet) ImmutableMap(com.google.common.collect.ImmutableMap) Collection(java.util.Collection) ScalablePushQueryMetadata(io.confluent.ksql.util.ScalablePushQueryMetadata) ScalablePushQueryMetrics(io.confluent.ksql.internal.ScalablePushQueryMetrics) KsqlConfig(io.confluent.ksql.util.KsqlConfig) ExecutableDdlStatement(io.confluent.ksql.parser.tree.ExecutableDdlStatement) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) Collectors(java.util.stream.Collectors) DdlCommand(io.confluent.ksql.execution.ddl.commands.DdlCommand) Objects(java.util.Objects) PullQueryExecutorMetrics(io.confluent.ksql.internal.PullQueryExecutorMetrics) QueryPlannerOptions(io.confluent.ksql.planner.QueryPlannerOptions) ConsistencyOffsetVector(io.confluent.ksql.util.ConsistencyOffsetVector) Optional(java.util.Optional) Statement(io.confluent.ksql.parser.tree.Statement) KsqlConstants(io.confluent.ksql.util.KsqlConstants) SuppressFBWarnings(edu.umd.cs.findbugs.annotations.SuppressFBWarnings) PullQueryQueuePopulator(io.confluent.ksql.physical.pull.PullQueryQueuePopulator) PullQueryQueue(io.confluent.ksql.query.PullQueryQueue) PushPhysicalPlan(io.confluent.ksql.physical.scalablepush.PushPhysicalPlan) HARouting(io.confluent.ksql.physical.pull.HARouting) PlanInfo(io.confluent.ksql.execution.plan.PlanInfo) PullPhysicalPlanBuilder(io.confluent.ksql.physical.pull.PullPhysicalPlanBuilder) KeyFormat(io.confluent.ksql.serde.KeyFormat) ResultType(io.confluent.ksql.util.PushQueryMetadata.ResultType) CompletableFuture(java.util.concurrent.CompletableFuture) DataSourceType(io.confluent.ksql.metastore.model.DataSource.DataSourceType) OptionalInt(java.util.OptionalInt) PushOffsetRange(io.confluent.ksql.util.PushOffsetRange) LogicalPlanner(io.confluent.ksql.planner.LogicalPlanner) Logger(org.slf4j.Logger) PhysicalPlan(io.confluent.ksql.physical.PhysicalPlan) PlanSummary(io.confluent.ksql.util.PlanSummary) PullPhysicalPlan(io.confluent.ksql.physical.pull.PullPhysicalPlan) PlanNode(io.confluent.ksql.planner.plan.PlanNode) QueryRegistry(io.confluent.ksql.query.QueryRegistry) Collections(java.util.Collections) CreateTableCommand(io.confluent.ksql.execution.ddl.commands.CreateTableCommand) Select(io.confluent.ksql.parser.tree.Select) PushQueryPreparer(io.confluent.ksql.physical.scalablepush.PushQueryPreparer) KsqlConstants(io.confluent.ksql.util.KsqlConstants) KsqlException(io.confluent.ksql.util.KsqlException) DataSource(io.confluent.ksql.metastore.model.DataSource)

Example 2 with ExecuteResult

use of io.confluent.ksql.KsqlExecutionContext.ExecuteResult in project ksql by confluentinc.

the class KsqlEngineTestUtil method execute.

@SuppressWarnings({ "rawtypes", "unchecked" })
private static ExecuteResult execute(final ServiceContext serviceContext, final KsqlExecutionContext executionContext, final ParsedStatement stmt, final KsqlConfig ksqlConfig, final Map<String, Object> overriddenProperties, final Optional<DefaultSchemaInjector> schemaInjector) {
    final PreparedStatement<?> prepared = executionContext.prepare(stmt);
    final ConfiguredStatement<?> configured = ConfiguredStatement.of(prepared, SessionConfig.of(ksqlConfig, overriddenProperties));
    final ConfiguredStatement<?> withFormats = new DefaultFormatInjector().inject(configured);
    final ConfiguredStatement<?> withSchema = schemaInjector.map(injector -> injector.inject(withFormats)).orElse((ConfiguredStatement) withFormats);
    final ConfiguredStatement<?> reformatted = new SqlFormatInjector(executionContext).inject(withSchema);
    try {
        return executionContext.execute(serviceContext, reformatted);
    } catch (final KsqlStatementException e) {
        // can easily check that the failed statement is the input statement
        throw new KsqlStatementException(e.getRawMessage(), stmt.getStatementText(), e.getCause());
    }
}
Also used : Query(io.confluent.ksql.parser.tree.Query) SequentialQueryIdGenerator(io.confluent.ksql.query.id.SequentialQueryIdGenerator) SchemaRegistryClient(io.confluent.kafka.schemaregistry.client.SchemaRegistryClient) ServiceContext(io.confluent.ksql.services.ServiceContext) SchemaRegistryTopicSchemaSupplier(io.confluent.ksql.schema.ksql.inference.SchemaRegistryTopicSchemaSupplier) ProcessingLogContext(io.confluent.ksql.logging.processing.ProcessingLogContext) MutableMetaStore(io.confluent.ksql.metastore.MutableMetaStore) MetricCollectors(io.confluent.ksql.metrics.MetricCollectors) Function(java.util.function.Function) SessionConfig(io.confluent.ksql.config.SessionConfig) TransientQueryMetadata(io.confluent.ksql.util.TransientQueryMetadata) ExecuteResult(io.confluent.ksql.KsqlExecutionContext.ExecuteResult) Map(java.util.Map) ParsedStatement(io.confluent.ksql.parser.KsqlParser.ParsedStatement) DefaultFormatInjector(io.confluent.ksql.format.DefaultFormatInjector) QueryIdGenerator(io.confluent.ksql.query.id.QueryIdGenerator) QueryMetadata(io.confluent.ksql.util.QueryMetadata) ConfiguredStatement(io.confluent.ksql.statement.ConfiguredStatement) KsqlConfig(io.confluent.ksql.util.KsqlConfig) DefaultSchemaInjector(io.confluent.ksql.schema.ksql.inference.DefaultSchemaInjector) Collectors(java.util.stream.Collectors) KsqlStatementException(io.confluent.ksql.util.KsqlStatementException) List(java.util.List) KsqlExecutionContext(io.confluent.ksql.KsqlExecutionContext) KsqlEngineMetrics(io.confluent.ksql.internal.KsqlEngineMetrics) Optional(java.util.Optional) Collections(java.util.Collections) PreparedStatement(io.confluent.ksql.parser.KsqlParser.PreparedStatement) DefaultFormatInjector(io.confluent.ksql.format.DefaultFormatInjector) KsqlStatementException(io.confluent.ksql.util.KsqlStatementException)

Example 3 with ExecuteResult

use of io.confluent.ksql.KsqlExecutionContext.ExecuteResult in project ksql by confluentinc.

the class KsqlEngineTest method shouldRegisterPersistentQueriesOnlyInSandbox.

@Test
public void shouldRegisterPersistentQueriesOnlyInSandbox() {
    // Given:
    final PreparedStatement<?> prepared = prepare(parse("create table bar as select * from test2;").get(0));
    // When:
    final ExecuteResult result = sandbox.execute(sandboxServiceContext, ConfiguredStatement.of(prepared, SessionConfig.of(ksqlConfig, new HashMap<>())));
    // Then:
    assertThat(result.getQuery(), is(not(Optional.empty())));
    assertThat(sandbox.getPersistentQuery(getQueryId(result.getQuery().get())), is(not(Optional.empty())));
    assertThat(ksqlEngine.getPersistentQuery(getQueryId(result.getQuery().get())), is(Optional.empty()));
}
Also used : ExecuteResult(io.confluent.ksql.KsqlExecutionContext.ExecuteResult) Test(org.junit.Test)

Example 4 with ExecuteResult

use of io.confluent.ksql.KsqlExecutionContext.ExecuteResult in project ksql by confluentinc.

the class KsqlEngineTest method shouldNotThrowWhenExecutingDuplicateTableWithIfNotExists.

@Test
public void shouldNotThrowWhenExecutingDuplicateTableWithIfNotExists() {
    // Given:
    final List<ParsedStatement> parsed = ksqlEngine.parse("CREATE TABLE FOO WITH (KAFKA_TOPIC='BAR') AS SELECT * FROM TEST2; " + "CREATE TABLE IF NOT EXISTS FOO WITH (KAFKA_TOPIC='BAR') AS SELECT * FROM TEST2;");
    givenStatementAlreadyExecuted(parsed.get(0));
    final PreparedStatement<?> prepared = prepare(parsed.get(1));
    // When:
    ExecuteResult executeResult = ksqlEngine.execute(serviceContext, ConfiguredStatement.of(prepared, SessionConfig.of(ksqlConfig, new HashMap<>())));
    // Then:
    assertThat(executeResult.getQuery(), is(Optional.empty()));
    assertThat(executeResult.getCommandResult(), is(Optional.of("Cannot add table `FOO`: A table with the same name already exists.")));
}
Also used : ParsedStatement(io.confluent.ksql.parser.KsqlParser.ParsedStatement) ExecuteResult(io.confluent.ksql.KsqlExecutionContext.ExecuteResult) Test(org.junit.Test)

Example 5 with ExecuteResult

use of io.confluent.ksql.KsqlExecutionContext.ExecuteResult in project ksql by confluentinc.

the class KsqlEngineTest method shouldExecuteDdlStatement.

@Test
public void shouldExecuteDdlStatement() {
    // Given:
    givenTopicsExist("foo");
    final PreparedStatement<?> statement = prepare(parse("CREATE STREAM FOO (a int) WITH (kafka_topic='foo', value_format='json', key_format='kafka');").get(0));
    // When:
    final ExecuteResult result = sandbox.execute(sandboxServiceContext, ConfiguredStatement.of(statement, SessionConfig.of(ksqlConfig, new HashMap<>())));
    // Then:
    assertThat(result.getCommandResult(), is(Optional.of("Stream created")));
}
Also used : ExecuteResult(io.confluent.ksql.KsqlExecutionContext.ExecuteResult) Test(org.junit.Test)

Aggregations

ExecuteResult (io.confluent.ksql.KsqlExecutionContext.ExecuteResult)14 Test (org.junit.Test)10 ParsedStatement (io.confluent.ksql.parser.KsqlParser.ParsedStatement)6 QueryId (io.confluent.ksql.query.QueryId)6 KsqlConfig (io.confluent.ksql.util.KsqlConfig)5 SessionConfig (io.confluent.ksql.config.SessionConfig)4 ServiceContext (io.confluent.ksql.services.ServiceContext)4 ConfiguredStatement (io.confluent.ksql.statement.ConfiguredStatement)4 PersistentQueryMetadata (io.confluent.ksql.util.PersistentQueryMetadata)4 Collections (java.util.Collections)4 Map (java.util.Map)4 Optional (java.util.Optional)4 Collectors (java.util.stream.Collectors)4 ImmutableMap (com.google.common.collect.ImmutableMap)3 Iterables (com.google.common.collect.Iterables)3 QueryMetadata (io.confluent.ksql.util.QueryMetadata)3 List (java.util.List)3 ImmutableList (com.google.common.collect.ImmutableList)2 MockSchemaRegistryClient (io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient)2 SchemaRegistryClient (io.confluent.kafka.schemaregistry.client.SchemaRegistryClient)2