Search in sources :

Example 1 with PlanBuildContext

use of io.confluent.ksql.planner.plan.PlanBuildContext in project ksql by confluentinc.

the class SchemaKSourceFactory method buildTable.

private static SchemaKTable<?> buildTable(final PlanBuildContext buildContext, final DataSource dataSource, final Stacker contextStacker) {
    final KeyFormat keyFormat = dataSource.getKsqlTopic().getKeyFormat();
    if (keyFormat.isWindowed()) {
        throw new IllegalArgumentException("windowed");
    }
    final SourceStep<KTableHolder<GenericKey>> step;
    final int pseudoColumnVersionToUse = determinePseudoColumnVersionToUse(buildContext);
    // If the old query has a v1 table step, continue to use it.
    // See https://github.com/confluentinc/ksql/pull/7990
    boolean useOldExecutionStepVersion = false;
    if (buildContext.getPlanInfo().isPresent()) {
        final Set<ExecutionStep<?>> sourceSteps = buildContext.getPlanInfo().get().getSources();
        useOldExecutionStepVersion = sourceSteps.stream().anyMatch(executionStep -> executionStep instanceof TableSourceV1);
    }
    if (useOldExecutionStepVersion && pseudoColumnVersionToUse != SystemColumns.LEGACY_PSEUDOCOLUMN_VERSION_NUMBER) {
        throw new IllegalStateException("TableSourceV2 was released in conjunction with pseudocolumn" + "version 1. Something has gone very wrong");
    }
    if (buildContext.getKsqlConfig().getBoolean(KsqlConfig.KSQL_ROWPARTITION_ROWOFFSET_ENABLED) && !useOldExecutionStepVersion) {
        step = ExecutionStepFactory.tableSource(contextStacker, dataSource.getSchema(), dataSource.getKafkaTopicName(), Formats.from(dataSource.getKsqlTopic()), dataSource.getTimestampColumn(), InternalFormats.of(keyFormat, Formats.from(dataSource.getKsqlTopic()).getValueFormat()), pseudoColumnVersionToUse);
    } else {
        step = ExecutionStepFactory.tableSourceV1(contextStacker, dataSource.getSchema(), dataSource.getKafkaTopicName(), Formats.from(dataSource.getKsqlTopic()), dataSource.getTimestampColumn(), pseudoColumnVersionToUse);
    }
    return schemaKTable(buildContext, resolveSchema(buildContext, step, dataSource), dataSource.getKsqlTopic().getKeyFormat(), step);
}
Also used : ExecutionStep(io.confluent.ksql.execution.plan.ExecutionStep) DataSource(io.confluent.ksql.metastore.model.DataSource) ExecutionStep(io.confluent.ksql.execution.plan.ExecutionStep) SystemColumns(io.confluent.ksql.schema.ksql.SystemColumns) PlanBuildContext(io.confluent.ksql.planner.plan.PlanBuildContext) KeyFormat(io.confluent.ksql.serde.KeyFormat) QueryContext(io.confluent.ksql.execution.context.QueryContext) Set(java.util.Set) KsqlConfig(io.confluent.ksql.util.KsqlConfig) Formats(io.confluent.ksql.execution.plan.Formats) InternalFormats(io.confluent.ksql.serde.InternalFormats) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) StreamSource(io.confluent.ksql.execution.plan.StreamSource) StepSchemaResolver(io.confluent.ksql.execution.streams.StepSchemaResolver) WindowInfo(io.confluent.ksql.serde.WindowInfo) KStreamHolder(io.confluent.ksql.execution.plan.KStreamHolder) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) SourceStep(io.confluent.ksql.execution.plan.SourceStep) WindowedStreamSource(io.confluent.ksql.execution.plan.WindowedStreamSource) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) Windowed(org.apache.kafka.streams.kstream.Windowed) Stacker(io.confluent.ksql.execution.context.QueryContext.Stacker) GenericKey(io.confluent.ksql.GenericKey) ExecutionStepFactory(io.confluent.ksql.execution.streams.ExecutionStepFactory) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) KeyFormat(io.confluent.ksql.serde.KeyFormat)

Example 2 with PlanBuildContext

use of io.confluent.ksql.planner.plan.PlanBuildContext in project ksql by confluentinc.

the class PhysicalPlanBuilder method buildPhysicalPlan.

public PhysicalPlan buildPhysicalPlan(final LogicalPlanNode logicalPlanNode, final QueryId queryId, final Optional<PlanInfo> oldPlanInfo) {
    final OutputNode outputNode = logicalPlanNode.getNode().orElseThrow(() -> new IllegalArgumentException("Need an output node to build a plan"));
    final PlanBuildContext buildContext = PlanBuildContext.of(ksqlConfig, serviceContext, functionRegistry, oldPlanInfo);
    final SchemaKStream<?> resultStream = outputNode.buildStream(buildContext);
    final LogicalSchema logicalSchema = outputNode.getSchema();
    final LogicalSchema physicalSchema = resultStream.getSchema();
    if (!logicalSchema.equals(physicalSchema)) {
        throw new IllegalStateException("Logical and Physical schemas do not match!" + System.lineSeparator() + "Logical : " + logicalSchema + System.lineSeparator() + "Physical: " + physicalSchema);
    }
    return new PhysicalPlan(queryId, resultStream.getSourceStep());
}
Also used : OutputNode(io.confluent.ksql.planner.plan.OutputNode) PlanBuildContext(io.confluent.ksql.planner.plan.PlanBuildContext) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema)

Aggregations

PlanBuildContext (io.confluent.ksql.planner.plan.PlanBuildContext)2 LogicalSchema (io.confluent.ksql.schema.ksql.LogicalSchema)2 GenericKey (io.confluent.ksql.GenericKey)1 QueryContext (io.confluent.ksql.execution.context.QueryContext)1 Stacker (io.confluent.ksql.execution.context.QueryContext.Stacker)1 ExecutionStep (io.confluent.ksql.execution.plan.ExecutionStep)1 Formats (io.confluent.ksql.execution.plan.Formats)1 KStreamHolder (io.confluent.ksql.execution.plan.KStreamHolder)1 KTableHolder (io.confluent.ksql.execution.plan.KTableHolder)1 SourceStep (io.confluent.ksql.execution.plan.SourceStep)1 StreamSource (io.confluent.ksql.execution.plan.StreamSource)1 TableSourceV1 (io.confluent.ksql.execution.plan.TableSourceV1)1 WindowedStreamSource (io.confluent.ksql.execution.plan.WindowedStreamSource)1 ExecutionStepFactory (io.confluent.ksql.execution.streams.ExecutionStepFactory)1 StepSchemaResolver (io.confluent.ksql.execution.streams.StepSchemaResolver)1 DataSource (io.confluent.ksql.metastore.model.DataSource)1 OutputNode (io.confluent.ksql.planner.plan.OutputNode)1 SystemColumns (io.confluent.ksql.schema.ksql.SystemColumns)1 InternalFormats (io.confluent.ksql.serde.InternalFormats)1 KeyFormat (io.confluent.ksql.serde.KeyFormat)1