Search in sources :

Example 1 with TableSourceV1

use of io.confluent.ksql.execution.plan.TableSourceV1 in project ksql by confluentinc.

the class SourceBuilderV1Test method givenUnwindowedSourceTableV1.

private void givenUnwindowedSourceTableV1(final Boolean forceChangelog, final int pseudoColumnVersion) {
    when(buildContext.buildKeySerde(any(), any(), any())).thenReturn(keySerde);
    givenConsumed(consumed, keySerde);
    tableSourceV1 = new TableSourceV1(new ExecutionStepPropertiesV1(ctx), TOPIC_NAME, Formats.of(keyFormatInfo, valueFormatInfo, KEY_FEATURES, VALUE_FEATURES), TIMESTAMP_COLUMN, SOURCE_SCHEMA, Optional.of(forceChangelog), OptionalInt.of(pseudoColumnVersion));
}
Also used : TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) ExecutionStepPropertiesV1(io.confluent.ksql.execution.plan.ExecutionStepPropertiesV1)

Example 2 with TableSourceV1

use of io.confluent.ksql.execution.plan.TableSourceV1 in project ksql by confluentinc.

the class StepSchemaResolverTest method shouldResolveSchemaForTableSourceV1.

@Test
public void shouldResolveSchemaForTableSourceV1() {
    // Given:
    final TableSourceV1 step = new TableSourceV1(PROPERTIES, "foo", formats, Optional.empty(), SCHEMA, Optional.of(true), OptionalInt.of(SystemColumns.CURRENT_PSEUDOCOLUMN_VERSION_NUMBER));
    // When:
    final LogicalSchema result = resolver.resolve(step, SCHEMA);
    // Then:
    assertThat(result, is(SCHEMA.withPseudoAndKeyColsInValue(false)));
}
Also used : TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) Test(org.junit.Test)

Example 3 with TableSourceV1

use of io.confluent.ksql.execution.plan.TableSourceV1 in project ksql by confluentinc.

the class SourceBuilderV1 method buildKTable.

@Override
<K> KTable<K, GenericRow> buildKTable(final SourceStep<?> streamSource, final RuntimeBuildContext buildContext, final Consumed<K, GenericRow> consumed, final Function<K, Collection<?>> keyGenerator, final Materialized<K, GenericRow, KeyValueStore<Bytes, byte[]>> materialized, final Serde<GenericRow> valueSerde, final String stateStoreName, final PlanInfo planInfo) {
    validateNotUsingOldExecutionStepWithNewQueries(streamSource);
    final boolean forceChangelog = streamSource instanceof TableSourceV1 && ((TableSourceV1) streamSource).isForceChangelog();
    final KTable<K, GenericRow> table;
    if (!forceChangelog) {
        final String changelogTopic = changelogTopic(buildContext, stateStoreName);
        final Callback onFailure = getRegisterCallback(buildContext, streamSource.getFormats().getValueFormat());
        table = buildContext.getStreamsBuilder().table(streamSource.getTopicName(), consumed.withValueSerde(StaticTopicSerde.wrap(changelogTopic, valueSerde, onFailure)), materialized);
    } else {
        final KTable<K, GenericRow> source = buildContext.getStreamsBuilder().table(streamSource.getTopicName(), consumed);
        final boolean forceMaterialization = !planInfo.isRepartitionedInPlan(streamSource);
        if (forceMaterialization) {
            // add this identity mapValues call to prevent the source-changelog
            // optimization in kafka streams - we don't want this optimization to
            // be enabled because we cannot require symmetric serialization between
            // producer and KSQL (see https://issues.apache.org/jira/browse/KAFKA-10179
            // and https://github.com/confluentinc/ksql/issues/5673 for more details)
            table = source.mapValues(row -> row, materialized);
        } else {
            // if we know this table source is repartitioned later in the topology,
            // we do not need to force a materialization at this source step since the
            // re-partitioned topic will be used for any subsequent state stores, in lieu
            // of the original source topic, thus avoiding the issues above.
            // See https://github.com/confluentinc/ksql/issues/6650
            table = source.mapValues(row -> row);
        }
    }
    return table.transformValues(new AddKeyAndPseudoColumns<>(keyGenerator, streamSource.getPseudoColumnVersion(), streamSource.getSourceSchema().headers()));
}
Also used : GenericRow(io.confluent.ksql.GenericRow) SourceBuilderUtils.getRegisterCallback(io.confluent.ksql.execution.streams.SourceBuilderUtils.getRegisterCallback) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) PlanInfo(io.confluent.ksql.execution.plan.PlanInfo) SourceBuilderUtils.getKeySerde(io.confluent.ksql.execution.streams.SourceBuilderUtils.getKeySerde) SourceBuilderUtils.getWindowedKeySerde(io.confluent.ksql.execution.streams.SourceBuilderUtils.getWindowedKeySerde) SourceBuilderUtils.buildSourceConsumed(io.confluent.ksql.execution.streams.SourceBuilderUtils.buildSourceConsumed) SourceBuilderUtils.getPhysicalSchema(io.confluent.ksql.execution.streams.SourceBuilderUtils.getPhysicalSchema) MaterializationInfo(io.confluent.ksql.execution.materialization.MaterializationInfo) AutoOffsetReset(org.apache.kafka.streams.Topology.AutoOffsetReset) KStream(org.apache.kafka.streams.kstream.KStream) Function(java.util.function.Function) WindowInfo(io.confluent.ksql.serde.WindowInfo) AddKeyAndPseudoColumns(io.confluent.ksql.execution.streams.SourceBuilderUtils.AddKeyAndPseudoColumns) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) SourceStep(io.confluent.ksql.execution.plan.SourceStep) SourceBuilderUtils.getValueSerde(io.confluent.ksql.execution.streams.SourceBuilderUtils.getValueSerde) Windowed(org.apache.kafka.streams.kstream.Windowed) Serde(org.apache.kafka.common.serialization.Serde) SourceBuilderUtils.changelogTopic(io.confluent.ksql.execution.streams.SourceBuilderUtils.changelogTopic) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) StaticTopicSerde(io.confluent.ksql.serde.StaticTopicSerde) KTable(org.apache.kafka.streams.kstream.KTable) RuntimeBuildContext(io.confluent.ksql.execution.runtime.RuntimeBuildContext) SourceBuilderUtils.buildSchema(io.confluent.ksql.execution.streams.SourceBuilderUtils.buildSchema) SourceBuilderUtils.windowedKeyGenerator(io.confluent.ksql.execution.streams.SourceBuilderUtils.windowedKeyGenerator) Consumed(org.apache.kafka.streams.kstream.Consumed) Collection(java.util.Collection) SourceBuilderUtils.tableChangeLogOpName(io.confluent.ksql.execution.streams.SourceBuilderUtils.tableChangeLogOpName) ExecutionKeyFactory(io.confluent.ksql.execution.plan.ExecutionKeyFactory) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) StreamSource(io.confluent.ksql.execution.plan.StreamSource) Bytes(org.apache.kafka.common.utils.Bytes) KStreamHolder(io.confluent.ksql.execution.plan.KStreamHolder) WindowedStreamSource(io.confluent.ksql.execution.plan.WindowedStreamSource) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) WindowedTableSource(io.confluent.ksql.execution.plan.WindowedTableSource) GenericRow(io.confluent.ksql.GenericRow) Callback(io.confluent.ksql.serde.StaticTopicSerde.Callback) Materialized(org.apache.kafka.streams.kstream.Materialized) KsqlException(io.confluent.ksql.util.KsqlException) GenericKey(io.confluent.ksql.GenericKey) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) SourceBuilderUtils.getRegisterCallback(io.confluent.ksql.execution.streams.SourceBuilderUtils.getRegisterCallback) Callback(io.confluent.ksql.serde.StaticTopicSerde.Callback)

Example 4 with TableSourceV1

use of io.confluent.ksql.execution.plan.TableSourceV1 in project ksql by confluentinc.

the class SchemaKSourceFactory method buildTable.

private static SchemaKTable<?> buildTable(final PlanBuildContext buildContext, final DataSource dataSource, final Stacker contextStacker) {
    final KeyFormat keyFormat = dataSource.getKsqlTopic().getKeyFormat();
    if (keyFormat.isWindowed()) {
        throw new IllegalArgumentException("windowed");
    }
    final SourceStep<KTableHolder<GenericKey>> step;
    final int pseudoColumnVersionToUse = determinePseudoColumnVersionToUse(buildContext);
    // If the old query has a v1 table step, continue to use it.
    // See https://github.com/confluentinc/ksql/pull/7990
    boolean useOldExecutionStepVersion = false;
    if (buildContext.getPlanInfo().isPresent()) {
        final Set<ExecutionStep<?>> sourceSteps = buildContext.getPlanInfo().get().getSources();
        useOldExecutionStepVersion = sourceSteps.stream().anyMatch(executionStep -> executionStep instanceof TableSourceV1);
    }
    if (useOldExecutionStepVersion && pseudoColumnVersionToUse != SystemColumns.LEGACY_PSEUDOCOLUMN_VERSION_NUMBER) {
        throw new IllegalStateException("TableSourceV2 was released in conjunction with pseudocolumn" + "version 1. Something has gone very wrong");
    }
    if (buildContext.getKsqlConfig().getBoolean(KsqlConfig.KSQL_ROWPARTITION_ROWOFFSET_ENABLED) && !useOldExecutionStepVersion) {
        step = ExecutionStepFactory.tableSource(contextStacker, dataSource.getSchema(), dataSource.getKafkaTopicName(), Formats.from(dataSource.getKsqlTopic()), dataSource.getTimestampColumn(), InternalFormats.of(keyFormat, Formats.from(dataSource.getKsqlTopic()).getValueFormat()), pseudoColumnVersionToUse);
    } else {
        step = ExecutionStepFactory.tableSourceV1(contextStacker, dataSource.getSchema(), dataSource.getKafkaTopicName(), Formats.from(dataSource.getKsqlTopic()), dataSource.getTimestampColumn(), pseudoColumnVersionToUse);
    }
    return schemaKTable(buildContext, resolveSchema(buildContext, step, dataSource), dataSource.getKsqlTopic().getKeyFormat(), step);
}
Also used : ExecutionStep(io.confluent.ksql.execution.plan.ExecutionStep) DataSource(io.confluent.ksql.metastore.model.DataSource) ExecutionStep(io.confluent.ksql.execution.plan.ExecutionStep) SystemColumns(io.confluent.ksql.schema.ksql.SystemColumns) PlanBuildContext(io.confluent.ksql.planner.plan.PlanBuildContext) KeyFormat(io.confluent.ksql.serde.KeyFormat) QueryContext(io.confluent.ksql.execution.context.QueryContext) Set(java.util.Set) KsqlConfig(io.confluent.ksql.util.KsqlConfig) Formats(io.confluent.ksql.execution.plan.Formats) InternalFormats(io.confluent.ksql.serde.InternalFormats) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) StreamSource(io.confluent.ksql.execution.plan.StreamSource) StepSchemaResolver(io.confluent.ksql.execution.streams.StepSchemaResolver) WindowInfo(io.confluent.ksql.serde.WindowInfo) KStreamHolder(io.confluent.ksql.execution.plan.KStreamHolder) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) SourceStep(io.confluent.ksql.execution.plan.SourceStep) WindowedStreamSource(io.confluent.ksql.execution.plan.WindowedStreamSource) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) Windowed(org.apache.kafka.streams.kstream.Windowed) Stacker(io.confluent.ksql.execution.context.QueryContext.Stacker) GenericKey(io.confluent.ksql.GenericKey) ExecutionStepFactory(io.confluent.ksql.execution.streams.ExecutionStepFactory) TableSourceV1(io.confluent.ksql.execution.plan.TableSourceV1) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) KeyFormat(io.confluent.ksql.serde.KeyFormat)

Aggregations

TableSourceV1 (io.confluent.ksql.execution.plan.TableSourceV1)4 LogicalSchema (io.confluent.ksql.schema.ksql.LogicalSchema)3 GenericKey (io.confluent.ksql.GenericKey)2 KStreamHolder (io.confluent.ksql.execution.plan.KStreamHolder)2 KTableHolder (io.confluent.ksql.execution.plan.KTableHolder)2 SourceStep (io.confluent.ksql.execution.plan.SourceStep)2 StreamSource (io.confluent.ksql.execution.plan.StreamSource)2 WindowedStreamSource (io.confluent.ksql.execution.plan.WindowedStreamSource)2 WindowInfo (io.confluent.ksql.serde.WindowInfo)2 GenericRow (io.confluent.ksql.GenericRow)1 QueryContext (io.confluent.ksql.execution.context.QueryContext)1 Stacker (io.confluent.ksql.execution.context.QueryContext.Stacker)1 MaterializationInfo (io.confluent.ksql.execution.materialization.MaterializationInfo)1 ExecutionKeyFactory (io.confluent.ksql.execution.plan.ExecutionKeyFactory)1 ExecutionStep (io.confluent.ksql.execution.plan.ExecutionStep)1 ExecutionStepPropertiesV1 (io.confluent.ksql.execution.plan.ExecutionStepPropertiesV1)1 Formats (io.confluent.ksql.execution.plan.Formats)1 PlanInfo (io.confluent.ksql.execution.plan.PlanInfo)1 WindowedTableSource (io.confluent.ksql.execution.plan.WindowedTableSource)1 RuntimeBuildContext (io.confluent.ksql.execution.runtime.RuntimeBuildContext)1