Search in sources :

Example 1 with TableSuppress

use of io.confluent.ksql.execution.plan.TableSuppress in project ksql by confluentinc.

the class TableSuppressBuilder method build.

@VisibleForTesting
@SuppressWarnings("unchecked")
<K> KTableHolder<K> build(final KTableHolder<K> table, final TableSuppress<K> step, final RuntimeBuildContext buildContext, final ExecutionKeyFactory<K> executionKeyFactory, final PhysicalSchemaFactory physicalSchemaFactory, final BiFunction<Serde<K>, Serde<GenericRow>, Materialized> materializedFactory) {
    final PhysicalSchema physicalSchema = physicalSchemaFactory.create(table.getSchema(), step.getInternalFormats().getKeyFeatures(), step.getInternalFormats().getValueFeatures());
    final QueryContext queryContext = QueryContext.Stacker.of(step.getProperties().getQueryContext()).push(SUPPRESS_OP_NAME).getQueryContext();
    final Serde<K> keySerde = executionKeyFactory.buildKeySerde(step.getInternalFormats().getKeyFormat(), physicalSchema, queryContext);
    final Serde<GenericRow> valueSerde = buildContext.buildValueSerde(step.getInternalFormats().getValueFormat(), physicalSchema, queryContext);
    final Materialized<K, GenericRow, KeyValueStore<Bytes, byte[]>> materialized = materializedFactory.apply(keySerde, valueSerde);
    final Suppressed.StrictBufferConfig strictBufferConfig;
    final long maxBytes = buildContext.getKsqlConfig().getLong(KsqlConfig.KSQL_SUPPRESS_BUFFER_SIZE_BYTES);
    if (maxBytes < 0) {
        strictBufferConfig = Suppressed.BufferConfig.unbounded();
    } else {
        strictBufferConfig = Suppressed.BufferConfig.maxBytes(maxBytes).shutDownWhenFull();
    }
    /* This is a dummy transformValues() call, we do this to ensure that the correct materialized
    with the correct key and val serdes is passed on when we call suppress
     */
    final KTable<K, GenericRow> suppressed = table.getTable().transformValues((() -> new KsTransformer<>((k, v, ctx) -> v)), materialized).suppress((Suppressed<? super K>) Suppressed.untilWindowCloses(strictBufferConfig).withName(SUPPRESS_OP_NAME));
    return table.withTable(suppressed, table.getSchema());
}
Also used : PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) KTable(org.apache.kafka.streams.kstream.KTable) TableSuppress(io.confluent.ksql.execution.plan.TableSuppress) RuntimeBuildContext(io.confluent.ksql.execution.runtime.RuntimeBuildContext) QueryContext(io.confluent.ksql.execution.context.QueryContext) BiFunction(java.util.function.BiFunction) Suppressed(org.apache.kafka.streams.kstream.Suppressed) KsqlConfig(io.confluent.ksql.util.KsqlConfig) ExecutionKeyFactory(io.confluent.ksql.execution.plan.ExecutionKeyFactory) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) Bytes(org.apache.kafka.common.utils.Bytes) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) KsTransformer(io.confluent.ksql.execution.streams.transform.KsTransformer) GenericRow(io.confluent.ksql.GenericRow) Serde(org.apache.kafka.common.serialization.Serde) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) Materialized(org.apache.kafka.streams.kstream.Materialized) VisibleForTesting(com.google.common.annotations.VisibleForTesting) SerdeFeatures(io.confluent.ksql.serde.SerdeFeatures) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) QueryContext(io.confluent.ksql.execution.context.QueryContext) GenericRow(io.confluent.ksql.GenericRow) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) Suppressed(org.apache.kafka.streams.kstream.Suppressed) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Aggregations

VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 GenericRow (io.confluent.ksql.GenericRow)1 QueryContext (io.confluent.ksql.execution.context.QueryContext)1 ExecutionKeyFactory (io.confluent.ksql.execution.plan.ExecutionKeyFactory)1 KTableHolder (io.confluent.ksql.execution.plan.KTableHolder)1 TableSuppress (io.confluent.ksql.execution.plan.TableSuppress)1 RuntimeBuildContext (io.confluent.ksql.execution.runtime.RuntimeBuildContext)1 KsTransformer (io.confluent.ksql.execution.streams.transform.KsTransformer)1 LogicalSchema (io.confluent.ksql.schema.ksql.LogicalSchema)1 PhysicalSchema (io.confluent.ksql.schema.ksql.PhysicalSchema)1 SerdeFeatures (io.confluent.ksql.serde.SerdeFeatures)1 KsqlConfig (io.confluent.ksql.util.KsqlConfig)1 BiFunction (java.util.function.BiFunction)1 Serde (org.apache.kafka.common.serialization.Serde)1 Bytes (org.apache.kafka.common.utils.Bytes)1 KTable (org.apache.kafka.streams.kstream.KTable)1 Materialized (org.apache.kafka.streams.kstream.Materialized)1 Suppressed (org.apache.kafka.streams.kstream.Suppressed)1 KeyValueStore (org.apache.kafka.streams.state.KeyValueStore)1