Search in sources :

Example 41 with GenericKey

use of io.confluent.ksql.GenericKey in project ksql by confluentinc.

the class TableGroupByBuilderBase method build.

public <K> KGroupedTableHolder build(final KTableHolder<K> table, final QueryContext queryContext, final Formats formats, final List<Expression> groupByExpressions) {
    final LogicalSchema sourceSchema = table.getSchema();
    final List<CompiledExpression> groupBy = CodeGenRunner.compileExpressions(groupByExpressions.stream(), "Group By", sourceSchema, buildContext.getKsqlConfig(), buildContext.getFunctionRegistry());
    final ProcessingLogger logger = buildContext.getProcessingLogger(queryContext);
    final GroupByParams params = paramsFactory.build(sourceSchema, groupBy, logger);
    final PhysicalSchema physicalSchema = PhysicalSchema.from(params.getSchema(), formats.getKeyFeatures(), formats.getValueFeatures());
    final Serde<GenericKey> keySerde = buildContext.buildKeySerde(formats.getKeyFormat(), physicalSchema, queryContext);
    final Serde<GenericRow> valSerde = buildContext.buildValueSerde(formats.getValueFormat(), physicalSchema, queryContext);
    final Grouped<GenericKey, GenericRow> grouped = groupedFactory.create(StreamsUtil.buildOpName(queryContext), keySerde, valSerde);
    final KGroupedTable<GenericKey, GenericRow> groupedTable = table.getTable().filter((k, v) -> v != null).groupBy(new TableKeyValueMapper<>(params.getMapper()), grouped);
    return KGroupedTableHolder.of(groupedTable, params.getSchema());
}
Also used : PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) KeyValueMapper(org.apache.kafka.streams.kstream.KeyValueMapper) RuntimeBuildContext(io.confluent.ksql.execution.runtime.RuntimeBuildContext) Expression(io.confluent.ksql.execution.expression.tree.Expression) QueryContext(io.confluent.ksql.execution.context.QueryContext) KeyValue(org.apache.kafka.streams.KeyValue) CompiledExpression(io.confluent.ksql.execution.codegen.CompiledExpression) Formats(io.confluent.ksql.execution.plan.Formats) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) Function(java.util.function.Function) CodeGenRunner(io.confluent.ksql.execution.codegen.CodeGenRunner) Grouped(org.apache.kafka.streams.kstream.Grouped) KGroupedTableHolder(io.confluent.ksql.execution.plan.KGroupedTableHolder) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) List(java.util.List) KGroupedTable(org.apache.kafka.streams.kstream.KGroupedTable) GenericRow(io.confluent.ksql.GenericRow) Serde(org.apache.kafka.common.serialization.Serde) Objects.requireNonNull(java.util.Objects.requireNonNull) ProcessingLogger(io.confluent.ksql.logging.processing.ProcessingLogger) GenericKey(io.confluent.ksql.GenericKey) ProcessingLogger(io.confluent.ksql.logging.processing.ProcessingLogger) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) CompiledExpression(io.confluent.ksql.execution.codegen.CompiledExpression) GenericRow(io.confluent.ksql.GenericRow) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) GenericKey(io.confluent.ksql.GenericKey)

Example 42 with GenericKey

use of io.confluent.ksql.GenericKey in project ksql by confluentinc.

the class WindowStoreCacheBypass method fetchRangeUncached.

/*
  This method is used for range queries. It is invoked by the fetchRange method
   */
private static KeyValueIterator<Windowed<GenericKey>, ValueAndTimestamp<GenericRow>> fetchRangeUncached(final ReadOnlyWindowStore<GenericKey, ValueAndTimestamp<GenericRow>> windowStore, final GenericKey keyFrom, final GenericKey keyTo, final Instant lower, final Instant upper) {
    if (!(windowStore instanceof MeteredWindowStore)) {
        throw new IllegalStateException("Expecting a MeteredWindowStore");
    }
    final StateSerdes<GenericKey, ValueAndTimestamp<GenericRow>> serdes = getSerdes(windowStore);
    final WindowStore<Bytes, byte[]> wrapped = getInnermostStore(windowStore);
    final Bytes rawKeyFrom = Bytes.wrap(serdes.rawKey(keyFrom));
    final Bytes rawKeyTo = Bytes.wrap(serdes.rawKey(keyTo));
    final KeyValueIterator<Windowed<Bytes>, byte[]> fetch = wrapped.fetch(rawKeyFrom, rawKeyTo, lower, upper);
    return new DeserializingKeyValueIterator(fetch, serdes);
}
Also used : ValueAndTimestamp(org.apache.kafka.streams.state.ValueAndTimestamp) Windowed(org.apache.kafka.streams.kstream.Windowed) Bytes(org.apache.kafka.common.utils.Bytes) MeteredWindowStore(org.apache.kafka.streams.state.internals.MeteredWindowStore) GenericKey(io.confluent.ksql.GenericKey)

Example 43 with GenericKey

use of io.confluent.ksql.GenericKey in project ksql by confluentinc.

the class GroupByParamsFactoryTest method shouldGenerateExpressionGroupByKeyForMultipleExpressions.

@Test
public void shouldGenerateExpressionGroupByKeyForMultipleExpressions() {
    // Given:
    when(groupBy0.evaluate(any(), any(), any(), any())).thenReturn(99);
    when(groupBy1.evaluate(any(), any(), any(), any())).thenReturn(-100L);
    // When:
    final GenericKey result = multiParams.getMapper().apply(value);
    // Then:
    assertThat(result, is(genericKey(99, -100L)));
}
Also used : GenericKey(io.confluent.ksql.GenericKey) Test(org.junit.Test)

Example 44 with GenericKey

use of io.confluent.ksql.GenericKey in project ksql by confluentinc.

the class GroupByParamsFactoryTest method shouldGenerateGroupByKeyForSingleExpression.

@Test
public void shouldGenerateGroupByKeyForSingleExpression() {
    // Given:
    when(groupBy0.evaluate(any(), any(), any(), any())).thenReturn(10);
    // When:
    final GenericKey result = singleParams.getMapper().apply(value);
    // Then:
    assertThat(result, is(genericKey(10)));
}
Also used : GenericKey(io.confluent.ksql.GenericKey) Test(org.junit.Test)

Example 45 with GenericKey

use of io.confluent.ksql.GenericKey in project ksql by confluentinc.

the class GroupByParamsFactoryTest method shouldReturnNullIfAnyMultiExpressionResolvesToNull.

@Test
public void shouldReturnNullIfAnyMultiExpressionResolvesToNull() {
    // Given:
    when(groupBy0.evaluate(any(), any(), any(), any())).thenReturn(null);
    // When:
    final GenericKey result = multiParams.getMapper().apply(value);
    // Then:
    assertThat(result, is(nullValue()));
}
Also used : GenericKey(io.confluent.ksql.GenericKey) Test(org.junit.Test)

Aggregations

GenericKey (io.confluent.ksql.GenericKey)147 GenericRow (io.confluent.ksql.GenericRow)100 Test (org.junit.Test)93 LogicalSchema (io.confluent.ksql.schema.ksql.LogicalSchema)24 Windowed (org.apache.kafka.streams.kstream.Windowed)20 WindowedRow (io.confluent.ksql.execution.streams.materialization.WindowedRow)14 PhysicalSchema (io.confluent.ksql.schema.ksql.PhysicalSchema)14 Materialized (org.apache.kafka.streams.kstream.Materialized)13 ValueAndTimestamp (org.apache.kafka.streams.state.ValueAndTimestamp)13 UnqualifiedColumnReferenceExp (io.confluent.ksql.execution.expression.tree.UnqualifiedColumnReferenceExp)12 MaterializationException (io.confluent.ksql.execution.streams.materialization.MaterializationException)9 IntegrationTest (io.confluent.common.utils.IntegrationTest)8 Materialization (io.confluent.ksql.execution.streams.materialization.Materialization)8 Row (io.confluent.ksql.execution.streams.materialization.Row)8 PersistentQueryMetadata (io.confluent.ksql.util.PersistentQueryMetadata)8 TimeWindow (org.apache.kafka.streams.kstream.internals.TimeWindow)8 IntegrationTest (org.apache.kafka.test.IntegrationTest)8 InOrder (org.mockito.InOrder)8 MaterializedTable (io.confluent.ksql.execution.streams.materialization.MaterializedTable)7 Objects (java.util.Objects)7