Search in sources :

Example 1 with PhysicalSchema

use of io.confluent.ksql.schema.ksql.PhysicalSchema in project ksql by confluentinc.

the class KafkaConsumerFactory method create.

public static KafkaConsumer<Object, GenericRow> create(final KsqlTopic ksqlTopic, final LogicalSchema logicalSchema, final ServiceContext serviceContext, final Map<String, Object> consumerProperties, final KsqlConfig ksqlConfig, final String consumerGroupId) {
    final PhysicalSchema physicalSchema = PhysicalSchema.from(logicalSchema, ksqlTopic.getKeyFormat().getFeatures(), ksqlTopic.getValueFormat().getFeatures());
    final KeySerdeFactory keySerdeFactory = new GenericKeySerDe();
    final Deserializer<Object> keyDeserializer;
    if (ksqlTopic.getKeyFormat().getWindowInfo().isPresent()) {
        final Serde<Windowed<GenericKey>> keySerde = keySerdeFactory.create(ksqlTopic.getKeyFormat().getFormatInfo(), ksqlTopic.getKeyFormat().getWindowInfo().get(), physicalSchema.keySchema(), ksqlConfig, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty());
        keyDeserializer = getDeserializer(keySerde.deserializer());
    } else {
        final Serde<GenericKey> keySerde = keySerdeFactory.create(ksqlTopic.getKeyFormat().getFormatInfo(), physicalSchema.keySchema(), ksqlConfig, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty());
        keyDeserializer = getDeserializer(keySerde.deserializer());
    }
    final ValueSerdeFactory valueSerdeFactory = new GenericRowSerDe();
    final Serde<GenericRow> valueSerde = valueSerdeFactory.create(ksqlTopic.getValueFormat().getFormatInfo(), physicalSchema.valueSchema(), ksqlConfig, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty());
    return new KafkaConsumer<>(consumerConfig(consumerProperties, ksqlConfig, consumerGroupId), keyDeserializer, valueSerde.deserializer());
}
Also used : ValueSerdeFactory(io.confluent.ksql.serde.ValueSerdeFactory) KafkaConsumer(org.apache.kafka.clients.consumer.KafkaConsumer) GenericKeySerDe(io.confluent.ksql.serde.GenericKeySerDe) GenericRowSerDe(io.confluent.ksql.serde.GenericRowSerDe) Windowed(org.apache.kafka.streams.kstream.Windowed) GenericRow(io.confluent.ksql.GenericRow) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) GenericKey(io.confluent.ksql.GenericKey) KeySerdeFactory(io.confluent.ksql.serde.KeySerdeFactory)

Example 2 with PhysicalSchema

use of io.confluent.ksql.schema.ksql.PhysicalSchema in project ksql by confluentinc.

the class InsertValuesExecutor method serializeValue.

private byte[] serializeValue(final GenericRow row, final DataSource dataSource, final KsqlConfig config, final ServiceContext serviceContext) {
    final PhysicalSchema physicalSchema = PhysicalSchema.from(dataSource.getSchema(), dataSource.getKsqlTopic().getKeyFormat().getFeatures(), dataSource.getKsqlTopic().getValueFormat().getFeatures());
    final Serde<GenericRow> valueSerde = valueSerdeFactory.create(dataSource.getKsqlTopic().getValueFormat().getFormatInfo(), physicalSchema.valueSchema(), config, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty());
    final String topicName = dataSource.getKafkaTopicName();
    try {
        return valueSerde.serializer().serialize(topicName, row);
    } catch (final Exception e) {
        maybeThrowSchemaRegistryAuthError(FormatFactory.fromName(dataSource.getKsqlTopic().getValueFormat().getFormat()), topicName, false, AclOperation.WRITE, e);
        LOG.error("Could not serialize value.", e);
        throw new KsqlException("Could not serialize value: " + row + ". " + e.getMessage(), e);
    }
}
Also used : GenericRow(io.confluent.ksql.GenericRow) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) KsqlException(io.confluent.ksql.util.KsqlException) KsqlTopicAuthorizationException(io.confluent.ksql.exception.KsqlTopicAuthorizationException) RestClientException(io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException) KsqlException(io.confluent.ksql.util.KsqlException) KsqlStatementException(io.confluent.ksql.util.KsqlStatementException) ExecutionException(java.util.concurrent.ExecutionException) KsqlSchemaAuthorizationException(io.confluent.ksql.exception.KsqlSchemaAuthorizationException) TopicAuthorizationException(org.apache.kafka.common.errors.TopicAuthorizationException)

Example 3 with PhysicalSchema

use of io.confluent.ksql.schema.ksql.PhysicalSchema in project ksql by confluentinc.

the class InsertValuesExecutor method serializeKey.

private byte[] serializeKey(final GenericKey keyValue, final DataSource dataSource, final KsqlConfig config, final ServiceContext serviceContext) {
    final PhysicalSchema physicalSchema = PhysicalSchema.from(dataSource.getSchema(), dataSource.getKsqlTopic().getKeyFormat().getFeatures(), dataSource.getKsqlTopic().getValueFormat().getFeatures());
    ensureKeySchemasMatch(physicalSchema.keySchema(), dataSource, serviceContext);
    final Serde<GenericKey> keySerde = keySerdeFactory.create(dataSource.getKsqlTopic().getKeyFormat().getFormatInfo(), physicalSchema.keySchema(), config, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty());
    final String topicName = dataSource.getKafkaTopicName();
    try {
        return keySerde.serializer().serialize(topicName, keyValue);
    } catch (final Exception e) {
        maybeThrowSchemaRegistryAuthError(FormatFactory.fromName(dataSource.getKsqlTopic().getKeyFormat().getFormat()), topicName, true, AclOperation.WRITE, e);
        LOG.error("Could not serialize key.", e);
        throw new KsqlException("Could not serialize key: " + keyValue, e);
    }
}
Also used : PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) GenericKey(io.confluent.ksql.GenericKey) KsqlException(io.confluent.ksql.util.KsqlException) KsqlTopicAuthorizationException(io.confluent.ksql.exception.KsqlTopicAuthorizationException) RestClientException(io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException) KsqlException(io.confluent.ksql.util.KsqlException) KsqlStatementException(io.confluent.ksql.util.KsqlStatementException) ExecutionException(java.util.concurrent.ExecutionException) KsqlSchemaAuthorizationException(io.confluent.ksql.exception.KsqlSchemaAuthorizationException) TopicAuthorizationException(org.apache.kafka.common.errors.TopicAuthorizationException)

Example 4 with PhysicalSchema

use of io.confluent.ksql.schema.ksql.PhysicalSchema in project ksql by confluentinc.

the class CreateSourceFactory method validateSerdesCanHandleSchemas.

private void validateSerdesCanHandleSchemas(final KsqlConfig ksqlConfig, final LogicalSchema schema, final Formats formats) {
    final PhysicalSchema physicalSchema = PhysicalSchema.from(schema, formats.getKeyFeatures(), formats.getValueFeatures());
    keySerdeFactory.create(formats.getKeyFormat(), physicalSchema.keySchema(), ksqlConfig, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty()).close();
    valueSerdeFactory.create(formats.getValueFormat(), physicalSchema.valueSchema(), ksqlConfig, serviceContext.getSchemaRegistryClientFactory(), "", NoopProcessingLogContext.INSTANCE, Optional.empty()).close();
}
Also used : PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema)

Example 5 with PhysicalSchema

use of io.confluent.ksql.schema.ksql.PhysicalSchema in project ksql by confluentinc.

the class TableGroupByBuilderBase method build.

public <K> KGroupedTableHolder build(final KTableHolder<K> table, final QueryContext queryContext, final Formats formats, final List<Expression> groupByExpressions) {
    final LogicalSchema sourceSchema = table.getSchema();
    final List<CompiledExpression> groupBy = CodeGenRunner.compileExpressions(groupByExpressions.stream(), "Group By", sourceSchema, buildContext.getKsqlConfig(), buildContext.getFunctionRegistry());
    final ProcessingLogger logger = buildContext.getProcessingLogger(queryContext);
    final GroupByParams params = paramsFactory.build(sourceSchema, groupBy, logger);
    final PhysicalSchema physicalSchema = PhysicalSchema.from(params.getSchema(), formats.getKeyFeatures(), formats.getValueFeatures());
    final Serde<GenericKey> keySerde = buildContext.buildKeySerde(formats.getKeyFormat(), physicalSchema, queryContext);
    final Serde<GenericRow> valSerde = buildContext.buildValueSerde(formats.getValueFormat(), physicalSchema, queryContext);
    final Grouped<GenericKey, GenericRow> grouped = groupedFactory.create(StreamsUtil.buildOpName(queryContext), keySerde, valSerde);
    final KGroupedTable<GenericKey, GenericRow> groupedTable = table.getTable().filter((k, v) -> v != null).groupBy(new TableKeyValueMapper<>(params.getMapper()), grouped);
    return KGroupedTableHolder.of(groupedTable, params.getSchema());
}
Also used : PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) KeyValueMapper(org.apache.kafka.streams.kstream.KeyValueMapper) RuntimeBuildContext(io.confluent.ksql.execution.runtime.RuntimeBuildContext) Expression(io.confluent.ksql.execution.expression.tree.Expression) QueryContext(io.confluent.ksql.execution.context.QueryContext) KeyValue(org.apache.kafka.streams.KeyValue) CompiledExpression(io.confluent.ksql.execution.codegen.CompiledExpression) Formats(io.confluent.ksql.execution.plan.Formats) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) Function(java.util.function.Function) CodeGenRunner(io.confluent.ksql.execution.codegen.CodeGenRunner) Grouped(org.apache.kafka.streams.kstream.Grouped) KGroupedTableHolder(io.confluent.ksql.execution.plan.KGroupedTableHolder) KTableHolder(io.confluent.ksql.execution.plan.KTableHolder) List(java.util.List) KGroupedTable(org.apache.kafka.streams.kstream.KGroupedTable) GenericRow(io.confluent.ksql.GenericRow) Serde(org.apache.kafka.common.serialization.Serde) Objects.requireNonNull(java.util.Objects.requireNonNull) ProcessingLogger(io.confluent.ksql.logging.processing.ProcessingLogger) GenericKey(io.confluent.ksql.GenericKey) ProcessingLogger(io.confluent.ksql.logging.processing.ProcessingLogger) LogicalSchema(io.confluent.ksql.schema.ksql.LogicalSchema) CompiledExpression(io.confluent.ksql.execution.codegen.CompiledExpression) GenericRow(io.confluent.ksql.GenericRow) PhysicalSchema(io.confluent.ksql.schema.ksql.PhysicalSchema) GenericKey(io.confluent.ksql.GenericKey)

Aggregations

PhysicalSchema (io.confluent.ksql.schema.ksql.PhysicalSchema)34 GenericRow (io.confluent.ksql.GenericRow)21 GenericKey (io.confluent.ksql.GenericKey)12 LogicalSchema (io.confluent.ksql.schema.ksql.LogicalSchema)11 QueryContext (io.confluent.ksql.execution.context.QueryContext)9 Test (org.junit.Test)7 Formats (io.confluent.ksql.execution.plan.Formats)6 RuntimeBuildContext (io.confluent.ksql.execution.runtime.RuntimeBuildContext)6 DataSource (io.confluent.ksql.metastore.model.DataSource)6 IntegrationTest (io.confluent.common.utils.IntegrationTest)5 ProcessingLogger (io.confluent.ksql.logging.processing.ProcessingLogger)4 Serde (org.apache.kafka.common.serialization.Serde)4 Windowed (org.apache.kafka.streams.kstream.Windowed)4 KeyValueStore (org.apache.kafka.streams.state.KeyValueStore)4 KTableHolder (io.confluent.ksql.execution.plan.KTableHolder)3 SourceBuilderUtils.getPhysicalSchema (io.confluent.ksql.execution.streams.SourceBuilderUtils.getPhysicalSchema)3 RestClientException (io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException)2 KsqlSchemaAuthorizationException (io.confluent.ksql.exception.KsqlSchemaAuthorizationException)2 KsqlTopicAuthorizationException (io.confluent.ksql.exception.KsqlTopicAuthorizationException)2 CodeGenRunner (io.confluent.ksql.execution.codegen.CodeGenRunner)2