Search in sources :

Example 6 with Deserializer

use of org.apache.kafka.common.serialization.Deserializer in project druid by druid-io.

the class KafkaRecordSupplier method getKafkaDeserializer.

private static Deserializer getKafkaDeserializer(Properties properties, String kafkaConfigKey) {
    Deserializer deserializerObject;
    try {
        Class deserializerClass = Class.forName(properties.getProperty(kafkaConfigKey, ByteArrayDeserializer.class.getTypeName()));
        Method deserializerMethod = deserializerClass.getMethod("deserialize", String.class, byte[].class);
        Type deserializerReturnType = deserializerMethod.getGenericReturnType();
        if (deserializerReturnType == byte[].class) {
            deserializerObject = (Deserializer) deserializerClass.getConstructor().newInstance();
        } else {
            throw new IllegalArgumentException("Kafka deserializers must return a byte array (byte[]), " + deserializerClass.getName() + " returns " + deserializerReturnType.getTypeName());
        }
    } catch (ClassNotFoundException | NoSuchMethodException | InstantiationException | IllegalAccessException | InvocationTargetException e) {
        throw new StreamException(e);
    }
    return deserializerObject;
}
Also used : Type(java.lang.reflect.Type) Deserializer(org.apache.kafka.common.serialization.Deserializer) ByteArrayDeserializer(org.apache.kafka.common.serialization.ByteArrayDeserializer) Method(java.lang.reflect.Method) InvocationTargetException(java.lang.reflect.InvocationTargetException) StreamException(org.apache.druid.indexing.seekablestream.common.StreamException)

Example 7 with Deserializer

use of org.apache.kafka.common.serialization.Deserializer in project ksql by confluentinc.

the class RecordFormatter method newProtobufDeserializer.

private static Deserializer<?> newProtobufDeserializer(final SchemaRegistryClient srClient) {
    final Printer printer = TextFormat.printer();
    final KafkaProtobufDeserializer<?> inner = new KafkaProtobufDeserializer<>(srClient);
    return (topic, data) -> {
        final Message msg = inner.deserialize(topic, data);
        if (msg == null) {
            return null;
        }
        return printer.shortDebugString(msg);
    };
}
Also used : Arrays(java.util.Arrays) SchemaRegistryClient(io.confluent.kafka.schemaregistry.client.SchemaRegistryClient) Function(java.util.function.Function) Supplier(java.util.function.Supplier) ImmutableList(com.google.common.collect.ImmutableList) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) Windowed(org.apache.kafka.streams.kstream.Windowed) Objects.requireNonNull(java.util.Objects.requireNonNull) StreamSupport(java.util.stream.StreamSupport) KafkaJsonSchemaDeserializer(io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer) ZoneOffset(java.time.ZoneOffset) Deserializer(org.apache.kafka.common.serialization.Deserializer) TextFormat(com.google.protobuf.TextFormat) TimeWindowedDeserializer(org.apache.kafka.streams.kstream.TimeWindowedDeserializer) KafkaProtobufDeserializer(io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer) Iterator(java.util.Iterator) SessionWindowedDeserializer(org.apache.kafka.streams.kstream.SessionWindowedDeserializer) UTF_8(java.nio.charset.StandardCharsets.UTF_8) KsqlJsonDeserializer(io.confluent.ksql.serde.json.KsqlJsonDeserializer) Printer(com.google.protobuf.TextFormat.Printer) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) IOException(java.io.IOException) Streams(com.google.common.collect.Streams) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) DoubleDeserializer(org.apache.kafka.common.serialization.DoubleDeserializer) KafkaAvroDeserializer(io.confluent.kafka.serializers.KafkaAvroDeserializer) Bytes(org.apache.kafka.common.utils.Bytes) List(java.util.List) Stream(java.util.stream.Stream) ConsumerRecord(org.apache.kafka.clients.consumer.ConsumerRecord) DateTimeFormatter(java.time.format.DateTimeFormatter) Message(com.google.protobuf.Message) BytesDeserializer(org.apache.kafka.common.serialization.BytesDeserializer) Optional(java.util.Optional) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Message(com.google.protobuf.Message) KafkaProtobufDeserializer(io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer) Printer(com.google.protobuf.TextFormat.Printer)

Example 8 with Deserializer

use of org.apache.kafka.common.serialization.Deserializer in project ksql by confluentinc.

the class RecordFormatter method newStringDeserializer.

private static Deserializer<?> newStringDeserializer() {
    final StringDeserializer deserializer = new StringDeserializer();
    final String replacement = UTF_8.newDecoder().replacement();
    return (Deserializer<Object>) (topic, data) -> {
        if (data.length == 0) {
            throw new DeserializationException("Empty data");
        }
        final String text = deserializer.deserialize("", data);
        if (text.contains(replacement)) {
            throw new DeserializationException("String contains replacement char");
        }
        return text;
    };
}
Also used : StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) KafkaJsonSchemaDeserializer(io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer) Deserializer(org.apache.kafka.common.serialization.Deserializer) TimeWindowedDeserializer(org.apache.kafka.streams.kstream.TimeWindowedDeserializer) KafkaProtobufDeserializer(io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer) SessionWindowedDeserializer(org.apache.kafka.streams.kstream.SessionWindowedDeserializer) KsqlJsonDeserializer(io.confluent.ksql.serde.json.KsqlJsonDeserializer) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) DoubleDeserializer(org.apache.kafka.common.serialization.DoubleDeserializer) KafkaAvroDeserializer(io.confluent.kafka.serializers.KafkaAvroDeserializer) BytesDeserializer(org.apache.kafka.common.serialization.BytesDeserializer) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer)

Aggregations

Deserializer (org.apache.kafka.common.serialization.Deserializer)8 Properties (java.util.Properties)5 StringDeserializer (org.apache.kafka.common.serialization.StringDeserializer)5 LongDeserializer (org.apache.kafka.common.serialization.LongDeserializer)4 Utils.mkProperties (org.apache.kafka.common.utils.Utils.mkProperties)3 KafkaAvroDeserializer (io.confluent.kafka.serializers.KafkaAvroDeserializer)2 KafkaJsonSchemaDeserializer (io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer)2 KafkaProtobufDeserializer (io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer)2 KsqlJsonDeserializer (io.confluent.ksql.serde.json.KsqlJsonDeserializer)2 BytesDeserializer (org.apache.kafka.common.serialization.BytesDeserializer)2 DoubleDeserializer (org.apache.kafka.common.serialization.DoubleDeserializer)2 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)2 SessionWindowedDeserializer (org.apache.kafka.streams.kstream.SessionWindowedDeserializer)2 TimeWindowedDeserializer (org.apache.kafka.streams.kstream.TimeWindowedDeserializer)2 JsonNode (com.fasterxml.jackson.databind.JsonNode)1 JsonNodeFactory (com.fasterxml.jackson.databind.node.JsonNodeFactory)1 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 ImmutableList (com.google.common.collect.ImmutableList)1 Streams (com.google.common.collect.Streams)1