Search in sources :

Example 26 with LongDeserializer

use of org.apache.kafka.common.serialization.LongDeserializer in project spring-cloud-stream by spring-cloud.

the class KafkaBinderTests method testConsumerCustomDeserializer.

@Test
@SuppressWarnings("unchecked")
void testConsumerCustomDeserializer() throws Exception {
    Binding<?> binding = null;
    try {
        KafkaBinderConfigurationProperties configurationProperties = createConfigurationProperties();
        Map<String, String> propertiesToOverride = configurationProperties.getConfiguration();
        propertiesToOverride.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        propertiesToOverride.put("value.deserializer", "org.apache.kafka.common.serialization.LongDeserializer");
        configurationProperties.setConfiguration(propertiesToOverride);
        String testTopicName = "existing" + System.currentTimeMillis();
        configurationProperties.setAutoCreateTopics(false);
        Binder binder = getBinder(configurationProperties);
        ExtendedConsumerProperties<KafkaConsumerProperties> consumerProperties = createConsumerProperties();
        DirectChannel input = createBindableChannel("input", createConsumerBindingProperties(consumerProperties));
        binding = binder.bindConsumer(testTopicName, "test", input, consumerProperties);
        DirectFieldAccessor consumerAccessor = new DirectFieldAccessor(getKafkaConsumer(binding));
        assertThat(consumerAccessor.getPropertyValue("keyDeserializer") instanceof StringDeserializer).isTrue();
        assertThat(consumerAccessor.getPropertyValue("valueDeserializer") instanceof LongDeserializer).isTrue();
    } finally {
        if (binding != null) {
            binding.unbind();
        }
    }
}
Also used : Binder(org.springframework.cloud.stream.binder.Binder) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) DirectChannel(org.springframework.integration.channel.DirectChannel) KafkaBinderConfigurationProperties(org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) DirectFieldAccessor(org.springframework.beans.DirectFieldAccessor) KafkaConsumerProperties(org.springframework.cloud.stream.binder.kafka.properties.KafkaConsumerProperties) Test(org.junit.jupiter.api.Test)

Example 27 with LongDeserializer

use of org.apache.kafka.common.serialization.LongDeserializer in project starlight-for-kafka by datastax.

the class KStreamAggregationTest method shouldCountHelper.

private void shouldCountHelper() throws Exception {
    startStreams();
    produceMessages(mockTime.milliseconds());
    final List<KeyValue<String, Long>> results = receiveMessages(new StringDeserializer(), new LongDeserializer(), 10);
    results.sort(KStreamAggregationTest::compare);
    assertThat(results, is(Arrays.asList(KeyValue.pair("A", 1L), KeyValue.pair("A", 2L), KeyValue.pair("B", 1L), KeyValue.pair("B", 2L), KeyValue.pair("C", 1L), KeyValue.pair("C", 2L), KeyValue.pair("D", 1L), KeyValue.pair("D", 2L), KeyValue.pair("E", 1L), KeyValue.pair("E", 2L))));
}
Also used : KeyValue(org.apache.kafka.streams.KeyValue) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer)

Example 28 with LongDeserializer

use of org.apache.kafka.common.serialization.LongDeserializer in project kop by streamnative.

the class KStreamAggregationTest method shouldCountHelper.

private void shouldCountHelper() throws Exception {
    startStreams();
    produceMessages(mockTime.milliseconds());
    final List<KeyValue<String, Long>> results = receiveMessages(new StringDeserializer(), new LongDeserializer(), 10);
    results.sort(KStreamAggregationTest::compare);
    assertThat(results, is(Arrays.asList(KeyValue.pair("A", 1L), KeyValue.pair("A", 2L), KeyValue.pair("B", 1L), KeyValue.pair("B", 2L), KeyValue.pair("C", 1L), KeyValue.pair("C", 2L), KeyValue.pair("D", 1L), KeyValue.pair("D", 2L), KeyValue.pair("E", 1L), KeyValue.pair("E", 2L))));
}
Also used : KeyValue(org.apache.kafka.streams.KeyValue) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer)

Example 29 with LongDeserializer

use of org.apache.kafka.common.serialization.LongDeserializer in project kop by streamnative.

the class KStreamAggregationTest method shouldGroupByKey.

@Test
public void shouldGroupByKey() throws Exception {
    final long timestamp = mockTime.milliseconds();
    produceMessages(timestamp);
    produceMessages(timestamp);
    stream.groupByKey(Serialized.with(Serdes.Integer(), Serdes.String())).windowedBy(TimeWindows.of(500L)).count().toStream((windowedKey, value) -> windowedKey.key() + "@" + windowedKey.window().start()).to(outputTopic, Produced.with(Serdes.String(), Serdes.Long()));
    startStreams();
    final List<KeyValue<String, Long>> results = receiveMessages(new StringDeserializer(), new LongDeserializer(), 10);
    results.sort(KStreamAggregationTest::compare);
    final long window = timestamp / 500 * 500;
    assertThat(results, is(Arrays.asList(KeyValue.pair("1@" + window, 1L), KeyValue.pair("1@" + window, 2L), KeyValue.pair("2@" + window, 1L), KeyValue.pair("2@" + window, 2L), KeyValue.pair("3@" + window, 1L), KeyValue.pair("3@" + window, 2L), KeyValue.pair("4@" + window, 1L), KeyValue.pair("4@" + window, 2L), KeyValue.pair("5@" + window, 1L), KeyValue.pair("5@" + window, 2L))));
}
Also used : Arrays(java.util.Arrays) Produced(org.apache.kafka.streams.kstream.Produced) Serialized(org.apache.kafka.streams.kstream.Serialized) Test(org.testng.annotations.Test) WindowedSerdes(org.apache.kafka.streams.kstream.WindowedSerdes) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) Serde(org.apache.kafka.common.serialization.Serde) Map(java.util.Map) KeyValueStore(org.apache.kafka.streams.state.KeyValueStore) Is.is(org.hamcrest.core.Is.is) Serdes(org.apache.kafka.common.serialization.Serdes) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) Aggregator(org.apache.kafka.streams.kstream.Aggregator) TimeWindowedDeserializer(org.apache.kafka.streams.kstream.TimeWindowedDeserializer) SessionWindowedDeserializer(org.apache.kafka.streams.kstream.SessionWindowedDeserializer) NonNull(lombok.NonNull) KeyValue(org.apache.kafka.streams.KeyValue) Set(java.util.Set) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) ConsumerConfig(org.apache.kafka.clients.consumer.ConsumerConfig) Bytes(org.apache.kafka.common.utils.Bytes) QueryableStoreTypes(org.apache.kafka.streams.state.QueryableStoreTypes) CountDownLatch(java.util.concurrent.CountDownLatch) ProcessorContext(org.apache.kafka.streams.processor.ProcessorContext) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) Materialized(org.apache.kafka.streams.kstream.Materialized) SessionWindow(org.apache.kafka.streams.kstream.internals.SessionWindow) TimeWindow(org.apache.kafka.streams.kstream.internals.TimeWindow) StreamsConfig(org.apache.kafka.streams.StreamsConfig) ByteArrayOutputStream(java.io.ByteArrayOutputStream) KGroupedStream(org.apache.kafka.streams.kstream.KGroupedStream) ConsoleConsumer(kafka.tools.ConsoleConsumer) SessionWindows(org.apache.kafka.streams.kstream.SessionWindows) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) HashMap(java.util.HashMap) KStream(org.apache.kafka.streams.kstream.KStream) HashSet(java.util.HashSet) Initializer(org.apache.kafka.streams.kstream.Initializer) Windowed(org.apache.kafka.streams.kstream.Windowed) IntegerSerializer(org.apache.kafka.common.serialization.IntegerSerializer) MatcherAssert.assertThat(org.hamcrest.MatcherAssert.assertThat) Deserializer(org.apache.kafka.common.serialization.Deserializer) PrintStream(java.io.PrintStream) KeyValueMapper(org.apache.kafka.streams.kstream.KeyValueMapper) Properties(java.util.Properties) Consumed(org.apache.kafka.streams.kstream.Consumed) Transformer(org.apache.kafka.streams.kstream.Transformer) Assert.assertTrue(org.junit.Assert.assertTrue) TimeUnit(java.util.concurrent.TimeUnit) KeyValueIterator(org.apache.kafka.streams.state.KeyValueIterator) TimeWindows(org.apache.kafka.streams.kstream.TimeWindows) Reducer(org.apache.kafka.streams.kstream.Reducer) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) Comparator(java.util.Comparator) Collections(java.util.Collections) ReadOnlySessionStore(org.apache.kafka.streams.state.ReadOnlySessionStore) KeyValue(org.apache.kafka.streams.KeyValue) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) Test(org.testng.annotations.Test)

Example 30 with LongDeserializer

use of org.apache.kafka.common.serialization.LongDeserializer in project kafka by apache.

the class WordCountDemoTest method setup.

@BeforeEach
public void setup() throws IOException {
    final StreamsBuilder builder = new StreamsBuilder();
    // Create Actual Stream Processing pipeline
    WordCountDemo.createWordCountStream(builder);
    testDriver = new TopologyTestDriver(builder.build(), WordCountDemo.getStreamsConfig(null));
    inputTopic = testDriver.createInputTopic(WordCountDemo.INPUT_TOPIC, new StringSerializer(), new StringSerializer());
    outputTopic = testDriver.createOutputTopic(WordCountDemo.OUTPUT_TOPIC, new StringDeserializer(), new LongDeserializer());
}
Also used : StreamsBuilder(org.apache.kafka.streams.StreamsBuilder) LongDeserializer(org.apache.kafka.common.serialization.LongDeserializer) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) TopologyTestDriver(org.apache.kafka.streams.TopologyTestDriver) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) BeforeEach(org.junit.jupiter.api.BeforeEach)

Aggregations

LongDeserializer (org.apache.kafka.common.serialization.LongDeserializer)38 StringDeserializer (org.apache.kafka.common.serialization.StringDeserializer)34 KeyValue (org.apache.kafka.streams.KeyValue)22 Test (org.junit.Test)22 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)20 List (java.util.List)19 Properties (java.util.Properties)18 StreamsBuilder (org.apache.kafka.streams.StreamsBuilder)16 HashMap (java.util.HashMap)14 Arrays (java.util.Arrays)13 Map (java.util.Map)13 Serdes (org.apache.kafka.common.serialization.Serdes)13 StreamsConfig (org.apache.kafka.streams.StreamsConfig)13 TopologyTestDriver (org.apache.kafka.streams.TopologyTestDriver)12 KStream (org.apache.kafka.streams.kstream.KStream)12 ArrayList (java.util.ArrayList)11 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)11 Consumed (org.apache.kafka.streams.kstream.Consumed)10 Produced (org.apache.kafka.streams.kstream.Produced)10 Deserializer (org.apache.kafka.common.serialization.Deserializer)9