Search in sources :

Example 61 with KafkaMapBasedConfig

use of io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class PerformanceProducerTest method testWithoutBackPressureAndNoWait.

@Test
public void testWithoutBackPressureAndNoWait() {
    String topic = UUID.randomUUID().toString();
    companion.topics().createAndWait(topic, 10);
    ConsumerTask<String, Integer> consumed = companion.consumeIntegers().fromTopics(topic, COUNT, Duration.ofMinutes(TIMEOUT_IN_MINUTES));
    KafkaMapBasedConfig config = kafkaConfig("mp.messaging.outgoing.kafka").put("topic", topic).put("max-inflight-messages", 0L).put("waitForWriteCompletion", false).put("value.serializer", IntegerSerializer.class.getName());
    GeneratorBean bean = runApplication(config, GeneratorBean.class);
    await().until(this::isReady);
    await().until(this::isAlive);
    long begin = System.currentTimeMillis();
    bean.run();
    await().atMost(Duration.ofMinutes(TIMEOUT_IN_MINUTES)).until(() -> bean.count() == COUNT);
    long end = System.currentTimeMillis();
    // Wait until all the messages are read.
    consumed.awaitCompletion(Duration.ofMinutes(TIMEOUT_IN_MINUTES));
    long duration = end - begin;
    System.out.println("Time " + duration + " ms");
    double speed = (COUNT * 1.0) / (duration / 1000.0);
    System.out.println(speed + " messages/ms");
}
Also used : IntegerSerializer(org.apache.kafka.common.serialization.IntegerSerializer) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 62 with KafkaMapBasedConfig

use of io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ConsumerRecordConverterTest method testBeanUsingConverterWithNullKeyAndValue.

@Test
public void testBeanUsingConverterWithNullKeyAndValue() {
    KafkaMapBasedConfig builder = kafkaConfig("mp.messaging.incoming.data");
    builder.put("value.deserializer", StringDeserializer.class.getName());
    builder.put("auto.offset.reset", "earliest");
    builder.put("topic", topic);
    addBeans(ConsumerRecordConverter.class, RecordConverter.class);
    RecordConverterTest.MyBean bean = runApplication(builder, RecordConverterTest.MyBean.class);
    companion.produceStrings().usingGenerator(i -> new ProducerRecord<>(topic, null, null), 10);
    await().until(() -> bean.list().size() == 10);
    assertThat(bean.list()).hasSize(10).allSatisfy(r -> {
        assertThat(r.value()).isNull();
        assertThat(r.value()).isNull();
    });
}
Also used : StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 63 with KafkaMapBasedConfig

use of io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ConsumerRecordsConverterTest method testBeanUsingConverterWithNullKeyAndValue.

@Test
public void testBeanUsingConverterWithNullKeyAndValue() {
    KafkaMapBasedConfig builder = kafkaConfig("mp.messaging.incoming.data");
    builder.put("value.deserializer", StringDeserializer.class.getName());
    builder.put("auto.offset.reset", "earliest");
    builder.put("topic", topic);
    builder.put("batch", true);
    addBeans(ConsumerRecordsConverter.class);
    MyBean bean = runApplication(builder, MyBean.class);
    companion.produceStrings().usingGenerator(i -> new ProducerRecord<>(topic, null, null), 10);
    await().until(() -> bean.list().stream().mapToInt(ConsumerRecords::count).sum() == 10);
    List<ConsumerRecord<String, String>> consumerRecords = bean.list().stream().flatMap(r -> StreamSupport.stream(r.records(topic).spliterator(), false)).collect(Collectors.toList());
    assertThat(consumerRecords).hasSize(10).allSatisfy(r -> {
        assertThat(r.value()).isNull();
        assertThat(r.value()).isNull();
    });
}
Also used : Awaitility.await(org.awaitility.Awaitility.await) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaCompanionTestBase(io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) ConsumerRecords(org.apache.kafka.clients.consumer.ConsumerRecords) Collectors(java.util.stream.Collectors) Test(org.junit.jupiter.api.Test) List(java.util.List) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) ConsumerRecord(org.apache.kafka.clients.consumer.ConsumerRecord) StreamSupport(java.util.stream.StreamSupport) ApplicationScoped(javax.enterprise.context.ApplicationScoped) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) Incoming(org.eclipse.microprofile.reactive.messaging.Incoming) StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) ConsumerRecord(org.apache.kafka.clients.consumer.ConsumerRecord) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 64 with KafkaMapBasedConfig

use of io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class RecordConverterTest method testBeanUsingConverter.

@Test
public void testBeanUsingConverter() {
    KafkaMapBasedConfig builder = kafkaConfig("mp.messaging.incoming.data");
    builder.put("value.deserializer", StringDeserializer.class.getName());
    builder.put("auto.offset.reset", "earliest");
    builder.put("topic", topic);
    addBeans(ConsumerRecordConverter.class, RecordConverter.class);
    MyBean bean = runApplication(builder, MyBean.class);
    companion.produceStrings().usingGenerator(i -> new ProducerRecord<>(topic, i % 2 == 0 ? "key" : "k", "v-" + i), 10);
    await().until(() -> bean.list().size() == 10);
    assertThat(bean.list()).hasSize(10).allSatisfy(r -> {
        assertThat(r.value()).startsWith("v-");
        assertThat(r.key()).startsWith("k");
        if (!r.key().equalsIgnoreCase("key")) {
            assertThat(r.key()).isEqualTo("k");
        }
    });
}
Also used : StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 65 with KafkaMapBasedConfig

use of io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class RecordConverterTest method testBeanUsingConverterWithNullKeyAndValue.

@Test
public void testBeanUsingConverterWithNullKeyAndValue() {
    KafkaMapBasedConfig builder = kafkaConfig("mp.messaging.incoming.data");
    builder.put("value.deserializer", StringDeserializer.class.getName());
    builder.put("auto.offset.reset", "earliest");
    builder.put("topic", topic);
    addBeans(ConsumerRecordConverter.class, RecordConverter.class);
    MyBean bean = runApplication(builder, MyBean.class);
    companion.produceStrings().usingGenerator(i -> new ProducerRecord<>(topic, null, null), 10);
    await().until(() -> bean.list().size() == 10);
    assertThat(bean.list()).hasSize(10).allSatisfy(r -> {
        assertThat(r.value()).isNull();
        assertThat(r.value()).isNull();
    });
}
Also used : StringDeserializer(org.apache.kafka.common.serialization.StringDeserializer) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Aggregations

KafkaMapBasedConfig (io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig)86 Test (org.junit.jupiter.api.Test)56 Message (org.eclipse.microprofile.reactive.messaging.Message)34 RecordHeader (org.apache.kafka.common.header.internals.RecordHeader)30 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)25 StringDeserializer (org.apache.kafka.common.serialization.StringDeserializer)22 CopyOnWriteArrayList (java.util.concurrent.CopyOnWriteArrayList)20 JsonObject (io.vertx.core.json.JsonObject)19 ArrayList (java.util.ArrayList)18 KafkaConnectorIncomingConfiguration (io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration)17 KafkaConnectorOutgoingConfiguration (io.smallrye.reactive.messaging.kafka.KafkaConnectorOutgoingConfiguration)17 KafkaSink (io.smallrye.reactive.messaging.kafka.impl.KafkaSink)17 IncomingKafkaCloudEventMetadata (io.smallrye.reactive.messaging.kafka.IncomingKafkaCloudEventMetadata)14 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)14 IntegerSerializer (org.apache.kafka.common.serialization.IntegerSerializer)13 JsonObjectDeserializer (io.vertx.kafka.client.serialization.JsonObjectDeserializer)5 Header (org.apache.kafka.common.header.Header)5 HealthReport (io.smallrye.reactive.messaging.health.HealthReport)4 KafkaCompanionTestBase (io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase)3 List (java.util.List)3