Search in sources :

Example 21 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testWithMultipleConsumersWithAMultipleConsumeGroup.

@Test
public void testWithMultipleConsumersWithAMultipleConsumeGroup() throws Exception {
    int count = 100;
    CountDownLatch latch = new CountDownLatch(count * partitions);
    List<KafkaSource<Integer, String>> sources = new ArrayList<>();
    for (int i = 0; i < partitions; i++) {
        String groupId = UUID.randomUUID().toString();
        MapBasedConfig config = createConsumerConfig(groupId).with("topic", topic).with(ConsumerConfig.CLIENT_ID_CONFIG, "client-" + i).with(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        KafkaSource<Integer, String> source = createSource(config, groupId);
        sources.add(source);
        source.getStream().invoke(rec -> {
            onReceive(rec);
            latch.countDown();
        }).subscribe().with(IncomingKafkaRecord::ack);
        await().until(() -> !source.getConsumer().getAssignments().await().indefinitely().isEmpty());
    }
    sendMessages(0, count);
    waitForMessages(latch);
    assertThat(sources).hasSize(4);
    assertThat(sources).allSatisfy(s -> assertThat(s.getConsumer().getAssignments().await().indefinitely()).hasSize(partitions));
    sources.forEach(KafkaSource::closeQuietly);
}
Also used : AtomicInteger(java.util.concurrent.atomic.AtomicInteger) KafkaSource(io.smallrye.reactive.messaging.kafka.impl.KafkaSource) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) IncomingKafkaRecord(io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord)

Example 22 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testAutoCommitAcknowledgement.

@Test
public void testAutoCommitAcknowledgement() throws Exception {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).with("topic", topic).with(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
    Multi<IncomingKafkaRecord<Integer, String>> stream = createSource(config, groupId).getStream().invoke(IncomingKafkaRecord::ack);
    sendReceive(stream, 0, 100, 0, 100);
    restartAndCheck(config, groupId, 1);
}
Also used : MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) IncomingKafkaRecord(io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord)

Example 23 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testMessageProcessingDelay.

/**
 * Tests that delays in message processing dont cause session timeouts.
 * Kafka consumer heartbeat thread should keep the session alive.
 */
@Test
public void testMessageProcessingDelay() throws Exception {
    int count = 3;
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).put("topic", topic);
    KafkaSource<Integer, String> source = createSource(config, groupId);
    AssertSubscriber<IncomingKafkaRecord<Integer, String>> subscriber = source.getStream().select().first(count).subscribe().withSubscriber(AssertSubscriber.create(0));
    sendMessages(0, count);
    for (int i = 0; i < count; i++) {
        subscriber.request(1);
        int l = i + 1;
        await().until(() -> subscriber.getItems().size() == l);
        Thread.sleep(sessionTimeoutMillis + 1000);
    }
    subscriber.awaitCompletion();
}
Also used : AtomicInteger(java.util.concurrent.atomic.AtomicInteger) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) IncomingKafkaRecord(io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord)

Example 24 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testAcknowledgementUsingThrottledStrategy.

@Test
public void testAcknowledgementUsingThrottledStrategy() throws Exception {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).with("topic", topic);
    Multi<IncomingKafkaRecord<Integer, String>> stream = createSource(config, groupId).getStream().invoke(IncomingKafkaRecord::ack);
    sendReceive(stream, 0, 100, 0, 100);
    waitForCommits(source, 100);
    // Close consumer and create another one. First consumer should commit final offset on close.
    // Second consumer should receive only new messages.
    cancelSubscriptions();
    clearReceivedMessages();
    config.with(ConsumerConfig.CLIENT_ID_CONFIG, "second-consumer");
    Multi<IncomingKafkaRecord<Integer, String>> stream2 = createSource(config, groupId).getStream();
    sendReceive(stream2, 100, 100, 100, 100);
}
Also used : MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) IncomingKafkaRecord(io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord)

Example 25 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testCommitBatchWithLatestStrategy.

@Test
public void testCommitBatchWithLatestStrategy() throws Exception {
    int count = 20;
    int commitIntervalMessages = 4;
    CountDownLatch commitLatch = new CountDownLatch(count);
    long[] committedOffsets = new long[partitions];
    Arrays.fill(committedOffsets, 0L);
    List<IncomingKafkaRecord<?, ?>> uncommitted = new ArrayList<>();
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).with("topic", topic).with("commit-strategy", "latest");
    Multi<IncomingKafkaRecord<Integer, String>> stream = createSource(config, groupId).getStream().onItem().call(record -> {
        uncommitted.add(record);
        if (uncommitted.size() == commitIntervalMessages) {
            return Uni.createFrom().completionStage(record.ack()).invoke(i -> onCommit(uncommitted, commitLatch, committedOffsets));
        }
        return Uni.createFrom().voidItem();
    });
    sendAndWaitForMessages(stream, count);
    checkCommitCallbacks(commitLatch, committedOffsets);
}
Also used : IncomingKafkaRecord(io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig)

Aggregations

MapBasedConfig (io.smallrye.reactive.messaging.test.common.config.MapBasedConfig)272 Test (org.junit.jupiter.api.Test)223 Message (org.eclipse.microprofile.reactive.messaging.Message)69 JsonObject (io.vertx.core.json.JsonObject)63 ArrayList (java.util.ArrayList)61 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)57 TopicPartition (org.apache.kafka.common.TopicPartition)43 Awaitility.await (org.awaitility.Awaitility.await)38 CopyOnWriteArrayList (java.util.concurrent.CopyOnWriteArrayList)37 Assertions.assertThat (org.assertj.core.api.Assertions.assertThat)34 Weld (org.jboss.weld.environment.se.Weld)32 KafkaMapBasedConfig (io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig)31 HashMap (java.util.HashMap)29 AfterEach (org.junit.jupiter.api.AfterEach)29 Collectors (java.util.stream.Collectors)28 HealthReport (io.smallrye.reactive.messaging.health.HealthReport)26 KafkaSource (io.smallrye.reactive.messaging.kafka.impl.KafkaSource)26 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)25 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)23 Duration (java.time.Duration)22