Examples with KafkaConnectorIncomingConfiguration io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration used on opensource projects

Example 11 with KafkaConnectorIncomingConfiguration

use of io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration in project smallrye-reactive-messaging by smallrye.

the class ReactiveKafkaConsumerTest method testRebalanceWhilePausedAndPendingCommit.

@Test
public void testRebalanceWhilePausedAndPendingCommit() throws Exception {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).put("topic", topic);
    MapBasedConfig config2 = createConsumerConfig(groupId).with(ConsumerConfig.CLIENT_ID_CONFIG, "consumer-" + groupId + "-2").with("topic", topic);
    List<IncomingKafkaRecord<Integer, String>> list = new CopyOnWriteArrayList<>();
    List<IncomingKafkaRecord<Integer, String>> list2 = new CopyOnWriteArrayList<>();
    // The first source do not commit for now.
    Multi<IncomingKafkaRecord<Integer, String>> stream = createSource(config, groupId).getStream().invoke(list::add);
    AssertSubscriber<IncomingKafkaRecord<Integer, String>> subscriber = stream.subscribe().withSubscriber(AssertSubscriber.create(0));
    waitForPartitionAssignment();
    sendMessages(0, 100);
    // Request 50 messages and wait for them to be received. They are not acknowledged
    subscriber.request(50);
    await().until(() -> list.size() == 50);
    // Create the second source acknowledging the message.
    // The rebalance will split the partitions between the 2 sources, but both will restaert from offset 0, as nothing
    // has been acked.
    KafkaSource<Integer, String> source2 = new KafkaSource<>(vertx, groupId, new KafkaConnectorIncomingConfiguration(config2), UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), 3);
    source2.getStream().invoke(i -> {
        list2.add(i);
        i.ack();
    }).subscribe().withSubscriber(AssertSubscriber.create(100));
    await().until(() -> !source2.getConsumer().getAssignments().await().indefinitely().isEmpty());
    // Verify rebalance
    await().until(() -> Uni.combine().all().unis(source.getConsumer().getAssignments(), source2.getConsumer().getAssignments()).combinedWith((tp1, tp2) -> tp1.size() + tp2.size()).await().indefinitely() == partitions);
    Set<TopicPartition> assignedToSource2 = source2.getConsumer().getAssignments().await().indefinitely();
    subscriber.request(100);
    await().until(() -> list.size() >= 100);
    await().until(() -> list2.size() == assignedToSource2.size() * 25);
    // Acknowledge messages, even these received before the rebalance.
    list.forEach(IncomingKafkaRecord::ack);
    // Verify that the 100 messages have been received.
    await().untilAsserted(() -> {
        List<String> receivedByFirstSource = list.stream().map(i -> i.getPartition() + "/" + i.getOffset()).collect(Collectors.toList());
        List<String> receivedBySecondSource = list2.stream().map(i -> i.getPartition() + "/" + i.getOffset()).collect(Collectors.toList());
        Set<String> set = new HashSet<>(receivedByFirstSource);
        set.addAll(receivedBySecondSource);
        assertThat(set).hasSize(100);
    });
    source2.closeQuietly();
}

Example 12 with KafkaConnectorIncomingConfiguration

use of io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration in project smallrye-reactive-messaging by smallrye.

the class ClientTestBase method createSourceSeekToOffset.

public KafkaSource<Integer, String> createSourceSeekToOffset() {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).put("topic", topic);
    SingletonInstance<KafkaConsumerRebalanceListener> listeners = new SingletonInstance<>(groupId, getKafkaConsumerRebalanceListenerAwaitingAssignationAndSeekToOffset());
    source = new KafkaSource<>(vertx, groupId, new KafkaConnectorIncomingConfiguration(config), listeners, CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), 0);
    return source;
}

Example 13 with KafkaConnectorIncomingConfiguration

use of io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration in project smallrye-reactive-messaging by smallrye.

the class ClientTestBase method createSourceSeekToEnd.

public KafkaSource<Integer, String> createSourceSeekToEnd() {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).put("topic", topic);
    SingletonInstance<KafkaConsumerRebalanceListener> listeners = new SingletonInstance<>(groupId, getKafkaConsumerRebalanceListenerAwaitingAssignationAndSeekToEnd());
    source = new KafkaSource<>(vertx, groupId, new KafkaConnectorIncomingConfiguration(config), listeners, CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), 0);
    return source;
}

Example 14 with KafkaConnectorIncomingConfiguration

use of io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration in project smallrye-reactive-messaging by smallrye.

the class KafkaClientReactiveStreamsPublisherTest method createSource.

public KafkaSource<String, String> createSource() {
    String groupId = UUID.randomUUID().toString();
    MapBasedConfig config = createConsumerConfig(groupId).put("topic", topic);
    source = new KafkaSource<>(vertx, groupId, new KafkaConnectorIncomingConfiguration(config), UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), 0);
    return source;
}

Example 15 with KafkaConnectorIncomingConfiguration

use of io.smallrye.reactive.messaging.kafka.KafkaConnectorIncomingConfiguration in project smallrye-reactive-messaging by smallrye.

the class DeprecatedCommitStrategiesTest method testFailureWhenNoRebalanceListenerMatchGivenName.

@Test
public void testFailureWhenNoRebalanceListenerMatchGivenName() {
    MapBasedConfig config = commonConfiguration();
    config.with("client.id", UUID.randomUUID().toString()).with("consumer-rebalance-listener.name", "my-missing-name");
    String group = UUID.randomUUID().toString();
    assertThatThrownBy(() -> {
        new KafkaSource<>(vertx, group, new KafkaConnectorIncomingConfiguration(config), getConsumerRebalanceListeners(), CountKafkaCdiEvents.noCdiEvents, getDeserializationFailureHandlers(), -1);
    }).isInstanceOf(UnsatisfiedResolutionException.class);
}