Search in sources :

Example 1 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class KafkaSourceTest method testSource.

@SuppressWarnings("unchecked")
@Test
public void testSource() {
    MapBasedConfig config = newCommonConfigForSource().with("value.deserializer", IntegerDeserializer.class.getName());
    KafkaConnectorIncomingConfiguration ic = new KafkaConnectorIncomingConfiguration(config);
    source = new KafkaSource<>(vertx, UUID.randomUUID().toString(), ic, UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), -1);
    List<Message<?>> messages = new ArrayList<>();
    source.getStream().subscribe().with(messages::add);
    companion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, "hello", i), 10);
    await().atMost(2, TimeUnit.MINUTES).until(() -> messages.size() >= 10);
    assertThat(messages.stream().map(m -> ((KafkaRecord<String, Integer>) m).getPayload()).collect(Collectors.toList())).containsExactly(0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
}
Also used : IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) IntStream(java.util.stream.IntStream) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaCompanionTestBase(io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) UnsatisfiedResolutionException(javax.enterprise.inject.UnsatisfiedResolutionException) HealthReport(io.smallrye.reactive.messaging.health.HealthReport) Random(java.util.Random) KafkaMetadataUtil(io.smallrye.reactive.messaging.kafka.api.KafkaMetadataUtil) Multi(io.smallrye.mutiny.Multi) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) Assertions.assertThatThrownBy(org.assertj.core.api.Assertions.assertThatThrownBy) Assert(org.testng.Assert) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Duration(java.time.Duration) KafkaBrokerExtension(io.smallrye.reactive.messaging.kafka.companion.test.KafkaBrokerExtension) Tag(org.junit.jupiter.api.Tag) ExecutorService(java.util.concurrent.ExecutorService) DeploymentException(org.jboss.weld.exceptions.DeploymentException) TopicPartition(org.apache.kafka.common.TopicPartition) ExecutionHolder(io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder) Awaitility.await(org.awaitility.Awaitility.await) StrimziKafkaContainer(io.strimzi.test.container.StrimziKafkaContainer) UUID(java.util.UUID) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) UnsatisfiedInstance(io.smallrye.reactive.messaging.kafka.base.UnsatisfiedInstance) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) Executors(java.util.concurrent.Executors) Channel(org.eclipse.microprofile.reactive.messaging.Channel) TimeUnit(java.util.concurrent.TimeUnit) Test(org.junit.jupiter.api.Test) List(java.util.List) Message(org.eclipse.microprofile.reactive.messaging.Message) AfterEach(org.junit.jupiter.api.AfterEach) KafkaCompanion(io.smallrye.reactive.messaging.kafka.companion.KafkaCompanion) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) ApplicationScoped(javax.enterprise.context.ApplicationScoped) KafkaSource(io.smallrye.reactive.messaging.kafka.impl.KafkaSource) PublisherBuilder(org.eclipse.microprofile.reactive.streams.operators.PublisherBuilder) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) Identifier(io.smallrye.common.annotation.Identifier) Message(org.eclipse.microprofile.reactive.messaging.Message) ArrayList(java.util.ArrayList) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 2 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class KafkaSourceTest method testRetry.

@SuppressWarnings({ "rawtypes" })
@Test
@Tag(TestTags.SLOW)
public void testRetry() {
    // This test need an individual Kafka container
    try (StrimziKafkaContainer kafka = KafkaBrokerExtension.createKafkaContainer()) {
        kafka.start();
        await().until(kafka::isRunning);
        MapBasedConfig config = newCommonConfigForSource().with("bootstrap.servers", kafka.getBootstrapServers()).with("value.deserializer", IntegerDeserializer.class.getName()).with("retry", true).with("retry-attempts", 100).with("retry-max-wait", 30);
        KafkaCompanion kafkaCompanion = new KafkaCompanion(kafka.getBootstrapServers());
        KafkaConnectorIncomingConfiguration ic = new KafkaConnectorIncomingConfiguration(config);
        source = new KafkaSource<>(vertx, UUID.randomUUID().toString(), ic, UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), -1);
        List<KafkaRecord> messages1 = new ArrayList<>();
        source.getStream().subscribe().with(messages1::add);
        AtomicInteger counter = new AtomicInteger();
        kafkaCompanion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 10);
        await().atMost(2, TimeUnit.MINUTES).until(() -> messages1.size() >= 10);
        try (@SuppressWarnings("unused") StrimziKafkaContainer container = KafkaBrokerExtension.restart(kafka, 2)) {
            kafkaCompanion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 10);
            await().atMost(2, TimeUnit.MINUTES).until(() -> messages1.size() >= 20);
            assertThat(messages1.size()).isGreaterThanOrEqualTo(20);
        }
    }
}
Also used : StrimziKafkaContainer(io.strimzi.test.container.StrimziKafkaContainer) KafkaCompanion(io.smallrye.reactive.messaging.kafka.companion.KafkaCompanion) ArrayList(java.util.ArrayList) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test) Tag(org.junit.jupiter.api.Tag)

Example 3 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class KafkaSourceWithLegacyMetadataTest method testBroadcastWithPartitions.

@SuppressWarnings({ "rawtypes", "unchecked" })
@Test
public void testBroadcastWithPartitions() {
    companion.topics().create(topic, 2);
    MapBasedConfig config = newCommonConfigForSource().with("value.deserializer", IntegerDeserializer.class.getName()).with("broadcast", true).with("partitions", 2);
    connector = new KafkaConnector();
    connector.executionHolder = new ExecutionHolder(vertx);
    connector.configurations = UnsatisfiedInstance.instance();
    connector.consumerRebalanceListeners = UnsatisfiedInstance.instance();
    connector.kafkaCDIEvents = new CountKafkaCdiEvents();
    connector.init();
    PublisherBuilder<? extends KafkaRecord> builder = (PublisherBuilder<? extends KafkaRecord>) connector.getPublisherBuilder(config);
    List<KafkaRecord> messages1 = new ArrayList<>();
    List<KafkaRecord> messages2 = new ArrayList<>();
    builder.forEach(messages1::add).run();
    builder.forEach(messages2::add).run();
    companion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 10);
    await().atMost(2, TimeUnit.MINUTES).until(() -> messages1.size() >= 10);
    await().atMost(2, TimeUnit.MINUTES).until(() -> messages2.size() >= 10);
    assertThat(messages1.stream().map(KafkaRecord::getPayload).collect(Collectors.toList())).containsExactlyInAnyOrder(0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
    assertThat(messages2.stream().map(KafkaRecord::getPayload).collect(Collectors.toList())).containsExactlyInAnyOrder(0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
}
Also used : IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) PublisherBuilder(org.eclipse.microprofile.reactive.streams.operators.PublisherBuilder) ArrayList(java.util.ArrayList) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) ExecutionHolder(io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder) Test(org.junit.jupiter.api.Test)

Example 4 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class KafkaSourceWithLegacyMetadataTest method testInvalidIncomingType.

@SuppressWarnings("unchecked")
@Test
public void testInvalidIncomingType() {
    MapBasedConfig config = newCommonConfigForSource().with("value.deserializer", IntegerDeserializer.class.getName());
    KafkaConnectorIncomingConfiguration ic = new KafkaConnectorIncomingConfiguration(config);
    source = new KafkaSource<>(vertx, UUID.randomUUID().toString(), ic, UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), -1);
    List<Message<?>> messages = new ArrayList<>();
    source.getStream().subscribe().with(messages::add);
    companion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 2);
    await().atMost(2, TimeUnit.MINUTES).until(() -> messages.size() >= 2);
    assertThat(messages.stream().map(m -> ((KafkaRecord<String, Integer>) m).getPayload()).collect(Collectors.toList())).containsExactly(0, 1);
    companion.produceStrings().fromRecords(new ProducerRecord<>(topic, "hello"));
    companion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 2);
    // no other message received
    assertThat(messages.stream().map(m -> ((KafkaRecord<String, Integer>) m).getPayload()).collect(Collectors.toList())).containsExactly(0, 1);
}
Also used : IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) IntStream(java.util.stream.IntStream) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaCompanionTestBase(io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) UnsatisfiedResolutionException(javax.enterprise.inject.UnsatisfiedResolutionException) HealthReport(io.smallrye.reactive.messaging.health.HealthReport) Random(java.util.Random) KafkaMetadataUtil(io.smallrye.reactive.messaging.kafka.api.KafkaMetadataUtil) Multi(io.smallrye.mutiny.Multi) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) Assertions.assertThatThrownBy(org.assertj.core.api.Assertions.assertThatThrownBy) Duration(java.time.Duration) KafkaBrokerExtension(io.smallrye.reactive.messaging.kafka.companion.test.KafkaBrokerExtension) Tag(org.junit.jupiter.api.Tag) ExecutorService(java.util.concurrent.ExecutorService) DeploymentException(org.jboss.weld.exceptions.DeploymentException) TopicPartition(org.apache.kafka.common.TopicPartition) ExecutionHolder(io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder) Awaitility.await(org.awaitility.Awaitility.await) StrimziKafkaContainer(io.strimzi.test.container.StrimziKafkaContainer) UUID(java.util.UUID) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) UnsatisfiedInstance(io.smallrye.reactive.messaging.kafka.base.UnsatisfiedInstance) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) Executors(java.util.concurrent.Executors) Channel(org.eclipse.microprofile.reactive.messaging.Channel) TimeUnit(java.util.concurrent.TimeUnit) Test(org.junit.jupiter.api.Test) List(java.util.List) Message(org.eclipse.microprofile.reactive.messaging.Message) AfterEach(org.junit.jupiter.api.AfterEach) KafkaCompanion(io.smallrye.reactive.messaging.kafka.companion.KafkaCompanion) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) ApplicationScoped(javax.enterprise.context.ApplicationScoped) KafkaSource(io.smallrye.reactive.messaging.kafka.impl.KafkaSource) KafkaBrokerExtension.restart(io.smallrye.reactive.messaging.kafka.companion.test.KafkaBrokerExtension.restart) PublisherBuilder(org.eclipse.microprofile.reactive.streams.operators.PublisherBuilder) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) Identifier(io.smallrye.common.annotation.Identifier) Message(org.eclipse.microprofile.reactive.messaging.Message) ArrayList(java.util.ArrayList) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Example 5 with MapBasedConfig

use of io.smallrye.reactive.messaging.test.common.config.MapBasedConfig in project smallrye-reactive-messaging by smallrye.

the class KafkaSourceWithLegacyMetadataTest method testSourceWithPartitions.

@SuppressWarnings("unchecked")
@Test
public void testSourceWithPartitions() {
    MapBasedConfig config = newCommonConfigForSource().with("value.deserializer", IntegerDeserializer.class.getName()).with("partitions", 4);
    companion.topics().createAndWait(topic, 3);
    KafkaConnectorIncomingConfiguration ic = new KafkaConnectorIncomingConfiguration(config);
    source = new KafkaSource<>(vertx, UUID.randomUUID().toString(), ic, UnsatisfiedInstance.instance(), CountKafkaCdiEvents.noCdiEvents, UnsatisfiedInstance.instance(), -1);
    List<Message<?>> messages = new ArrayList<>();
    source.getStream().subscribe().with(messages::add);
    companion.produceIntegers().usingGenerator(i -> new ProducerRecord<>(topic, i), 1000);
    await().atMost(2, TimeUnit.MINUTES).until(() -> messages.size() >= 1000);
    List<Integer> expected = IntStream.range(0, 1000).boxed().collect(Collectors.toList());
    // Because of partitions we cannot enforce the order.
    assertThat(messages.stream().map(m -> ((KafkaRecord<String, Integer>) m).getPayload()).collect(Collectors.toList())).containsExactlyInAnyOrderElementsOf(expected);
}
Also used : IntStream(java.util.stream.IntStream) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaCompanionTestBase(io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) UnsatisfiedResolutionException(javax.enterprise.inject.UnsatisfiedResolutionException) HealthReport(io.smallrye.reactive.messaging.health.HealthReport) Random(java.util.Random) KafkaMetadataUtil(io.smallrye.reactive.messaging.kafka.api.KafkaMetadataUtil) Multi(io.smallrye.mutiny.Multi) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) Assertions.assertThatThrownBy(org.assertj.core.api.Assertions.assertThatThrownBy) Duration(java.time.Duration) KafkaBrokerExtension(io.smallrye.reactive.messaging.kafka.companion.test.KafkaBrokerExtension) Tag(org.junit.jupiter.api.Tag) ExecutorService(java.util.concurrent.ExecutorService) DeploymentException(org.jboss.weld.exceptions.DeploymentException) TopicPartition(org.apache.kafka.common.TopicPartition) ExecutionHolder(io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder) Awaitility.await(org.awaitility.Awaitility.await) StrimziKafkaContainer(io.strimzi.test.container.StrimziKafkaContainer) UUID(java.util.UUID) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) UnsatisfiedInstance(io.smallrye.reactive.messaging.kafka.base.UnsatisfiedInstance) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) Executors(java.util.concurrent.Executors) Channel(org.eclipse.microprofile.reactive.messaging.Channel) TimeUnit(java.util.concurrent.TimeUnit) Test(org.junit.jupiter.api.Test) List(java.util.List) Message(org.eclipse.microprofile.reactive.messaging.Message) AfterEach(org.junit.jupiter.api.AfterEach) KafkaCompanion(io.smallrye.reactive.messaging.kafka.companion.KafkaCompanion) IntegerDeserializer(org.apache.kafka.common.serialization.IntegerDeserializer) ApplicationScoped(javax.enterprise.context.ApplicationScoped) KafkaSource(io.smallrye.reactive.messaging.kafka.impl.KafkaSource) KafkaBrokerExtension.restart(io.smallrye.reactive.messaging.kafka.companion.test.KafkaBrokerExtension.restart) PublisherBuilder(org.eclipse.microprofile.reactive.streams.operators.PublisherBuilder) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) Identifier(io.smallrye.common.annotation.Identifier) Message(org.eclipse.microprofile.reactive.messaging.Message) ArrayList(java.util.ArrayList) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) MapBasedConfig(io.smallrye.reactive.messaging.test.common.config.MapBasedConfig) KafkaMapBasedConfig(io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig) Test(org.junit.jupiter.api.Test)

Aggregations

MapBasedConfig (io.smallrye.reactive.messaging.test.common.config.MapBasedConfig)272 Test (org.junit.jupiter.api.Test)223 Message (org.eclipse.microprofile.reactive.messaging.Message)69 JsonObject (io.vertx.core.json.JsonObject)63 ArrayList (java.util.ArrayList)61 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)57 TopicPartition (org.apache.kafka.common.TopicPartition)43 Awaitility.await (org.awaitility.Awaitility.await)38 CopyOnWriteArrayList (java.util.concurrent.CopyOnWriteArrayList)37 Assertions.assertThat (org.assertj.core.api.Assertions.assertThat)34 Weld (org.jboss.weld.environment.se.Weld)32 KafkaMapBasedConfig (io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig)31 HashMap (java.util.HashMap)29 AfterEach (org.junit.jupiter.api.AfterEach)29 Collectors (java.util.stream.Collectors)28 HealthReport (io.smallrye.reactive.messaging.health.HealthReport)26 KafkaSource (io.smallrye.reactive.messaging.kafka.impl.KafkaSource)26 IntegerDeserializer (org.apache.kafka.common.serialization.IntegerDeserializer)25 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)23 Duration (java.time.Duration)22