Search in sources :

Example 1 with KafkaConsumerService

use of io.kestra.runner.kafka.services.KafkaConsumerService in project kestra by kestra-io.

the class KafkaElasticIndexerTest method run.

@Test
void run() throws IOException, InterruptedException {
    String topic = this.topicsConfig.stream().filter(indicesConfig -> indicesConfig.getCls() == Execution.class).findFirst().orElseThrow().getName();
    CountDownLatch countDownLatch = new CountDownLatch(1);
    RestHighLevelClient elasticClientSpy = spy(elasticClient);
    doAnswer(invocation -> {
        countDownLatch.countDown();
        return invocation.callRealMethod();
    }).when(elasticClientSpy).bulk(any(), any());
    KafkaConsumerService kafkaConsumerServiceSpy = mock(KafkaConsumerService.class);
    MockConsumer<String, String> mockConsumer = mockConsumer(topic);
    doReturn(mockConsumer).when(kafkaConsumerServiceSpy).of(any(), any(), any());
    ConsumerRecord<String, String> first = buildExecutionRecord(topic, 0);
    mockConsumer.addRecord(first);
    mockConsumer.addRecord(buildExecutionRecord(topic, 1));
    mockConsumer.addRecord(buildExecutionRecord(topic, 2));
    mockConsumer.addRecord(buildExecutionRecord(topic, 3));
    mockConsumer.addRecord(buildExecutionRecord(topic, 4));
    mockConsumer.addRecord(buildRecord(topic, first.key(), null, 5));
    KafkaElasticIndexer indexer = new KafkaElasticIndexer(metricRegistry, elasticClientSpy, indexerConfig, topicsConfig, indicesConfigs, elasticSearchIndicesService, kafkaConsumerServiceSpy, executorsUtils);
    Thread thread = new Thread(indexer);
    thread.start();
    countDownLatch.await();
    assertThat(countDownLatch.getCount(), is(0L));
}
Also used : ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) MockConsumer(org.apache.kafka.clients.consumer.MockConsumer) JacksonMapper(io.kestra.core.serializers.JacksonMapper) TestsUtils(io.kestra.core.utils.TestsUtils) HashMap(java.util.HashMap) OffsetResetStrategy(org.apache.kafka.clients.consumer.OffsetResetStrategy) TopicsConfig(io.kestra.runner.kafka.configs.TopicsConfig) MatcherAssert.assertThat(org.hamcrest.MatcherAssert.assertThat) RestHighLevelClient(org.opensearch.client.RestHighLevelClient) MicronautTest(io.micronaut.test.extensions.junit5.annotation.MicronautTest) TopicPartition(org.apache.kafka.common.TopicPartition) ImmutableMap(com.google.common.collect.ImmutableMap) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) IOException(java.io.IOException) Execution(io.kestra.core.models.executions.Execution) KafkaConsumerService(io.kestra.runner.kafka.services.KafkaConsumerService) ElasticSearchIndicesService(io.kestra.repository.elasticsearch.ElasticSearchIndicesService) IndicesConfig(io.kestra.repository.elasticsearch.configs.IndicesConfig) Test(org.junit.jupiter.api.Test) CountDownLatch(java.util.concurrent.CountDownLatch) Mockito(org.mockito.Mockito) List(java.util.List) ExecutorsUtils(io.kestra.core.utils.ExecutorsUtils) ConsumerRecord(org.apache.kafka.clients.consumer.ConsumerRecord) Matchers.is(org.hamcrest.Matchers.is) Flow(io.kestra.core.models.flows.Flow) MetricRegistry(io.kestra.core.metrics.MetricRegistry) Inject(jakarta.inject.Inject) Collections(java.util.Collections) KafkaConsumerService(io.kestra.runner.kafka.services.KafkaConsumerService) RestHighLevelClient(org.opensearch.client.RestHighLevelClient) CountDownLatch(java.util.concurrent.CountDownLatch) MicronautTest(io.micronaut.test.extensions.junit5.annotation.MicronautTest) Test(org.junit.jupiter.api.Test)

Aggregations

JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)1 ImmutableMap (com.google.common.collect.ImmutableMap)1 MetricRegistry (io.kestra.core.metrics.MetricRegistry)1 Execution (io.kestra.core.models.executions.Execution)1 Flow (io.kestra.core.models.flows.Flow)1 JacksonMapper (io.kestra.core.serializers.JacksonMapper)1 ExecutorsUtils (io.kestra.core.utils.ExecutorsUtils)1 TestsUtils (io.kestra.core.utils.TestsUtils)1 ElasticSearchIndicesService (io.kestra.repository.elasticsearch.ElasticSearchIndicesService)1 IndicesConfig (io.kestra.repository.elasticsearch.configs.IndicesConfig)1 TopicsConfig (io.kestra.runner.kafka.configs.TopicsConfig)1 KafkaConsumerService (io.kestra.runner.kafka.services.KafkaConsumerService)1 MicronautTest (io.micronaut.test.extensions.junit5.annotation.MicronautTest)1 Inject (jakarta.inject.Inject)1 IOException (java.io.IOException)1 Collections (java.util.Collections)1 HashMap (java.util.HashMap)1 List (java.util.List)1 CountDownLatch (java.util.concurrent.CountDownLatch)1 ConsumerRecord (org.apache.kafka.clients.consumer.ConsumerRecord)1