Search in sources :

Example 31 with MockTime

use of org.apache.kafka.common.utils.MockTime in project apache-kafka-on-k8s by banzaicloud.

the class KafkaConsumerTest method verifyNoCoordinatorLookupForManualAssignmentWithSeek.

@Test
public void verifyNoCoordinatorLookupForManualAssignmentWithSeek() {
    Time time = new MockTime();
    Cluster cluster = TestUtils.singletonCluster(topic, 1);
    Node node = cluster.nodes().get(0);
    Metadata metadata = createMetadata();
    metadata.update(cluster, Collections.<String>emptySet(), time.milliseconds());
    MockClient client = new MockClient(time, metadata);
    client.setNode(node);
    PartitionAssignor assignor = new RoundRobinAssignor();
    KafkaConsumer<String, String> consumer = newConsumer(time, client, metadata, assignor, true);
    consumer.assign(singleton(tp0));
    consumer.seekToBeginning(singleton(tp0));
    // there shouldn't be any need to lookup the coordinator or fetch committed offsets.
    // we just lookup the starting position and send the record fetch.
    client.prepareResponse(listOffsetsResponse(Collections.singletonMap(tp0, 50L)));
    client.prepareResponse(fetchResponse(tp0, 50L, 5));
    ConsumerRecords<String, String> records = consumer.poll(5);
    assertEquals(5, records.count());
    assertEquals(55L, consumer.position(tp0));
    consumer.close(0, TimeUnit.MILLISECONDS);
}
Also used : Node(org.apache.kafka.common.Node) Metadata(org.apache.kafka.clients.Metadata) Cluster(org.apache.kafka.common.Cluster) MockTime(org.apache.kafka.common.utils.MockTime) Time(org.apache.kafka.common.utils.Time) PartitionAssignor(org.apache.kafka.clients.consumer.internals.PartitionAssignor) MockTime(org.apache.kafka.common.utils.MockTime) MockClient(org.apache.kafka.clients.MockClient) Test(org.junit.Test)

Example 32 with MockTime

use of org.apache.kafka.common.utils.MockTime in project apache-kafka-on-k8s by banzaicloud.

the class KafkaConsumerTest method testManualAssignmentChangeWithAutoCommitEnabled.

@Test
public void testManualAssignmentChangeWithAutoCommitEnabled() {
    Time time = new MockTime();
    Map<String, Integer> tpCounts = new HashMap<>();
    tpCounts.put(topic, 1);
    tpCounts.put(topic2, 1);
    Cluster cluster = TestUtils.singletonCluster(tpCounts);
    Node node = cluster.nodes().get(0);
    Metadata metadata = createMetadata();
    metadata.update(cluster, Collections.<String>emptySet(), time.milliseconds());
    MockClient client = new MockClient(time, metadata);
    client.setNode(node);
    PartitionAssignor assignor = new RangeAssignor();
    KafkaConsumer<String, String> consumer = newConsumer(time, client, metadata, assignor, true);
    // lookup coordinator
    client.prepareResponseFrom(new FindCoordinatorResponse(Errors.NONE, node), node);
    Node coordinator = new Node(Integer.MAX_VALUE - node.id(), node.host(), node.port());
    // manual assignment
    consumer.assign(singleton(tp0));
    consumer.seekToBeginning(singleton(tp0));
    // fetch offset for one topic
    client.prepareResponseFrom(offsetResponse(Collections.singletonMap(tp0, 0L), Errors.NONE), coordinator);
    assertEquals(0, consumer.committed(tp0).offset());
    // verify that assignment immediately changes
    assertTrue(consumer.assignment().equals(singleton(tp0)));
    // there shouldn't be any need to lookup the coordinator or fetch committed offsets.
    // we just lookup the starting position and send the record fetch.
    client.prepareResponse(listOffsetsResponse(Collections.singletonMap(tp0, 10L)));
    client.prepareResponse(fetchResponse(tp0, 10L, 1));
    ConsumerRecords<String, String> records = consumer.poll(5);
    assertEquals(1, records.count());
    assertEquals(11L, consumer.position(tp0));
    // mock the offset commit response for to be revoked partitions
    AtomicBoolean commitReceived = prepareOffsetCommitResponse(client, coordinator, tp0, 11);
    // new manual assignment
    consumer.assign(singleton(t2p0));
    // verify that assignment immediately changes
    assertTrue(consumer.assignment().equals(singleton(t2p0)));
    // verify that the offset commits occurred as expected
    assertTrue(commitReceived.get());
    client.requests().clear();
    consumer.close();
}
Also used : FindCoordinatorResponse(org.apache.kafka.common.requests.FindCoordinatorResponse) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) Node(org.apache.kafka.common.Node) Metadata(org.apache.kafka.clients.Metadata) Cluster(org.apache.kafka.common.Cluster) MockTime(org.apache.kafka.common.utils.MockTime) Time(org.apache.kafka.common.utils.Time) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) PartitionAssignor(org.apache.kafka.clients.consumer.internals.PartitionAssignor) MockTime(org.apache.kafka.common.utils.MockTime) MockClient(org.apache.kafka.clients.MockClient) Test(org.junit.Test)

Example 33 with MockTime

use of org.apache.kafka.common.utils.MockTime in project apache-kafka-on-k8s by banzaicloud.

the class KafkaConsumerTest method testChangingRegexSubscription.

@Test
public void testChangingRegexSubscription() {
    PartitionAssignor assignor = new RoundRobinAssignor();
    String otherTopic = "other";
    TopicPartition otherTopicPartition = new TopicPartition(otherTopic, 0);
    Time time = new MockTime();
    Map<String, Integer> topicMetadata = new HashMap<>();
    topicMetadata.put(topic, 1);
    topicMetadata.put(otherTopic, 1);
    Cluster cluster = TestUtils.clusterWith(1, topicMetadata);
    Metadata metadata = createMetadata();
    Node node = cluster.nodes().get(0);
    MockClient client = new MockClient(time, metadata);
    client.setNode(node);
    client.cluster(cluster);
    metadata.update(cluster, Collections.<String>emptySet(), time.milliseconds());
    KafkaConsumer<String, String> consumer = newConsumer(time, client, metadata, assignor, false);
    Node coordinator = prepareRebalance(client, node, singleton(topic), assignor, singletonList(tp0), null);
    consumer.subscribe(Pattern.compile(topic), getConsumerRebalanceListener(consumer));
    consumer.poll(0);
    assertEquals(singleton(topic), consumer.subscription());
    consumer.subscribe(Pattern.compile(otherTopic), getConsumerRebalanceListener(consumer));
    prepareRebalance(client, node, singleton(otherTopic), assignor, singletonList(otherTopicPartition), coordinator);
    consumer.poll(0);
    assertEquals(singleton(otherTopic), consumer.subscription());
    consumer.close(0, TimeUnit.MILLISECONDS);
}
Also used : HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) Node(org.apache.kafka.common.Node) Metadata(org.apache.kafka.clients.Metadata) Cluster(org.apache.kafka.common.Cluster) MockTime(org.apache.kafka.common.utils.MockTime) Time(org.apache.kafka.common.utils.Time) TopicPartition(org.apache.kafka.common.TopicPartition) PartitionAssignor(org.apache.kafka.clients.consumer.internals.PartitionAssignor) MockTime(org.apache.kafka.common.utils.MockTime) MockClient(org.apache.kafka.clients.MockClient) Test(org.junit.Test)

Example 34 with MockTime

use of org.apache.kafka.common.utils.MockTime in project apache-kafka-on-k8s by banzaicloud.

the class KafkaConsumerTest method consumerCloseTest.

private void consumerCloseTest(final long closeTimeoutMs, List<? extends AbstractResponse> responses, long waitMs, boolean interrupt) throws Exception {
    Time time = new MockTime();
    Cluster cluster = TestUtils.singletonCluster(topic, 1);
    Node node = cluster.nodes().get(0);
    Metadata metadata = createMetadata();
    metadata.update(cluster, Collections.<String>emptySet(), time.milliseconds());
    MockClient client = new MockClient(time, metadata);
    client.setNode(node);
    PartitionAssignor assignor = new RoundRobinAssignor();
    final KafkaConsumer<String, String> consumer = newConsumer(time, client, metadata, assignor, false);
    consumer.subscribe(singleton(topic), getConsumerRebalanceListener(consumer));
    Node coordinator = prepareRebalance(client, node, assignor, singletonList(tp0), null);
    client.prepareMetadataUpdate(cluster, Collections.<String>emptySet());
    // Poll with responses
    client.prepareResponseFrom(fetchResponse(tp0, 0, 1), node);
    client.prepareResponseFrom(fetchResponse(tp0, 1, 0), node);
    consumer.poll(0);
    // Initiate close() after a commit request on another thread.
    // Kafka consumer is single-threaded, but the implementation allows calls on a
    // different thread as long as the calls are not executed concurrently. So this is safe.
    ExecutorService executor = Executors.newSingleThreadExecutor();
    final AtomicReference<Exception> closeException = new AtomicReference<>();
    try {
        Future<?> future = executor.submit(new Runnable() {

            @Override
            public void run() {
                consumer.commitAsync();
                try {
                    consumer.close(closeTimeoutMs, TimeUnit.MILLISECONDS);
                } catch (Exception e) {
                    closeException.set(e);
                }
            }
        });
        // if close timeout is not zero.
        try {
            future.get(100, TimeUnit.MILLISECONDS);
            if (closeTimeoutMs != 0)
                fail("Close completed without waiting for commit or leave response");
        } catch (TimeoutException e) {
        // Expected exception
        }
        // Ensure close has started and queued at least one more request after commitAsync
        client.waitForRequests(2, 1000);
        // In non-graceful mode, close() times out without an exception even though commit response is pending
        for (int i = 0; i < responses.size(); i++) {
            client.waitForRequests(1, 1000);
            client.respondFrom(responses.get(i), coordinator);
            if (i != responses.size() - 1) {
                try {
                    future.get(100, TimeUnit.MILLISECONDS);
                    fail("Close completed without waiting for response");
                } catch (TimeoutException e) {
                // Expected exception
                }
            }
        }
        if (waitMs > 0)
            time.sleep(waitMs);
        if (interrupt) {
            assertTrue("Close terminated prematurely", future.cancel(true));
            TestUtils.waitForCondition(new TestCondition() {

                @Override
                public boolean conditionMet() {
                    return closeException.get() != null;
                }
            }, "InterruptException did not occur within timeout.");
            assertTrue("Expected exception not thrown " + closeException, closeException.get() instanceof InterruptException);
        } else {
            // Should succeed without TimeoutException or ExecutionException
            future.get(500, TimeUnit.MILLISECONDS);
            assertNull("Unexpected exception during close", closeException.get());
        }
    } finally {
        executor.shutdownNow();
    }
}
Also used : Node(org.apache.kafka.common.Node) Metadata(org.apache.kafka.clients.Metadata) InterruptException(org.apache.kafka.common.errors.InterruptException) Cluster(org.apache.kafka.common.Cluster) MockTime(org.apache.kafka.common.utils.MockTime) Time(org.apache.kafka.common.utils.Time) AtomicReference(java.util.concurrent.atomic.AtomicReference) KafkaException(org.apache.kafka.common.KafkaException) TimeoutException(java.util.concurrent.TimeoutException) WakeupException(org.apache.kafka.common.errors.WakeupException) AuthenticationException(org.apache.kafka.common.errors.AuthenticationException) InterruptException(org.apache.kafka.common.errors.InterruptException) ExpectedException(org.junit.rules.ExpectedException) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) ExecutorService(java.util.concurrent.ExecutorService) TestCondition(org.apache.kafka.test.TestCondition) PartitionAssignor(org.apache.kafka.clients.consumer.internals.PartitionAssignor) MockTime(org.apache.kafka.common.utils.MockTime) MockClient(org.apache.kafka.clients.MockClient) TimeoutException(java.util.concurrent.TimeoutException)

Example 35 with MockTime

use of org.apache.kafka.common.utils.MockTime in project apache-kafka-on-k8s by banzaicloud.

the class KafkaConsumerTest method testConsumerWithinBlackoutPeriodAfterAuthenticationFailure.

@Test
public void testConsumerWithinBlackoutPeriodAfterAuthenticationFailure() {
    Time time = new MockTime();
    Map<String, Integer> tpCounts = new HashMap<>();
    tpCounts.put(topic, 1);
    Cluster cluster = TestUtils.singletonCluster(tpCounts);
    Node node = cluster.nodes().get(0);
    Metadata metadata = createMetadata();
    metadata.update(cluster, Collections.<String>emptySet(), time.milliseconds());
    MockClient client = new MockClient(time, metadata);
    client.setNode(node);
    client.authenticationFailed(node, 300);
    PartitionAssignor assignor = new RangeAssignor();
    final KafkaConsumer<String, String> consumer = newConsumer(time, client, metadata, assignor, true);
    consumer.subscribe(Collections.singleton(topic));
    callConsumerApisAndExpectAnAuthenticationError(consumer, tp0);
    // wait less than the blackout period
    time.sleep(30);
    assertTrue(client.connectionFailed(node));
    callConsumerApisAndExpectAnAuthenticationError(consumer, tp0);
    client.requests().clear();
    consumer.close(0, TimeUnit.MILLISECONDS);
}
Also used : HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) Node(org.apache.kafka.common.Node) Metadata(org.apache.kafka.clients.Metadata) Cluster(org.apache.kafka.common.Cluster) MockTime(org.apache.kafka.common.utils.MockTime) Time(org.apache.kafka.common.utils.Time) PartitionAssignor(org.apache.kafka.clients.consumer.internals.PartitionAssignor) MockTime(org.apache.kafka.common.utils.MockTime) MockClient(org.apache.kafka.clients.MockClient) Test(org.junit.Test)

Aggregations

MockTime (org.apache.kafka.common.utils.MockTime)363 Test (org.junit.Test)156 Test (org.junit.jupiter.api.Test)134 HashMap (java.util.HashMap)104 Time (org.apache.kafka.common.utils.Time)98 MockClient (org.apache.kafka.clients.MockClient)73 Cluster (org.apache.kafka.common.Cluster)73 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)65 Node (org.apache.kafka.common.Node)56 Metrics (org.apache.kafka.common.metrics.Metrics)56 TopicPartition (org.apache.kafka.common.TopicPartition)55 LogContext (org.apache.kafka.common.utils.LogContext)50 StreamsConfig (org.apache.kafka.streams.StreamsConfig)49 Before (org.junit.Before)38 Metadata (org.apache.kafka.clients.Metadata)32 MockScheduler (org.apache.kafka.common.utils.MockScheduler)31 MetadataResponse (org.apache.kafka.common.requests.MetadataResponse)30 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)30 ProducerMetadata (org.apache.kafka.clients.producer.internals.ProducerMetadata)29 Properties (java.util.Properties)27