Search in sources :

Example 76 with Consumer

use of com.yahoo.pulsar.client.api.Consumer in project pulsar by yahoo.

the class BrokerServiceThrottlingTest method testLookupThrottlingForClientByBrokerInternalRetry.

/**
     * This testcase make sure that once consumer lost connection with broker, it always reconnects with broker by
     * retrying on throttling-error exception also.
     * 
     * <pre>
     * 1. all consumers get connected 
     * 2. broker restarts with maxConcurrentLookupRequest = 1 
     * 3. consumers reconnect and some get TooManyRequestException and again retries
     * 4. eventually all consumers will successfully connect to broker
     * </pre>
     * 
     * @throws Exception
     */
@Test
public void testLookupThrottlingForClientByBrokerInternalRetry() throws Exception {
    // create configuration znode
    ZkUtils.createFullPathOptimistic(mockZookKeeper, BROKER_SERVICE_CONFIGURATION_PATH, "{}".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
    // Now, znode is created: set the watch and listener on the znode
    setWatchOnThrottlingZnode();
    final String topicName = "persistent://prop/usw/my-ns/newTopic";
    com.yahoo.pulsar.client.api.ClientConfiguration clientConf = new com.yahoo.pulsar.client.api.ClientConfiguration();
    clientConf.setStatsInterval(0, TimeUnit.SECONDS);
    clientConf.setIoThreads(20);
    clientConf.setConnectionsPerBroker(20);
    String lookupUrl = new URI("pulsar://localhost:" + BROKER_PORT).toString();
    PulsarClient pulsarClient = PulsarClient.create(lookupUrl, clientConf);
    upsertLookupPermits(100);
    ConsumerConfiguration consumerConfig = new ConsumerConfiguration();
    consumerConfig.setSubscriptionType(SubscriptionType.Shared);
    List<Consumer> consumers = Lists.newArrayList();
    ExecutorService executor = Executors.newFixedThreadPool(10);
    final int totalConsumers = 8;
    CountDownLatch latch = new CountDownLatch(totalConsumers);
    for (int i = 0; i < totalConsumers; i++) {
        executor.execute(() -> {
            try {
                consumers.add(pulsarClient.subscribe(topicName, "mysub", consumerConfig));
            } catch (PulsarClientException.TooManyLookupRequestException e) {
            // ok
            } catch (Exception e) {
                fail("it shouldn't failed");
            }
            latch.countDown();
        });
    }
    latch.await();
    stopBroker();
    conf.setMaxConcurrentLookupRequest(1);
    startBroker();
    // wait strategically for all consumers to reconnect
    for (int i = 0; i < 5; i++) {
        if (!areAllConsumersConnected(consumers)) {
            Thread.sleep(1000 + (i * 500));
        } else {
            break;
        }
    }
    int totalConnectedConsumers = 0;
    for (int i = 0; i < consumers.size(); i++) {
        if (((ConsumerImpl) consumers.get(i)).isConnected()) {
            totalConnectedConsumers++;
        }
        consumers.get(i).close();
    }
    assertEquals(totalConnectedConsumers, totalConsumers);
    pulsarClient.close();
}
Also used : CountDownLatch(java.util.concurrent.CountDownLatch) URI(java.net.URI) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) ConsumerImpl(com.yahoo.pulsar.client.impl.ConsumerImpl) Consumer(com.yahoo.pulsar.client.api.Consumer) ConsumerConfiguration(com.yahoo.pulsar.client.api.ConsumerConfiguration) ExecutorService(java.util.concurrent.ExecutorService) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) PulsarClient(com.yahoo.pulsar.client.api.PulsarClient) Test(org.testng.annotations.Test)

Example 77 with Consumer

use of com.yahoo.pulsar.client.api.Consumer in project pulsar by yahoo.

the class BrokerServiceThrottlingTest method testLookupThrottlingForClientByBroker.

/**
     * Verifies: Broker side throttling:
     * 
     * <pre>
     * 1. concurrent_consumer_creation > maxConcurrentLookupRequest at broker 
     * 2. few of the consumer creation must fail with TooManyLookupRequestException.
     * </pre>
     * 
     * @throws Exception
     */
@Test
public void testLookupThrottlingForClientByBroker() throws Exception {
    // create configuration znode
    ZkUtils.createFullPathOptimistic(mockZookKeeper, BROKER_SERVICE_CONFIGURATION_PATH, "{}".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
    // Now, znode is created: set the watch and listener on the znode
    setWatchOnThrottlingZnode();
    final String topicName = "persistent://prop/usw/my-ns/newTopic";
    com.yahoo.pulsar.client.api.ClientConfiguration clientConf = new com.yahoo.pulsar.client.api.ClientConfiguration();
    clientConf.setStatsInterval(0, TimeUnit.SECONDS);
    clientConf.setIoThreads(20);
    clientConf.setConnectionsPerBroker(20);
    String lookupUrl = new URI("pulsar://localhost:" + BROKER_PORT).toString();
    PulsarClient pulsarClient = PulsarClient.create(lookupUrl, clientConf);
    ConsumerConfiguration consumerConfig = new ConsumerConfiguration();
    consumerConfig.setSubscriptionType(SubscriptionType.Shared);
    int newPermits = 1;
    admin.brokers().updateDynamicConfiguration("maxConcurrentLookupRequest", Integer.toString(newPermits));
    // wait config to be updated
    for (int i = 0; i < 5; i++) {
        if (pulsar.getConfiguration().getMaxConcurrentLookupRequest() != newPermits) {
            Thread.sleep(100 + (i * 10));
        } else {
            break;
        }
    }
    List<Consumer> successfulConsumers = Lists.newArrayList();
    ExecutorService executor = Executors.newFixedThreadPool(10);
    final int totalConsumers = 20;
    CountDownLatch latch = new CountDownLatch(totalConsumers);
    for (int i = 0; i < totalConsumers; i++) {
        executor.execute(() -> {
            try {
                successfulConsumers.add(pulsarClient.subscribe(topicName, "mysub", consumerConfig));
            } catch (PulsarClientException.TooManyLookupRequestException e) {
            // ok
            } catch (Exception e) {
                fail("it shouldn't failed");
            }
            latch.countDown();
        });
    }
    latch.await();
    for (int i = 0; i < successfulConsumers.size(); i++) {
        successfulConsumers.get(i).close();
    }
    pulsarClient.close();
    assertNotEquals(successfulConsumers.size(), totalConsumers);
}
Also used : CountDownLatch(java.util.concurrent.CountDownLatch) URI(java.net.URI) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) Consumer(com.yahoo.pulsar.client.api.Consumer) ConsumerConfiguration(com.yahoo.pulsar.client.api.ConsumerConfiguration) ExecutorService(java.util.concurrent.ExecutorService) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) PulsarClient(com.yahoo.pulsar.client.api.PulsarClient) Test(org.testng.annotations.Test)

Example 78 with Consumer

use of com.yahoo.pulsar.client.api.Consumer in project pulsar by yahoo.

the class PersistentFailoverE2ETest method testSimpleConsumerEventsWithPartition.

@Test(enabled = false)
public void testSimpleConsumerEventsWithPartition() throws Exception {
    int numPartitions = 4;
    final String topicName = "persistent://prop/use/ns-abc/failover-topic2";
    final DestinationName destName = DestinationName.get(topicName);
    final String subName = "sub1";
    final int numMsgs = 100;
    Set<String> uniqueMessages = new HashSet<>();
    admin.persistentTopics().createPartitionedTopic(topicName, numPartitions);
    ProducerConfiguration producerConf = new ProducerConfiguration();
    producerConf.setMessageRoutingMode(MessageRoutingMode.RoundRobinPartition);
    ConsumerConfiguration consumerConf1 = new ConsumerConfiguration();
    consumerConf1.setSubscriptionType(SubscriptionType.Failover);
    consumerConf1.setConsumerName("1");
    ConsumerConfiguration consumerConf2 = new ConsumerConfiguration();
    consumerConf2.setSubscriptionType(SubscriptionType.Failover);
    consumerConf2.setConsumerName("2");
    // 1. two consumers on the same subscription
    Consumer consumer1 = pulsarClient.subscribe(topicName, subName, consumerConf1);
    Consumer consumer2 = pulsarClient.subscribe(topicName, subName, consumerConf2);
    PersistentTopic topicRef;
    topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(destName.getPartition(0).toString());
    PersistentDispatcherSingleActiveConsumer disp0 = (PersistentDispatcherSingleActiveConsumer) topicRef.getPersistentSubscription(subName).getDispatcher();
    topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(destName.getPartition(1).toString());
    PersistentDispatcherSingleActiveConsumer disp1 = (PersistentDispatcherSingleActiveConsumer) topicRef.getPersistentSubscription(subName).getDispatcher();
    topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(destName.getPartition(2).toString());
    PersistentDispatcherSingleActiveConsumer disp2 = (PersistentDispatcherSingleActiveConsumer) topicRef.getPersistentSubscription(subName).getDispatcher();
    topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(destName.getPartition(3).toString());
    PersistentDispatcherSingleActiveConsumer disp3 = (PersistentDispatcherSingleActiveConsumer) topicRef.getPersistentSubscription(subName).getDispatcher();
    List<CompletableFuture<MessageId>> futures = Lists.newArrayListWithCapacity(numMsgs);
    Producer producer = pulsarClient.createProducer(topicName, producerConf);
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // equal distribution between both consumers
    int totalMessages = 0;
    Message msg = null;
    while (true) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            break;
        }
        totalMessages++;
        consumer1.acknowledge(msg);
    }
    Assert.assertEquals(totalMessages, numMsgs / 2);
    while (true) {
        msg = consumer2.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            break;
        }
        totalMessages++;
        consumer2.acknowledge(msg);
    }
    Assert.assertEquals(totalMessages, numMsgs);
    Assert.assertEquals(disp0.getActiveConsumer().consumerName(), consumerConf1.getConsumerName());
    Assert.assertEquals(disp1.getActiveConsumer().consumerName(), consumerConf2.getConsumerName());
    Assert.assertEquals(disp2.getActiveConsumer().consumerName(), consumerConf1.getConsumerName());
    Assert.assertEquals(disp3.getActiveConsumer().consumerName(), consumerConf2.getConsumerName());
    totalMessages = 0;
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // add a consumer
    ConsumerConfiguration consumerConf3 = new ConsumerConfiguration();
    consumerConf3.setSubscriptionType(SubscriptionType.Failover);
    consumerConf3.setConsumerName("3");
    for (int i = 0; i < 20; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        uniqueMessages.add(new String(msg.getData()));
        consumer1.acknowledge(msg);
    }
    Consumer consumer3 = pulsarClient.subscribe(topicName, subName, consumerConf3);
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    int consumer1Messages = 0;
    while (true) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            Assert.assertEquals(consumer1Messages, 55);
            break;
        }
        consumer1Messages++;
        uniqueMessages.add(new String(msg.getData()));
        consumer1.acknowledge(msg);
    }
    int consumer2Messages = 0;
    while (true) {
        msg = consumer2.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            Assert.assertEquals(consumer2Messages, 50);
            break;
        }
        consumer2Messages++;
        uniqueMessages.add(new String(msg.getData()));
        consumer2.acknowledge(msg);
    }
    int consumer3Messages = 0;
    while (true) {
        msg = consumer3.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            Assert.assertEquals(consumer3Messages, 15, 10);
            break;
        }
        consumer3Messages++;
        uniqueMessages.add(new String(msg.getData()));
        consumer3.acknowledge(msg);
    }
    Assert.assertEquals(uniqueMessages.size(), numMsgs);
    Assert.assertEquals(disp0.getActiveConsumer().consumerName(), consumerConf1.getConsumerName());
    Assert.assertEquals(disp1.getActiveConsumer().consumerName(), consumerConf2.getConsumerName());
    Assert.assertEquals(disp2.getActiveConsumer().consumerName(), consumerConf3.getConsumerName());
    Assert.assertEquals(disp3.getActiveConsumer().consumerName(), consumerConf1.getConsumerName());
    uniqueMessages.clear();
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // remove a consumer
    for (int i = 0; i < 10; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        uniqueMessages.add(new String(msg.getData()));
        consumer1.acknowledge(msg);
    }
    consumer1.close();
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    consumer2Messages = 0;
    while (true) {
        msg = consumer2.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            Assert.assertEquals(consumer2Messages, 70, 5);
            break;
        }
        consumer2Messages++;
        uniqueMessages.add(new String(msg.getData()));
        consumer2.acknowledge(msg);
    }
    consumer3Messages = 0;
    while (true) {
        msg = consumer3.receive(1, TimeUnit.SECONDS);
        if (msg == null) {
            Assert.assertEquals(consumer3Messages, 70, 5);
            break;
        }
        consumer3Messages++;
        uniqueMessages.add(new String(msg.getData()));
        consumer3.acknowledge(msg);
    }
    Assert.assertEquals(uniqueMessages.size(), numMsgs);
    Assert.assertEquals(disp0.getActiveConsumer().consumerName(), consumerConf2.getConsumerName());
    Assert.assertEquals(disp1.getActiveConsumer().consumerName(), consumerConf3.getConsumerName());
    Assert.assertEquals(disp2.getActiveConsumer().consumerName(), consumerConf2.getConsumerName());
    Assert.assertEquals(disp3.getActiveConsumer().consumerName(), consumerConf3.getConsumerName());
    producer.close();
    consumer2.close();
    consumer3.unsubscribe();
    admin.persistentTopics().deletePartitionedTopic(topicName);
}
Also used : Message(com.yahoo.pulsar.client.api.Message) ProducerConfiguration(com.yahoo.pulsar.client.api.ProducerConfiguration) PersistentDispatcherSingleActiveConsumer(com.yahoo.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer) CompletableFuture(java.util.concurrent.CompletableFuture) Consumer(com.yahoo.pulsar.client.api.Consumer) PersistentDispatcherSingleActiveConsumer(com.yahoo.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer) Producer(com.yahoo.pulsar.client.api.Producer) PersistentTopic(com.yahoo.pulsar.broker.service.persistent.PersistentTopic) DestinationName(com.yahoo.pulsar.common.naming.DestinationName) ConsumerConfiguration(com.yahoo.pulsar.client.api.ConsumerConfiguration) HashSet(java.util.HashSet) Test(org.testng.annotations.Test)

Example 79 with Consumer

use of com.yahoo.pulsar.client.api.Consumer in project pulsar by yahoo.

the class PersistentFailoverE2ETest method testSimpleConsumerEventsWithoutPartition.

@Test
public void testSimpleConsumerEventsWithoutPartition() throws Exception {
    final String topicName = "persistent://prop/use/ns-abc/failover-topic1";
    final String subName = "sub1";
    final int numMsgs = 100;
    ConsumerConfiguration consumerConf1 = new ConsumerConfiguration();
    consumerConf1.setSubscriptionType(SubscriptionType.Failover);
    consumerConf1.setConsumerName("1");
    ConsumerConfiguration consumerConf2 = new ConsumerConfiguration();
    consumerConf2.setSubscriptionType(SubscriptionType.Failover);
    consumerConf2.setConsumerName("2");
    // 1. two consumers on the same subscription
    Consumer consumer1 = pulsarClient.subscribe(topicName, subName, consumerConf1);
    Consumer consumer2 = pulsarClient.subscribe(topicName, subName, consumerConf2);
    PersistentTopic topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(topicName);
    PersistentSubscription subRef = topicRef.getPersistentSubscription(subName);
    assertNotNull(topicRef);
    assertNotNull(subRef);
    // 2. validate basic dispatcher state
    assertTrue(subRef.getDispatcher().isConsumerConnected());
    assertEquals(subRef.getDispatcher().getType(), SubType.Failover);
    List<CompletableFuture<MessageId>> futures = Lists.newArrayListWithCapacity(numMsgs);
    Producer producer = pulsarClient.createProducer(topicName);
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    rolloverPerIntervalStats();
    assertEquals(subRef.getNumberOfEntriesInBacklog(), numMsgs);
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    // 3. consumer1 should have all the messages while consumer2 should have no messages
    Message msg = null;
    Assert.assertNull(consumer2.receive(1, TimeUnit.SECONDS));
    for (int i = 0; i < numMsgs; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer1.acknowledge(msg);
    }
    rolloverPerIntervalStats();
    // 4. messages deleted on individual acks
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    assertEquals(subRef.getNumberOfEntriesInBacklog(), 0);
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // 5. master consumer failure should resend unacked messages and new messages to another consumer
    for (int i = 0; i < 5; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer1.acknowledge(msg);
    }
    for (int i = 5; i < 10; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
    // do not ack
    }
    consumer1.close();
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    for (int i = 5; i < numMsgs; i++) {
        msg = consumer2.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer2.acknowledge(msg);
    }
    Assert.assertNull(consumer2.receive(1, TimeUnit.SECONDS));
    rolloverPerIntervalStats();
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    assertEquals(subRef.getNumberOfEntriesInBacklog(), 0);
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // 6. consumer subscription should send messages to the new consumer if its name is highest in the list
    for (int i = 0; i < 5; i++) {
        msg = consumer2.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer2.acknowledge(msg);
    }
    consumer1 = pulsarClient.subscribe(topicName, subName, consumerConf1);
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    for (int i = 5; i < numMsgs; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer1.acknowledge(msg);
    }
    Assert.assertNull(consumer1.receive(1, TimeUnit.SECONDS));
    rolloverPerIntervalStats();
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    assertEquals(subRef.getNumberOfEntriesInBacklog(), 0);
    for (int i = 0; i < numMsgs; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    futures.clear();
    // 7. consumer subscription should not send messages to the new consumer if its name is not highest in the list
    ConsumerConfiguration consumerConf3 = new ConsumerConfiguration();
    consumerConf3.setSubscriptionType(SubscriptionType.Failover);
    consumerConf3.setConsumerName("3");
    for (int i = 0; i < 5; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer1.acknowledge(msg);
    }
    Consumer consumer3 = pulsarClient.subscribe(topicName, subName, consumerConf3);
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    Assert.assertNull(consumer3.receive(1, TimeUnit.SECONDS));
    for (int i = 5; i < numMsgs; i++) {
        msg = consumer1.receive(1, TimeUnit.SECONDS);
        Assert.assertNotNull(msg);
        Assert.assertEquals(new String(msg.getData()), "my-message-" + i);
        consumer1.acknowledge(msg);
    }
    rolloverPerIntervalStats();
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    assertEquals(subRef.getNumberOfEntriesInBacklog(), 0);
    // 8. unsubscribe not allowed if multiple consumers connected
    try {
        consumer1.unsubscribe();
        fail("should fail");
    } catch (PulsarClientException e) {
    // ok
    }
    // 9. unsubscribe allowed if there is a lone consumer
    consumer1.close();
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    consumer2.close();
    Thread.sleep(CONSUMER_ADD_OR_REMOVE_WAIT_TIME);
    try {
        consumer3.unsubscribe();
    } catch (PulsarClientException e) {
        fail("Should not fail", e);
    }
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    subRef = topicRef.getPersistentSubscription(subName);
    assertNull(subRef);
    producer.close();
    consumer3.close();
    admin.persistentTopics().delete(topicName);
}
Also used : CompletableFuture(java.util.concurrent.CompletableFuture) Consumer(com.yahoo.pulsar.client.api.Consumer) PersistentDispatcherSingleActiveConsumer(com.yahoo.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer) Producer(com.yahoo.pulsar.client.api.Producer) Message(com.yahoo.pulsar.client.api.Message) PersistentTopic(com.yahoo.pulsar.broker.service.persistent.PersistentTopic) ConsumerConfiguration(com.yahoo.pulsar.client.api.ConsumerConfiguration) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) PersistentSubscription(com.yahoo.pulsar.broker.service.persistent.PersistentSubscription) Test(org.testng.annotations.Test)

Example 80 with Consumer

use of com.yahoo.pulsar.client.api.Consumer in project pulsar by yahoo.

the class PersistentQueueE2ETest method testSimpleConsumerEvents.

@Test
public void testSimpleConsumerEvents() throws Exception {
    final String topicName = "persistent://prop/use/ns-abc/shared-topic1";
    final String subName = "sub1";
    final int numMsgs = 100;
    ConsumerConfiguration conf = new ConsumerConfiguration();
    conf.setSubscriptionType(SubscriptionType.Shared);
    // 1. two consumers on the same subscription
    Consumer consumer1 = pulsarClient.subscribe(topicName, subName, conf);
    Consumer consumer2 = pulsarClient.subscribe(topicName, subName, conf);
    PersistentTopic topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(topicName);
    PersistentSubscription subRef = topicRef.getPersistentSubscription(subName);
    assertNotNull(topicRef);
    assertNotNull(subRef);
    // 2. validate basic dispatcher state
    assertTrue(subRef.getDispatcher().isConsumerConnected());
    assertEquals(subRef.getDispatcher().getType(), SubType.Shared);
    List<CompletableFuture<MessageId>> futures = Lists.newArrayListWithCapacity(numMsgs * 2);
    Producer producer = pulsarClient.createProducer(topicName);
    for (int i = 0; i < numMsgs * 2; i++) {
        String message = "my-message-" + i;
        futures.add(producer.sendAsync(message.getBytes()));
    }
    FutureUtil.waitForAll(futures).get();
    rolloverPerIntervalStats();
    assertEquals(subRef.getNumberOfEntriesInBacklog(), numMsgs * 2);
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    // both consumers will together consumer all messages
    Message msg;
    Consumer c = consumer1;
    while (true) {
        try {
            msg = c.receive(1, TimeUnit.SECONDS);
            c.acknowledge(msg);
        } catch (PulsarClientException e) {
            if (c.equals(consumer1)) {
                c = consumer2;
            } else {
                break;
            }
        }
    }
    rolloverPerIntervalStats();
    // 3. messages deleted on individual acks
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    assertEquals(subRef.getNumberOfEntriesInBacklog(), 0);
    // 4. shared consumer unsubscribe not allowed
    try {
        consumer1.unsubscribe();
        fail("should fail");
    } catch (PulsarClientException e) {
    // ok
    }
    // 5. cumulative acks disabled
    consumer1.close();
    producer.send("message".getBytes());
    msg = consumer2.receive();
    try {
        consumer2.acknowledgeCumulative(msg);
        fail("Should fail");
    } catch (PulsarClientException e) {
        assertTrue(e instanceof PulsarClientException.InvalidConfigurationException);
    }
    // 6. unsubscribe allowed if this is the lone consumer
    try {
        consumer2.unsubscribe();
    } catch (PulsarClientException e) {
        fail("Should not fail");
    }
    Thread.sleep(ASYNC_EVENT_COMPLETION_WAIT);
    subRef = topicRef.getPersistentSubscription(subName);
    assertNull(subRef);
    producer.close();
    consumer2.close();
    admin.persistentTopics().delete(topicName);
}
Also used : CompletableFuture(java.util.concurrent.CompletableFuture) Consumer(com.yahoo.pulsar.client.api.Consumer) Producer(com.yahoo.pulsar.client.api.Producer) Message(com.yahoo.pulsar.client.api.Message) PersistentTopic(com.yahoo.pulsar.broker.service.persistent.PersistentTopic) ConsumerConfiguration(com.yahoo.pulsar.client.api.ConsumerConfiguration) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) PersistentSubscription(com.yahoo.pulsar.broker.service.persistent.PersistentSubscription) Test(org.testng.annotations.Test)

Aggregations

Consumer (com.yahoo.pulsar.client.api.Consumer)109 Test (org.testng.annotations.Test)99 ConsumerConfiguration (com.yahoo.pulsar.client.api.ConsumerConfiguration)75 Producer (com.yahoo.pulsar.client.api.Producer)71 Message (com.yahoo.pulsar.client.api.Message)62 PersistentTopic (com.yahoo.pulsar.broker.service.persistent.PersistentTopic)34 PulsarClientException (com.yahoo.pulsar.client.api.PulsarClientException)34 PulsarClient (com.yahoo.pulsar.client.api.PulsarClient)33 ProducerConfiguration (com.yahoo.pulsar.client.api.ProducerConfiguration)26 CompletableFuture (java.util.concurrent.CompletableFuture)21 ClientConfiguration (com.yahoo.pulsar.client.api.ClientConfiguration)20 PulsarAdminException (com.yahoo.pulsar.client.admin.PulsarAdminException)14 MockedPulsarServiceBaseTest (com.yahoo.pulsar.broker.auth.MockedPulsarServiceBaseTest)13 PersistentTopicStats (com.yahoo.pulsar.common.policies.data.PersistentTopicStats)13 HashSet (java.util.HashSet)12 PersistentSubscription (com.yahoo.pulsar.broker.service.persistent.PersistentSubscription)11 MessageId (com.yahoo.pulsar.client.api.MessageId)11 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)10 CountDownLatch (java.util.concurrent.CountDownLatch)8 BacklogQuota (com.yahoo.pulsar.common.policies.data.BacklogQuota)7