Search in sources :

Example 86 with TopicName

use of org.apache.pulsar.common.naming.TopicName in project incubator-pulsar by apache.

the class PulsarClientImpl method patternTopicSubscribeAsync.

private <T> CompletableFuture<Consumer<T>> patternTopicSubscribeAsync(ConsumerConfigurationData<T> conf, Schema<T> schema) {
    String regex = conf.getTopicsPattern().pattern();
    TopicName destination = TopicName.get(regex);
    NamespaceName namespaceName = destination.getNamespaceObject();
    CompletableFuture<Consumer<T>> consumerSubscribedFuture = new CompletableFuture<>();
    lookup.getTopicsUnderNamespace(namespaceName).thenAccept(topics -> {
        if (log.isDebugEnabled()) {
            log.debug("Get topics under namespace {}, topics.size: {}", namespaceName.toString(), topics.size());
            topics.forEach(topicName -> log.debug("Get topics under namespace {}, topic: {}", namespaceName.toString(), topicName));
        }
        List<String> topicsList = topicsPatternFilter(topics, conf.getTopicsPattern());
        conf.getTopicNames().addAll(topicsList);
        ConsumerBase<T> consumer = new PatternTopicsConsumerImpl<>(conf.getTopicsPattern(), PulsarClientImpl.this, conf, externalExecutorProvider.getExecutor(), consumerSubscribedFuture, schema);
        synchronized (consumers) {
            consumers.put(consumer, Boolean.TRUE);
        }
    }).exceptionally(ex -> {
        log.warn("[{}] Failed to get topics under namespace", namespaceName);
        consumerSubscribedFuture.completeExceptionally(ex);
        return null;
    });
    return consumerSubscribedFuture;
}
Also used : DefaultThreadFactory(io.netty.util.concurrent.DefaultThreadFactory) TopicName(org.apache.pulsar.common.naming.TopicName) ConsumerConfigurationData(org.apache.pulsar.client.impl.conf.ConsumerConfigurationData) ReaderConfigurationData(org.apache.pulsar.client.impl.conf.ReaderConfigurationData) ClientConfiguration(org.apache.pulsar.client.api.ClientConfiguration) Producer(org.apache.pulsar.client.api.Producer) LoggerFactory(org.slf4j.LoggerFactory) CompletableFuture(java.util.concurrent.CompletableFuture) ConsumerBuilder(org.apache.pulsar.client.api.ConsumerBuilder) PartitionedTopicMetadata(org.apache.pulsar.common.partition.PartitionedTopicMetadata) AtomicReference(java.util.concurrent.atomic.AtomicReference) ProducerBuilder(org.apache.pulsar.client.api.ProducerBuilder) Lists(com.google.common.collect.Lists) ExecutorProvider(org.apache.pulsar.client.util.ExecutorProvider) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) ThreadFactory(java.util.concurrent.ThreadFactory) PulsarClient(org.apache.pulsar.client.api.PulsarClient) TopicDomain(org.apache.pulsar.common.naming.TopicDomain) PulsarClientException(org.apache.pulsar.client.api.PulsarClientException) ExecutorService(java.util.concurrent.ExecutorService) EventLoopGroup(io.netty.channel.EventLoopGroup) IdentityHashMap(java.util.IdentityHashMap) EventLoopUtil(org.apache.pulsar.common.util.netty.EventLoopUtil) Logger(org.slf4j.Logger) ReaderConfiguration(org.apache.pulsar.client.api.ReaderConfiguration) Reader(org.apache.pulsar.client.api.Reader) ConsumerConfiguration(org.apache.pulsar.client.api.ConsumerConfiguration) SubscriptionType(org.apache.pulsar.client.api.SubscriptionType) Maps(com.google.common.collect.Maps) Collectors(java.util.stream.Collectors) ProducerConfiguration(org.apache.pulsar.client.api.ProducerConfiguration) ProducerConfigurationData(org.apache.pulsar.client.impl.conf.ProducerConfigurationData) Schema(org.apache.pulsar.client.api.Schema) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) AtomicLong(java.util.concurrent.atomic.AtomicLong) Consumer(org.apache.pulsar.client.api.Consumer) List(java.util.List) FutureUtil(org.apache.pulsar.common.util.FutureUtil) MessageId(org.apache.pulsar.client.api.MessageId) StringUtils.isBlank(org.apache.commons.lang3.StringUtils.isBlank) ClientConfigurationData(org.apache.pulsar.client.impl.conf.ClientConfigurationData) HashedWheelTimer(io.netty.util.HashedWheelTimer) Timer(io.netty.util.Timer) ReaderBuilder(org.apache.pulsar.client.api.ReaderBuilder) Pattern(java.util.regex.Pattern) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) CompletableFuture(java.util.concurrent.CompletableFuture) Consumer(org.apache.pulsar.client.api.Consumer) List(java.util.List) TopicName(org.apache.pulsar.common.naming.TopicName)

Example 87 with TopicName

use of org.apache.pulsar.common.naming.TopicName in project incubator-pulsar by apache.

the class TopicsConsumerImpl method subscribeAsync.

// subscribe one more given topic
public CompletableFuture<Void> subscribeAsync(String topicName) {
    if (!topicNameValid(topicName)) {
        return FutureUtil.failedFuture(new PulsarClientException.AlreadyClosedException("Topic name not valid"));
    }
    if (getState() == State.Closing || getState() == State.Closed) {
        return FutureUtil.failedFuture(new PulsarClientException.AlreadyClosedException("Topics Consumer was already closed"));
    }
    CompletableFuture<Void> subscribeResult = new CompletableFuture<>();
    final AtomicInteger partitionNumber = new AtomicInteger(0);
    client.getPartitionedTopicMetadata(topicName).thenAccept(metadata -> {
        if (log.isDebugEnabled()) {
            log.debug("Received topic {} metadata.partitions: {}", topicName, metadata.partitions);
        }
        List<CompletableFuture<Consumer<T>>> futureList;
        if (metadata.partitions > 1) {
            this.topics.putIfAbsent(topicName, metadata.partitions);
            numberTopicPartitions.addAndGet(metadata.partitions);
            partitionNumber.addAndGet(metadata.partitions);
            futureList = IntStream.range(0, partitionNumber.get()).mapToObj(partitionIndex -> {
                String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
                ConsumerImpl<T> newConsumer = new ConsumerImpl<>(client, partitionName, internalConfig, client.externalExecutorProvider().getExecutor(), partitionIndex, subFuture, schema);
                consumers.putIfAbsent(newConsumer.getTopic(), newConsumer);
                return subFuture;
            }).collect(Collectors.toList());
        } else {
            this.topics.putIfAbsent(topicName, 1);
            numberTopicPartitions.incrementAndGet();
            partitionNumber.incrementAndGet();
            CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
            ConsumerImpl<T> newConsumer = new ConsumerImpl<>(client, topicName, internalConfig, client.externalExecutorProvider().getExecutor(), 0, subFuture, schema);
            consumers.putIfAbsent(newConsumer.getTopic(), newConsumer);
            futureList = Collections.singletonList(subFuture);
        }
        FutureUtil.waitForAll(futureList).thenAccept(finalFuture -> {
            try {
                if (numberTopicPartitions.get() > maxReceiverQueueSize) {
                    setMaxReceiverQueueSize(numberTopicPartitions.get());
                }
                int numTopics = this.topics.values().stream().mapToInt(Integer::intValue).sum();
                checkState(numberTopicPartitions.get() == numTopics, "numberTopicPartitions " + numberTopicPartitions.get() + " not equals expected: " + numTopics);
                // We have successfully created new consumers, so we can start receiving messages for them
                startReceivingMessages(consumers.values().stream().filter(consumer1 -> {
                    String consumerTopicName = consumer1.getTopic();
                    if (TopicName.get(consumerTopicName).getPartitionedTopicName().equals(TopicName.get(topicName).getPartitionedTopicName().toString())) {
                        return true;
                    } else {
                        return false;
                    }
                }).collect(Collectors.toList()));
                subscribeResult.complete(null);
                log.info("[{}] [{}] Success subscribe new topic {} in topics consumer, numberTopicPartitions {}", topic, subscription, topicName, numberTopicPartitions.get());
                if (this.namespaceName == null) {
                    this.namespaceName = TopicName.get(topicName).getNamespaceObject();
                }
                return;
            } catch (PulsarClientException e) {
                handleSubscribeOneTopicError(topicName, e);
                subscribeResult.completeExceptionally(e);
            }
        }).exceptionally(ex -> {
            handleSubscribeOneTopicError(topicName, ex);
            subscribeResult.completeExceptionally(ex);
            return null;
        });
    }).exceptionally(ex1 -> {
        log.warn("[{}] Failed to get partitioned topic metadata: {}", topicName, ex1.getMessage());
        subscribeResult.completeExceptionally(ex1);
        return null;
    });
    return subscribeResult;
}
Also used : IntStream(java.util.stream.IntStream) TopicName(org.apache.pulsar.common.naming.TopicName) ConsumerConfigurationData(org.apache.pulsar.client.impl.conf.ConsumerConfigurationData) LoggerFactory(org.slf4j.LoggerFactory) CompletableFuture(java.util.concurrent.CompletableFuture) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) Message(org.apache.pulsar.client.api.Message) HashSet(java.util.HashSet) ConsumerStats(org.apache.pulsar.client.api.ConsumerStats) ConsumerName(org.apache.pulsar.client.util.ConsumerName) AckType(org.apache.pulsar.common.api.proto.PulsarApi.CommandAck.AckType) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Map(java.util.Map) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) PulsarClientException(org.apache.pulsar.client.api.PulsarClientException) ExecutorService(java.util.concurrent.ExecutorService) ReadWriteLock(java.util.concurrent.locks.ReadWriteLock) Logger(org.slf4j.Logger) Collection(java.util.Collection) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) SubscriptionType(org.apache.pulsar.client.api.SubscriptionType) Collectors(java.util.stream.Collectors) Preconditions.checkState(com.google.common.base.Preconditions.checkState) Schema(org.apache.pulsar.client.api.Schema) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) Consumer(org.apache.pulsar.client.api.Consumer) List(java.util.List) FutureUtil(org.apache.pulsar.common.util.FutureUtil) MessageId(org.apache.pulsar.client.api.MessageId) Optional(java.util.Optional) Collections(java.util.Collections) ConcurrentLinkedQueue(java.util.concurrent.ConcurrentLinkedQueue) CompletableFuture(java.util.concurrent.CompletableFuture) Consumer(org.apache.pulsar.client.api.Consumer) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) PulsarClientException(org.apache.pulsar.client.api.PulsarClientException) List(java.util.List)

Example 88 with TopicName

use of org.apache.pulsar.common.naming.TopicName in project incubator-pulsar by apache.

the class ReplicatorTest method testResetCursorNotFail.

@Test(enabled = true, timeOut = 30000)
public void testResetCursorNotFail() throws Exception {
    log.info("--- Starting ReplicatorTest::testResetCursorNotFail ---");
    // This test is to verify that reset cursor fails on global topic
    SortedSet<String> testDests = new TreeSet<String>();
    List<Future<Void>> results = Lists.newArrayList();
    for (int i = 0; i < 1; i++) {
        final TopicName dest = TopicName.get(String.format("persistent://pulsar/global/ns/resetrepltopic-%d", i));
        testDests.add(dest.toString());
        results.add(executor.submit(new Callable<Void>() {

            @Override
            public Void call() throws Exception {
                MessageProducer producer1 = new MessageProducer(url1, dest);
                log.info("--- Starting producer --- " + url1);
                MessageConsumer consumer1 = new MessageConsumer(url1, dest);
                log.info("--- Starting Consumer --- " + url1);
                // Produce from cluster1 and consume from the rest
                producer1.produce(2);
                consumer1.receive(2);
                producer1.close();
                consumer1.close();
                return null;
            }
        }));
    }
    for (Future<Void> result : results) {
        try {
            result.get();
        } catch (Exception e) {
            log.error("exception in getting future result ", e);
            fail(String.format("replication test failed with %s exception", e.getMessage()));
        }
    }
    admin1.persistentTopics().resetCursor(testDests.first(), "sub-id", System.currentTimeMillis());
}
Also used : Callable(java.util.concurrent.Callable) NamingException(org.apache.pulsar.broker.service.BrokerServiceException.NamingException) PreconditionFailedException(org.apache.pulsar.client.admin.PulsarAdminException.PreconditionFailedException) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) CursorAlreadyClosedException(org.apache.bookkeeper.mledger.ManagedLedgerException.CursorAlreadyClosedException) TopicName(org.apache.pulsar.common.naming.TopicName) TreeSet(java.util.TreeSet) Future(java.util.concurrent.Future) CompletableFuture(java.util.concurrent.CompletableFuture) Test(org.testng.annotations.Test)

Example 89 with TopicName

use of org.apache.pulsar.common.naming.TopicName in project incubator-pulsar by apache.

the class ReplicatorTest method testReplicatePeekAndSkip.

@Test(timeOut = 30000)
public void testReplicatePeekAndSkip() throws Exception {
    SortedSet<String> testDests = new TreeSet<String>();
    final TopicName dest = TopicName.get("persistent://pulsar/global/ns/peekAndSeekTopic");
    testDests.add(dest.toString());
    MessageProducer producer1 = new MessageProducer(url1, dest);
    MessageConsumer consumer1 = new MessageConsumer(url3, dest);
    // Produce from cluster1 and consume from the rest
    producer1.produce(2);
    producer1.close();
    PersistentTopic topic = (PersistentTopic) pulsar1.getBrokerService().getTopicReference(dest.toString());
    PersistentReplicator replicator = (PersistentReplicator) topic.getReplicators().get(topic.getReplicators().keys().get(0));
    replicator.skipMessages(2);
    CompletableFuture<Entry> result = replicator.peekNthMessage(1);
    Entry entry = result.get(50, TimeUnit.MILLISECONDS);
    assertNull(entry);
    consumer1.close();
}
Also used : Entry(org.apache.bookkeeper.mledger.Entry) PersistentReplicator(org.apache.pulsar.broker.service.persistent.PersistentReplicator) TreeSet(java.util.TreeSet) PersistentTopic(org.apache.pulsar.broker.service.persistent.PersistentTopic) TopicName(org.apache.pulsar.common.naming.TopicName) Test(org.testng.annotations.Test)

Example 90 with TopicName

use of org.apache.pulsar.common.naming.TopicName in project incubator-pulsar by apache.

the class ReplicatorTest method testConfigChange.

@Test(enabled = true, timeOut = 30000)
public void testConfigChange() throws Exception {
    log.info("--- Starting ReplicatorTest::testConfigChange ---");
    // This test is to verify that the config change on global namespace is successfully applied in broker during
    // runtime.
    // Run a set of producer tasks to create the topics
    List<Future<Void>> results = Lists.newArrayList();
    for (int i = 0; i < 10; i++) {
        final TopicName dest = TopicName.get(String.format("persistent://pulsar/global/ns/topic-%d", i));
        results.add(executor.submit(new Callable<Void>() {

            @Override
            public Void call() throws Exception {
                MessageProducer producer = new MessageProducer(url1, dest);
                log.info("--- Starting producer --- " + url1);
                MessageConsumer consumer = new MessageConsumer(url1, dest);
                log.info("--- Starting Consumer --- " + url1);
                producer.produce(2);
                consumer.receive(2);
                producer.close();
                consumer.close();
                return null;
            }
        }));
    }
    for (Future<Void> result : results) {
        try {
            result.get();
        } catch (Exception e) {
            log.error("exception in getting future result ", e);
            fail(String.format("replication test failed with %s exception", e.getMessage()));
        }
    }
    Thread.sleep(1000L);
    // Make sure that the internal replicators map contains remote cluster info
    ConcurrentOpenHashMap<String, PulsarClient> replicationClients1 = ns1.getReplicationClients();
    ConcurrentOpenHashMap<String, PulsarClient> replicationClients2 = ns2.getReplicationClients();
    ConcurrentOpenHashMap<String, PulsarClient> replicationClients3 = ns3.getReplicationClients();
    Assert.assertNotNull(replicationClients1.get("r2"));
    Assert.assertNotNull(replicationClients1.get("r3"));
    Assert.assertNotNull(replicationClients2.get("r1"));
    Assert.assertNotNull(replicationClients2.get("r3"));
    Assert.assertNotNull(replicationClients3.get("r1"));
    Assert.assertNotNull(replicationClients3.get("r2"));
    // Case 1: Update the global namespace replication configuration to only contains the local cluster itself
    admin1.namespaces().setNamespaceReplicationClusters("pulsar/global/ns", Lists.newArrayList("r1"));
    // Wait for config changes to be updated.
    Thread.sleep(1000L);
    // Make sure that the internal replicators map still contains remote cluster info
    Assert.assertNotNull(replicationClients1.get("r2"));
    Assert.assertNotNull(replicationClients1.get("r3"));
    Assert.assertNotNull(replicationClients2.get("r1"));
    Assert.assertNotNull(replicationClients2.get("r3"));
    Assert.assertNotNull(replicationClients3.get("r1"));
    Assert.assertNotNull(replicationClients3.get("r2"));
    // Case 2: Update the configuration back
    admin1.namespaces().setNamespaceReplicationClusters("pulsar/global/ns", Lists.newArrayList("r1", "r2", "r3"));
    // Wait for config changes to be updated.
    Thread.sleep(1000L);
    // Make sure that the internal replicators map still contains remote cluster info
    Assert.assertNotNull(replicationClients1.get("r2"));
    Assert.assertNotNull(replicationClients1.get("r3"));
    Assert.assertNotNull(replicationClients2.get("r1"));
    Assert.assertNotNull(replicationClients2.get("r3"));
    Assert.assertNotNull(replicationClients3.get("r1"));
    Assert.assertNotNull(replicationClients3.get("r2"));
// Case 3: TODO: Once automatic cleanup is implemented, add tests case to verify auto removal of clusters
}
Also used : Callable(java.util.concurrent.Callable) NamingException(org.apache.pulsar.broker.service.BrokerServiceException.NamingException) PreconditionFailedException(org.apache.pulsar.client.admin.PulsarAdminException.PreconditionFailedException) ManagedLedgerException(org.apache.bookkeeper.mledger.ManagedLedgerException) CursorAlreadyClosedException(org.apache.bookkeeper.mledger.ManagedLedgerException.CursorAlreadyClosedException) TopicName(org.apache.pulsar.common.naming.TopicName) Future(java.util.concurrent.Future) CompletableFuture(java.util.concurrent.CompletableFuture) PulsarClient(org.apache.pulsar.client.api.PulsarClient) Test(org.testng.annotations.Test)

Aggregations

TopicName (org.apache.pulsar.common.naming.TopicName)127 Test (org.testng.annotations.Test)54 CompletableFuture (java.util.concurrent.CompletableFuture)43 WebTarget (javax.ws.rs.client.WebTarget)32 NamespaceBundle (org.apache.pulsar.common.naming.NamespaceBundle)23 NamespaceName (org.apache.pulsar.common.naming.NamespaceName)23 Logger (org.slf4j.Logger)23 LoggerFactory (org.slf4j.LoggerFactory)23 List (java.util.List)22 ManagedLedgerException (org.apache.bookkeeper.mledger.ManagedLedgerException)22 Map (java.util.Map)20 ExecutionException (java.util.concurrent.ExecutionException)20 TimeUnit (java.util.concurrent.TimeUnit)20 NamingException (org.apache.pulsar.broker.service.BrokerServiceException.NamingException)18 Field (java.lang.reflect.Field)17 PersistentTopic (org.apache.pulsar.broker.service.persistent.PersistentTopic)17 PulsarClientException (org.apache.pulsar.client.api.PulsarClientException)17 ByteBuf (io.netty.buffer.ByteBuf)15 Set (java.util.Set)15 ServerMetadataException (org.apache.pulsar.broker.service.BrokerServiceException.ServerMetadataException)14