Search in sources :

Example 1 with ZooKeeperClientFactory

use of org.apache.pulsar.zookeeper.ZooKeeperClientFactory in project incubator-pulsar by apache.

the class ZookeeperClientFactoryImplTest method testZKCreationRW.

@Test
void testZKCreationRW() throws Exception {
    ZooKeeperClientFactory zkf = new ZookeeperClientFactoryImpl();
    CompletableFuture<ZooKeeper> zkFuture = zkf.create("127.0.0.1:" + LOCAL_ZOOKEEPER_PORT, SessionType.ReadWrite, (int) ZOOKEEPER_SESSION_TIMEOUT_MILLIS);
    localZkc = zkFuture.get(ZOOKEEPER_SESSION_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS);
    assertTrue(localZkc.getState().isConnected());
    assertNotEquals(localZkc.getState(), States.CONNECTEDREADONLY);
    localZkc.close();
}
Also used : ZooKeeper(org.apache.zookeeper.ZooKeeper) ZookeeperClientFactoryImpl(org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl) ZooKeeperClientFactory(org.apache.pulsar.zookeeper.ZooKeeperClientFactory) Test(org.testng.annotations.Test)

Example 2 with ZooKeeperClientFactory

use of org.apache.pulsar.zookeeper.ZooKeeperClientFactory in project incubator-pulsar by apache.

the class ZookeeperClientFactoryImplTest method testZKCreationRO.

@Test
void testZKCreationRO() throws Exception {
    ZooKeeperClientFactory zkf = new ZookeeperClientFactoryImpl();
    CompletableFuture<ZooKeeper> zkFuture = zkf.create("127.0.0.1:" + LOCAL_ZOOKEEPER_PORT, SessionType.AllowReadOnly, (int) ZOOKEEPER_SESSION_TIMEOUT_MILLIS);
    localZkc = zkFuture.get(ZOOKEEPER_SESSION_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS);
    assertTrue(localZkc.getState().isConnected());
    localZkc.close();
}
Also used : ZooKeeper(org.apache.zookeeper.ZooKeeper) ZookeeperClientFactoryImpl(org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl) ZooKeeperClientFactory(org.apache.pulsar.zookeeper.ZooKeeperClientFactory) Test(org.testng.annotations.Test)

Example 3 with ZooKeeperClientFactory

use of org.apache.pulsar.zookeeper.ZooKeeperClientFactory in project incubator-pulsar by apache.

the class ZooKeeperClientAspectJTest method testZkConnected.

@Test
public void testZkConnected() throws Exception {
    OrderedScheduler executor = OrderedScheduler.newSchedulerBuilder().build();
    try {
        ZooKeeperClientFactory zkf = new ZookeeperBkClientFactoryImpl(executor);
        CompletableFuture<ZooKeeper> zkFuture = zkf.create("127.0.0.1:" + LOCAL_ZOOKEEPER_PORT, SessionType.ReadWrite, (int) ZOOKEEPER_SESSION_TIMEOUT_MILLIS);
        localZkc = zkFuture.get(ZOOKEEPER_SESSION_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS);
        assertTrue(localZkc.getState().isConnected());
        assertNotEquals(localZkc.getState(), States.CONNECTEDREADONLY);
    } finally {
        if (localZkc != null) {
            localZkc.close();
        }
        executor.shutdown();
    }
}
Also used : ZooKeeper(org.apache.zookeeper.ZooKeeper) ZookeeperBkClientFactoryImpl(org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl) OrderedScheduler(org.apache.bookkeeper.common.util.OrderedScheduler) ZooKeeperClientFactory(org.apache.pulsar.zookeeper.ZooKeeperClientFactory) Test(org.testng.annotations.Test)

Example 4 with ZooKeeperClientFactory

use of org.apache.pulsar.zookeeper.ZooKeeperClientFactory in project incubator-pulsar by apache.

the class ZooKeeperClientAspectJTest method testZkOpStatsMetrics.

/**
 * Verifies that aspect-advice calculates the latency of of zk-operation and updates PulsarStats
 *
 * @throws Exception
 */
@Test(enabled = false, timeOut = 7000)
public void testZkOpStatsMetrics() throws Exception {
    OrderedScheduler executor = OrderedScheduler.newSchedulerBuilder().build();
    ZooKeeperClientFactory zkf = new ZookeeperBkClientFactoryImpl(executor);
    CompletableFuture<ZooKeeper> zkFuture = zkf.create("127.0.0.1:" + LOCAL_ZOOKEEPER_PORT, SessionType.ReadWrite, (int) ZOOKEEPER_SESSION_TIMEOUT_MILLIS);
    localZkc = zkFuture.get(ZOOKEEPER_SESSION_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS);
    MockPulsar mockPulsar = new MockPulsar(localZkc);
    mockPulsar.setup();
    try {
        PulsarClient pulsarClient = mockPulsar.getClient();
        PulsarService pulsar = mockPulsar.getPulsar();
        pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic1").create();
        Metrics zkOpMetric = getMetric(pulsar, "zk_write_latency");
        Assert.assertNotNull(zkOpMetric);
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_rate_s"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_95percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_99_99_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_99_9_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_99_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_mean_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_write_time_median_ms"));
        zkOpMetric = getMetric(pulsar, "zk_read_latency");
        Assert.assertNotNull(zkOpMetric);
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_rate_s"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_95percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_99_99_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_99_9_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_99_percentile_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_mean_ms"));
        Assert.assertTrue(zkOpMetric.getMetrics().containsKey("brk_zk_read_time_median_ms"));
        CountDownLatch createLatch = new CountDownLatch(1);
        CountDownLatch deleteLatch = new CountDownLatch(1);
        CountDownLatch readLatch = new CountDownLatch(1);
        CountDownLatch existLatch = new CountDownLatch(1);
        localZkc.create("/createTest", "data".getBytes(), Acl, CreateMode.EPHEMERAL, (rc, path, ctx, name) -> {
            createLatch.countDown();
        }, "create");
        localZkc.delete("/deleteTest", -1, (rc, path, ctx) -> {
            deleteLatch.countDown();
        }, "delete");
        localZkc.exists("/createTest", null, (int rc, String path, Object ctx, Stat stat) -> {
            existLatch.countDown();
        }, null);
        localZkc.getData("/createTest", null, (int rc, String path, Object ctx, byte[] data, Stat stat) -> {
            readLatch.countDown();
        }, null);
        createLatch.await();
        deleteLatch.await();
        existLatch.await();
        readLatch.await();
        Thread.sleep(10);
        BrokerService brokerService = pulsar.getBrokerService();
        brokerService.updateRates();
        List<Metrics> metrics = brokerService.getTopicMetrics();
        AtomicDouble writeRate = new AtomicDouble();
        AtomicDouble readRate = new AtomicDouble();
        metrics.forEach(m -> {
            if ("zk_write_latency".equalsIgnoreCase(m.getDimension("metric"))) {
                writeRate.set((double) m.getMetrics().get("brk_zk_write_latency_rate_s"));
            } else if ("zk_read_latency".equalsIgnoreCase(m.getDimension("metric"))) {
                readRate.set((double) m.getMetrics().get("brk_zk_read_latency_rate_s"));
            }
        });
        Assert.assertTrue(readRate.get() > 0);
        Assert.assertTrue(writeRate.get() > 0);
    } finally {
        mockPulsar.cleanup();
        if (localZkc != null) {
            localZkc.close();
        }
        executor.shutdown();
    }
}
Also used : AtomicDouble(com.google.common.util.concurrent.AtomicDouble) CountDownLatch(java.util.concurrent.CountDownLatch) ZooKeeperClientFactory(org.apache.pulsar.zookeeper.ZooKeeperClientFactory) Metrics(org.apache.pulsar.common.stats.Metrics) ZooKeeper(org.apache.zookeeper.ZooKeeper) Stat(org.apache.zookeeper.data.Stat) PulsarService(org.apache.pulsar.broker.PulsarService) ZookeeperBkClientFactoryImpl(org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl) PulsarClient(org.apache.pulsar.client.api.PulsarClient) BrokerService(org.apache.pulsar.broker.service.BrokerService) OrderedScheduler(org.apache.bookkeeper.common.util.OrderedScheduler) Test(org.testng.annotations.Test)

Example 5 with ZooKeeperClientFactory

use of org.apache.pulsar.zookeeper.ZooKeeperClientFactory in project incubator-pulsar by apache.

the class CompactorTool method main.

public static void main(String[] args) throws Exception {
    Arguments arguments = new Arguments();
    JCommander jcommander = new JCommander(arguments);
    jcommander.setProgramName("PulsarTopicCompactor");
    // parse args by JCommander
    jcommander.parse(args);
    if (arguments.help) {
        jcommander.usage();
        System.exit(-1);
    }
    // init broker config
    ServiceConfiguration brokerConfig;
    if (isBlank(arguments.brokerConfigFile)) {
        jcommander.usage();
        throw new IllegalArgumentException("Need to specify a configuration file for broker");
    } else {
        brokerConfig = PulsarConfigurationLoader.create(arguments.brokerConfigFile, ServiceConfiguration.class);
    }
    String pulsarServiceUrl = PulsarService.brokerUrl(brokerConfig);
    ClientConfiguration clientConfig = new ClientConfiguration();
    if (isNotBlank(brokerConfig.getBrokerClientAuthenticationPlugin())) {
        clientConfig.setAuthentication(brokerConfig.getBrokerClientAuthenticationPlugin(), brokerConfig.getBrokerClientAuthenticationParameters());
    }
    clientConfig.setUseTls(brokerConfig.isTlsEnabled());
    clientConfig.setTlsAllowInsecureConnection(brokerConfig.isTlsAllowInsecureConnection());
    clientConfig.setTlsTrustCertsFilePath(brokerConfig.getTlsCertificateFilePath());
    ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder().setNameFormat("compaction-%d").setDaemon(true).build());
    OrderedScheduler executor = OrderedScheduler.newSchedulerBuilder().build();
    ZooKeeperClientFactory zkClientFactory = new ZookeeperBkClientFactoryImpl(executor);
    ZooKeeper zk = zkClientFactory.create(brokerConfig.getZookeeperServers(), ZooKeeperClientFactory.SessionType.ReadWrite, (int) brokerConfig.getZooKeeperSessionTimeoutMillis()).get();
    BookKeeperClientFactory bkClientFactory = new BookKeeperClientFactoryImpl();
    BookKeeper bk = bkClientFactory.create(brokerConfig, zk);
    try (PulsarClient pulsar = PulsarClient.create(pulsarServiceUrl, clientConfig)) {
        Compactor compactor = new TwoPhaseCompactor(brokerConfig, pulsar, bk, scheduler);
        long ledgerId = compactor.compact(arguments.topic).get();
        log.info("Compaction of topic {} complete. Compacted to ledger {}", arguments.topic, ledgerId);
    } finally {
        bk.close();
        bkClientFactory.close();
        zk.close();
        scheduler.shutdownNow();
        executor.shutdown();
    }
}
Also used : ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) BookKeeperClientFactoryImpl(org.apache.pulsar.broker.BookKeeperClientFactoryImpl) BookKeeper(org.apache.bookkeeper.client.BookKeeper) ZooKeeperClientFactory(org.apache.pulsar.zookeeper.ZooKeeperClientFactory) BookKeeperClientFactory(org.apache.pulsar.broker.BookKeeperClientFactory) ZooKeeper(org.apache.zookeeper.ZooKeeper) ServiceConfiguration(org.apache.pulsar.broker.ServiceConfiguration) JCommander(com.beust.jcommander.JCommander) ZookeeperBkClientFactoryImpl(org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl) ThreadFactoryBuilder(com.google.common.util.concurrent.ThreadFactoryBuilder) PulsarClient(org.apache.pulsar.client.api.PulsarClient) ClientConfiguration(org.apache.pulsar.client.api.ClientConfiguration) OrderedScheduler(org.apache.bookkeeper.common.util.OrderedScheduler)

Aggregations

ZooKeeperClientFactory (org.apache.pulsar.zookeeper.ZooKeeperClientFactory)9 ZooKeeper (org.apache.zookeeper.ZooKeeper)8 Test (org.testng.annotations.Test)6 ZookeeperClientFactoryImpl (org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl)5 OrderedScheduler (org.apache.bookkeeper.common.util.OrderedScheduler)4 ZookeeperBkClientFactoryImpl (org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl)4 JCommander (com.beust.jcommander.JCommander)2 CountDownLatch (java.util.concurrent.CountDownLatch)2 PulsarClient (org.apache.pulsar.client.api.PulsarClient)2 Stat (org.apache.zookeeper.data.Stat)2 AtomicDouble (com.google.common.util.concurrent.AtomicDouble)1 ThreadFactoryBuilder (com.google.common.util.concurrent.ThreadFactoryBuilder)1 IOException (java.io.IOException)1 ScheduledExecutorService (java.util.concurrent.ScheduledExecutorService)1 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)1 ServletException (javax.servlet.ServletException)1 BookKeeper (org.apache.bookkeeper.client.BookKeeper)1 ServerConfiguration (org.apache.bookkeeper.conf.ServerConfiguration)1 BookKeeperClientFactory (org.apache.pulsar.broker.BookKeeperClientFactory)1 BookKeeperClientFactoryImpl (org.apache.pulsar.broker.BookKeeperClientFactoryImpl)1