Search in sources :

Example 1 with ExponentialBackoffRetry

use of org.apache.flink.shaded.curator5.org.apache.curator.retry.ExponentialBackoffRetry in project druid by druid-io.

the class RemoteTaskRunnerFactoryTest method setUp.

@Before
public void setUp() throws Exception {
    TestUtils testUtils = new TestUtils();
    jsonMapper = testUtils.getTestObjectMapper();
    testingCluster = new TestingCluster(1);
    testingCluster.start();
    cf = CuratorFrameworkFactory.builder().connectString(testingCluster.getConnectString()).retryPolicy(new ExponentialBackoffRetry(1, 10)).compressionProvider(new PotentiallyGzippedCompressionProvider(false)).build();
    cf.start();
    cf.blockUntilConnected();
}
Also used : TestUtils(io.druid.indexing.common.TestUtils) TestingCluster(org.apache.curator.test.TestingCluster) ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry) PotentiallyGzippedCompressionProvider(io.druid.curator.PotentiallyGzippedCompressionProvider) Before(org.junit.Before)

Example 2 with ExponentialBackoffRetry

use of org.apache.flink.shaded.curator5.org.apache.curator.retry.ExponentialBackoffRetry in project druid by druid-io.

the class WorkerTaskMonitorTest method setUp.

@Before
public void setUp() throws Exception {
    testingCluster = new TestingCluster(1);
    testingCluster.start();
    cf = CuratorFrameworkFactory.builder().connectString(testingCluster.getConnectString()).retryPolicy(new ExponentialBackoffRetry(1, 10)).compressionProvider(new PotentiallyGzippedCompressionProvider(false)).build();
    cf.start();
    cf.blockUntilConnected();
    cf.create().creatingParentsIfNeeded().forPath(basePath);
    worker = new Worker("worker", "localhost", 3, "0");
    workerCuratorCoordinator = new WorkerCuratorCoordinator(jsonMapper, new IndexerZkConfig(new ZkPathsConfig() {

        @Override
        public String getBase() {
            return basePath;
        }
    }, null, null, null, null, null), new TestRemoteTaskRunnerConfig(new Period("PT1S")), cf, worker);
    workerCuratorCoordinator.start();
    // Start a task monitor
    workerTaskMonitor = createTaskMonitor();
    TestTasks.registerSubtypes(jsonMapper);
    jsonMapper.registerSubtypes(new NamedType(TestRealtimeTask.class, "test_realtime"));
    workerTaskMonitor.start();
    task = TestTasks.immediateSuccess("test");
}
Also used : IndexerZkConfig(io.druid.server.initialization.IndexerZkConfig) TestRemoteTaskRunnerConfig(io.druid.indexing.overlord.TestRemoteTaskRunnerConfig) ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry) NamedType(com.fasterxml.jackson.databind.jsontype.NamedType) Period(org.joda.time.Period) PotentiallyGzippedCompressionProvider(io.druid.curator.PotentiallyGzippedCompressionProvider) TestRealtimeTask(io.druid.indexing.common.TestRealtimeTask) TestingCluster(org.apache.curator.test.TestingCluster) ZkPathsConfig(io.druid.server.initialization.ZkPathsConfig) Before(org.junit.Before)

Example 3 with ExponentialBackoffRetry

use of org.apache.flink.shaded.curator5.org.apache.curator.retry.ExponentialBackoffRetry in project druid by druid-io.

the class WorkerResourceTest method setUp.

@Before
public void setUp() throws Exception {
    testingCluster = new TestingCluster(1);
    testingCluster.start();
    cf = CuratorFrameworkFactory.builder().connectString(testingCluster.getConnectString()).retryPolicy(new ExponentialBackoffRetry(1, 10)).compressionProvider(new PotentiallyGzippedCompressionProvider(false)).build();
    cf.start();
    cf.blockUntilConnected();
    cf.create().creatingParentsIfNeeded().forPath(basePath);
    worker = new Worker("host", "ip", 3, "v1");
    curatorCoordinator = new WorkerCuratorCoordinator(jsonMapper, new IndexerZkConfig(new ZkPathsConfig() {

        @Override
        public String getBase() {
            return basePath;
        }
    }, null, null, null, null, null), new RemoteTaskRunnerConfig(), cf, worker);
    curatorCoordinator.start();
    workerResource = new WorkerResource(worker, curatorCoordinator, null);
}
Also used : IndexerZkConfig(io.druid.server.initialization.IndexerZkConfig) TestingCluster(org.apache.curator.test.TestingCluster) ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry) WorkerCuratorCoordinator(io.druid.indexing.worker.WorkerCuratorCoordinator) ZkPathsConfig(io.druid.server.initialization.ZkPathsConfig) Worker(io.druid.indexing.worker.Worker) PotentiallyGzippedCompressionProvider(io.druid.curator.PotentiallyGzippedCompressionProvider) RemoteTaskRunnerConfig(io.druid.indexing.overlord.config.RemoteTaskRunnerConfig) Before(org.junit.Before)

Example 4 with ExponentialBackoffRetry

use of org.apache.flink.shaded.curator5.org.apache.curator.retry.ExponentialBackoffRetry in project hive by apache.

the class HiveServer2 method addServerInstanceToZooKeeper.

/**
   * Adds a server instance to ZooKeeper as a znode.
   *
   * @param hiveConf
   * @throws Exception
   */
private void addServerInstanceToZooKeeper(HiveConf hiveConf) throws Exception {
    String zooKeeperEnsemble = ZooKeeperHiveHelper.getQuorumServers(hiveConf);
    String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
    String instanceURI = getServerInstanceURI();
    setUpZooKeeperAuth(hiveConf);
    int sessionTimeout = (int) hiveConf.getTimeVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_SESSION_TIMEOUT, TimeUnit.MILLISECONDS);
    int baseSleepTime = (int) hiveConf.getTimeVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME, TimeUnit.MILLISECONDS);
    int maxRetries = hiveConf.getIntVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES);
    // Create a CuratorFramework instance to be used as the ZooKeeper client
    // Use the zooKeeperAclProvider to create appropriate ACLs
    zooKeeperClient = CuratorFrameworkFactory.builder().connectString(zooKeeperEnsemble).sessionTimeoutMs(sessionTimeout).aclProvider(zooKeeperAclProvider).retryPolicy(new ExponentialBackoffRetry(baseSleepTime, maxRetries)).build();
    zooKeeperClient.start();
    // Create the parent znodes recursively; ignore if the parent already exists.
    try {
        zooKeeperClient.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT).forPath(ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + rootNamespace);
        LOG.info("Created the root name space: " + rootNamespace + " on ZooKeeper for HiveServer2");
    } catch (KeeperException e) {
        if (e.code() != KeeperException.Code.NODEEXISTS) {
            LOG.error("Unable to create HiveServer2 namespace: " + rootNamespace + " on ZooKeeper", e);
            throw e;
        }
    }
    // Znode name: serverUri=host:port;version=versionInfo;sequence=sequenceNumber
    try {
        String pathPrefix = ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + rootNamespace + ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + "serverUri=" + instanceURI + ";" + "version=" + HiveVersionInfo.getVersion() + ";" + "sequence=";
        String znodeData = "";
        if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_PUBLISH_CONFIGS)) {
            // HiveServer2 configs that this instance will publish to ZooKeeper,
            // so that the clients can read these and configure themselves properly.
            Map<String, String> confsToPublish = new HashMap<String, String>();
            addConfsToPublish(hiveConf, confsToPublish);
            // Publish configs for this instance as the data on the node
            znodeData = Joiner.on(';').withKeyValueSeparator("=").join(confsToPublish);
        } else {
            znodeData = instanceURI;
        }
        byte[] znodeDataUTF8 = znodeData.getBytes(Charset.forName("UTF-8"));
        znode = new PersistentEphemeralNode(zooKeeperClient, PersistentEphemeralNode.Mode.EPHEMERAL_SEQUENTIAL, pathPrefix, znodeDataUTF8);
        znode.start();
        // We'll wait for 120s for node creation
        long znodeCreationTimeout = 120;
        if (!znode.waitForInitialCreate(znodeCreationTimeout, TimeUnit.SECONDS)) {
            throw new Exception("Max znode creation wait time: " + znodeCreationTimeout + "s exhausted");
        }
        setDeregisteredWithZooKeeper(false);
        znodePath = znode.getActualPath();
        // Set a watch on the znode
        if (zooKeeperClient.checkExists().usingWatcher(new DeRegisterWatcher()).forPath(znodePath) == null) {
            // No node exists, throw exception
            throw new Exception("Unable to create znode for this HiveServer2 instance on ZooKeeper.");
        }
        LOG.info("Created a znode on ZooKeeper for HiveServer2 uri: " + instanceURI);
    } catch (Exception e) {
        LOG.error("Unable to create a znode for this server instance", e);
        if (znode != null) {
            znode.close();
        }
        throw (e);
    }
}
Also used : HashMap(java.util.HashMap) ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry) PersistentEphemeralNode(org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode) KeeperException(org.apache.zookeeper.KeeperException) ServiceException(org.apache.hive.service.ServiceException) ParseException(org.apache.commons.cli.ParseException) HiveException(org.apache.hadoop.hive.ql.metadata.HiveException) LogInitializationException(org.apache.hadoop.hive.common.LogUtils.LogInitializationException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException)

Example 5 with ExponentialBackoffRetry

use of org.apache.flink.shaded.curator5.org.apache.curator.retry.ExponentialBackoffRetry in project hive by apache.

the class ZooKeeperTokenStore method getSession.

private CuratorFramework getSession() {
    if (zkSession == null || zkSession.getState() == CuratorFrameworkState.STOPPED) {
        synchronized (this) {
            if (zkSession == null || zkSession.getState() == CuratorFrameworkState.STOPPED) {
                zkSession = CuratorFrameworkFactory.builder().connectString(zkConnectString).connectionTimeoutMs(connectTimeoutMillis).aclProvider(aclDefaultProvider).retryPolicy(new ExponentialBackoffRetry(1000, 3)).build();
                zkSession.start();
            }
        }
    }
    return zkSession;
}
Also used : ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry)

Aggregations

ExponentialBackoffRetry (org.apache.curator.retry.ExponentialBackoffRetry)189 CuratorFramework (org.apache.curator.framework.CuratorFramework)113 RetryPolicy (org.apache.curator.RetryPolicy)46 Before (org.junit.Before)31 TestingCluster (org.apache.curator.test.TestingCluster)28 Test (org.testng.annotations.Test)23 TestingServer (org.apache.curator.test.TestingServer)19 IOException (java.io.IOException)18 Timing (org.apache.curator.test.Timing)16 CountDownLatch (java.util.concurrent.CountDownLatch)15 ArrayList (java.util.ArrayList)14 CuratorFrameworkFactory (org.apache.curator.framework.CuratorFrameworkFactory)12 ACLProvider (org.apache.curator.framework.api.ACLProvider)12 Test (org.junit.Test)12 ConnectionState (org.apache.curator.framework.state.ConnectionState)11 ExecutorService (java.util.concurrent.ExecutorService)10 ConnectionStateListener (org.apache.curator.framework.state.ConnectionStateListener)10 TestingServerStarter (io.pravega.test.common.TestingServerStarter)9 HashMap (java.util.HashMap)8 KeeperException (org.apache.zookeeper.KeeperException)8