Examples with LocalHBaseCluster org.apache.hadoop.hbase.LocalHBaseCluster used on opensource projects

Example 1 with LocalHBaseCluster

use of org.apache.hadoop.hbase.LocalHBaseCluster in project hbase by apache.

the class TestShadeSaslAuthenticationProvider method createCluster.

static LocalHBaseCluster createCluster(HBaseTestingUtil util, File keytabFile, MiniKdc kdc, Map<String, char[]> userDatabase) throws Exception {
    String servicePrincipal = "hbase/localhost";
    String spnegoPrincipal = "HTTP/localhost";
    kdc.createPrincipal(keytabFile, servicePrincipal);
    util.startMiniZKCluster();
    HBaseKerberosUtils.setSecuredConfiguration(util.getConfiguration(), servicePrincipal + "@" + kdc.getRealm(), spnegoPrincipal + "@" + kdc.getRealm());
    HBaseKerberosUtils.setSSLConfiguration(util, TestShadeSaslAuthenticationProvider.class);
    util.getConfiguration().setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, TokenProvider.class.getName());
    util.startMiniDFSCluster(1);
    Path testDir = util.getDataTestDirOnTestFS("TestShadeSaslAuthenticationProvider");
    USER_DATABASE_FILE = new Path(testDir, "user-db.txt");
    createUserDBFile(USER_DATABASE_FILE.getFileSystem(CONF), USER_DATABASE_FILE, userDatabase);
    CONF.set(ShadeSaslServerAuthenticationProvider.PASSWORD_FILE_KEY, USER_DATABASE_FILE.toString());
    Path rootdir = new Path(testDir, "hbase-root");
    CommonFSUtils.setRootDir(CONF, rootdir);
    LocalHBaseCluster cluster = new LocalHBaseCluster(CONF, 1);
    return cluster;
}

Example 2 with LocalHBaseCluster

use of org.apache.hadoop.hbase.LocalHBaseCluster in project hbase by apache.

the class TestMasterShutdown method testMasterShutdownBeforeStartingAnyRegionServer.

/**
 * This test appears to be an intentional race between a thread that issues a shutdown RPC to the
 * master, while the master is concurrently realizing it cannot initialize because there are no
 * region servers available to it. The expected behavior is that master initialization is
 * interruptable via said shutdown RPC.
 */
@Test
public void testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
    LocalHBaseCluster hbaseCluster = null;
    try {
        htu = new HBaseTestingUtil(createMasterShutdownBeforeStartingAnyRegionServerConfiguration());
        // configure a cluster with
        final StartTestingClusterOption options = StartTestingClusterOption.builder().numDataNodes(1).numMasters(1).numRegionServers(0).masterClass(HMaster.class).rsClass(SingleProcessHBaseCluster.MiniHBaseClusterRegionServer.class).createRootDir(true).build();
        // Can't simply `htu.startMiniCluster(options)` because that method waits for the master to
        // start completely. However, this test's premise is that a partially started master should
        // still respond to a shutdown RPC. So instead, we manage each component lifecycle
        // independently.
        // I think it's not worth refactoring HTU's helper methods just for this class.
        htu.startMiniDFSCluster(options.getNumDataNodes());
        htu.startMiniZKCluster(options.getNumZkServers());
        htu.createRootDir();
        hbaseCluster = new LocalHBaseCluster(htu.getConfiguration(), options.getNumMasters(), options.getNumRegionServers(), options.getMasterClass(), options.getRsClass());
        final MasterThread masterThread = hbaseCluster.getMasters().get(0);
        masterThread.start();
        // Switching to master registry exacerbated a race in the master bootstrap that can result
        // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
        // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
        // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
        // is because the connection creation with ZK registry is so slow that by then the server
        // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
        // wait() in the test, waiting for the server manager to become available.
        final long timeout = TimeUnit.MINUTES.toMillis(10);
        assertNotEquals("timeout waiting for server manager to become available.", -1, htu.waitFor(timeout, () -> masterThread.getMaster().getServerManager() != null));
        // Master has come up far enough that we can terminate it without creating a zombie.
        try {
            // HBASE-24327 : (Resolve Flaky connection issues)
            // shutdown() RPC can have flaky ZK connection issues.
            // e.g
            // ERROR [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=53033]
            // master.HMaster(2878): ZooKeeper exception trying to set cluster as down in ZK
            // org.apache.zookeeper.KeeperException$SystemErrorException:
            // KeeperErrorCode = SystemError
            // 
            // However, even when above flakes happen, shutdown call does get completed even if
            // RPC call has failure. Hence, subsequent retries will never succeed as HMaster is
            // already shutdown. Hence, it can fail. To resolve it, after making one shutdown()
            // call, we are ignoring IOException.
            htu.getConnection().getAdmin().shutdown();
        } catch (RetriesExhaustedException e) {
            if (e.getCause() instanceof ConnectionClosedException) {
                LOG.info("Connection is Closed to the cluster. The cluster is already down.", e);
            } else {
                throw e;
            }
        }
        LOG.info("Shutdown RPC sent.");
        masterThread.join();
    } finally {
        if (hbaseCluster != null) {
            hbaseCluster.shutdown();
        }
        if (htu != null) {
            htu.shutdownMiniCluster();
            htu = null;
        }
    }
}

Example 3 with LocalHBaseCluster

use of org.apache.hadoop.hbase.LocalHBaseCluster in project hbase by apache.

the class CustomSaslAuthenticationProviderTestBase method startCluster.

protected static void startCluster(String rpcServerImpl) throws Exception {
    KEYTAB_FILE = new File(UTIL.getDataTestDir("keytab").toUri().getPath());
    final MiniKdc kdc = UTIL.setupMiniKdc(KEYTAB_FILE);
    // Adds our test impls instead of creating service loader entries which
    // might inadvertently get them loaded on a real cluster.
    CONF.setStrings(SaslClientAuthenticationProviders.EXTRA_PROVIDERS_KEY, InMemoryClientProvider.class.getName());
    CONF.setStrings(SaslServerAuthenticationProviders.EXTRA_PROVIDERS_KEY, InMemoryServerProvider.class.getName());
    CONF.set(SaslClientAuthenticationProviders.SELECTOR_KEY, InMemoryProviderSelector.class.getName());
    createBaseCluster(UTIL, KEYTAB_FILE, kdc);
    CONF.set(RpcServerFactory.CUSTOM_RPC_SERVER_IMPL_CONF_KEY, rpcServerImpl);
    CLUSTER = new LocalHBaseCluster(CONF, 1);
    CLUSTER.startup();
}

Example 4 with LocalHBaseCluster

use of org.apache.hadoop.hbase.LocalHBaseCluster in project hbase by apache.

the class SecureTestCluster method setUp.

/**
 * Setup and start kerberos, hbase
 */
@BeforeClass
public static void setUp() throws Exception {
    // Can take a long time for the mini kdc to come up on loaded test cluster. Tolerate this in
    // test by upping the skew time allowed from 30s to 90s.
    TEST_UTIL.getConfiguration().setLong(ServerManager.MAX_CLOCK_SKEW_MS, 90000);
    KDC = TEST_UTIL.setupMiniKdc(KEYTAB_FILE);
    USERNAME = UserGroupInformation.getLoginUser().getShortUserName();
    PRINCIPAL = USERNAME + "/" + HOST;
    HTTP_PRINCIPAL = "HTTP/" + HOST;
    KDC.createPrincipal(KEYTAB_FILE, PRINCIPAL, HTTP_PRINCIPAL);
    TEST_UTIL.startMiniZKCluster();
    HBaseKerberosUtils.setSecuredConfiguration(TEST_UTIL.getConfiguration(), PRINCIPAL + "@" + KDC.getRealm(), HTTP_PRINCIPAL + "@" + KDC.getRealm());
    HBaseKerberosUtils.setSSLConfiguration(TEST_UTIL, testRunnerClass);
    TEST_UTIL.getConfiguration().setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, TokenProvider.class.getName());
    TEST_UTIL.startMiniDFSCluster(1);
    Path rootdir = TEST_UTIL.getDataTestDirOnTestFS("TestGenerateDelegationToken");
    CommonFSUtils.setRootDir(TEST_UTIL.getConfiguration(), rootdir);
    CLUSTER = new LocalHBaseCluster(TEST_UTIL.getConfiguration(), 1);
    CLUSTER.startup();
}

Example 5 with LocalHBaseCluster

use of org.apache.hadoop.hbase.LocalHBaseCluster in project apex-malhar by apache.

the class HBaseTestHelper method startLocalCluster.

public static void startLocalCluster() throws IOException, InterruptedException {
    startZooKeeperServer();
    // Configuration conf = HBaseConfiguration.create();
    Configuration conf = getConfiguration();
    LocalHBaseCluster lc = new LocalHBaseCluster(conf);
    lc.startup();
}