Search in sources :

Example 1 with StorageContainerLocationProtocolClientSideTranslatorPB

use of org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB in project ozone by apache.

the class TestFailoverWithSCMHA method testFailover.

@Test
public void testFailover() throws Exception {
    SCMClientConfig scmClientConfig = conf.getObject(SCMClientConfig.class);
    scmClientConfig.setRetryCount(1);
    scmClientConfig.setRetryInterval(100);
    scmClientConfig.setMaxRetryTimeout(1500);
    Assert.assertEquals(scmClientConfig.getRetryCount(), 15);
    conf.setFromObject(scmClientConfig);
    StorageContainerManager scm = getLeader(cluster);
    Assert.assertNotNull(scm);
    SCMBlockLocationFailoverProxyProvider failoverProxyProvider = new SCMBlockLocationFailoverProxyProvider(conf);
    failoverProxyProvider.changeCurrentProxy(scm.getSCMNodeId());
    ScmBlockLocationProtocolClientSideTranslatorPB scmBlockLocationClient = new ScmBlockLocationProtocolClientSideTranslatorPB(failoverProxyProvider);
    GenericTestUtils.setLogLevel(SCMBlockLocationFailoverProxyProvider.LOG, Level.DEBUG);
    GenericTestUtils.LogCapturer logCapture = GenericTestUtils.LogCapturer.captureLogs(SCMBlockLocationFailoverProxyProvider.LOG);
    ScmBlockLocationProtocol scmBlockLocationProtocol = TracingUtil.createProxy(scmBlockLocationClient, ScmBlockLocationProtocol.class, conf);
    scmBlockLocationProtocol.getScmInfo();
    Assert.assertTrue(logCapture.getOutput().contains("Performing failover to suggested leader"));
    scm = getLeader(cluster);
    SCMContainerLocationFailoverProxyProvider proxyProvider = new SCMContainerLocationFailoverProxyProvider(conf, null);
    GenericTestUtils.setLogLevel(SCMContainerLocationFailoverProxyProvider.LOG, Level.DEBUG);
    logCapture = GenericTestUtils.LogCapturer.captureLogs(SCMContainerLocationFailoverProxyProvider.LOG);
    proxyProvider.changeCurrentProxy(scm.getSCMNodeId());
    StorageContainerLocationProtocol scmContainerClient = TracingUtil.createProxy(new StorageContainerLocationProtocolClientSideTranslatorPB(proxyProvider), StorageContainerLocationProtocol.class, conf);
    scmContainerClient.allocateContainer(HddsProtos.ReplicationType.RATIS, HddsProtos.ReplicationFactor.ONE, "ozone");
    Assert.assertTrue(logCapture.getOutput().contains("Performing failover to suggested leader"));
}
Also used : SCMBlockLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMBlockLocationFailoverProxyProvider) ScmBlockLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB) ScmBlockLocationProtocol(org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol) StorageContainerManager(org.apache.hadoop.hdds.scm.server.StorageContainerManager) SCMClientConfig(org.apache.hadoop.hdds.scm.proxy.SCMClientConfig) StorageContainerLocationProtocol(org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol) GenericTestUtils(org.apache.ozone.test.GenericTestUtils) StorageContainerLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB) SCMContainerLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider) Test(org.junit.jupiter.api.Test)

Example 2 with StorageContainerLocationProtocolClientSideTranslatorPB

use of org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB in project ozone by apache.

the class HAUtils method getScmContainerClient.

@VisibleForTesting
public static StorageContainerLocationProtocol getScmContainerClient(ConfigurationSource conf, UserGroupInformation userGroupInformation) {
    SCMContainerLocationFailoverProxyProvider proxyProvider = new SCMContainerLocationFailoverProxyProvider(conf, userGroupInformation);
    StorageContainerLocationProtocol scmContainerClient = TracingUtil.createProxy(new StorageContainerLocationProtocolClientSideTranslatorPB(proxyProvider), StorageContainerLocationProtocol.class, conf);
    return scmContainerClient;
}
Also used : StorageContainerLocationProtocol(org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol) StorageContainerLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB) SCMContainerLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Example 3 with StorageContainerLocationProtocolClientSideTranslatorPB

use of org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB in project ozone by apache.

the class HAUtils method getScmContainerClient.

public static StorageContainerLocationProtocol getScmContainerClient(ConfigurationSource conf) {
    SCMContainerLocationFailoverProxyProvider proxyProvider = new SCMContainerLocationFailoverProxyProvider(conf, null);
    StorageContainerLocationProtocol scmContainerClient = TracingUtil.createProxy(new StorageContainerLocationProtocolClientSideTranslatorPB(proxyProvider), StorageContainerLocationProtocol.class, conf);
    return scmContainerClient;
}
Also used : StorageContainerLocationProtocol(org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol) StorageContainerLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB) SCMContainerLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider)

Example 4 with StorageContainerLocationProtocolClientSideTranslatorPB

use of org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB in project ozone by apache.

the class MiniOzoneClusterImpl method getStorageContainerLocationClient.

/**
 * Returns an RPC proxy connected to this cluster's StorageContainerManager
 * for accessing container location information.  Callers take ownership of
 * the proxy and must close it when done.
 *
 * @return RPC proxy for accessing container location information
 * @throws IOException if there is an I/O error
 */
@Override
public StorageContainerLocationProtocolClientSideTranslatorPB getStorageContainerLocationClient() throws IOException {
    InetSocketAddress address = scm.getClientRpcAddress();
    LOG.info("Creating StorageContainerLocationProtocol RPC client with address {}", address);
    SCMContainerLocationFailoverProxyProvider proxyProvider = new SCMContainerLocationFailoverProxyProvider(conf, null);
    return new StorageContainerLocationProtocolClientSideTranslatorPB(proxyProvider);
}
Also used : InetSocketAddress(java.net.InetSocketAddress) StorageContainerLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB) SCMContainerLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider)

Example 5 with StorageContainerLocationProtocolClientSideTranslatorPB

use of org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB in project ozone by apache.

the class TestFailoverWithSCMHA method testMoveFailover.

@Test
public void testMoveFailover() throws Exception {
    SCMClientConfig scmClientConfig = conf.getObject(SCMClientConfig.class);
    scmClientConfig.setRetryCount(1);
    scmClientConfig.setRetryInterval(100);
    scmClientConfig.setMaxRetryTimeout(1500);
    Assert.assertEquals(scmClientConfig.getRetryCount(), 15);
    conf.setFromObject(scmClientConfig);
    StorageContainerManager scm = getLeader(cluster);
    Assert.assertNotNull(scm);
    final ContainerID id = getContainer(HddsProtos.LifeCycleState.CLOSED).containerID();
    DatanodeDetails dn1 = randomDatanodeDetails();
    DatanodeDetails dn2 = randomDatanodeDetails();
    // here we just want to test whether the new leader will get the same
    // inflight move after failover, so no need to create container and datanode,
    // just mock them bypassing all the pre checks.
    scm.getReplicationManager().getMoveScheduler().startMove(id.getProtobuf(), (new MoveDataNodePair(dn1, dn2)).getProtobufMessage(CURRENT_VERSION));
    SCMBlockLocationFailoverProxyProvider failoverProxyProvider = new SCMBlockLocationFailoverProxyProvider(conf);
    failoverProxyProvider.changeCurrentProxy(scm.getSCMNodeId());
    ScmBlockLocationProtocolClientSideTranslatorPB scmBlockLocationClient = new ScmBlockLocationProtocolClientSideTranslatorPB(failoverProxyProvider);
    GenericTestUtils.setLogLevel(SCMBlockLocationFailoverProxyProvider.LOG, Level.DEBUG);
    GenericTestUtils.LogCapturer logCapture = GenericTestUtils.LogCapturer.captureLogs(SCMBlockLocationFailoverProxyProvider.LOG);
    ScmBlockLocationProtocol scmBlockLocationProtocol = TracingUtil.createProxy(scmBlockLocationClient, ScmBlockLocationProtocol.class, conf);
    scmBlockLocationProtocol.getScmInfo();
    Assert.assertTrue(logCapture.getOutput().contains("Performing failover to suggested leader"));
    scm = getLeader(cluster);
    Assert.assertNotNull(scm);
    // switch to the new leader successfully, new leader should
    // get the same inflightMove
    Map<ContainerID, MoveDataNodePair> inflightMove = scm.getReplicationManager().getMoveScheduler().getInflightMove();
    Assert.assertTrue(inflightMove.containsKey(id));
    MoveDataNodePair mp = inflightMove.get(id);
    Assert.assertTrue(dn2.equals(mp.getTgt()));
    Assert.assertTrue(dn1.equals(mp.getSrc()));
    // complete move in the new leader
    scm.getReplicationManager().getMoveScheduler().completeMove(id.getProtobuf());
    SCMContainerLocationFailoverProxyProvider proxyProvider = new SCMContainerLocationFailoverProxyProvider(conf, null);
    GenericTestUtils.setLogLevel(SCMContainerLocationFailoverProxyProvider.LOG, Level.DEBUG);
    logCapture = GenericTestUtils.LogCapturer.captureLogs(SCMContainerLocationFailoverProxyProvider.LOG);
    proxyProvider.changeCurrentProxy(scm.getSCMNodeId());
    StorageContainerLocationProtocol scmContainerClient = TracingUtil.createProxy(new StorageContainerLocationProtocolClientSideTranslatorPB(proxyProvider), StorageContainerLocationProtocol.class, conf);
    scmContainerClient.allocateContainer(HddsProtos.ReplicationType.RATIS, HddsProtos.ReplicationFactor.ONE, "ozone");
    Assert.assertTrue(logCapture.getOutput().contains("Performing failover to suggested leader"));
    // switch to the new leader successfully, new leader should
    // get the same inflightMove , which should not contains
    // that container.
    scm = getLeader(cluster);
    Assert.assertNotNull(scm);
    inflightMove = scm.getReplicationManager().getMoveScheduler().getInflightMove();
    Assert.assertFalse(inflightMove.containsKey(id));
}
Also used : SCMBlockLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMBlockLocationFailoverProxyProvider) StorageContainerManager(org.apache.hadoop.hdds.scm.server.StorageContainerManager) SCMClientConfig(org.apache.hadoop.hdds.scm.proxy.SCMClientConfig) GenericTestUtils(org.apache.ozone.test.GenericTestUtils) SCMContainerLocationFailoverProxyProvider(org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider) ScmBlockLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB) ScmBlockLocationProtocol(org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol) MoveDataNodePair(org.apache.hadoop.hdds.scm.container.common.helpers.MoveDataNodePair) ContainerID(org.apache.hadoop.hdds.scm.container.ContainerID) MockDatanodeDetails.randomDatanodeDetails(org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails) DatanodeDetails(org.apache.hadoop.hdds.protocol.DatanodeDetails) StorageContainerLocationProtocol(org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol) StorageContainerLocationProtocolClientSideTranslatorPB(org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB) Test(org.junit.jupiter.api.Test)

Aggregations

StorageContainerLocationProtocolClientSideTranslatorPB (org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB)5 SCMContainerLocationFailoverProxyProvider (org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider)5 StorageContainerLocationProtocol (org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol)4 ScmBlockLocationProtocol (org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol)2 ScmBlockLocationProtocolClientSideTranslatorPB (org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB)2 SCMBlockLocationFailoverProxyProvider (org.apache.hadoop.hdds.scm.proxy.SCMBlockLocationFailoverProxyProvider)2 SCMClientConfig (org.apache.hadoop.hdds.scm.proxy.SCMClientConfig)2 StorageContainerManager (org.apache.hadoop.hdds.scm.server.StorageContainerManager)2 GenericTestUtils (org.apache.ozone.test.GenericTestUtils)2 Test (org.junit.jupiter.api.Test)2 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 InetSocketAddress (java.net.InetSocketAddress)1 DatanodeDetails (org.apache.hadoop.hdds.protocol.DatanodeDetails)1 MockDatanodeDetails.randomDatanodeDetails (org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails)1 ContainerID (org.apache.hadoop.hdds.scm.container.ContainerID)1 MoveDataNodePair (org.apache.hadoop.hdds.scm.container.common.helpers.MoveDataNodePair)1