Examples with CheckpointStorage org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.CheckpointStorage used on opensource projects

Example 1 with CheckpointStorage

use of org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.CheckpointStorage in project hadoop by apache.

the class TestCheckpoint method testMultipleSecondaryNNsAgainstSameNN.

/**
   * Test case where two secondary namenodes are checkpointing the same
   * NameNode. This differs from {@link #testMultipleSecondaryNamenodes()}
   * since that test runs against two distinct NNs.
   * 
   * This case tests the following interleaving:
   * - 2NN A downloads image (up to txid 2)
   * - 2NN A about to save its own checkpoint
   * - 2NN B downloads image (up to txid 4)
   * - 2NN B uploads checkpoint (txid 4)
   * - 2NN A uploads checkpoint (txid 2)
   * 
   * It verifies that this works even though the earlier-txid checkpoint gets
   * uploaded after the later-txid checkpoint.
   */
@Test
public void testMultipleSecondaryNNsAgainstSameNN() throws Exception {
    Configuration conf = new HdfsConfiguration();
    MiniDFSCluster cluster = null;
    SecondaryNameNode secondary1 = null, secondary2 = null;
    try {
        cluster = new MiniDFSCluster.Builder(conf).numDataNodes(0).format(true).build();
        // Start 2NNs
        secondary1 = startSecondaryNameNode(conf, 1);
        secondary2 = startSecondaryNameNode(conf, 2);
        // Make the first 2NN's checkpoint process delayable - we can pause it
        // right before it saves its checkpoint image.
        CheckpointStorage spyImage1 = spyOnSecondaryImage(secondary1);
        DelayAnswer delayer = new DelayAnswer(LOG);
        Mockito.doAnswer(delayer).when(spyImage1).saveFSImageInAllDirs(Mockito.<FSNamesystem>any(), Mockito.anyLong());
        // Set up a thread to do a checkpoint from the first 2NN
        DoCheckpointThread checkpointThread = new DoCheckpointThread(secondary1);
        checkpointThread.start();
        // Wait for the first checkpointer to get to where it should save its image.
        delayer.waitForCall();
        // Now make the second checkpointer run an entire checkpoint
        secondary2.doCheckpoint();
        // Let the first one finish
        delayer.proceed();
        // It should have succeeded even though another checkpoint raced with it.
        checkpointThread.join();
        checkpointThread.propagateExceptions();
        // primary should record "last checkpoint" as the higher txid (even though
        // a checkpoint with a lower txid finished most recently)
        NNStorage storage = cluster.getNameNode().getFSImage().getStorage();
        assertEquals(4, storage.getMostRecentCheckpointTxId());
        // Should have accepted both checkpoints
        assertNNHasCheckpoints(cluster, ImmutableList.of(2, 4));
        // Now have second one checkpoint one more time just to make sure that
        // the NN isn't left in a broken state
        secondary2.doCheckpoint();
        // NN should have received new checkpoint
        assertEquals(6, storage.getMostRecentCheckpointTxId());
        // Validate invariant that files named the same are the same.
        assertParallelFilesInvariant(cluster, ImmutableList.of(secondary1, secondary2));
        // NN should have removed the checkpoint at txid 2 at this point, but has
        // one at txid 6
        assertNNHasCheckpoints(cluster, ImmutableList.of(4, 6));
    } finally {
        cleanup(secondary1);
        secondary1 = null;
        cleanup(secondary2);
        secondary2 = null;
        if (cluster != null) {
            cluster.shutdown();
            cluster = null;
        }
    }
}

Example 2 with CheckpointStorage

use of org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.CheckpointStorage in project hadoop by apache.

the class TestCheckpoint method spyOnSecondaryImage.

private static CheckpointStorage spyOnSecondaryImage(SecondaryNameNode secondary1) {
    CheckpointStorage spy = Mockito.spy((CheckpointStorage) secondary1.getFSImage());
    ;
    secondary1.setFSImage(spy);
    return spy;
}