Search in sources :

Example 1 with ReconOMMetadataManager

use of org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager in project ozone by apache.

the class TestReconWithOzoneManagerFSO method testNamespaceSummaryAPI.

@Test
public void testNamespaceSummaryAPI() throws Exception {
    // add a vol, bucket and key
    addKeys(0, 10, "dir");
    OzoneManagerServiceProviderImpl impl = (OzoneManagerServiceProviderImpl) cluster.getReconServer().getOzoneManagerServiceProvider();
    impl.syncDataFromOM();
    ReconNamespaceSummaryManager namespaceSummaryManager = cluster.getReconServer().getReconNamespaceSummaryManager();
    ReconOMMetadataManager omMetadataManagerInstance = (ReconOMMetadataManager) cluster.getReconServer().getOzoneManagerServiceProvider().getOMMetadataManagerInstance();
    OzoneStorageContainerManager reconSCM = cluster.getReconServer().getReconStorageContainerManager();
    NSSummaryEndpoint endpoint = new NSSummaryEndpoint(namespaceSummaryManager, omMetadataManagerInstance, reconSCM);
    Response basicInfo = endpoint.getBasicInfo("/vol1/bucket1/dir1");
    NamespaceSummaryResponse entity = (NamespaceSummaryResponse) basicInfo.getEntity();
    Assert.assertSame(entity.getEntityType(), EntityType.DIRECTORY);
    Assert.assertEquals(1, entity.getNumTotalKey());
    Assert.assertEquals(0, entity.getNumTotalDir());
    for (int i = 0; i < 10; i++) {
        Assert.assertNotNull(impl.getOMMetadataManagerInstance().getVolumeTable().get("/vol" + i));
    }
    addKeys(10, 12, "dir");
    impl.syncDataFromOM();
    // test Recon is sync'ed with OM.
    for (int i = 10; i < 12; i++) {
        Assert.assertNotNull(impl.getOMMetadataManagerInstance().getVolumeTable().getSkipCache("/vol" + i));
    }
    // test root response
    Response rootBasicRes = endpoint.getBasicInfo("/");
    NamespaceSummaryResponse rootBasicEntity = (NamespaceSummaryResponse) rootBasicRes.getEntity();
    Assert.assertSame(EntityType.ROOT, rootBasicEntity.getEntityType());
    // one additional dummy volume at creation
    Assert.assertEquals(13, rootBasicEntity.getNumVolume());
    Assert.assertEquals(12, rootBasicEntity.getNumBucket());
    Assert.assertEquals(12, rootBasicEntity.getNumTotalDir());
    Assert.assertEquals(12, rootBasicEntity.getNumTotalKey());
}
Also used : NamespaceSummaryResponse(org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse) Response(javax.ws.rs.core.Response) NSSummaryEndpoint(org.apache.hadoop.ozone.recon.api.NSSummaryEndpoint) ReconNamespaceSummaryManager(org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager) OzoneManagerServiceProviderImpl(org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager) OzoneStorageContainerManager(org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager) NSSummaryEndpoint(org.apache.hadoop.ozone.recon.api.NSSummaryEndpoint) NamespaceSummaryResponse(org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse) Test(org.junit.Test)

Example 2 with ReconOMMetadataManager

use of org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager in project ozone by apache.

the class MockOzoneServiceProvider method testUpdateReconOmDBWithNewSnapshot.

@Test
public void testUpdateReconOmDBWithNewSnapshot() throws Exception {
    OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(temporaryFolder.newFolder());
    ReconOMMetadataManager reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager, temporaryFolder.newFolder());
    writeDataToOm(omMetadataManager, "key_one");
    writeDataToOm(omMetadataManager, "key_two");
    DBCheckpoint checkpoint = omMetadataManager.getStore().getCheckpoint(true);
    File tarFile = createTarFile(checkpoint.getCheckpointLocation());
    InputStream inputStream = new FileInputStream(tarFile);
    ReconUtils reconUtilsMock = getMockReconUtils();
    HttpURLConnection httpURLConnectionMock = mock(HttpURLConnection.class);
    when(httpURLConnectionMock.getInputStream()).thenReturn(inputStream);
    when(reconUtilsMock.makeHttpCall(any(), anyString(), anyBoolean())).thenReturn(httpURLConnectionMock);
    ReconTaskController reconTaskController = getMockTaskController();
    OzoneManagerServiceProviderImpl ozoneManagerServiceProvider = new OzoneManagerServiceProviderImpl(configuration, reconOMMetadataManager, reconTaskController, reconUtilsMock, ozoneManagerProtocol);
    Assert.assertNull(reconOMMetadataManager.getKeyTable(getBucketLayout()).get("/sampleVol/bucketOne/key_one"));
    Assert.assertNull(reconOMMetadataManager.getKeyTable(getBucketLayout()).get("/sampleVol/bucketOne/key_two"));
    assertTrue(ozoneManagerServiceProvider.updateReconOmDBWithNewSnapshot());
    assertNotNull(reconOMMetadataManager.getKeyTable(getBucketLayout()).get("/sampleVol/bucketOne/key_one"));
    assertNotNull(reconOMMetadataManager.getKeyTable(getBucketLayout()).get("/sampleVol/bucketOne/key_two"));
}
Also used : HttpURLConnection(java.net.HttpURLConnection) ReconUtils(org.apache.hadoop.ozone.recon.ReconUtils) DBCheckpoint(org.apache.hadoop.hdds.utils.db.DBCheckpoint) FileInputStream(java.io.FileInputStream) InputStream(java.io.InputStream) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager) OMMetadataManager(org.apache.hadoop.ozone.om.OMMetadataManager) ReconTaskController(org.apache.hadoop.ozone.recon.tasks.ReconTaskController) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager) ReconUtils.createTarFile(org.apache.hadoop.ozone.recon.ReconUtils.createTarFile) File(java.io.File) FileInputStream(java.io.FileInputStream) Test(org.junit.Test)

Example 3 with ReconOMMetadataManager

use of org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager in project ozone by apache.

the class MockOzoneServiceProvider method testSyncDataFromOMFullSnapshot.

@Test
public void testSyncDataFromOMFullSnapshot() throws Exception {
    // Empty OM DB to start with.
    ReconOMMetadataManager omMetadataManager = getTestReconOmMetadataManager(initializeEmptyOmMetadataManager(temporaryFolder.newFolder()), temporaryFolder.newFolder());
    ReconTaskStatusDao reconTaskStatusDaoMock = mock(ReconTaskStatusDao.class);
    doNothing().when(reconTaskStatusDaoMock).update(any(ReconTaskStatus.class));
    ReconTaskController reconTaskControllerMock = getMockTaskController();
    when(reconTaskControllerMock.getReconTaskStatusDao()).thenReturn(reconTaskStatusDaoMock);
    doNothing().when(reconTaskControllerMock).reInitializeTasks(omMetadataManager);
    OzoneManagerServiceProviderImpl ozoneManagerServiceProvider = new MockOzoneServiceProvider(configuration, omMetadataManager, reconTaskControllerMock, new ReconUtils(), ozoneManagerProtocol);
    OzoneManagerSyncMetrics metrics = ozoneManagerServiceProvider.getMetrics();
    assertEquals(0, metrics.getNumSnapshotRequests().value());
    // Should trigger full snapshot request.
    ozoneManagerServiceProvider.syncDataFromOM();
    ArgumentCaptor<ReconTaskStatus> captor = ArgumentCaptor.forClass(ReconTaskStatus.class);
    verify(reconTaskStatusDaoMock, times(1)).update(captor.capture());
    assertTrue(captor.getValue().getTaskName().equals(OmSnapshotRequest.name()));
    verify(reconTaskControllerMock, times(1)).reInitializeTasks(omMetadataManager);
    assertEquals(1, metrics.getNumSnapshotRequests().value());
}
Also used : ReconUtils(org.apache.hadoop.ozone.recon.ReconUtils) OzoneManagerSyncMetrics(org.apache.hadoop.ozone.recon.metrics.OzoneManagerSyncMetrics) ReconTaskController(org.apache.hadoop.ozone.recon.tasks.ReconTaskController) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager) ReconTaskStatusDao(org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao) ReconTaskStatus(org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus) Test(org.junit.Test)

Example 4 with ReconOMMetadataManager

use of org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager in project ozone by apache.

the class MockOzoneServiceProvider method testGetOzoneManagerDBSnapshot.

@Test
public void testGetOzoneManagerDBSnapshot() throws Exception {
    File reconOmSnapshotDbDir = temporaryFolder.newFolder();
    File checkpointDir = Paths.get(reconOmSnapshotDbDir.getAbsolutePath(), "testGetOzoneManagerDBSnapshot").toFile();
    checkpointDir.mkdir();
    File file1 = Paths.get(checkpointDir.getAbsolutePath(), "file1").toFile();
    String str = "File1 Contents";
    BufferedWriter writer = null;
    try {
        writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file1), UTF_8));
        writer.write(str);
    } finally {
        if (writer != null) {
            writer.close();
        }
    }
    File file2 = Paths.get(checkpointDir.getAbsolutePath(), "file2").toFile();
    str = "File2 Contents";
    try {
        writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file2), UTF_8));
        writer.write(str);
    } finally {
        writer.close();
    }
    // Create test tar file.
    File tarFile = createTarFile(checkpointDir.toPath());
    InputStream fileInputStream = new FileInputStream(tarFile);
    ReconUtils reconUtilsMock = getMockReconUtils();
    HttpURLConnection httpURLConnectionMock = mock(HttpURLConnection.class);
    when(httpURLConnectionMock.getInputStream()).thenReturn(fileInputStream);
    when(reconUtilsMock.makeHttpCall(any(), anyString(), anyBoolean())).thenReturn(httpURLConnectionMock);
    ReconOMMetadataManager reconOMMetadataManager = mock(ReconOMMetadataManager.class);
    ReconTaskController reconTaskController = getMockTaskController();
    OzoneManagerServiceProviderImpl ozoneManagerServiceProvider = new OzoneManagerServiceProviderImpl(configuration, reconOMMetadataManager, reconTaskController, reconUtilsMock, ozoneManagerProtocol);
    DBCheckpoint checkpoint = ozoneManagerServiceProvider.getOzoneManagerDBSnapshot();
    assertNotNull(checkpoint);
    assertTrue(checkpoint.getCheckpointLocation().toFile().isDirectory());
    assertTrue(checkpoint.getCheckpointLocation().toFile().listFiles().length == 2);
}
Also used : ReconUtils(org.apache.hadoop.ozone.recon.ReconUtils) FileInputStream(java.io.FileInputStream) InputStream(java.io.InputStream) DBCheckpoint(org.apache.hadoop.hdds.utils.db.DBCheckpoint) ReconTaskController(org.apache.hadoop.ozone.recon.tasks.ReconTaskController) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) FileInputStream(java.io.FileInputStream) BufferedWriter(java.io.BufferedWriter) HttpURLConnection(java.net.HttpURLConnection) FileOutputStream(java.io.FileOutputStream) OutputStreamWriter(java.io.OutputStreamWriter) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager) ReconUtils.createTarFile(org.apache.hadoop.ozone.recon.ReconUtils.createTarFile) File(java.io.File) Test(org.junit.Test)

Example 5 with ReconOMMetadataManager

use of org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager in project ozone by apache.

the class TestTableCountTask method initializeInjector.

private void initializeInjector() throws IOException {
    ReconOMMetadataManager omMetadataManager = getTestReconOmMetadataManager(initializeNewOmMetadataManager(temporaryFolder.newFolder()), temporaryFolder.newFolder());
    globalStatsDao = getDao(GlobalStatsDao.class);
    tableCountTask = new TableCountTask(globalStatsDao, getConfiguration(), omMetadataManager);
    dslContext = getDslContext();
}
Also used : GlobalStatsDao(org.hadoop.ozone.recon.schema.tables.daos.GlobalStatsDao) ReconOMMetadataManager(org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager)

Aggregations

ReconOMMetadataManager (org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager)9 Test (org.junit.Test)6 ReconUtils (org.apache.hadoop.ozone.recon.ReconUtils)4 ReconTaskController (org.apache.hadoop.ozone.recon.tasks.ReconTaskController)4 DBCheckpoint (org.apache.hadoop.hdds.utils.db.DBCheckpoint)3 OMMetadataManager (org.apache.hadoop.ozone.om.OMMetadataManager)3 ReconTaskStatus (org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus)3 File (java.io.File)2 FileInputStream (java.io.FileInputStream)2 InputStream (java.io.InputStream)2 HttpURLConnection (java.net.HttpURLConnection)2 ReconUtils.createTarFile (org.apache.hadoop.ozone.recon.ReconUtils.createTarFile)2 OzoneManagerSyncMetrics (org.apache.hadoop.ozone.recon.metrics.OzoneManagerSyncMetrics)2 ReconTaskStatusDao (org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao)2 BufferedWriter (java.io.BufferedWriter)1 FileOutputStream (java.io.FileOutputStream)1 OutputStreamWriter (java.io.OutputStreamWriter)1 ArrayList (java.util.ArrayList)1 Response (javax.ws.rs.core.Response)1 BlockID (org.apache.hadoop.hdds.client.BlockID)1