Search in sources :

Example 21 with StoreFileInfo

use of org.apache.hadoop.hbase.regionserver.StoreFileInfo in project hbase by apache.

the class TestMajorCompactionRequest method mockStoreFiles.

protected List<StoreFileInfo> mockStoreFiles(Path regionStoreDir, int howMany, long timestamp) throws IOException {
    List<StoreFileInfo> infos = Lists.newArrayList();
    int i = 0;
    while (i < howMany) {
        StoreFileInfo storeFileInfo = mock(StoreFileInfo.class);
        doReturn(timestamp).doReturn(timestamp).when(storeFileInfo).getModificationTime();
        doReturn(new Path(regionStoreDir, RandomStringUtils.randomAlphabetic(10))).when(storeFileInfo).getPath();
        infos.add(storeFileInfo);
        i++;
    }
    return infos;
}
Also used : Path(org.apache.hadoop.fs.Path) StoreFileInfo(org.apache.hadoop.hbase.regionserver.StoreFileInfo)

Example 22 with StoreFileInfo

use of org.apache.hadoop.hbase.regionserver.StoreFileInfo in project hbase by apache.

the class TestMajorCompactionRequest method testIfWeHaveNewReferenceFilesButOldStoreFiles.

@Test
public void testIfWeHaveNewReferenceFilesButOldStoreFiles() throws Exception {
    // this tests that reference files that are new, but have older timestamps for the files
    // they reference still will get compacted.
    TableName table = TableName.valueOf("TestMajorCompactor");
    TableDescriptor htd = UTILITY.createTableDescriptor(table, Bytes.toBytes(FAMILY));
    RegionInfo hri = RegionInfoBuilder.newBuilder(htd.getTableName()).build();
    HRegion region = HBaseTestingUtil.createRegionAndWAL(hri, rootRegionDir, UTILITY.getConfiguration(), htd);
    Connection connection = mock(Connection.class);
    // the reference file timestamp is newer
    List<StoreFileInfo> storeFiles = mockStoreFiles(regionStoreDir, 4, 101);
    List<Path> paths = storeFiles.stream().map(StoreFileInfo::getPath).collect(Collectors.toList());
    // the files that are referenced are older, thus we still compact.
    HRegionFileSystem fileSystem = mockFileSystem(region.getRegionInfo(), true, storeFiles, 50);
    MajorCompactionRequest majorCompactionRequest = spy(new MajorCompactionRequest(connection, region.getRegionInfo(), Sets.newHashSet(FAMILY)));
    doReturn(paths).when(majorCompactionRequest).getReferenceFilePaths(any(FileSystem.class), any(Path.class));
    doReturn(fileSystem).when(majorCompactionRequest).getFileSystem();
    Set<String> result = majorCompactionRequest.getStoresRequiringCompaction(Sets.newHashSet("a"), 100);
    assertEquals(FAMILY, Iterables.getOnlyElement(result));
}
Also used : Path(org.apache.hadoop.fs.Path) Connection(org.apache.hadoop.hbase.client.Connection) RegionInfo(org.apache.hadoop.hbase.client.RegionInfo) Matchers.anyString(org.mockito.Matchers.anyString) TableDescriptor(org.apache.hadoop.hbase.client.TableDescriptor) TableName(org.apache.hadoop.hbase.TableName) HRegion(org.apache.hadoop.hbase.regionserver.HRegion) HRegionFileSystem(org.apache.hadoop.hbase.regionserver.HRegionFileSystem) FileSystem(org.apache.hadoop.fs.FileSystem) HRegionFileSystem(org.apache.hadoop.hbase.regionserver.HRegionFileSystem) StoreFileInfo(org.apache.hadoop.hbase.regionserver.StoreFileInfo) Test(org.junit.Test)

Aggregations

StoreFileInfo (org.apache.hadoop.hbase.regionserver.StoreFileInfo)22 Path (org.apache.hadoop.fs.Path)14 HRegionFileSystem (org.apache.hadoop.hbase.regionserver.HRegionFileSystem)9 ArrayList (java.util.ArrayList)8 IOException (java.io.IOException)6 Configuration (org.apache.hadoop.conf.Configuration)6 StoreFileTracker (org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTracker)6 ColumnFamilyDescriptor (org.apache.hadoop.hbase.client.ColumnFamilyDescriptor)5 SnapshotRegionManifest (org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest)5 HashMap (java.util.HashMap)4 StoreFileList (org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList)4 InterruptedIOException (java.io.InterruptedIOException)3 Map (java.util.Map)3 FileSystem (org.apache.hadoop.fs.FileSystem)3 HColumnDescriptor (org.apache.hadoop.hbase.HColumnDescriptor)3 TableDescriptor (org.apache.hadoop.hbase.client.TableDescriptor)3 Collection (java.util.Collection)2 List (java.util.List)2 ExecutionException (java.util.concurrent.ExecutionException)2 Future (java.util.concurrent.Future)2