Search in sources :

Example 21 with Stoppable

use of org.apache.hadoop.hbase.Stoppable in project hbase by apache.

the class TestCleanerChore method testDeletesEmptyDirectories.

@Test
public void testDeletesEmptyDirectories() throws Exception {
    Stoppable stop = new StoppableImplementation();
    Configuration conf = UTIL.getConfiguration();
    Path testDir = UTIL.getDataTestDir();
    FileSystem fs = UTIL.getTestFileSystem();
    String confKey = "hbase.test.cleaner.delegates";
    conf.set(confKey, AlwaysDelete.class.getName());
    AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey, POOL);
    // create the directory layout in the directory to clean
    Path parent = new Path(testDir, "parent");
    Path child = new Path(parent, "child");
    Path emptyChild = new Path(parent, "emptyChild");
    Path file = new Path(child, "someFile");
    fs.mkdirs(child);
    fs.mkdirs(emptyChild);
    // touch a new file
    fs.create(file).close();
    // also create a file in the top level directory
    Path topFile = new Path(testDir, "topFile");
    fs.create(topFile).close();
    assertTrue("Test file didn't get created.", fs.exists(file));
    assertTrue("Test file didn't get created.", fs.exists(topFile));
    // run the chore
    chore.chore();
    // verify all the files got deleted
    assertFalse("File didn't get deleted", fs.exists(topFile));
    assertFalse("File didn't get deleted", fs.exists(file));
    assertFalse("Empty directory didn't get deleted", fs.exists(child));
    assertFalse("Empty directory didn't get deleted", fs.exists(parent));
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) StoppableImplementation(org.apache.hadoop.hbase.util.StoppableImplementation) FileSystem(org.apache.hadoop.fs.FileSystem) FilterFileSystem(org.apache.hadoop.fs.FilterFileSystem) Stoppable(org.apache.hadoop.hbase.Stoppable) Test(org.junit.Test)

Example 22 with Stoppable

use of org.apache.hadoop.hbase.Stoppable in project hbase by apache.

the class TestCleanerChore method testDoesNotCheckDirectories.

/**
 * Test to make sure that we don't attempt to ask the delegate whether or not we should preserve a
 * directory.
 * @throws Exception on failure
 */
@Test
public void testDoesNotCheckDirectories() throws Exception {
    Stoppable stop = new StoppableImplementation();
    Configuration conf = UTIL.getConfiguration();
    Path testDir = UTIL.getDataTestDir();
    FileSystem fs = UTIL.getTestFileSystem();
    String confKey = "hbase.test.cleaner.delegates";
    conf.set(confKey, AlwaysDelete.class.getName());
    AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey, POOL);
    // spy on the delegate to ensure that we don't check for directories
    AlwaysDelete delegate = (AlwaysDelete) chore.cleanersChain.get(0);
    AlwaysDelete spy = Mockito.spy(delegate);
    chore.cleanersChain.set(0, spy);
    // create the directory layout in the directory to clean
    Path parent = new Path(testDir, "parent");
    Path file = new Path(parent, "someFile");
    fs.mkdirs(parent);
    assertTrue("Test parent didn't get created.", fs.exists(parent));
    // touch a new file
    fs.create(file).close();
    assertTrue("Test file didn't get created.", fs.exists(file));
    FileStatus fStat = fs.getFileStatus(parent);
    chore.chore();
    // make sure we never checked the directory
    Mockito.verify(spy, Mockito.never()).isFileDeletable(fStat);
    Mockito.reset(spy);
}
Also used : Path(org.apache.hadoop.fs.Path) FileStatus(org.apache.hadoop.fs.FileStatus) Configuration(org.apache.hadoop.conf.Configuration) StoppableImplementation(org.apache.hadoop.hbase.util.StoppableImplementation) FileSystem(org.apache.hadoop.fs.FileSystem) FilterFileSystem(org.apache.hadoop.fs.FilterFileSystem) Stoppable(org.apache.hadoop.hbase.Stoppable) Test(org.junit.Test)

Example 23 with Stoppable

use of org.apache.hadoop.hbase.Stoppable in project hbase by apache.

the class TestCleanerChore method testDeleteFileWithCleanerDisabled.

@Test
public void testDeleteFileWithCleanerDisabled() throws Exception {
    Stoppable stop = new StoppableImplementation();
    Configuration conf = UTIL.getConfiguration();
    Path testDir = UTIL.getDataTestDir();
    FileSystem fs = UTIL.getTestFileSystem();
    String confKey = "hbase.test.cleaner.delegates";
    conf.set(confKey, AlwaysDelete.class.getName());
    AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey, POOL);
    // Disable cleaner
    chore.setEnabled(false);
    // create the directory layout in the directory to clean
    Path parent = new Path(testDir, "parent");
    Path child = new Path(parent, "child");
    Path file = new Path(child, "someFile");
    fs.mkdirs(child);
    // touch a new file
    fs.create(file).close();
    assertTrue("Test file didn't get created.", fs.exists(file));
    // run the chore
    chore.chore();
    // verify all the files exist
    assertTrue("File got deleted with cleaner disabled", fs.exists(file));
    assertTrue("Directory got deleted", fs.exists(child));
    assertTrue("Directory got deleted", fs.exists(parent));
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) StoppableImplementation(org.apache.hadoop.hbase.util.StoppableImplementation) FileSystem(org.apache.hadoop.fs.FileSystem) FilterFileSystem(org.apache.hadoop.fs.FilterFileSystem) Stoppable(org.apache.hadoop.hbase.Stoppable) Test(org.junit.Test)

Example 24 with Stoppable

use of org.apache.hadoop.hbase.Stoppable in project hbase by apache.

the class TestCleanerChore method testDeleteFileWithCleanerEnabled.

@Test
public void testDeleteFileWithCleanerEnabled() throws Exception {
    Stoppable stop = new StoppableImplementation();
    Configuration conf = UTIL.getConfiguration();
    Path testDir = UTIL.getDataTestDir();
    FileSystem fs = UTIL.getTestFileSystem();
    String confKey = "hbase.test.cleaner.delegates";
    conf.set(confKey, AlwaysDelete.class.getName());
    AllValidPaths chore = new AllValidPaths("test-file-cleaner", stop, conf, fs, testDir, confKey, POOL);
    // Enable cleaner
    chore.setEnabled(true);
    // create the directory layout in the directory to clean
    Path parent = new Path(testDir, "parent");
    Path child = new Path(parent, "child");
    Path file = new Path(child, "someFile");
    fs.mkdirs(child);
    // touch a new file
    fs.create(file).close();
    assertTrue("Test file didn't get created.", fs.exists(file));
    // run the chore
    chore.chore();
    // verify all the files got deleted
    assertFalse("File didn't get deleted", fs.exists(file));
    assertFalse("Empty directory didn't get deleted", fs.exists(child));
    assertFalse("Empty directory didn't get deleted", fs.exists(parent));
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) StoppableImplementation(org.apache.hadoop.hbase.util.StoppableImplementation) FileSystem(org.apache.hadoop.fs.FileSystem) FilterFileSystem(org.apache.hadoop.fs.FilterFileSystem) Stoppable(org.apache.hadoop.hbase.Stoppable) Test(org.junit.Test)

Example 25 with Stoppable

use of org.apache.hadoop.hbase.Stoppable in project phoenix by apache.

the class TestIndexWriter method testSynchronouslyCompletesAllWrites.

/**
 * With the move to using a pool of threads to write, we need to ensure that we still block until
 * all index writes for a mutation/batch are completed.
 * @throws Exception on failure
 */
@SuppressWarnings({ "unchecked", "deprecation" })
@Test
public void testSynchronouslyCompletesAllWrites() throws Exception {
    LOG.info("Starting " + testName.getTableNameString());
    LOG.info("Current thread is interrupted: " + Thread.interrupted());
    Abortable abort = new StubAbortable();
    Stoppable stop = Mockito.mock(Stoppable.class);
    RegionCoprocessorEnvironment e = Mockito.mock(RegionCoprocessorEnvironment.class);
    Configuration conf = new Configuration();
    Mockito.when(e.getConfiguration()).thenReturn(conf);
    Mockito.when(e.getSharedData()).thenReturn(new ConcurrentHashMap<String, Object>());
    Region mockRegion = Mockito.mock(Region.class);
    Mockito.when(e.getRegion()).thenReturn(mockRegion);
    HTableDescriptor mockTableDesc = Mockito.mock(HTableDescriptor.class);
    Mockito.when(mockRegion.getTableDesc()).thenReturn(mockTableDesc);
    ExecutorService exec = Executors.newFixedThreadPool(1);
    Map<ImmutableBytesPtr, HTableInterface> tables = new HashMap<ImmutableBytesPtr, HTableInterface>();
    FakeTableFactory factory = new FakeTableFactory(tables);
    byte[] tableName = this.testName.getTableName();
    Put m = new Put(row);
    m.add(Bytes.toBytes("family"), Bytes.toBytes("qual"), null);
    Collection<Pair<Mutation, byte[]>> indexUpdates = Arrays.asList(new Pair<Mutation, byte[]>(m, tableName));
    HTableInterface table = Mockito.mock(HTableInterface.class);
    final boolean[] completed = new boolean[] { false };
    Mockito.when(table.batch(Mockito.anyList())).thenAnswer(new Answer<Void>() {

        @Override
        public Void answer(InvocationOnMock invocation) throws Throwable {
            // just keep track that it was called
            completed[0] = true;
            return null;
        }
    });
    Mockito.when(table.getTableName()).thenReturn(testName.getTableName());
    // add the table to the set of tables, so its returned to the writer
    tables.put(new ImmutableBytesPtr(tableName), table);
    // setup the writer and failure policy
    TrackingParallelWriterIndexCommitter committer = new TrackingParallelWriterIndexCommitter(VersionInfo.getVersion());
    committer.setup(factory, exec, abort, stop, e);
    KillServerOnFailurePolicy policy = new KillServerOnFailurePolicy();
    policy.setup(stop, abort);
    IndexWriter writer = new IndexWriter(committer, policy);
    writer.write(indexUpdates);
    assertTrue("Writer returned before the table batch completed! Likely a race condition tripped", completed[0]);
    writer.stop(this.testName.getTableNameString() + " finished");
    assertTrue("Factory didn't get shutdown after writer#stop!", factory.shutdown);
    assertTrue("ExectorService isn't terminated after writer#stop!", exec.isShutdown());
}
Also used : StubAbortable(org.apache.phoenix.hbase.index.StubAbortable) Configuration(org.apache.hadoop.conf.Configuration) HashMap(java.util.HashMap) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) HTableInterface(org.apache.hadoop.hbase.client.HTableInterface) RegionCoprocessorEnvironment(org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment) Abortable(org.apache.hadoop.hbase.Abortable) StubAbortable(org.apache.phoenix.hbase.index.StubAbortable) Pair(org.apache.hadoop.hbase.util.Pair) ImmutableBytesPtr(org.apache.phoenix.hbase.index.util.ImmutableBytesPtr) Stoppable(org.apache.hadoop.hbase.Stoppable) Put(org.apache.hadoop.hbase.client.Put) HTableDescriptor(org.apache.hadoop.hbase.HTableDescriptor) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ExecutorService(java.util.concurrent.ExecutorService) Region(org.apache.hadoop.hbase.regionserver.Region) Mutation(org.apache.hadoop.hbase.client.Mutation) Test(org.junit.Test)

Aggregations

Stoppable (org.apache.hadoop.hbase.Stoppable)34 Configuration (org.apache.hadoop.conf.Configuration)31 Test (org.junit.Test)31 FileSystem (org.apache.hadoop.fs.FileSystem)16 Path (org.apache.hadoop.fs.Path)16 StoppableImplementation (org.apache.hadoop.hbase.util.StoppableImplementation)15 FilterFileSystem (org.apache.hadoop.fs.FilterFileSystem)11 InvocationOnMock (org.mockito.invocation.InvocationOnMock)8 ArrayList (java.util.ArrayList)7 ExecutorService (java.util.concurrent.ExecutorService)7 Abortable (org.apache.hadoop.hbase.Abortable)7 RegionCoprocessorEnvironment (org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment)7 StubAbortable (org.apache.phoenix.hbase.index.StubAbortable)7 HTableInterface (org.apache.hadoop.hbase.client.HTableInterface)5 Mutation (org.apache.hadoop.hbase.client.Mutation)5 Put (org.apache.hadoop.hbase.client.Put)5 SnapshotManager (org.apache.hadoop.hbase.master.snapshot.SnapshotManager)5 ImmutableBytesPtr (org.apache.phoenix.hbase.index.util.ImmutableBytesPtr)5 CountDownLatch (java.util.concurrent.CountDownLatch)4 HTableDescriptor (org.apache.hadoop.hbase.HTableDescriptor)4