Search in sources :

Example 1 with NotEnoughPermitsException

use of org.apache.gobblin.util.limiter.NotEnoughPermitsException in project incubator-gobblin by apache.

the class ThrottledFileSystemTest method testListing.

@Test
public void testListing() throws Exception {
    FileSystem fs = Mockito.mock(FileSystem.class);
    Mockito.when(fs.listStatus(Mockito.any(Path.class))).thenAnswer(new Answer<FileStatus[]>() {

        @Override
        public FileStatus[] answer(InvocationOnMock invocation) throws Throwable {
            Path path = (Path) invocation.getArguments()[0];
            int files = Integer.parseInt(path.getName());
            FileStatus status = new FileStatus(0, false, 0, 0, 0, new Path("/"));
            FileStatus[] out = new FileStatus[files];
            for (int i = 0; i < files; i++) {
                out[i] = status;
            }
            return out;
        }
    });
    Limiter limiter = new CountBasedLimiter(5);
    ThrottledFileSystem throttledFileSystem = new ThrottledFileSystem(fs, limiter, "testService");
    Assert.assertEquals(throttledFileSystem.getServiceName(), "testService");
    // use 1 permit
    Assert.assertNotNull(throttledFileSystem.listStatus(new Path("/files/99")));
    // use 3 permits
    Assert.assertNotNull(throttledFileSystem.listStatus(new Path("/files/250")));
    try {
        // requires 2 permits
        throttledFileSystem.listStatus(new Path("/files/150"));
        Assert.fail();
    } catch (NotEnoughPermitsException expected) {
    // Expected
    }
    // requires 1 permit
    Assert.assertNotNull(throttledFileSystem.listStatus(new Path("/files/99")));
}
Also used : Path(org.apache.hadoop.fs.Path) CountBasedLimiter(org.apache.gobblin.util.limiter.CountBasedLimiter) FileStatus(org.apache.hadoop.fs.FileStatus) InvocationOnMock(org.mockito.invocation.InvocationOnMock) FileSystem(org.apache.hadoop.fs.FileSystem) NotEnoughPermitsException(org.apache.gobblin.util.limiter.NotEnoughPermitsException) Limiter(org.apache.gobblin.util.limiter.Limiter) CountBasedLimiter(org.apache.gobblin.util.limiter.CountBasedLimiter) Test(org.testng.annotations.Test)

Example 2 with NotEnoughPermitsException

use of org.apache.gobblin.util.limiter.NotEnoughPermitsException in project incubator-gobblin by apache.

the class ThrottledFileSystemTest method testSimpleCalls.

@Test
public void testSimpleCalls() throws Exception {
    FileSystem fs = Mockito.mock(FileSystem.class);
    Mockito.when(fs.getFileStatus(Mockito.any(Path.class))).thenReturn(new FileStatus(0, false, 0, 0, 0, new Path("/")));
    Limiter limiter = new CountBasedLimiter(2);
    ThrottledFileSystem throttledFileSystem = new ThrottledFileSystem(fs, limiter, "testService");
    Assert.assertNotNull(throttledFileSystem.getFileStatus(new Path("/myFile")));
    Assert.assertNotNull(throttledFileSystem.getFileStatus(new Path("/myFile")));
    try {
        throttledFileSystem.getFileStatus(new Path("/myFile"));
        Assert.fail();
    } catch (NotEnoughPermitsException expected) {
    // Expected
    }
}
Also used : Path(org.apache.hadoop.fs.Path) CountBasedLimiter(org.apache.gobblin.util.limiter.CountBasedLimiter) FileStatus(org.apache.hadoop.fs.FileStatus) FileSystem(org.apache.hadoop.fs.FileSystem) NotEnoughPermitsException(org.apache.gobblin.util.limiter.NotEnoughPermitsException) Limiter(org.apache.gobblin.util.limiter.Limiter) CountBasedLimiter(org.apache.gobblin.util.limiter.CountBasedLimiter) Test(org.testng.annotations.Test)

Aggregations

CountBasedLimiter (org.apache.gobblin.util.limiter.CountBasedLimiter)2 Limiter (org.apache.gobblin.util.limiter.Limiter)2 NotEnoughPermitsException (org.apache.gobblin.util.limiter.NotEnoughPermitsException)2 FileStatus (org.apache.hadoop.fs.FileStatus)2 FileSystem (org.apache.hadoop.fs.FileSystem)2 Path (org.apache.hadoop.fs.Path)2 Test (org.testng.annotations.Test)2 InvocationOnMock (org.mockito.invocation.InvocationOnMock)1