Search in sources :

Example 36 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by axbaretto.

the class TestDynamicUDFSupport method testSuccessfulUnregistrationAfterSeveralRetryAttempts.

@Test
public void testSuccessfulUnregistrationAfterSeveralRetryAttempts() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    copyDefaultJarsToStagingArea();
    test("create function using jar '%s'", default_binary_name);
    reset(remoteFunctionRegistry);
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).doThrow(new VersionMismatchException("Version mismatch detected", 1)).doCallRealMethod().when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "The following UDFs in jar %s have been unregistered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("drop function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, default_binary_name)).go();
    verify(remoteFunctionRegistry, times(3)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertFalse("Registry area should be empty", fs.listFiles(remoteFunctionRegistry.getRegistryArea(), false).hasNext());
    assertEquals("Registry should be empty", remoteFunctionRegistry.getRegistry(new DataChangeVersion()).getJarList().size(), 0);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) FileSystem(org.apache.hadoop.fs.FileSystem) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) Matchers.anyString(org.mockito.Matchers.anyString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) SlowTest(org.apache.drill.categories.SlowTest) SqlFunctionTest(org.apache.drill.categories.SqlFunctionTest) Test(org.junit.Test)

Example 37 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by axbaretto.

the class TestDynamicUDFSupport method testSuccessfulRegistrationAfterSeveralRetryAttempts.

@Test
public void testSuccessfulRegistrationAfterSeveralRetryAttempts() throws Exception {
    final RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    final Path registryPath = hadoopToJavaPath(remoteFunctionRegistry.getRegistryArea());
    final Path stagingPath = hadoopToJavaPath(remoteFunctionRegistry.getStagingArea());
    final Path tmpPath = hadoopToJavaPath(remoteFunctionRegistry.getTmpArea());
    copyDefaultJarsToStagingArea();
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).doThrow(new VersionMismatchException("Version mismatch detected", 1)).doCallRealMethod().when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "The following UDFs in jar %s have been registered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("create function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, default_binary_name)).go();
    verify(remoteFunctionRegistry, times(3)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    assertTrue("Staging area should be empty", ArrayUtils.isEmpty(stagingPath.toFile().listFiles()));
    assertTrue("Temporary area should be empty", ArrayUtils.isEmpty(tmpPath.toFile().listFiles()));
    assertTrue("Binary should be present in registry area", registryPath.resolve(default_binary_name).toFile().exists());
    assertTrue("Source should be present in registry area", registryPath.resolve(default_source_name).toFile().exists());
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), default_binary_name);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) HadoopUtils.hadoopToJavaPath(org.apache.drill.test.HadoopUtils.hadoopToJavaPath) Path(java.nio.file.Path) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) Matchers.anyString(org.mockito.Matchers.anyString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) SlowTest(org.apache.drill.categories.SlowTest) SqlFunctionTest(org.apache.drill.categories.SqlFunctionTest) Test(org.junit.Test)

Example 38 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testSuccessfulRegistrationAfterSeveralRetryAttempts.

@Test
public void testSuccessfulRegistrationAfterSeveralRetryAttempts() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    Path registryPath = hadoopToJavaPath(remoteFunctionRegistry.getRegistryArea());
    Path stagingPath = hadoopToJavaPath(remoteFunctionRegistry.getStagingArea());
    Path tmpPath = hadoopToJavaPath(remoteFunctionRegistry.getTmpArea());
    copyDefaultJarsToStagingArea();
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).doThrow(new VersionMismatchException("Version mismatch detected", 1)).doCallRealMethod().when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "The following UDFs in jar %s have been registered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("create function using jar '%s'", defaultBinaryJar).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, defaultBinaryJar)).go();
    verify(remoteFunctionRegistry, times(3)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    assertTrue("Staging area should be empty", ArrayUtils.isEmpty(stagingPath.toFile().listFiles()));
    assertTrue("Temporary area should be empty", ArrayUtils.isEmpty(tmpPath.toFile().listFiles()));
    assertTrue("Binary should be present in registry area", registryPath.resolve(defaultBinaryJar).toFile().exists());
    assertTrue("Source should be present in registry area", registryPath.resolve(defaultSourceJar).toFile().exists());
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), defaultBinaryJar);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) HadoopUtils.hadoopToJavaPath(org.apache.drill.test.HadoopUtils.hadoopToJavaPath) Path(java.nio.file.Path) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) SlowTest(org.apache.drill.categories.SlowTest) SqlFunctionTest(org.apache.drill.categories.SqlFunctionTest) Test(org.junit.Test)

Example 39 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testExceedRetryAttemptsDuringUnregistration.

@Test
public void testExceedRetryAttemptsDuringUnregistration() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    Path registryPath = hadoopToJavaPath(remoteFunctionRegistry.getRegistryArea());
    copyDefaultJarsToStagingArea();
    test("create function using jar '%s'", defaultBinaryJar);
    reset(remoteFunctionRegistry);
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "Failed to update remote function registry. Exceeded retry attempts limit.";
    testBuilder().sqlQuery("drop function using jar '%s'", defaultBinaryJar).unOrdered().baselineColumns("ok", "summary").baselineValues(false, summary).go();
    verify(remoteFunctionRegistry, times(remoteFunctionRegistry.getRetryAttempts() + 1)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    assertTrue("Binary should be present in registry area", registryPath.resolve(defaultBinaryJar).toFile().exists());
    assertTrue("Source should be present in registry area", registryPath.resolve(defaultSourceJar).toFile().exists());
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), defaultBinaryJar);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) HadoopUtils.hadoopToJavaPath(org.apache.drill.test.HadoopUtils.hadoopToJavaPath) Path(java.nio.file.Path) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) SlowTest(org.apache.drill.categories.SlowTest) SqlFunctionTest(org.apache.drill.categories.SqlFunctionTest) Test(org.junit.Test)

Example 40 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testSuccessfulRegistration.

@Test
public void testSuccessfulRegistration() throws Exception {
    copyDefaultJarsToStagingArea();
    String summary = "The following UDFs in jar %s have been registered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("create function using jar '%s'", defaultBinaryJar).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, defaultBinaryJar)).go();
    RemoteFunctionRegistry remoteFunctionRegistry = getDrillbitContext().getRemoteFunctionRegistry();
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertFalse("Staging area should be empty", fs.listFiles(remoteFunctionRegistry.getStagingArea(), false).hasNext());
    assertFalse("Temporary area should be empty", fs.listFiles(remoteFunctionRegistry.getTmpArea(), false).hasNext());
    Path path = hadoopToJavaPath(remoteFunctionRegistry.getRegistryArea());
    assertTrue("Binary should be present in registry area", path.resolve(defaultBinaryJar).toFile().exists());
    assertTrue("Source should be present in registry area", path.resolve(defaultBinaryJar).toFile().exists());
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), defaultBinaryJar);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) HadoopUtils.hadoopToJavaPath(org.apache.drill.test.HadoopUtils.hadoopToJavaPath) Path(java.nio.file.Path) FileSystem(org.apache.hadoop.fs.FileSystem) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) SlowTest(org.apache.drill.categories.SlowTest) SqlFunctionTest(org.apache.drill.categories.SqlFunctionTest) Test(org.junit.Test)

Aggregations

DataChangeVersion (org.apache.drill.exec.store.sys.store.DataChangeVersion)43 Test (org.junit.Test)34 RemoteFunctionRegistry (org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry)28 Registry (org.apache.drill.exec.proto.UserBitShared.Registry)25 CoreMatchers.containsString (org.hamcrest.CoreMatchers.containsString)24 FunctionImplementationRegistry (org.apache.drill.exec.expr.fn.FunctionImplementationRegistry)23 LocalFunctionRegistry (org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry)21 VersionMismatchException (org.apache.drill.exec.exception.VersionMismatchException)19 SlowTest (org.apache.drill.categories.SlowTest)16 SqlFunctionTest (org.apache.drill.categories.SqlFunctionTest)16 Matchers.anyString (org.mockito.Matchers.anyString)16 Path (java.nio.file.Path)11 HadoopUtils.hadoopToJavaPath (org.apache.drill.test.HadoopUtils.hadoopToJavaPath)11 FileSystem (org.apache.hadoop.fs.FileSystem)11 Jar (org.apache.drill.exec.proto.UserBitShared.Jar)10 DrillRuntimeException (org.apache.drill.common.exceptions.DrillRuntimeException)9 IOException (java.io.IOException)8 Path (org.apache.hadoop.fs.Path)8 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)8 CountDownLatch (java.util.concurrent.CountDownLatch)6