Search in sources :

Example 1 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testExceedRetryAttemptsDuringUnregistration.

@Test
public void testExceedRetryAttemptsDuringUnregistration() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    copyDefaultJarsToStagingArea();
    test("create function using jar '%s'", default_binary_name);
    reset(remoteFunctionRegistry);
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "Failed to update remote function registry. Exceeded retry attempts limit.";
    testBuilder().sqlQuery("drop function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(false, summary).go();
    verify(remoteFunctionRegistry, times(remoteFunctionRegistry.getRetryAttempts() + 1)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertTrue("Binary should be present in registry area", fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), default_binary_name)));
    assertTrue("Source should be present in registry area", fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), default_source_name)));
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), default_binary_name);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Path(org.apache.hadoop.fs.Path) FileSystem(org.apache.hadoop.fs.FileSystem) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) Matchers.anyString(org.mockito.Matchers.anyString) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) Test(org.junit.Test)

Example 2 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class FunctionImplementationRegistry method syncWithRemoteRegistry.

/**
   * Purpose of this method is to synchronize remote and local function registries if needed
   * and to inform if function registry was changed after given version.
   *
   * To make synchronization as much light-weigh as possible, first only versions of both registries are checked
   * without any locking. If synchronization is needed, enters synchronized block to prevent others loading the same jars.
   * The need of synchronization is checked again (double-check lock) before comparing jars.
   * If any missing jars are found, they are downloaded to local udf area, each is wrapped into {@link JarScan}.
   * Once jar download is finished, all missing jars are registered in one batch.
   * In case if any errors during jars download / registration, these errors are logged.
   *
   * During registration local function registry is updated with remote function registry version it is synced with.
   * When at least one jar of the missing jars failed to download / register,
   * local function registry version are not updated but jars that where successfully downloaded / registered
   * are added to local function registry.
   *
   * If synchronization between remote and local function registry was not needed,
   * checks if given registry version matches latest sync version
   * to inform if function registry was changed after given version.
   *
   * @param version remote function registry local function registry was based on
   * @return true if remote and local function registries were synchronized after given version
   */
@SuppressWarnings("resource")
public boolean syncWithRemoteRegistry(long version) {
    if (isRegistrySyncNeeded(remoteFunctionRegistry.getRegistryVersion(), localFunctionRegistry.getVersion())) {
        synchronized (this) {
            long localRegistryVersion = localFunctionRegistry.getVersion();
            if (isRegistrySyncNeeded(remoteFunctionRegistry.getRegistryVersion(), localRegistryVersion)) {
                DataChangeVersion remoteVersion = new DataChangeVersion();
                List<String> missingJars = getMissingJars(this.remoteFunctionRegistry, localFunctionRegistry, remoteVersion);
                List<JarScan> jars = Lists.newArrayList();
                if (!missingJars.isEmpty()) {
                    logger.info("Starting dynamic UDFs lazy-init process.\n" + "The following jars are going to be downloaded and registered locally: " + missingJars);
                    for (String jarName : missingJars) {
                        Path binary = null;
                        Path source = null;
                        URLClassLoader classLoader = null;
                        try {
                            binary = copyJarToLocal(jarName, this.remoteFunctionRegistry);
                            source = copyJarToLocal(JarUtil.getSourceName(jarName), this.remoteFunctionRegistry);
                            URL[] urls = { binary.toUri().toURL(), source.toUri().toURL() };
                            classLoader = new URLClassLoader(urls);
                            ScanResult scanResult = scan(classLoader, binary, urls);
                            localFunctionRegistry.validate(jarName, scanResult);
                            jars.add(new JarScan(jarName, scanResult, classLoader));
                        } catch (Exception e) {
                            deleteQuietlyLocalJar(binary);
                            deleteQuietlyLocalJar(source);
                            if (classLoader != null) {
                                try {
                                    classLoader.close();
                                } catch (Exception ex) {
                                    logger.warn("Problem during closing class loader for {}", jarName, e);
                                }
                            }
                            logger.error("Problem during remote functions load from {}", jarName, e);
                        }
                    }
                }
                long latestRegistryVersion = jars.size() != missingJars.size() ? localRegistryVersion : remoteVersion.getVersion();
                localFunctionRegistry.register(jars, latestRegistryVersion);
                return true;
            }
        }
    }
    return version != localFunctionRegistry.getVersion();
}
Also used : Path(org.apache.hadoop.fs.Path) ScanResult(org.apache.drill.common.scanner.persistence.ScanResult) URLClassLoader(java.net.URLClassLoader) JarScan(org.apache.drill.exec.expr.fn.registry.JarScan) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) URL(java.net.URL) FunctionValidationException(org.apache.drill.exec.exception.FunctionValidationException) InvocationTargetException(java.lang.reflect.InvocationTargetException) DrillRuntimeException(org.apache.drill.common.exceptions.DrillRuntimeException) JarValidationException(org.apache.drill.exec.exception.JarValidationException) IOException(java.io.IOException)

Example 3 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testExceedRetryAttemptsDuringRegistration.

@Test
public void testExceedRetryAttemptsDuringRegistration() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    copyDefaultJarsToStagingArea();
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "Failed to update remote function registry. Exceeded retry attempts limit.";
    testBuilder().sqlQuery("create function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(false, summary).go();
    verify(remoteFunctionRegistry, times(remoteFunctionRegistry.getRetryAttempts() + 1)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertTrue("Binary should be present in staging area", fs.exists(new Path(remoteFunctionRegistry.getStagingArea(), default_binary_name)));
    assertTrue("Source should be present in staging area", fs.exists(new Path(remoteFunctionRegistry.getStagingArea(), default_source_name)));
    assertFalse("Registry area should be empty", fs.listFiles(remoteFunctionRegistry.getRegistryArea(), false).hasNext());
    assertFalse("Temporary area should be empty", fs.listFiles(remoteFunctionRegistry.getTmpArea(), false).hasNext());
    assertEquals("Registry should be empty", remoteFunctionRegistry.getRegistry(new DataChangeVersion()).getJarList().size(), 0);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Path(org.apache.hadoop.fs.Path) FileSystem(org.apache.hadoop.fs.FileSystem) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) Matchers.anyString(org.mockito.Matchers.anyString) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) Test(org.junit.Test)

Example 4 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testSuccessfulRegistration.

@Test
public void testSuccessfulRegistration() throws Exception {
    copyDefaultJarsToStagingArea();
    String summary = "The following UDFs in jar %s have been registered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("create function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, default_binary_name)).go();
    RemoteFunctionRegistry remoteFunctionRegistry = getDrillbitContext().getRemoteFunctionRegistry();
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertFalse("Staging area should be empty", fs.listFiles(remoteFunctionRegistry.getStagingArea(), false).hasNext());
    assertFalse("Temporary area should be empty", fs.listFiles(remoteFunctionRegistry.getTmpArea(), false).hasNext());
    assertTrue("Binary should be present in registry area", fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), default_binary_name)));
    assertTrue("Source should be present in registry area", fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), default_source_name)));
    Registry registry = remoteFunctionRegistry.getRegistry(new DataChangeVersion());
    assertEquals("Registry should contain one jar", registry.getJarList().size(), 1);
    assertEquals(registry.getJar(0).getName(), default_binary_name);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Path(org.apache.hadoop.fs.Path) FileSystem(org.apache.hadoop.fs.FileSystem) Matchers.anyString(org.mockito.Matchers.anyString) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) Test(org.junit.Test)

Example 5 with DataChangeVersion

use of org.apache.drill.exec.store.sys.store.DataChangeVersion in project drill by apache.

the class TestDynamicUDFSupport method testSuccessfulUnregistrationAfterSeveralRetryAttempts.

@Test
public void testSuccessfulUnregistrationAfterSeveralRetryAttempts() throws Exception {
    RemoteFunctionRegistry remoteFunctionRegistry = spyRemoteFunctionRegistry();
    copyDefaultJarsToStagingArea();
    test("create function using jar '%s'", default_binary_name);
    reset(remoteFunctionRegistry);
    doThrow(new VersionMismatchException("Version mismatch detected", 1)).doThrow(new VersionMismatchException("Version mismatch detected", 1)).doCallRealMethod().when(remoteFunctionRegistry).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    String summary = "The following UDFs in jar %s have been unregistered:\n" + "[custom_lower(VARCHAR-REQUIRED)]";
    testBuilder().sqlQuery("drop function using jar '%s'", default_binary_name).unOrdered().baselineColumns("ok", "summary").baselineValues(true, String.format(summary, default_binary_name)).go();
    verify(remoteFunctionRegistry, times(3)).updateRegistry(any(Registry.class), any(DataChangeVersion.class));
    FileSystem fs = remoteFunctionRegistry.getFs();
    assertFalse("Registry area should be empty", fs.listFiles(remoteFunctionRegistry.getRegistryArea(), false).hasNext());
    assertEquals("Registry should be empty", remoteFunctionRegistry.getRegistry(new DataChangeVersion()).getJarList().size(), 0);
}
Also used : RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) FileSystem(org.apache.hadoop.fs.FileSystem) LocalFunctionRegistry(org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry) FunctionImplementationRegistry(org.apache.drill.exec.expr.fn.FunctionImplementationRegistry) RemoteFunctionRegistry(org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry) Registry(org.apache.drill.exec.proto.UserBitShared.Registry) Matchers.anyString(org.mockito.Matchers.anyString) CoreMatchers.containsString(org.hamcrest.CoreMatchers.containsString) DataChangeVersion(org.apache.drill.exec.store.sys.store.DataChangeVersion) VersionMismatchException(org.apache.drill.exec.exception.VersionMismatchException) Test(org.junit.Test)

Aggregations

DataChangeVersion (org.apache.drill.exec.store.sys.store.DataChangeVersion)43 Test (org.junit.Test)34 RemoteFunctionRegistry (org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry)28 Registry (org.apache.drill.exec.proto.UserBitShared.Registry)25 CoreMatchers.containsString (org.hamcrest.CoreMatchers.containsString)24 FunctionImplementationRegistry (org.apache.drill.exec.expr.fn.FunctionImplementationRegistry)23 LocalFunctionRegistry (org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry)21 VersionMismatchException (org.apache.drill.exec.exception.VersionMismatchException)19 SlowTest (org.apache.drill.categories.SlowTest)16 SqlFunctionTest (org.apache.drill.categories.SqlFunctionTest)16 Matchers.anyString (org.mockito.Matchers.anyString)16 Path (java.nio.file.Path)11 HadoopUtils.hadoopToJavaPath (org.apache.drill.test.HadoopUtils.hadoopToJavaPath)11 FileSystem (org.apache.hadoop.fs.FileSystem)11 Jar (org.apache.drill.exec.proto.UserBitShared.Jar)10 DrillRuntimeException (org.apache.drill.common.exceptions.DrillRuntimeException)9 IOException (java.io.IOException)8 Path (org.apache.hadoop.fs.Path)8 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)8 CountDownLatch (java.util.concurrent.CountDownLatch)6