use of org.apache.drill.common.scanner.persistence.ScanResult in project drill by apache.
the class Drillbit method start.
public static Drillbit start(final DrillConfig config, final RemoteServiceSet remoteServiceSet) throws DrillbitStartupException {
logger.debug("Starting new Drillbit.");
// TODO: allow passing as a parameter
ScanResult classpathScan = ClassPathScanner.fromPrescan(config);
Drillbit bit;
try {
bit = new Drillbit(config, remoteServiceSet, classpathScan);
} catch (final Exception ex) {
throw new DrillbitStartupException("Failure while initializing values in Drillbit.", ex);
}
try {
bit.run();
} catch (final Exception e) {
logger.error("Failure during initial startup of Drillbit.", e);
bit.close();
throw new DrillbitStartupException("Failure during initial startup of Drillbit.", e);
}
logger.debug("Started new Drillbit.");
return bit;
}
use of org.apache.drill.common.scanner.persistence.ScanResult in project drill by apache.
the class TestClassPathScanner method test.
@Test
public void test() throws Exception {
ScanResult result = ClassPathScanner.fromPrescan(DrillConfig.create());
// if the build has run properly. BuildTimeScan.REGISTRY_FILE was created with a prescan
// assertListEqualsUnordered(result.getPrescannedPackages(),
// "org.apache.drill.common.logical",
// "org.apache.drill.exec.expr",
// "org.apache.drill.exec.physical.base",
// "org.apache.drill.exec.expr.fn.impl",
// "org.apache.drill.exec.physical.impl",
// "org.apache.drill.exec.rpc.user.security",
// "org.apache.drill.exec.store",
// "org.apache.drill.exec.store.mock",
// "org.apache.drill.exec.physical.config",
// "org.apache.drill.storage"
// );
// // this is added in the unit test folder that was not scanned so far
// assertListEqualsUnordered(result.getScannedPackages(),
// "org.apache.drill.exec.testing",
// "org.apache.drill.exec.fn.impl.testing",
// "org.apache.drill.exec.rpc.user.security.testing"
// );
List<AnnotatedClassDescriptor> functions = result.getAnnotatedClasses();
Set<String> scanned = new HashSet<>();
AnnotatedClassDescriptor functionRandomBigIntGauss = null;
for (AnnotatedClassDescriptor function : functions) {
assertTrue(function.getClassName() + " scanned twice", scanned.add(function.getClassName()));
if (function.getClassName().equals(RandomBigIntGauss.class.getName())) {
functionRandomBigIntGauss = function;
}
}
if (functionRandomBigIntGauss == null) {
Assert.fail("functionRandomBigIntGauss not found");
}
// TODO: use Andrew's randomized test framework to verify a subset of the functions
for (AnnotatedClassDescriptor function : functions) {
Class<?> c = Class.forName(function.getClassName(), false, this.getClass().getClassLoader());
Field[] fields = c.getDeclaredFields();
assertEquals("fields count for " + function, fields.length, function.getFields().size());
for (int i = 0; i < fields.length; i++) {
FieldDescriptor fieldDescriptor = function.getFields().get(i);
Field field = fields[i];
assertEquals("Class fields:\n" + Arrays.toString(fields) + "\n != \nDescriptor fields:\n" + function.getFields(), field.getName(), fieldDescriptor.getName());
verifyAnnotations(field.getDeclaredAnnotations(), fieldDescriptor.getAnnotations());
assertEquals(field.getType(), fieldDescriptor.getFieldClass());
}
Annotation[] annotations = c.getDeclaredAnnotations();
List<AnnotationDescriptor> scannedAnnotations = function.getAnnotations();
verifyAnnotations(annotations, scannedAnnotations);
FunctionTemplate bytecodeAnnotation = function.getAnnotationProxy(FunctionTemplate.class);
FunctionTemplate reflectionAnnotation = c.getAnnotation(FunctionTemplate.class);
assertEquals(reflectionAnnotation.name(), bytecodeAnnotation.name());
assertArrayEquals(reflectionAnnotation.names(), bytecodeAnnotation.names());
assertEquals(reflectionAnnotation.scope(), bytecodeAnnotation.scope());
assertEquals(reflectionAnnotation.nulls(), bytecodeAnnotation.nulls());
assertEquals(reflectionAnnotation.isBinaryCommutative(), bytecodeAnnotation.isBinaryCommutative());
assertEquals(reflectionAnnotation.desc(), bytecodeAnnotation.desc());
assertEquals(reflectionAnnotation.costCategory(), bytecodeAnnotation.costCategory());
}
for (String baseType : result.getScannedClasses()) {
validateType(result, baseType);
}
assertTrue(result.getImplementations(PhysicalOperator.class).size() > 0);
assertTrue(result.getImplementations(DrillFunc.class).size() > 0);
}
use of org.apache.drill.common.scanner.persistence.ScanResult in project drill by apache.
the class FunctionImplementationRegistry method syncWithRemoteRegistry.
/**
* Purpose of this method is to synchronize remote and local function registries if needed
* and to inform if function registry was changed after given version.
*
* To make synchronization as much light-weigh as possible, first only versions of both registries are checked
* without any locking. If synchronization is needed, enters synchronized block to prevent others loading the same jars.
* The need of synchronization is checked again (double-check lock) before comparing jars.
* If any missing jars are found, they are downloaded to local udf area, each is wrapped into {@link JarScan}.
* Once jar download is finished, all missing jars are registered in one batch.
* In case if any errors during jars download / registration, these errors are logged.
*
* During registration local function registry is updated with remote function registry version it is synced with.
* When at least one jar of the missing jars failed to download / register,
* local function registry version are not updated but jars that where successfully downloaded / registered
* are added to local function registry.
*
* If synchronization between remote and local function registry was not needed,
* checks if given registry version matches latest sync version
* to inform if function registry was changed after given version.
*
* @param version remote function registry local function registry was based on
* @return true if remote and local function registries were synchronized after given version
*/
@SuppressWarnings("resource")
public boolean syncWithRemoteRegistry(long version) {
if (isRegistrySyncNeeded(remoteFunctionRegistry.getRegistryVersion(), localFunctionRegistry.getVersion())) {
synchronized (this) {
long localRegistryVersion = localFunctionRegistry.getVersion();
if (isRegistrySyncNeeded(remoteFunctionRegistry.getRegistryVersion(), localRegistryVersion)) {
DataChangeVersion remoteVersion = new DataChangeVersion();
List<String> missingJars = getMissingJars(this.remoteFunctionRegistry, localFunctionRegistry, remoteVersion);
List<JarScan> jars = Lists.newArrayList();
if (!missingJars.isEmpty()) {
logger.info("Starting dynamic UDFs lazy-init process.\n" + "The following jars are going to be downloaded and registered locally: " + missingJars);
for (String jarName : missingJars) {
Path binary = null;
Path source = null;
URLClassLoader classLoader = null;
try {
binary = copyJarToLocal(jarName, this.remoteFunctionRegistry);
source = copyJarToLocal(JarUtil.getSourceName(jarName), this.remoteFunctionRegistry);
URL[] urls = { binary.toUri().toURL(), source.toUri().toURL() };
classLoader = new URLClassLoader(urls);
ScanResult scanResult = scan(classLoader, binary, urls);
localFunctionRegistry.validate(jarName, scanResult);
jars.add(new JarScan(jarName, scanResult, classLoader));
} catch (Exception e) {
deleteQuietlyLocalJar(binary);
deleteQuietlyLocalJar(source);
if (classLoader != null) {
try {
classLoader.close();
} catch (Exception ex) {
logger.warn("Problem during closing class loader for {}", jarName, e);
}
}
logger.error("Problem during remote functions load from {}", jarName, e);
}
}
}
long latestRegistryVersion = jars.size() != missingJars.size() ? localRegistryVersion : remoteVersion.getVersion();
localFunctionRegistry.register(jars, latestRegistryVersion);
return true;
}
}
}
return version != localFunctionRegistry.getVersion();
}
use of org.apache.drill.common.scanner.persistence.ScanResult in project drill by apache.
the class StoragePlugins method main.
public static void main(String[] args) throws Exception {
DrillConfig config = DrillConfig.create();
ScanResult scanResult = ClassPathScanner.fromPrescan(config);
LogicalPlanPersistence lpp = new LogicalPlanPersistence(config, scanResult);
String data = Resources.toString(Resources.getResource("storage-engines.json"), Charsets.UTF_8);
StoragePlugins se = lpp.getMapper().readValue(data, StoragePlugins.class);
ByteArrayOutputStream os = new ByteArrayOutputStream();
lpp.getMapper().writeValue(System.out, se);
lpp.getMapper().writeValue(os, se);
se = lpp.getMapper().readValue(new ByteArrayInputStream(os.toByteArray()), StoragePlugins.class);
System.out.println(se);
}
use of org.apache.drill.common.scanner.persistence.ScanResult in project drill by axbaretto.
the class RunTimeScan method fromPrescan.
/**
* loads prescanned classpath info and scans for extra ones based on configuration.
* (unless prescan is disabled with {@see ClassPathScanner#IMPLEMENTATIONS_SCAN_CACHE}=falses)
* @param config to retrieve the packages to scan
* @return the scan result
*/
public static ScanResult fromPrescan(DrillConfig config) {
List<String> packagePrefixes = ClassPathScanner.getPackagePrefixes(config);
List<String> scannedBaseClasses = ClassPathScanner.getScannedBaseClasses(config);
List<String> scannedAnnotations = ClassPathScanner.getScannedAnnotations(config);
if (ClassPathScanner.isScanBuildTimeCacheEnabled(config)) {
// scan only locations that have not been scanned yet
ScanResult runtimeScan = ClassPathScanner.scan(NON_PRESCANNED_MARKED_PATHS, packagePrefixes, scannedBaseClasses, scannedAnnotations, PRESCANNED);
return runtimeScan.merge(PRESCANNED);
} else {
// scan everything
return ClassPathScanner.scan(ClassPathScanner.getMarkedPaths(), packagePrefixes, scannedBaseClasses, scannedAnnotations, ClassPathScanner.emptyResult());
}
}
Aggregations