Search in sources :

Example 11 with DatasetModule

use of co.cask.cdap.api.dataset.module.DatasetModule in project cdap by caskdata.

the class DatasetTypeService method getExtensionModules.

private Map<String, DatasetModule> getExtensionModules(CConfiguration cConf) {
    Map<String, DatasetModule> modules = new LinkedHashMap<>();
    String moduleStr = cConf.get(Constants.Dataset.Extensions.MODULES);
    if (moduleStr != null) {
        for (String moduleName : Splitter.on(',').omitEmptyStrings().split(moduleStr)) {
            // create DatasetModule object
            try {
                Class tableModuleClass = Class.forName(moduleName);
                DatasetModule module = (DatasetModule) tableModuleClass.newInstance();
                modules.put(moduleName, module);
            } catch (ClassCastException | ClassNotFoundException | InstantiationException | IllegalAccessException ex) {
                LOG.error("Failed to add {} extension module: {}", moduleName, ex.toString());
            }
        }
    }
    return modules;
}
Also used : DatasetModule(co.cask.cdap.api.dataset.module.DatasetModule) LinkedHashMap(java.util.LinkedHashMap)

Example 12 with DatasetModule

use of co.cask.cdap.api.dataset.module.DatasetModule in project cdap by caskdata.

the class AbstractDatasetFrameworkTest method testNamespaceModuleIsolation.

@Test
public void testNamespaceModuleIsolation() throws Exception {
    DatasetFramework framework = getFramework();
    // create 2 namespaces
    NamespaceId namespace1 = new NamespaceId("ns1");
    NamespaceId namespace2 = new NamespaceId("ns2");
    namespaceAdmin.create(new NamespaceMeta.Builder().setName(namespace1).build());
    namespaceAdmin.create(new NamespaceMeta.Builder().setName(namespace2).build());
    namespacedLocationFactory.get(namespace1).mkdirs();
    namespacedLocationFactory.get(namespace2).mkdirs();
    // add modules in each namespace, with one module that shares the same name
    DatasetModuleId simpleModuleNs1 = namespace1.datasetModule(SimpleKVTable.class.getName());
    DatasetModuleId simpleModuleNs2 = namespace2.datasetModule(SimpleKVTable.class.getName());
    DatasetModuleId doubleModuleNs2 = namespace2.datasetModule(DoubleWrappedKVTable.class.getName());
    DatasetModule module1 = new SingleTypeModule(SimpleKVTable.class);
    DatasetModule module2 = new SingleTypeModule(DoubleWrappedKVTable.class);
    framework.addModule(simpleModuleNs1, module1);
    framework.addModule(simpleModuleNs2, module1);
    framework.addModule(doubleModuleNs2, module2);
    // check that we can add instances of datasets in those modules
    framework.addInstance(SimpleKVTable.class.getName(), namespace1.dataset("kv1"), DatasetProperties.EMPTY);
    framework.addInstance(SimpleKVTable.class.getName(), namespace2.dataset("kv1"), DatasetProperties.EMPTY);
    // check that only namespace2 can add an instance of this type, since the module should only be in namespace2
    framework.addInstance(DoubleWrappedKVTable.class.getName(), namespace2.dataset("kv2"), DatasetProperties.EMPTY);
    try {
        framework.addInstance(DoubleWrappedKVTable.class.getName(), namespace2.dataset("kv2"), DatasetProperties.EMPTY);
        Assert.fail();
    } catch (Exception e) {
    // expected
    }
    // check that deleting all modules from namespace2 does not affect namespace1
    framework.deleteAllInstances(namespace2);
    framework.deleteAllModules(namespace2);
    // should still be able to add an instance in namespace1
    framework.addInstance(SimpleKVTable.class.getName(), namespace1.dataset("kv3"), DatasetProperties.EMPTY);
    // but not in namespace2
    try {
        framework.addInstance(SimpleKVTable.class.getName(), namespace2.dataset("kv3"), DatasetProperties.EMPTY);
        Assert.fail();
    } catch (Exception e) {
    // expected
    }
    // add back modules to namespace2
    framework.addModule(simpleModuleNs2, module1);
    framework.addModule(doubleModuleNs2, module2);
    // check that deleting a single module from namespace1 does not affect namespace2
    framework.deleteAllInstances(namespace1);
    framework.deleteModule(simpleModuleNs1);
    // should still be able to add an instance in namespace2
    framework.addInstance(DoubleWrappedKVTable.class.getName(), namespace2.dataset("kv1"), DatasetProperties.EMPTY);
    // but not in namespace1
    try {
        framework.addInstance(SimpleKVTable.class.getName(), namespace1.dataset("kv1"), DatasetProperties.EMPTY);
        Assert.fail();
    } catch (Exception e) {
    // expected
    }
}
Also used : LineageWriterDatasetFramework(co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework) DatasetModuleId(co.cask.cdap.proto.id.DatasetModuleId) NamespaceMeta(co.cask.cdap.proto.NamespaceMeta) NamespaceId(co.cask.cdap.proto.id.NamespaceId) DatasetModule(co.cask.cdap.api.dataset.module.DatasetModule) InstanceConflictException(co.cask.cdap.api.dataset.InstanceConflictException) DatasetManagementException(co.cask.cdap.api.dataset.DatasetManagementException) IOException(java.io.IOException) Test(org.junit.Test)

Aggregations

DatasetModule (co.cask.cdap.api.dataset.module.DatasetModule)12 AbstractModule (com.google.inject.AbstractModule)6 DatasetModuleId (co.cask.cdap.proto.id.DatasetModuleId)4 Constants (co.cask.cdap.common.conf.Constants)3 ConfigModule (co.cask.cdap.common.guice.ConfigModule)3 ExternalDatasetModule (co.cask.cdap.data2.dataset2.lib.external.ExternalDatasetModule)3 MetadataDatasetModule (co.cask.cdap.data2.metadata.dataset.MetadataDatasetModule)3 LineageDatasetModule (co.cask.cdap.data2.metadata.lineage.LineageDatasetModule)3 UsageDatasetModule (co.cask.cdap.data2.registry.UsageDatasetModule)3 HBaseQueueDatasetModule (co.cask.cdap.data2.transaction.queue.hbase.HBaseQueueDatasetModule)3 MetricsCollectionService (co.cask.cdap.api.metrics.MetricsCollectionService)2 DiscoveryRuntimeModule (co.cask.cdap.common.guice.DiscoveryRuntimeModule)2 NoOpMetricsCollectionService (co.cask.cdap.common.metrics.NoOpMetricsCollectionService)2 SystemDatasetInstantiatorFactory (co.cask.cdap.data.dataset.SystemDatasetInstantiatorFactory)2 DynamicTransactionExecutorFactory (co.cask.cdap.data.runtime.DynamicTransactionExecutorFactory)2 DatasetInstanceManager (co.cask.cdap.data2.datafabric.dataset.instance.DatasetInstanceManager)2 DatasetAdminOpHTTPHandler (co.cask.cdap.data2.datafabric.dataset.service.executor.DatasetAdminOpHTTPHandler)2 DatasetAdminService (co.cask.cdap.data2.datafabric.dataset.service.executor.DatasetAdminService)2 DatasetOpExecutor (co.cask.cdap.data2.datafabric.dataset.service.executor.DatasetOpExecutor)2 DatasetOpExecutorService (co.cask.cdap.data2.datafabric.dataset.service.executor.DatasetOpExecutorService)2