Search in sources :

Example 1 with PartitionedFileSetModule

use of io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule in project cdap by caskdata.

the class AbstractDatasetFrameworkTest method testMultipleTransitiveDependencies.

@Test
public void testMultipleTransitiveDependencies() throws DatasetManagementException, IOException, UnauthorizedException {
    // Adding modules
    DatasetFramework framework = getFramework();
    try {
        framework.addModule(IN_MEMORY, new InMemoryTableModule());
        framework.addModule(CORE, new CoreDatasetsModule());
        framework.addModule(FILE, new FileSetModule());
        framework.addModule(PFS, new PartitionedFileSetModule());
        framework.addModule(TWICE, new SingleTypeModule(EmbedsTableTwiceDataset.class));
        // Creating an instances
        framework.addInstance(EmbedsTableTwiceDataset.class.getName(), MY_DS, PartitionedFileSetProperties.builder().setPartitioning(Partitioning.builder().addStringField("x").build()).build());
        Assert.assertTrue(framework.hasInstance(MY_DS));
        framework.getDataset(MY_DS, DatasetProperties.EMPTY.getProperties(), null);
    } finally {
        framework.deleteAllInstances(NAMESPACE_ID);
        framework.deleteAllModules(NAMESPACE_ID);
    }
}
Also used : LineageWriterDatasetFramework(io.cdap.cdap.data2.metadata.writer.LineageWriterDatasetFramework) InMemoryTableModule(io.cdap.cdap.data2.dataset2.module.lib.inmemory.InMemoryTableModule) CoreDatasetsModule(io.cdap.cdap.data2.dataset2.lib.table.CoreDatasetsModule) FileSetModule(io.cdap.cdap.data2.dataset2.lib.file.FileSetModule) PartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) PartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) Test(org.junit.Test)

Example 2 with PartitionedFileSetModule

use of io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule in project cdap by caskdata.

the class SystemDatasetRuntimeModule method bindDefaultModules.

/**
 * Add bindings for Dataset modules that are available by default
 */
private void bindDefaultModules(MapBinder<String, DatasetModule> mapBinder) {
    mapBinder.addBinding("core").toInstance(new CoreDatasetsModule());
    mapBinder.addBinding(FileSetDataset.TYPE).toInstance(new FileSetModule());
    mapBinder.addBinding(TimePartitionedFileSet.TYPE).toInstance(new TimePartitionedFileSetModule());
    mapBinder.addBinding(PartitionedFileSet.TYPE).toInstance(new PartitionedFileSetModule());
    mapBinder.addBinding(ObjectMappedTable.TYPE).toInstance(new ObjectMappedTableModule());
    mapBinder.addBinding(Cube.TYPE).toInstance(new CubeModule());
    mapBinder.addBinding(ExternalDataset.TYPE).toInstance(new ExternalDatasetModule());
}
Also used : ExternalDatasetModule(io.cdap.cdap.data2.dataset2.lib.external.ExternalDatasetModule) TimePartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule) ObjectMappedTableModule(io.cdap.cdap.data2.dataset2.lib.table.ObjectMappedTableModule) CoreDatasetsModule(io.cdap.cdap.data2.dataset2.lib.table.CoreDatasetsModule) CubeModule(io.cdap.cdap.data2.dataset2.lib.table.CubeModule) FileSetModule(io.cdap.cdap.data2.dataset2.lib.file.FileSetModule) PartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) TimePartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule) PartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) TimePartitionedFileSetModule(io.cdap.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule)

Example 3 with PartitionedFileSetModule

use of io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule in project cdap by caskdata.

the class SystemDatasetRuntimeModule method bindDefaultModules.

/**
 * Add bindings for Dataset modules that are available by default
 */
private void bindDefaultModules(MapBinder<String, DatasetModule> mapBinder) {
    mapBinder.addBinding("core").toInstance(new CoreDatasetsModule());
    mapBinder.addBinding("fileSet").toInstance(new FileSetModule());
    mapBinder.addBinding("timePartitionedFileSet").toInstance(new TimePartitionedFileSetModule());
    mapBinder.addBinding("partitionedFileSet").toInstance(new PartitionedFileSetModule());
    mapBinder.addBinding("objectMappedTable").toInstance(new ObjectMappedTableModule());
    mapBinder.addBinding("cube").toInstance(new CubeModule());
    mapBinder.addBinding("usage").toInstance(new UsageDatasetModule());
    mapBinder.addBinding("metadata").toInstance(new MetadataDatasetModule());
    mapBinder.addBinding("lineage").toInstance(new LineageDatasetModule());
    mapBinder.addBinding("externalDataset").toInstance(new ExternalDatasetModule());
}
Also used : MetadataDatasetModule(co.cask.cdap.data2.metadata.dataset.MetadataDatasetModule) LineageDatasetModule(co.cask.cdap.data2.metadata.lineage.LineageDatasetModule) ExternalDatasetModule(co.cask.cdap.data2.dataset2.lib.external.ExternalDatasetModule) TimePartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule) ObjectMappedTableModule(co.cask.cdap.data2.dataset2.lib.table.ObjectMappedTableModule) CoreDatasetsModule(co.cask.cdap.data2.dataset2.lib.table.CoreDatasetsModule) UsageDatasetModule(co.cask.cdap.data2.registry.UsageDatasetModule) CubeModule(co.cask.cdap.data2.dataset2.lib.table.CubeModule) PartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) FileSetModule(co.cask.cdap.data2.dataset2.lib.file.FileSetModule) TimePartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule) PartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) TimePartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule)

Example 4 with PartitionedFileSetModule

use of io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule in project cdap by caskdata.

the class AbstractDatasetFrameworkTest method testMultipleTransitiveDependencies.

@Test
public void testMultipleTransitiveDependencies() throws DatasetManagementException, IOException {
    // Adding modules
    DatasetFramework framework = getFramework();
    try {
        framework.addModule(IN_MEMORY, new InMemoryTableModule());
        framework.addModule(CORE, new CoreDatasetsModule());
        framework.addModule(FILE, new FileSetModule());
        framework.addModule(PFS, new PartitionedFileSetModule());
        framework.addModule(TWICE, new SingleTypeModule(EmbedsTableTwiceDataset.class));
        // Creating an instances
        framework.addInstance(EmbedsTableTwiceDataset.class.getName(), MY_DS, PartitionedFileSetProperties.builder().setPartitioning(Partitioning.builder().addStringField("x").build()).build());
        Assert.assertTrue(framework.hasInstance(MY_DS));
        framework.getDataset(MY_DS, DatasetProperties.EMPTY.getProperties(), null);
    } finally {
        framework.deleteAllInstances(NAMESPACE_ID);
        framework.deleteAllModules(NAMESPACE_ID);
    }
}
Also used : LineageWriterDatasetFramework(co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework) InMemoryTableModule(co.cask.cdap.data2.dataset2.module.lib.inmemory.InMemoryTableModule) CoreDatasetsModule(co.cask.cdap.data2.dataset2.lib.table.CoreDatasetsModule) PartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) FileSetModule(co.cask.cdap.data2.dataset2.lib.file.FileSetModule) PartitionedFileSetModule(co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule) Test(org.junit.Test)

Aggregations

FileSetModule (co.cask.cdap.data2.dataset2.lib.file.FileSetModule)2 PartitionedFileSetModule (co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule)2 CoreDatasetsModule (co.cask.cdap.data2.dataset2.lib.table.CoreDatasetsModule)2 FileSetModule (io.cdap.cdap.data2.dataset2.lib.file.FileSetModule)2 PartitionedFileSetModule (io.cdap.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule)2 CoreDatasetsModule (io.cdap.cdap.data2.dataset2.lib.table.CoreDatasetsModule)2 Test (org.junit.Test)2 ExternalDatasetModule (co.cask.cdap.data2.dataset2.lib.external.ExternalDatasetModule)1 TimePartitionedFileSetModule (co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule)1 CubeModule (co.cask.cdap.data2.dataset2.lib.table.CubeModule)1 ObjectMappedTableModule (co.cask.cdap.data2.dataset2.lib.table.ObjectMappedTableModule)1 InMemoryTableModule (co.cask.cdap.data2.dataset2.module.lib.inmemory.InMemoryTableModule)1 MetadataDatasetModule (co.cask.cdap.data2.metadata.dataset.MetadataDatasetModule)1 LineageDatasetModule (co.cask.cdap.data2.metadata.lineage.LineageDatasetModule)1 LineageWriterDatasetFramework (co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework)1 UsageDatasetModule (co.cask.cdap.data2.registry.UsageDatasetModule)1 ExternalDatasetModule (io.cdap.cdap.data2.dataset2.lib.external.ExternalDatasetModule)1 TimePartitionedFileSetModule (io.cdap.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule)1 CubeModule (io.cdap.cdap.data2.dataset2.lib.table.CubeModule)1 ObjectMappedTableModule (io.cdap.cdap.data2.dataset2.lib.table.ObjectMappedTableModule)1