Search in sources :

Example 1 with IndexedTable

use of io.cdap.cdap.api.dataset.lib.IndexedTable in project cdap by caskdata.

the class PartitionedFileSetDefinition method getDataset.

@Override
public PartitionedFileSet getDataset(DatasetContext datasetContext, DatasetSpecification spec, Map<String, String> arguments, ClassLoader classLoader) throws IOException {
    // properties must contain the partitioning
    Partitioning partitioning = PartitionedFileSetProperties.getPartitioning(spec.getProperties());
    // make any necessary updates to the arguments
    arguments = updateArgumentsIfNeeded(arguments, partitioning);
    FileSet fileset = filesetDef.getDataset(datasetContext, spec.getSpecification(FILESET_NAME), arguments, classLoader);
    IndexedTable table = indexedTableDef.getDataset(datasetContext, spec.getSpecification(PARTITION_TABLE_NAME), arguments, classLoader);
    return new PartitionedFileSetDataset(datasetContext, spec.getName(), partitioning, fileset, table, spec, arguments, getExploreProvider());
}
Also used : Partitioning(io.cdap.cdap.api.dataset.lib.Partitioning) PartitionedFileSet(io.cdap.cdap.api.dataset.lib.PartitionedFileSet) FileSet(io.cdap.cdap.api.dataset.lib.FileSet) IndexedTable(io.cdap.cdap.api.dataset.lib.IndexedTable)

Example 2 with IndexedTable

use of io.cdap.cdap.api.dataset.lib.IndexedTable in project cdap by caskdata.

the class TimePartitionedFileSetDefinition method getDataset.

@Override
public PartitionedFileSet getDataset(DatasetContext datasetContext, DatasetSpecification spec, Map<String, String> arguments, ClassLoader classLoader) throws IOException {
    // make any necessary updates to the arguments
    arguments = updateArgumentsIfNeeded(arguments);
    FileSet fileset = filesetDef.getDataset(datasetContext, spec.getSpecification(FILESET_NAME), arguments, classLoader);
    IndexedTable table = indexedTableDef.getDataset(datasetContext, spec.getSpecification(PARTITION_TABLE_NAME), arguments, classLoader);
    return new TimePartitionedFileSetDataset(datasetContext, spec.getName(), fileset, table, spec, arguments, getExploreProvider());
}
Also used : FileSet(io.cdap.cdap.api.dataset.lib.FileSet) PartitionedFileSet(io.cdap.cdap.api.dataset.lib.PartitionedFileSet) IndexedTable(io.cdap.cdap.api.dataset.lib.IndexedTable)

Aggregations

FileSet (io.cdap.cdap.api.dataset.lib.FileSet)2 IndexedTable (io.cdap.cdap.api.dataset.lib.IndexedTable)2 PartitionedFileSet (io.cdap.cdap.api.dataset.lib.PartitionedFileSet)2 Partitioning (io.cdap.cdap.api.dataset.lib.Partitioning)1