Search in sources :

Example 1 with DatasetType

use of io.cdap.cdap.data2.datafabric.dataset.DatasetType in project cdap by caskdata.

the class InMemoryDatasetOpExecutor method create.

@Override
public DatasetSpecification create(DatasetId datasetInstanceId, DatasetTypeMeta typeMeta, DatasetProperties props) throws Exception {
    DatasetType type = client.getDatasetType(typeMeta, null, new ConstantClassLoaderProvider());
    if (type == null) {
        throw new IllegalArgumentException("Dataset type cannot be instantiated for provided type meta: " + typeMeta);
    }
    DatasetSpecification spec = type.configure(datasetInstanceId.getEntityName(), props);
    DatasetAdmin admin = type.getAdmin(DatasetContext.from(datasetInstanceId.getNamespace()), spec);
    admin.create();
    return spec;
}
Also used : ConstantClassLoaderProvider(co.cask.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider) DatasetSpecification(co.cask.cdap.api.dataset.DatasetSpecification) DatasetAdmin(co.cask.cdap.api.dataset.DatasetAdmin) DatasetType(co.cask.cdap.data2.datafabric.dataset.DatasetType)

Example 2 with DatasetType

use of io.cdap.cdap.data2.datafabric.dataset.DatasetType in project cdap by caskdata.

the class InMemoryDatasetOpExecutor method drop.

@Override
public void drop(DatasetId datasetInstanceId, DatasetTypeMeta typeMeta, DatasetSpecification spec) throws Exception {
    DatasetType type = client.getDatasetType(typeMeta, null, new ConstantClassLoaderProvider());
    if (type == null) {
        throw new IllegalArgumentException("Dataset type cannot be instantiated for provided type meta: " + typeMeta);
    }
    DatasetAdmin admin = type.getAdmin(DatasetContext.from(datasetInstanceId.getNamespace()), spec);
    admin.drop();
}
Also used : ConstantClassLoaderProvider(io.cdap.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin) DatasetType(io.cdap.cdap.data2.datafabric.dataset.DatasetType)

Example 3 with DatasetType

use of io.cdap.cdap.data2.datafabric.dataset.DatasetType in project cdap by caskdata.

the class InMemoryDatasetOpExecutor method create.

@Override
public DatasetCreationResponse create(DatasetId datasetInstanceId, DatasetTypeMeta typeMeta, DatasetProperties props) throws Exception {
    DatasetType type = client.getDatasetType(typeMeta, null, new ConstantClassLoaderProvider());
    if (type == null) {
        throw new IllegalArgumentException("Dataset type cannot be instantiated for provided type meta: " + typeMeta);
    }
    DatasetSpecification spec = type.configure(datasetInstanceId.getEntityName(), props);
    DatasetAdmin admin = type.getAdmin(DatasetContext.from(datasetInstanceId.getNamespace()), spec);
    admin.create();
    return new DatasetCreationResponse(spec, null);
}
Also used : ConstantClassLoaderProvider(io.cdap.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider) DatasetSpecification(io.cdap.cdap.api.dataset.DatasetSpecification) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin) DatasetType(io.cdap.cdap.data2.datafabric.dataset.DatasetType)

Example 4 with DatasetType

use of io.cdap.cdap.data2.datafabric.dataset.DatasetType in project cdap by caskdata.

the class InMemoryDatasetOpExecutor method update.

@Override
public DatasetCreationResponse update(DatasetId datasetInstanceId, DatasetTypeMeta typeMeta, DatasetProperties props, DatasetSpecification existing) throws Exception {
    DatasetType type = client.getDatasetType(typeMeta, null, new ConstantClassLoaderProvider());
    if (type == null) {
        throw new IllegalArgumentException("Dataset type cannot be instantiated for provided type meta: " + typeMeta);
    }
    try {
        DatasetSpecification spec = type.reconfigure(datasetInstanceId.getEntityName(), props, existing);
        DatasetAdmin admin = type.getAdmin(DatasetContext.from(datasetInstanceId.getNamespace()), spec);
        if (admin instanceof Updatable) {
            ((Updatable) admin).update(existing);
        } else {
            admin.create();
        }
        if (spec.getDescription() == null && existing.getDescription() != null) {
            spec.setDescription(existing.getDescription());
        }
        return new DatasetCreationResponse(spec, null);
    } catch (IncompatibleUpdateException e) {
        throw new ConflictException(e.getMessage());
    }
}
Also used : ConflictException(io.cdap.cdap.common.ConflictException) Updatable(io.cdap.cdap.api.dataset.Updatable) ConstantClassLoaderProvider(io.cdap.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider) DatasetSpecification(io.cdap.cdap.api.dataset.DatasetSpecification) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin) DatasetType(io.cdap.cdap.data2.datafabric.dataset.DatasetType) IncompatibleUpdateException(io.cdap.cdap.api.dataset.IncompatibleUpdateException)

Example 5 with DatasetType

use of io.cdap.cdap.data2.datafabric.dataset.DatasetType in project cdap by caskdata.

the class DatasetAdminService method createOrUpdate.

/**
 * Configures and creates a Dataset
 *
 * @param datasetInstanceId dataset instance to be created
 * @param typeMeta type meta for the dataset
 * @param props dataset instance properties
 * @param existing if dataset already exists (in case of update), the existing properties
 * @return dataset specification
 */
public DatasetCreationResponse createOrUpdate(final DatasetId datasetInstanceId, final DatasetTypeMeta typeMeta, final DatasetProperties props, @Nullable final DatasetSpecification existing) throws Exception {
    if (existing == null) {
        LOG.info("Creating dataset instance {}, type meta: {}", datasetInstanceId, typeMeta);
    } else {
        LOG.info("Updating dataset instance {}, type meta: {}, existing: {}", datasetInstanceId, typeMeta, existing);
    }
    try (DatasetClassLoaderProvider classLoaderProvider = new DirectoryClassLoaderProvider(cConf, locationFactory)) {
        final DatasetContext context = DatasetContext.from(datasetInstanceId.getNamespace());
        UserGroupInformation ugi = getUgiForDataset(impersonator, datasetInstanceId);
        final DatasetType type = ImpersonationUtils.doAs(ugi, () -> {
            LOG.trace("Getting dataset type {}", typeMeta.getName());
            DatasetType type1 = dsFramework.getDatasetType(typeMeta, null, classLoaderProvider);
            if (type1 == null) {
                throw new BadRequestException(String.format("Cannot instantiate dataset type using provided type meta: %s", typeMeta));
            }
            LOG.trace("Got dataset type {}", typeMeta.getName());
            return type1;
        });
        DatasetSpecification spec = ImpersonationUtils.doAs(ugi, () -> {
            LOG.trace("Configuring dataset {} of type {}", datasetInstanceId.getDataset(), typeMeta.getName());
            DatasetSpecification spec1 = existing == null ? type.configure(datasetInstanceId.getEntityName(), props) : type.reconfigure(datasetInstanceId.getEntityName(), props, existing);
            LOG.trace("Configured dataset {} of type {}", datasetInstanceId.getDataset(), typeMeta.getName());
            DatasetAdmin admin = type.getAdmin(context, spec1);
            try {
                if (existing != null) {
                    if (admin instanceof Updatable) {
                        ((Updatable) admin).update(existing);
                    } else {
                        admin.upgrade();
                    }
                } else {
                    LOG.trace("Creating dataset {} of type {}", datasetInstanceId.getDataset(), typeMeta.getName());
                    admin.create();
                    LOG.trace("Created dataset {} of type {}", datasetInstanceId.getDataset(), typeMeta.getName());
                }
            } finally {
                Closeables.closeQuietly(admin);
            }
            return spec1;
        });
        // Writing system metadata should be done without impersonation since user may not have access to system tables.
        LOG.trace("Computing metadata for dataset {}", datasetInstanceId.getDataset());
        SystemMetadata metadata = computeSystemMetadata(datasetInstanceId, spec, props, typeMeta, type, context, existing != null, ugi);
        LOG.trace("Computed metadata for dataset {}", datasetInstanceId.getDataset());
        return new DatasetCreationResponse(spec, metadata);
    } catch (Exception e) {
        if (e instanceof IncompatibleUpdateException) {
            // this is expected to happen if user provides bad update properties, so we log this as debug
            LOG.debug("Incompatible update for dataset '{}'", datasetInstanceId, e);
        } else {
            LOG.error("Error {} dataset '{}': {}", existing == null ? "creating" : "updating", datasetInstanceId, e.getMessage(), e);
        }
        throw e;
    }
}
Also used : DatasetSpecification(io.cdap.cdap.api.dataset.DatasetSpecification) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin) DatasetType(io.cdap.cdap.data2.datafabric.dataset.DatasetType) IncompatibleUpdateException(io.cdap.cdap.api.dataset.IncompatibleUpdateException) AccessException(io.cdap.cdap.api.security.AccessException) IOException(java.io.IOException) BadRequestException(io.cdap.cdap.common.BadRequestException) NotFoundException(io.cdap.cdap.common.NotFoundException) DirectoryClassLoaderProvider(io.cdap.cdap.data2.datafabric.dataset.type.DirectoryClassLoaderProvider) Updatable(io.cdap.cdap.api.dataset.Updatable) SystemMetadata(io.cdap.cdap.data2.metadata.system.SystemMetadata) BadRequestException(io.cdap.cdap.common.BadRequestException) DatasetClassLoaderProvider(io.cdap.cdap.data2.datafabric.dataset.type.DatasetClassLoaderProvider) DatasetContext(io.cdap.cdap.api.dataset.DatasetContext) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation) IncompatibleUpdateException(io.cdap.cdap.api.dataset.IncompatibleUpdateException)

Aggregations

DatasetAdmin (co.cask.cdap.api.dataset.DatasetAdmin)5 DatasetType (co.cask.cdap.data2.datafabric.dataset.DatasetType)5 DatasetAdmin (io.cdap.cdap.api.dataset.DatasetAdmin)5 DatasetType (io.cdap.cdap.data2.datafabric.dataset.DatasetType)5 UserGroupInformation (org.apache.hadoop.security.UserGroupInformation)4 DatasetSpecification (co.cask.cdap.api.dataset.DatasetSpecification)3 ConstantClassLoaderProvider (co.cask.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider)3 DatasetSpecification (io.cdap.cdap.api.dataset.DatasetSpecification)3 IncompatibleUpdateException (io.cdap.cdap.api.dataset.IncompatibleUpdateException)3 BadRequestException (io.cdap.cdap.common.BadRequestException)3 ConstantClassLoaderProvider (io.cdap.cdap.data2.datafabric.dataset.type.ConstantClassLoaderProvider)3 IOException (java.io.IOException)3 IncompatibleUpdateException (co.cask.cdap.api.dataset.IncompatibleUpdateException)2 Updatable (co.cask.cdap.api.dataset.Updatable)2 BadRequestException (co.cask.cdap.common.BadRequestException)2 DatasetClassLoaderProvider (co.cask.cdap.data2.datafabric.dataset.type.DatasetClassLoaderProvider)2 DirectoryClassLoaderProvider (co.cask.cdap.data2.datafabric.dataset.type.DirectoryClassLoaderProvider)2 Updatable (io.cdap.cdap.api.dataset.Updatable)2 AccessException (io.cdap.cdap.api.security.AccessException)2 NotFoundException (io.cdap.cdap.common.NotFoundException)2