Search in sources :

Example 11 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class InMemoryFeedManagerFeedService method getFeedSummaryForCategory.

public List<FeedSummary> getFeedSummaryForCategory(String categoryId) {
    List<FeedSummary> summaryList = new ArrayList<>();
    FeedCategory category = categoryProvider.getCategoryById(categoryId);
    if (category != null && category.getFeeds() != null) {
        summaryList.addAll(category.getFeeds());
    }
    return summaryList;
}
Also used : FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) ArrayList(java.util.ArrayList) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)

Example 12 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class FeedImporter method validateEntityAccess.

private boolean validateEntityAccess(FeedMetadata existingFeed, String feedCategory) {
    FeedMetadata importingFeed = importFeed.getFeedToImport();
    if (existingFeed != null) {
        FeedMetadata userAccessFeed = metadataAccess.read(() -> {
            return metadataService.getFeedByName(feedCategory, importingFeed.getSystemFeedName());
        });
        if (userAccessFeed == null || !userAccessFeed.hasAction(FeedAccessControl.EDIT_DETAILS.getSystemName())) {
            // error
            importFeed.setValid(false);
            if (importFeed.getTemplate() == null) {
                ImportTemplate importTemplate = new ImportTemplate(importFeed.getFileName());
                importFeed.setTemplate(importTemplate);
            }
            String msg = "Access Denied.  You do not have access to edit this feed.";
            importFeed.getImportOptions().addErrorMessage(ImportComponent.FEED_DATA, msg);
            importFeed.addErrorMessage(existingFeed, msg);
            importFeed.setValid(false);
            return false;
        } else {
            return true;
        }
    } else {
        // ensure the user can create under the category
        Category category = metadataAccess.read(() -> {
            return categoryProvider.findBySystemName(feedCategory);
        }, MetadataAccess.SERVICE);
        if (category == null) {
            // ensure the user has functional access to create categories
            boolean hasPermission = accessController.hasPermission(AccessController.SERVICES, FeedServicesAccessControl.EDIT_CATEGORIES);
            if (!hasPermission) {
                String msg = "Access Denied. The category for this feed," + feedCategory + ", doesn't exist and you do not have access to create a new category.";
                importFeed.getImportOptions().addErrorMessage(ImportComponent.FEED_DATA, msg);
                importFeed.addErrorMessage(existingFeed, msg);
                importFeed.setValid(false);
                return false;
            }
            return true;
        } else {
            // if the feed is new ensure the user has write access to create feeds
            return metadataAccess.read(() -> {
                // Query for Category and ensure the user has access to create feeds on that category
                Category domainCategory = categoryProvider.findBySystemName(feedCategory);
                if (domainCategory == null || (!domainCategory.getAllowedActions().hasPermission(CategoryAccessControl.CREATE_FEED))) {
                    String msg = "Access Denied. You do not have access to create feeds under the category " + feedCategory + ". Attempt made to create feed " + FeedNameUtil.fullName(feedCategory, importingFeed.getSystemFeedName()) + ".";
                    importFeed.getImportOptions().addErrorMessage(ImportComponent.FEED_DATA, msg);
                    importFeed.addErrorMessage(existingFeed, msg);
                    importFeed.setValid(false);
                    return false;
                }
                /*
                       TemplateAccessControl.CREATE_FEED permission is not being used right now.
                       Uncomment this code once/if we should be checking it

                    // Query for Template and ensure the user has access to create feeds
                    final RegisteredTemplate domainTemplate = registeredTemplateService.findRegisteredTemplate(
                        new RegisteredTemplateRequest.Builder().templateName(importingFeed.getTemplateName()).isFeedEdit(true).build());
                    if (domainTemplate != null && !registeredTemplateService.hasTemplatePermission(domainTemplate.getId(), TemplateAccessControl.CREATE_FEED)) {
                        final String msg = "Access Denied. You do not have access to create feeds using the template " + importingFeed.getTemplateName()
                                           + ". Attempt made to create feed " + FeedNameUtil.fullName(feedCategory, importingFeed.getSystemFeedName()) + ".";
                        feed.getImportOptions().addErrorMessage(ImportComponent.FEED_DATA, msg);
                        feed.addErrorMessage(existingFeed, msg);
                        feed.setValid(false);
                        return false;
                    }
                    */
                return true;
            });
        }
    }
}
Also used : Category(com.thinkbiganalytics.metadata.api.category.Category) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ImportTemplate(com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate)

Example 13 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class FeedImporter method validate.

public ImportFeed validate() {
    UploadProgressMessage feedImportStatusMessage = uploadProgressService.addUploadStatus(importFeedOptions.getUploadKey(), "Validating Feed import.");
    try {
        init();
        // read the JSON into the Feed object
        FeedMetadata metadata = importFeed.getFeedToImport();
        // validate the incoming category exists
        validateFeedCategory();
        // verify if we should overwrite the feed if it already exists
        String feedCategory = StringUtils.isNotBlank(importFeedOptions.getCategorySystemName()) ? importFeedOptions.getCategorySystemName() : metadata.getSystemCategoryName();
        // query for this feed.
        // first read in the feed as a service account
        FeedMetadata existingFeed = metadataAccess.read(() -> {
            return metadataService.getFeedByName(feedCategory, metadata.getSystemFeedName());
        }, MetadataAccess.SERVICE);
        if (!validateOverwriteExistingFeed(existingFeed)) {
            // exit
            return importFeed;
        }
        if (accessController.isEntityAccessControlled()) {
            if (!validateEntityAccess(existingFeed, feedCategory)) {
                return importFeed;
            }
        }
        // sensitive properties
        if (!validateSensitiveProperties()) {
            return importFeed;
        }
        // Valid data sources
        if (!validateUserDatasources()) {
            return importFeed;
        }
        // UploadProgressMessage statusMessage = uploadProgressService.addUploadStatus(options.getUploadKey(),"Validating the template data");
        TemplateImporter templateImporter = templateImporterFactory.apply(importFeed.getFileName(), file, importFeedOptions);
        ImportTemplate importTemplate = templateImporter.validate();
        // need to set the importOptions back to the feed options
        // find importOptions for the Template and add them back to the set of options
        // importFeed.getImportOptions().updateOptions(importTemplate.getImportOptions().getImportComponentOptions());
        importFeed.setTemplate(importTemplate);
        // statusMessage.update("Validated the template data",importTemplate.isValid());
        if (!importTemplate.isValid()) {
            importFeed.setValid(false);
            List<String> errorMessages = importTemplate.getTemplateResults().getAllErrors().stream().map(nifiError -> nifiError.getMessage()).collect(Collectors.toList());
            if (!errorMessages.isEmpty()) {
                for (String msg : errorMessages) {
                    importFeed.addErrorMessage(metadata, msg);
                }
            }
        }
    // statusMessage = uploadProgressService.addUploadStatus(options.getUploadKey(),"Validation complete: the feed is "+(importFeed.isValid() ? "valid" : "invalid"),true,importFeed.isValid());
    } catch (Exception e) {
        feedImportStatusMessage.update("Validation error. Feed import error: " + e.getMessage(), false);
        throw new UnsupportedOperationException("Error importing template  " + fileName + ".  " + e.getMessage());
    }
    feedImportStatusMessage.update("Validated Feed import.", importFeed.isValid());
    return this.importFeed;
}
Also used : UploadProgressMessage(com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage) UploadProgressService(com.thinkbiganalytics.feedmgr.service.UploadProgressService) RegisteredTemplateService(com.thinkbiganalytics.feedmgr.service.template.RegisteredTemplateService) Category(com.thinkbiganalytics.metadata.api.category.Category) LoggerFactory(org.slf4j.LoggerFactory) DatasourceModelTransform(com.thinkbiganalytics.feedmgr.service.datasource.DatasourceModelTransform) TemplateImporter(com.thinkbiganalytics.feedmgr.service.template.importing.TemplateImporter) StringUtils(org.apache.commons.lang3.StringUtils) ByteArrayInputStream(java.io.ByteArrayInputStream) FeedAccessControl(com.thinkbiganalytics.metadata.api.feed.security.FeedAccessControl) NifiPropertyUtil(com.thinkbiganalytics.nifi.rest.support.NifiPropertyUtil) ImportUtil(com.thinkbiganalytics.feedmgr.util.ImportUtil) AccessController(com.thinkbiganalytics.security.AccessController) NifiFeed(com.thinkbiganalytics.feedmgr.rest.model.NifiFeed) CategoryAccessControl(com.thinkbiganalytics.metadata.api.category.security.CategoryAccessControl) FeedServicesAccessControl(com.thinkbiganalytics.feedmgr.security.FeedServicesAccessControl) MetadataAccess(com.thinkbiganalytics.metadata.api.MetadataAccess) ZipEntry(java.util.zip.ZipEntry) MetadataService(com.thinkbiganalytics.feedmgr.service.MetadataService) PolicyPropertyTypes(com.thinkbiganalytics.policy.PolicyPropertyTypes) MetadataFieldAnnotationFieldNameResolver(com.thinkbiganalytics.feedmgr.MetadataFieldAnnotationFieldNameResolver) ImportTemplate(com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate) NifiProperty(com.thinkbiganalytics.nifi.rest.model.NifiProperty) Set(java.util.Set) UploadProgress(com.thinkbiganalytics.feedmgr.rest.model.UploadProgress) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) UploadProgressMessage(com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage) ZipFileUtil(com.thinkbiganalytics.feedmgr.support.ZipFileUtil) List(java.util.List) CategoryProvider(com.thinkbiganalytics.metadata.api.category.CategoryProvider) ImportTemplateOptions(com.thinkbiganalytics.feedmgr.rest.model.ImportTemplateOptions) ImportFeed(com.thinkbiganalytics.feedmgr.service.feed.importing.model.ImportFeed) Optional(java.util.Optional) ImportFeedException(com.thinkbiganalytics.feedmgr.service.feed.ImportFeedException) ImportComponent(com.thinkbiganalytics.feedmgr.rest.ImportComponent) ZipInputStream(java.util.zip.ZipInputStream) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ImportSection(com.thinkbiganalytics.feedmgr.rest.ImportSection) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) Inject(javax.inject.Inject) ImportProperty(com.thinkbiganalytics.feedmgr.rest.model.ImportProperty) ImportType(com.thinkbiganalytics.feedmgr.rest.ImportType) ImportFeedOptions(com.thinkbiganalytics.feedmgr.rest.model.ImportFeedOptions) RegisteredTemplate(com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate) TemplateImporterFactory(com.thinkbiganalytics.feedmgr.service.template.importing.TemplateImporterFactory) UserDatasource(com.thinkbiganalytics.metadata.api.datasource.UserDatasource) ImportTemplateRoutineFactory(com.thinkbiganalytics.feedmgr.service.template.importing.importprocess.ImportTemplateRoutineFactory) Logger(org.slf4j.Logger) FeedNameUtil(com.thinkbiganalytics.support.FeedNameUtil) Datasource(com.thinkbiganalytics.metadata.rest.model.data.Datasource) ImportTemplateRoutine(com.thinkbiganalytics.feedmgr.service.template.importing.importprocess.ImportTemplateRoutine) IOException(java.io.IOException) ImportException(com.thinkbiganalytics.feedmgr.service.template.importing.ImportException) DatasourceProvider(com.thinkbiganalytics.metadata.api.datasource.DatasourceProvider) Collections(java.util.Collections) LegacyNifiRestClient(com.thinkbiganalytics.nifi.rest.client.LegacyNifiRestClient) ImportComponentOption(com.thinkbiganalytics.feedmgr.rest.model.ImportComponentOption) InputStream(java.io.InputStream) TemplateImporter(com.thinkbiganalytics.feedmgr.service.template.importing.TemplateImporter) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ImportTemplate(com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate) ImportFeedException(com.thinkbiganalytics.feedmgr.service.feed.ImportFeedException) IOException(java.io.IOException) ImportException(com.thinkbiganalytics.feedmgr.service.template.importing.ImportException)

Example 14 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class FeedImporter method validateFeedCategory.

private boolean validateFeedCategory() {
    FeedMetadata metadata = importFeed.getFeedToImport();
    boolean valid = true;
    if (StringUtils.isNotBlank(importFeedOptions.getCategorySystemName())) {
        UploadProgressMessage statusMessage = uploadProgressService.addUploadStatus(importFeedOptions.getUploadKey(), "Validating the newly specified category. Ensure " + importFeedOptions.getCategorySystemName() + " exists.");
        FeedCategory optionsCategory = metadataService.getCategoryBySystemName(importFeedOptions.getCategorySystemName());
        if (optionsCategory == null) {
            importFeed.setValid(false);
            statusMessage.update("Validation Error. The category " + importFeedOptions.getCategorySystemName() + " does not exist, or you dont have access to it.", false);
            valid = false;
        } else {
            if (valid) {
                metadata.getCategory().setSystemName(importFeedOptions.getCategorySystemName());
                statusMessage.update("Validated. The category " + importFeedOptions.getCategorySystemName() + " exists.", true);
            }
        }
    }
    uploadProgressService.completeSection(importFeedOptions, ImportSection.Section.VALIDATE_FEED_CATEGORY);
    return valid;
}
Also used : UploadProgressMessage(com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata)

Example 15 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class PropertyExpressionResolverTest method testFeedMetadataProperties.

@Test
public void testFeedMetadataProperties() {
    FeedMetadata metadata = new FeedMetadata();
    metadata.setSystemFeedName("feedSystemName");
    metadata.setCategory(new FeedCategory());
    metadata.setTable(new TableSetup());
    metadata.getTable().setSourceTableSchema(new DefaultTableSchema());
    metadata.getTable().setTableSchema(new DefaultTableSchema());
    metadata.getTable().getSourceTableSchema().setName("sourceTableName");
    metadata.getTable().getTableSchema().setName("tableSchemaName");
    final NifiProperty prop1 = createProperty("${metadata.table.sourceTableSchema.name}");
    Assert.assertTrue(resolver.resolveExpression(metadata, prop1));
    Assert.assertEquals("sourceTableName", prop1.getValue());
}
Also used : FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) TableSetup(com.thinkbiganalytics.feedmgr.rest.model.schema.TableSetup) NifiProperty(com.thinkbiganalytics.nifi.rest.model.NifiProperty) DefaultTableSchema(com.thinkbiganalytics.discovery.model.DefaultTableSchema) Test(org.junit.Test)

Aggregations

FeedCategory (com.thinkbiganalytics.feedmgr.rest.model.FeedCategory)32 FeedMetadata (com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata)18 ImportTemplate (com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate)8 Category (com.thinkbiganalytics.metadata.api.category.Category)8 ArrayList (java.util.ArrayList)7 FeedSummary (com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)5 NifiProperty (com.thinkbiganalytics.nifi.rest.model.NifiProperty)5 Test (org.junit.Test)5 RegisteredTemplate (com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate)4 IOException (java.io.IOException)4 PropertyExpressionResolver (com.thinkbiganalytics.feedmgr.nifi.PropertyExpressionResolver)3 NifiFeed (com.thinkbiganalytics.feedmgr.rest.model.NifiFeed)3 UploadProgressMessage (com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage)3 HadoopSecurityGroup (com.thinkbiganalytics.metadata.api.security.HadoopSecurityGroup)3 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)2 Sets (com.google.common.collect.Sets)2 DefaultTableSchema (com.thinkbiganalytics.discovery.model.DefaultTableSchema)2 DefaultTag (com.thinkbiganalytics.discovery.model.DefaultTag)2 MetadataFieldAnnotationFieldNameResolver (com.thinkbiganalytics.feedmgr.MetadataFieldAnnotationFieldNameResolver)2 ImportComponent (com.thinkbiganalytics.feedmgr.rest.ImportComponent)2