Search in sources :

Example 21 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class TableSetupTest method test.

@Test
public void test() throws Exception {
    ObjectMapper mapper = new ObjectMapper();
    FeedMetadata feedMetadata = new FeedMetadata();
    feedMetadata.setCategory(new FeedCategory());
    feedMetadata.setTable(new TableSetup());
    feedMetadata.getTable().setTableSchema(new DefaultTableSchema());
    feedMetadata.getTable().getTableSchema().setName("test");
    DefaultField f1 = new DefaultField();
    f1.setName("field1");
    feedMetadata.getTable().getTableSchema().getFields().add(f1);
    String json = mapper.writeValueAsString(feedMetadata);
    FeedMetadata feedMetadata2 = mapper.readValue(json, FeedMetadata.class);
    assertEquals(feedMetadata2.getTable().getTableSchema().getName(), feedMetadata.getTable().getTableSchema().getName());
}
Also used : FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) DefaultTableSchema(com.thinkbiganalytics.discovery.model.DefaultTableSchema) DefaultField(com.thinkbiganalytics.discovery.model.DefaultField) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test)

Example 22 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class CategoryModelTransform method domainToFeedCategory.

/**
 * Transforms the specified Metadata category into a Feed Manager category.
 *
 * @param domainCategory the Metadata category
 * @param userFields     the user-defined fields
 * @return the Feed Manager category
 */
@Nullable
private FeedCategory domainToFeedCategory(@Nullable final Category domainCategory, @Nonnull final Set<UserFieldDescriptor> userFields, boolean includeFeedDetails) {
    if (domainCategory != null) {
        FeedCategory category = new FeedCategory();
        category.setId(domainCategory.getId().toString());
        if (includeFeedDetails && domainCategory.getFeeds() != null) {
            List<FeedSummary> summaries = feedModelTransform.domainToFeedSummary(domainCategory.getFeeds());
            category.setFeeds(summaries);
            category.setRelatedFeeds(summaries.size());
        }
        category.setIconColor(domainCategory.getIconColor());
        category.setIcon(domainCategory.getIcon());
        category.setName(domainCategory.getDisplayName());
        // in pre-0.8.4 version of Kylo there was no system name stored for domain categories
        category.setSystemName(domainCategory.getSystemName() == null ? domainCategory.getDisplayName() : domainCategory.getSystemName());
        category.setDescription(domainCategory.getDescription());
        category.setAllowIndexing(domainCategory.isAllowIndexing());
        category.setCreateDate(domainCategory.getCreatedTime() != null ? domainCategory.getCreatedTime().toDate() : null);
        category.setUpdateDate(domainCategory.getModifiedTime() != null ? domainCategory.getModifiedTime().toDate() : null);
        // Transform user-defined fields and properties
        category.setUserFields(UserPropertyTransform.toUserFields(categoryProvider.getFeedUserFields(domainCategory.getId()).orElse(Collections.emptySet())));
        category.setUserProperties(UserPropertyTransform.toUserProperties(domainCategory.getUserProperties(), userFields));
        // Convert JCR securitygroup to DTO
        List<com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup> restSecurityGroups = new ArrayList<>();
        if (domainCategory.getSecurityGroups() != null && domainCategory.getSecurityGroups().size() > 0) {
            for (Object group : domainCategory.getSecurityGroups()) {
                HadoopSecurityGroup hadoopSecurityGroup = (HadoopSecurityGroup) group;
                com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup restSecurityGroup = new com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup();
                restSecurityGroup.setDescription(hadoopSecurityGroup.getDescription());
                restSecurityGroup.setId(hadoopSecurityGroup.getGroupId());
                restSecurityGroup.setName(hadoopSecurityGroup.getName());
                restSecurityGroups.add(restSecurityGroup);
            }
        }
        category.setSecurityGroups(restSecurityGroups);
        securityTransform.applyAccessControl(domainCategory, category);
        return category;
    } else {
        return null;
    }
}
Also used : FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) ArrayList(java.util.ArrayList) HadoopSecurityGroup(com.thinkbiganalytics.metadata.api.security.HadoopSecurityGroup) JcrHadoopSecurityGroup(com.thinkbiganalytics.metadata.modeshape.security.JcrHadoopSecurityGroup) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary) Nullable(javax.annotation.Nullable)

Example 23 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class DefaultFeedManagerCategoryService method getCategoryBySystemName.

@Override
public FeedCategory getCategoryBySystemName(final String name) {
    return metadataAccess.read(() -> {
        this.accessController.checkPermission(AccessController.SERVICES, FeedServicesAccessControl.ACCESS_CATEGORIES);
        final Category domainCategory = categoryProvider.findBySystemName(name);
        return categoryModelTransform.domainToFeedCategory(domainCategory, true);
    });
}
Also used : Category(com.thinkbiganalytics.metadata.api.category.Category) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory)

Example 24 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class InMemoryFeedManagerCategoryService method saveCategory.

@Override
public void saveCategory(final FeedCategory category) {
    if (category.getId() == null) {
        category.setId(UUID.randomUUID().toString());
        category.setSystemName(SystemNamingService.generateSystemName(category.getName()));
    } else {
        FeedCategory oldCategory = categories.get(category.getId());
        if (oldCategory != null && !oldCategory.getName().equalsIgnoreCase(category.getName())) {
            // only regenerate the system name if there are no related feeds
            if (oldCategory.getRelatedFeeds() == 0) {
                category.setSystemName(SystemNamingService.generateSystemName(category.getName()));
            }
        }
        List<FeedSummary> feeds = categories.get(category.getId()).getFeeds();
        category.setFeeds(feeds);
    }
    categories.put(category.getId(), category);
    FileObjectPersistence.getInstance().writeCategoriesToFile(categories.values());
}
Also used : FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)

Example 25 with FeedCategory

use of com.thinkbiganalytics.feedmgr.rest.model.FeedCategory in project kylo by Teradata.

the class FeedImporter method importFeed.

// Import
/**
 * Import a feed zip file
 */
private ImportFeed importFeed() throws Exception {
    // read the JSON into the Feed object
    FeedMetadata metadata = importFeed.getFeedToImport();
    // query for this feed.
    String feedCategory = StringUtils.isNotBlank(importFeedOptions.getCategorySystemName()) ? importFeedOptions.getCategorySystemName() : metadata.getSystemCategoryName();
    FeedMetadata existingFeed = metadataAccess.read(() -> metadataService.getFeedByName(feedCategory, metadata.getSystemFeedName()));
    metadata.getCategory().setSystemName(feedCategory);
    ImportTemplateOptions importTemplateOptions = new ImportTemplateOptions();
    importTemplateOptions.setImportComponentOptions(importFeedOptions.getImportComponentOptions());
    importTemplateOptions.findImportComponentOption(ImportComponent.TEMPLATE_DATA).setContinueIfExists(true);
    ImportTemplate importTemplate = importFeed.getTemplate();
    importTemplate.setImportOptions(importTemplateOptions);
    importTemplateOptions.setUploadKey(importFeedOptions.getUploadKey());
    importTemplate.setValid(true);
    importTemplateOptions.setDeferCleanup(true);
    // Import the Template
    ImportTemplateRoutine importTemplateRoutine = importTemplateRoutineFactory.apply(importTemplate, importTemplateOptions, ImportTemplate.TYPE.ARCHIVE);
    importTemplateRoutine.importTemplate();
    if (importTemplate.isSuccess()) {
        // import the feed
        importFeed.setTemplate(importTemplate);
        // now that we have the Feed object we need to create the instance of the feed
        UploadProgressMessage uploadProgressMessage = uploadProgressService.addUploadStatus(importFeedOptions.getUploadKey(), "Saving  and creating feed instance in NiFi");
        metadata.setIsNew(existingFeed == null ? true : false);
        metadata.setFeedId(existingFeed != null ? existingFeed.getFeedId() : null);
        metadata.setId(existingFeed != null ? existingFeed.getId() : null);
        // reassign the templateId to the newly registered template id
        metadata.setTemplateId(importTemplate.getTemplateId());
        if (metadata.getRegisteredTemplate() != null) {
            metadata.getRegisteredTemplate().setNifiTemplateId(importTemplate.getNifiTemplateId());
            metadata.getRegisteredTemplate().setId(importTemplate.getTemplateId());
        }
        // get/create category
        FeedCategory category = metadataService.getCategoryBySystemName(metadata.getCategory().getSystemName());
        if (category == null) {
            metadata.getCategory().setId(null);
            metadataService.saveCategory(metadata.getCategory());
        } else {
            metadata.setCategory(category);
        }
        if (importFeedOptions.isDisableUponImport()) {
            metadata.setActive(false);
            metadata.setState(FeedMetadata.STATE.DISABLED.name());
        }
        // remap any preconditions to this new feed/category name.
        if (metadata.getSchedule().hasPreconditions()) {
            metadata.getSchedule().getPreconditions().stream().flatMap(preconditionRule -> preconditionRule.getProperties().stream()).filter(fieldRuleProperty -> PolicyPropertyTypes.PROPERTY_TYPE.currentFeed.name().equals(fieldRuleProperty.getType())).forEach(fieldRuleProperty -> fieldRuleProperty.setValue(metadata.getCategoryAndFeedName()));
        }
        // //for all those properties where the template value is != userEditable and the template value has a metadata. property, remove that property from the feed properties so it can be imported and assigned correctly
        RegisteredTemplate template1 = registeredTemplateService.findRegisteredTemplateById(importTemplate.getTemplateId());
        if (template1 != null) {
            // Find all the properties in the template that have ${metadata. and are not userEditable.
            // These are the properties we need to replace on the feed metadata
            List<NifiProperty> metadataProperties = template1.getProperties().stream().filter(nifiProperty -> {
                return nifiProperty != null && StringUtils.isNotBlank(nifiProperty.getValue()) && !nifiProperty.isUserEditable() && nifiProperty.getValue().contains("${" + MetadataFieldAnnotationFieldNameResolver.metadataPropertyPrefix);
            }).collect(Collectors.toList());
            // Replace the Feed Metadata properties with those that match the template ones from above.
            List<NifiProperty> updatedProperties = metadata.getProperties().stream().map(nifiProperty -> {
                NifiProperty p = NifiPropertyUtil.findPropertyByProcessorName(metadataProperties, nifiProperty);
                return p != null ? p : nifiProperty;
            }).collect(Collectors.toList());
            metadata.setProperties(updatedProperties);
        }
        NifiFeed nifiFeed = metadataService.createFeed(metadata);
        if (nifiFeed != null) {
            importFeed.setFeedName(nifiFeed.getFeedMetadata().getCategoryAndFeedName());
            if (nifiFeed.isSuccess()) {
                uploadProgressMessage.update("Successfully saved the feed " + importFeed.getFeedName(), true);
            } else {
                if (nifiFeed.getFeedProcessGroup() != null && nifiFeed.getFeedProcessGroup().isRolledBack()) {
                    if (importTemplateRoutine != null) {
                        importTemplateRoutine.rollback();
                    }
                }
                uploadProgressMessage.update("Errors were found importing the feed " + importFeed.getFeedName(), false);
            }
            importTemplateRoutine.cleanup();
        }
        importFeed.setNifiFeed(nifiFeed);
        importFeed.setSuccess(nifiFeed != null && nifiFeed.isSuccess());
    } else {
        importFeed.setSuccess(false);
        importFeed.setTemplate(importTemplate);
        importFeed.addErrorMessage(existingFeed, "The feed " + FeedNameUtil.fullName(feedCategory, metadata.getSystemFeedName()) + " needs additional properties to be supplied before importing.");
    }
    uploadProgressService.completeSection(importFeedOptions, ImportSection.Section.IMPORT_FEED_DATA);
    return importFeed;
}
Also used : UploadProgressMessage(com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage) UploadProgressService(com.thinkbiganalytics.feedmgr.service.UploadProgressService) RegisteredTemplateService(com.thinkbiganalytics.feedmgr.service.template.RegisteredTemplateService) Category(com.thinkbiganalytics.metadata.api.category.Category) LoggerFactory(org.slf4j.LoggerFactory) DatasourceModelTransform(com.thinkbiganalytics.feedmgr.service.datasource.DatasourceModelTransform) TemplateImporter(com.thinkbiganalytics.feedmgr.service.template.importing.TemplateImporter) StringUtils(org.apache.commons.lang3.StringUtils) ByteArrayInputStream(java.io.ByteArrayInputStream) FeedAccessControl(com.thinkbiganalytics.metadata.api.feed.security.FeedAccessControl) NifiPropertyUtil(com.thinkbiganalytics.nifi.rest.support.NifiPropertyUtil) ImportUtil(com.thinkbiganalytics.feedmgr.util.ImportUtil) AccessController(com.thinkbiganalytics.security.AccessController) NifiFeed(com.thinkbiganalytics.feedmgr.rest.model.NifiFeed) CategoryAccessControl(com.thinkbiganalytics.metadata.api.category.security.CategoryAccessControl) FeedServicesAccessControl(com.thinkbiganalytics.feedmgr.security.FeedServicesAccessControl) MetadataAccess(com.thinkbiganalytics.metadata.api.MetadataAccess) ZipEntry(java.util.zip.ZipEntry) MetadataService(com.thinkbiganalytics.feedmgr.service.MetadataService) PolicyPropertyTypes(com.thinkbiganalytics.policy.PolicyPropertyTypes) MetadataFieldAnnotationFieldNameResolver(com.thinkbiganalytics.feedmgr.MetadataFieldAnnotationFieldNameResolver) ImportTemplate(com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate) NifiProperty(com.thinkbiganalytics.nifi.rest.model.NifiProperty) Set(java.util.Set) UploadProgress(com.thinkbiganalytics.feedmgr.rest.model.UploadProgress) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) UploadProgressMessage(com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage) ZipFileUtil(com.thinkbiganalytics.feedmgr.support.ZipFileUtil) List(java.util.List) CategoryProvider(com.thinkbiganalytics.metadata.api.category.CategoryProvider) ImportTemplateOptions(com.thinkbiganalytics.feedmgr.rest.model.ImportTemplateOptions) ImportFeed(com.thinkbiganalytics.feedmgr.service.feed.importing.model.ImportFeed) Optional(java.util.Optional) ImportFeedException(com.thinkbiganalytics.feedmgr.service.feed.ImportFeedException) ImportComponent(com.thinkbiganalytics.feedmgr.rest.ImportComponent) ZipInputStream(java.util.zip.ZipInputStream) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ImportSection(com.thinkbiganalytics.feedmgr.rest.ImportSection) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) Inject(javax.inject.Inject) ImportProperty(com.thinkbiganalytics.feedmgr.rest.model.ImportProperty) ImportType(com.thinkbiganalytics.feedmgr.rest.ImportType) ImportFeedOptions(com.thinkbiganalytics.feedmgr.rest.model.ImportFeedOptions) RegisteredTemplate(com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate) TemplateImporterFactory(com.thinkbiganalytics.feedmgr.service.template.importing.TemplateImporterFactory) UserDatasource(com.thinkbiganalytics.metadata.api.datasource.UserDatasource) ImportTemplateRoutineFactory(com.thinkbiganalytics.feedmgr.service.template.importing.importprocess.ImportTemplateRoutineFactory) Logger(org.slf4j.Logger) FeedNameUtil(com.thinkbiganalytics.support.FeedNameUtil) Datasource(com.thinkbiganalytics.metadata.rest.model.data.Datasource) ImportTemplateRoutine(com.thinkbiganalytics.feedmgr.service.template.importing.importprocess.ImportTemplateRoutine) IOException(java.io.IOException) ImportException(com.thinkbiganalytics.feedmgr.service.template.importing.ImportException) DatasourceProvider(com.thinkbiganalytics.metadata.api.datasource.DatasourceProvider) Collections(java.util.Collections) LegacyNifiRestClient(com.thinkbiganalytics.nifi.rest.client.LegacyNifiRestClient) ImportComponentOption(com.thinkbiganalytics.feedmgr.rest.model.ImportComponentOption) InputStream(java.io.InputStream) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ImportTemplateOptions(com.thinkbiganalytics.feedmgr.rest.model.ImportTemplateOptions) NifiProperty(com.thinkbiganalytics.nifi.rest.model.NifiProperty) ImportTemplate(com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate) RegisteredTemplate(com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate) ImportTemplateRoutine(com.thinkbiganalytics.feedmgr.service.template.importing.importprocess.ImportTemplateRoutine) NifiFeed(com.thinkbiganalytics.feedmgr.rest.model.NifiFeed)

Aggregations

FeedCategory (com.thinkbiganalytics.feedmgr.rest.model.FeedCategory)32 FeedMetadata (com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata)18 ImportTemplate (com.thinkbiganalytics.feedmgr.service.template.importing.model.ImportTemplate)8 Category (com.thinkbiganalytics.metadata.api.category.Category)8 ArrayList (java.util.ArrayList)7 FeedSummary (com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)5 NifiProperty (com.thinkbiganalytics.nifi.rest.model.NifiProperty)5 Test (org.junit.Test)5 RegisteredTemplate (com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate)4 IOException (java.io.IOException)4 PropertyExpressionResolver (com.thinkbiganalytics.feedmgr.nifi.PropertyExpressionResolver)3 NifiFeed (com.thinkbiganalytics.feedmgr.rest.model.NifiFeed)3 UploadProgressMessage (com.thinkbiganalytics.feedmgr.rest.model.UploadProgressMessage)3 HadoopSecurityGroup (com.thinkbiganalytics.metadata.api.security.HadoopSecurityGroup)3 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)2 Sets (com.google.common.collect.Sets)2 DefaultTableSchema (com.thinkbiganalytics.discovery.model.DefaultTableSchema)2 DefaultTag (com.thinkbiganalytics.discovery.model.DefaultTag)2 MetadataFieldAnnotationFieldNameResolver (com.thinkbiganalytics.feedmgr.MetadataFieldAnnotationFieldNameResolver)2 ImportComponent (com.thinkbiganalytics.feedmgr.rest.ImportComponent)2