Search in sources :

Example 1 with FeedSummary

use of com.thinkbiganalytics.feedmgr.rest.model.FeedSummary in project kylo by Teradata.

the class FeedModelTransform method domainToFeedMetadata.

/**
 * Transforms the specified Metadata feed to a Feed Manager feed.
 *
 * @param domain       the Metadata feed
 * @param userFieldMap cache map from category to user-defined fields, or {@code null}
 * @return the Feed Manager feed
 */
@Nonnull
private FeedMetadata domainToFeedMetadata(@Nonnull final Feed domain, @Nullable final Map<Category, Set<UserFieldDescriptor>> userFieldMap) {
    FeedMetadata feed = deserializeFeedMetadata(domain, false);
    feed.setId(domain.getId().toString());
    feed.setFeedId(domain.getId().toString());
    feed.setFeedName(domain.getDisplayName());
    feed.setSystemFeedName(domain.getName());
    feed.setDescription(domain.getDescription());
    feed.setAllowIndexing(domain.isAllowIndexing());
    feed.setHistoryReindexingStatus(domain.getCurrentHistoryReindexingStatus().getHistoryReindexingState().toString());
    feed.setOwner(domain.getOwner() != null ? new User(domain.getOwner().getName()) : null);
    if (domain.getCreatedTime() != null) {
        feed.setCreateDate(domain.getCreatedTime().toDate());
    }
    if (domain.getModifiedTime() != null) {
        feed.setUpdateDate(domain.getModifiedTime().toDate());
    }
    FeedManagerTemplate template = domain.getTemplate();
    if (template != null) {
        RegisteredTemplate registeredTemplate = templateModelTransform.DOMAIN_TO_REGISTERED_TEMPLATE.apply(template);
        feed.setRegisteredTemplate(registeredTemplate);
        feed.setTemplateId(registeredTemplate.getId());
        feed.setTemplateName(registeredTemplate.getTemplateName());
    }
    Category category = domain.getCategory();
    if (category != null) {
        feed.setCategory(categoryModelTransform.domainToFeedCategorySimple(category));
    }
    feed.setState(domain.getState() != null ? domain.getState().name() : null);
    feed.setVersionName(domain.getVersionName() != null ? domain.getVersionName() : null);
    // Set user-defined properties
    final Set<UserFieldDescriptor> userFields;
    if (userFieldMap == null) {
        userFields = getUserFields(category);
    } else if (userFieldMap.containsKey(category)) {
        userFields = userFieldMap.get(category);
    } else {
        userFields = getUserFields(category);
        userFieldMap.put(category, userFields);
    }
    @SuppressWarnings("unchecked") final Set<UserProperty> userProperties = UserPropertyTransform.toUserProperties(domain.getUserProperties(), userFields);
    feed.setUserProperties(userProperties);
    // Convert JCR securitygroup to DTO
    List<com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup> restSecurityGroups = new ArrayList<>();
    if (domain.getSecurityGroups() != null && domain.getSecurityGroups().size() > 0) {
        for (Object group : domain.getSecurityGroups()) {
            HadoopSecurityGroup hadoopSecurityGroup = (HadoopSecurityGroup) group;
            com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup restSecurityGroup = new com.thinkbiganalytics.feedmgr.rest.model.HadoopSecurityGroup();
            restSecurityGroup.setDescription(hadoopSecurityGroup.getDescription());
            restSecurityGroup.setId(hadoopSecurityGroup.getGroupId());
            restSecurityGroup.setName(hadoopSecurityGroup.getName());
            restSecurityGroups.add(restSecurityGroup);
        }
    }
    feed.setSecurityGroups(restSecurityGroups);
    feed.setTags(domain.getTags().stream().map(name -> new DefaultTag(name)).collect(Collectors.toList()));
    if (domain.getUsedByFeeds() != null) {
        final List<FeedSummary> usedByFeeds = domain.getUsedByFeeds().stream().map(this::domainToFeedSummary).collect(Collectors.toList());
        feed.setUsedByFeeds(usedByFeeds);
    }
    // add in access control items
    securityTransform.applyAccessControl(domain, feed);
    return feed;
}
Also used : User(com.thinkbiganalytics.security.rest.model.User) Category(com.thinkbiganalytics.metadata.api.category.Category) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) UserFieldDescriptor(com.thinkbiganalytics.metadata.api.extension.UserFieldDescriptor) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) ArrayList(java.util.ArrayList) HadoopSecurityGroup(com.thinkbiganalytics.metadata.api.security.HadoopSecurityGroup) JcrHadoopSecurityGroup(com.thinkbiganalytics.metadata.modeshape.security.JcrHadoopSecurityGroup) UserProperty(com.thinkbiganalytics.feedmgr.rest.model.UserProperty) RegisteredTemplate(com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate) FeedManagerTemplate(com.thinkbiganalytics.metadata.api.template.FeedManagerTemplate) DefaultTag(com.thinkbiganalytics.discovery.model.DefaultTag) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary) Nonnull(javax.annotation.Nonnull)

Example 2 with FeedSummary

use of com.thinkbiganalytics.feedmgr.rest.model.FeedSummary in project kylo by Teradata.

the class FeedRestController method startFeed.

@POST
@Path("/start/{feedId}")
@Produces(MediaType.APPLICATION_JSON)
@ApiOperation("Starts a feed.")
@ApiResponses({ @ApiResponse(code = 200, message = "The feed was started.", response = FeedSummary.class), @ApiResponse(code = 500, message = "The feed could not be started.", response = RestResponseStatus.class) })
public Response startFeed(@PathParam("feedId") String feedId) {
    try {
        FeedSummary feed = getMetadataService().startFeed(feedId);
        RestResponseStatus.ResponseStatusBuilder builder = new RestResponseStatus.ResponseStatusBuilder();
        RestResponseStatus status = null;
        if (feed != null) {
            status = builder.message("Feed " + feed.getCategoryAndFeedDisplayName() + " started successfully").buildSuccess();
        } else {
            status = builder.message("Error starting feed for id " + feedId).buildError();
        }
        return Response.ok(status).build();
    } catch (Exception e) {
        log.error("Exception starting the feed ", e);
        throw new InternalServerErrorException("Unexpected exception starting the feed " + feedId + " " + e.getMessage());
    }
}
Also used : InternalServerErrorException(javax.ws.rs.InternalServerErrorException) NifiClientRuntimeException(com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException) FeedCleanupTimeoutException(com.thinkbiganalytics.feedmgr.service.FeedCleanupTimeoutException) FeedCleanupFailedException(com.thinkbiganalytics.feedmgr.service.FeedCleanupFailedException) WebApplicationException(javax.ws.rs.WebApplicationException) AccessDeniedException(java.nio.file.AccessDeniedException) DeployFeedException(com.thinkbiganalytics.feedmgr.service.feed.DeployFeedException) DuplicateFeedNameException(com.thinkbiganalytics.feedmgr.service.feed.DuplicateFeedNameException) IOException(java.io.IOException) FeedCurrentlyRunningException(com.thinkbiganalytics.feedmgr.service.feed.reindexing.FeedCurrentlyRunningException) ClientErrorException(javax.ws.rs.ClientErrorException) JDBCException(org.hibernate.JDBCException) FeedHistoryDataReindexingNotEnabledException(com.thinkbiganalytics.feedmgr.service.feed.reindexing.FeedHistoryDataReindexingNotEnabledException) InternalServerErrorException(javax.ws.rs.InternalServerErrorException) NotFoundException(javax.ws.rs.NotFoundException) AccessControlException(java.security.AccessControlException) DataAccessException(org.springframework.dao.DataAccessException) VersionNotFoundException(com.thinkbiganalytics.metadata.api.versioning.VersionNotFoundException) FeedNotFoundException(com.thinkbiganalytics.metadata.api.feed.FeedNotFoundException) NifiConnectionException(com.thinkbiganalytics.nifi.rest.client.NifiConnectionException) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary) RestResponseStatus(com.thinkbiganalytics.rest.model.RestResponseStatus) Path(javax.ws.rs.Path) POST(javax.ws.rs.POST) Produces(javax.ws.rs.Produces) ApiOperation(io.swagger.annotations.ApiOperation) ApiResponses(io.swagger.annotations.ApiResponses)

Example 3 with FeedSummary

use of com.thinkbiganalytics.feedmgr.rest.model.FeedSummary in project kylo by Teradata.

the class FeedManagerMetadataService method enableFeed.

public FeedSummary enableFeed(String feedId) {
    return metadataAccess.commit(() -> {
        this.accessController.checkPermission(AccessController.SERVICES, FeedServicesAccessControl.EDIT_FEEDS);
        FeedMetadata feedMetadata = feedProvider.getFeedById(feedId);
        if (feedMetadata == null) {
            // feed will not be found when user is allowed to export feeds but has no entity access to feed with feed id
            throw new NotFoundException("Feed not found for id " + feedId);
        }
        if (!feedMetadata.getState().equals(Feed.State.ENABLED.name())) {
            FeedSummary feedSummary = feedProvider.enableFeed(feedId);
            boolean updatedNifi = updateNifiFeedRunningStatus(feedSummary, Feed.State.ENABLED);
            if (!updatedNifi) {
                // rollback
                throw new RuntimeException("Unable to enable Feed " + feedId);
            }
            return feedSummary;
        }
        return new FeedSummary(feedMetadata);
    });
}
Also used : NifiClientRuntimeException(com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) NotFoundException(javax.ws.rs.NotFoundException) FeedNotFoundException(com.thinkbiganalytics.metadata.api.feed.FeedNotFoundException) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)

Example 4 with FeedSummary

use of com.thinkbiganalytics.feedmgr.rest.model.FeedSummary in project kylo by Teradata.

the class FeedManagerMetadataService method startFeed.

@Override
@SuppressWarnings("deprecation")
public FeedSummary startFeed(String feedId) {
    FeedMetadata feedMetadata = this.metadataAccess.read(() -> {
        this.accessController.checkPermission(AccessController.SERVICES, FeedServicesAccessControl.ADMIN_FEEDS);
        Feed.ID domainId = domainFeedProvider.resolveId(feedId);
        Feed domainFeed = domainFeedProvider.findById(domainId);
        if (domainFeed != null) {
            this.accessController.checkPermission(domainFeed, FeedAccessControl.START);
            return feedModelTransform.domainToFeedMetadata(domainFeed);
        } else {
            throw new FeedNotFoundException(domainId);
        }
    });
    Optional<ProcessorDTO> feedInputProcessor = nifiClient.processGroups().findByName("root", feedMetadata.getSystemCategoryName(), false, false).flatMap(categoryGroup -> nifiClient.processGroups().findByName(categoryGroup.getId(), feedMetadata.getSystemFeedName(), false, true)).map(feedGroup -> {
        List<ProcessorDTO> inputProcessors = this.nifiRestClient.getInputProcessors(feedGroup.getId());
        ProcessorDTO inputProcessor = NifiProcessUtil.findFirstProcessorsByTypeAndName(inputProcessors, feedMetadata.getInputProcessorType(), feedMetadata.getInputProcessorName());
        if (inputProcessor != null) {
            this.nifiClient.processors().wakeUp(inputProcessor);
        }
        return inputProcessor;
    });
    if (!feedInputProcessor.isPresent()) {
        log.error("Unable to start the feed {}.  Could not find the input processor to start.", feedMetadata.getCategoryAndFeedName());
        throw new RuntimeException("Unable to start Feed " + feedMetadata.getCategoryAndFeedName() + ".  Could not find the input processor to start the feed");
    }
    return new FeedSummary(feedMetadata);
}
Also used : Action(com.thinkbiganalytics.security.action.Action) HadoopAuthorizationService(com.thinkbiganalytics.datalake.authorization.service.HadoopAuthorizationService) CleanupTriggerEvent(com.thinkbiganalytics.metadata.api.event.feed.CleanupTriggerEvent) LoggerFactory(org.slf4j.LoggerFactory) ConnectionDTO(org.apache.nifi.web.api.dto.ConnectionDTO) Autowired(org.springframework.beans.factory.annotation.Autowired) NifiProcessUtil(com.thinkbiganalytics.nifi.rest.support.NifiProcessUtil) FeedModelTransform(com.thinkbiganalytics.feedmgr.service.feed.FeedModelTransform) NifiClientRuntimeException(com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException) FeedProvider(com.thinkbiganalytics.metadata.api.feed.FeedProvider) ProcessGroupDTO(org.apache.nifi.web.api.dto.ProcessGroupDTO) FeedAccessControl(com.thinkbiganalytics.metadata.api.feed.security.FeedAccessControl) FeedManagerCategoryService(com.thinkbiganalytics.feedmgr.service.category.FeedManagerCategoryService) Map(java.util.Map) AccessController(com.thinkbiganalytics.security.AccessController) NifiFeed(com.thinkbiganalytics.feedmgr.rest.model.NifiFeed) NiFiRestClient(com.thinkbiganalytics.nifi.rest.client.NiFiRestClient) CategoryAccessControl(com.thinkbiganalytics.metadata.api.category.security.CategoryAccessControl) Pageable(org.springframework.data.domain.Pageable) FeedServicesAccessControl(com.thinkbiganalytics.feedmgr.security.FeedServicesAccessControl) MetadataAccess(com.thinkbiganalytics.metadata.api.MetadataAccess) FeedManagerTemplateService(com.thinkbiganalytics.feedmgr.service.template.FeedManagerTemplateService) FeedVersions(com.thinkbiganalytics.feedmgr.rest.model.FeedVersions) DeployResponseEntityVersion(com.thinkbiganalytics.feedmgr.rest.model.DeployResponseEntityVersion) MetadataEventListener(com.thinkbiganalytics.metadata.api.event.MetadataEventListener) NifiProperty(com.thinkbiganalytics.nifi.rest.model.NifiProperty) Collection(java.util.Collection) Set(java.util.Set) Page(org.springframework.data.domain.Page) EntityVersionDifference(com.thinkbiganalytics.feedmgr.rest.model.EntityVersionDifference) MetadataEventService(com.thinkbiganalytics.metadata.api.event.MetadataEventService) Collectors(java.util.stream.Collectors) NotFoundException(javax.ws.rs.NotFoundException) EntityVersion(com.thinkbiganalytics.feedmgr.rest.model.EntityVersion) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary) PortDTO(org.apache.nifi.web.api.dto.PortDTO) List(java.util.List) ProcessorDTO(org.apache.nifi.web.api.dto.ProcessorDTO) Optional(java.util.Optional) NiFiComponentState(com.thinkbiganalytics.nifi.rest.client.NiFiComponentState) InvalidOperationException(com.thinkbiganalytics.feedmgr.InvalidOperationException) FeedOperationStatusEvent(com.thinkbiganalytics.metadata.api.event.feed.FeedOperationStatusEvent) FeedManagerFeedService(com.thinkbiganalytics.feedmgr.service.feed.FeedManagerFeedService) Feed(com.thinkbiganalytics.metadata.api.feed.Feed) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) UserProperty(com.thinkbiganalytics.feedmgr.rest.model.UserProperty) ArrayList(java.util.ArrayList) FeedCategory(com.thinkbiganalytics.feedmgr.rest.model.FeedCategory) Value(org.springframework.beans.factory.annotation.Value) HashSet(java.util.HashSet) Inject(javax.inject.Inject) UIFeed(com.thinkbiganalytics.feedmgr.rest.model.UIFeed) ServiceLevelAgreementService(com.thinkbiganalytics.feedmgr.sla.ServiceLevelAgreementService) Qualifier(org.springframework.beans.factory.annotation.Qualifier) ProcessGroupFlowDTO(org.apache.nifi.web.api.dto.flow.ProcessGroupFlowDTO) RegisteredTemplate(com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate) Nonnull(javax.annotation.Nonnull) NifiConstants(com.thinkbiganalytics.nifi.rest.support.NifiConstants) Nullable(javax.annotation.Nullable) FeedNotFoundException(com.thinkbiganalytics.metadata.api.feed.FeedNotFoundException) FeedOperation(com.thinkbiganalytics.metadata.api.op.FeedOperation) Logger(org.slf4j.Logger) UserFieldCollection(com.thinkbiganalytics.feedmgr.rest.model.UserFieldCollection) LegacyNifiRestClient(com.thinkbiganalytics.nifi.rest.client.LegacyNifiRestClient) NifiClientRuntimeException(com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException) ProcessorDTO(org.apache.nifi.web.api.dto.ProcessorDTO) FeedMetadata(com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata) FeedNotFoundException(com.thinkbiganalytics.metadata.api.feed.FeedNotFoundException) NifiFeed(com.thinkbiganalytics.feedmgr.rest.model.NifiFeed) Feed(com.thinkbiganalytics.metadata.api.feed.Feed) UIFeed(com.thinkbiganalytics.feedmgr.rest.model.UIFeed) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)

Example 5 with FeedSummary

use of com.thinkbiganalytics.feedmgr.rest.model.FeedSummary in project kylo by Teradata.

the class DefaultFeedManagerFeedService method applyFeedSelectOptions.

@Override
public /**
 * Applies new LableValue array to the FieldProperty.selectableValues {label = Category.Display Feed Name, value=category.system_feed_name}
 */
void applyFeedSelectOptions(List<FieldRuleProperty> properties) {
    if (properties != null && !properties.isEmpty()) {
        List<FeedSummary> feedSummaries = getFeedSummaryData();
        List<LabelValue> feedSelection = new ArrayList<>();
        for (FeedSummary feedSummary : feedSummaries) {
            boolean isDisabled = feedSummary.getState() == Feed.State.DISABLED.name();
            boolean canEditDetails = accessController.isEntityAccessControlled() ? feedSummary.hasAction(FeedAccessControl.EDIT_DETAILS.getSystemName()) : accessController.hasPermission(AccessController.SERVICES, FeedServicesAccessControl.EDIT_FEEDS);
            Map<String, Object> labelValueProperties = new HashMap<>();
            labelValueProperties.put("feed:disabled", isDisabled);
            labelValueProperties.put("feed:editDetails", canEditDetails);
            feedSelection.add(new LabelValue(feedSummary.getCategoryAndFeedDisplayName() + (isDisabled ? " (DISABLED) " : ""), feedSummary.getCategoryAndFeedSystemName(), isDisabled ? "This feed is currently disabled" : "", labelValueProperties));
        }
        feedSelection.sort(Comparator.comparing(LabelValue::getLabel, String.CASE_INSENSITIVE_ORDER));
        for (FieldRuleProperty property : properties) {
            property.setSelectableValues(feedSelection);
            if (property.getValues() == null) {
                // reset the intial values to be an empty arraylist
                property.setValues(new ArrayList<>());
            }
        }
    }
}
Also used : LabelValue(com.thinkbiganalytics.rest.model.LabelValue) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) FieldRuleProperty(com.thinkbiganalytics.policy.rest.model.FieldRuleProperty) FeedSummary(com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)

Aggregations

FeedSummary (com.thinkbiganalytics.feedmgr.rest.model.FeedSummary)22 FeedMetadata (com.thinkbiganalytics.feedmgr.rest.model.FeedMetadata)11 FeedCategory (com.thinkbiganalytics.feedmgr.rest.model.FeedCategory)10 ArrayList (java.util.ArrayList)9 Feed (com.thinkbiganalytics.metadata.api.feed.Feed)6 UIFeed (com.thinkbiganalytics.feedmgr.rest.model.UIFeed)5 NifiFeed (com.thinkbiganalytics.feedmgr.rest.model.NifiFeed)4 RegisteredTemplate (com.thinkbiganalytics.feedmgr.rest.model.RegisteredTemplate)4 UserProperty (com.thinkbiganalytics.feedmgr.rest.model.UserProperty)4 Category (com.thinkbiganalytics.metadata.api.category.Category)4 FeedNotFoundException (com.thinkbiganalytics.metadata.api.feed.FeedNotFoundException)4 NifiClientRuntimeException (com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException)4 NotFoundException (javax.ws.rs.NotFoundException)4 Response (com.jayway.restassured.response.Response)3 DefaultTag (com.thinkbiganalytics.discovery.model.DefaultTag)3 EntityVersionDifference (com.thinkbiganalytics.feedmgr.rest.model.EntityVersionDifference)3 FeedVersions (com.thinkbiganalytics.feedmgr.rest.model.FeedVersions)3 HadoopSecurityGroup (com.thinkbiganalytics.metadata.api.security.HadoopSecurityGroup)3 JcrHadoopSecurityGroup (com.thinkbiganalytics.metadata.modeshape.security.JcrHadoopSecurityGroup)3 FieldRuleProperty (com.thinkbiganalytics.policy.rest.model.FieldRuleProperty)3