Search in sources :

Example 6 with TopologyEditorMetadata

use of com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata in project streamline by hortonworks.

the class TopologyEditorMetadataResource method addOrUpdateTopologyEditorMetadata.

@PUT
@Path("/system/topologyeditormetadata/{id}")
@Timed
public Response addOrUpdateTopologyEditorMetadata(@PathParam("id") Long topologyId, TopologyEditorMetadata topologyEditorMetadata, @Context SecurityContext securityContext) {
    SecurityUtil.checkRoleOrPermissions(authorizer, securityContext, Roles.ROLE_TOPOLOGY_SUPER_ADMIN, Topology.NAMESPACE, topologyId, WRITE);
    TopologyEditorMetadata newTopologyEditorMetadata = catalogService.addOrUpdateTopologyEditorMetadata(topologyId, topologyEditorMetadata);
    return WSUtils.respondEntity(newTopologyEditorMetadata, OK);
}
Also used : TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata) Path(javax.ws.rs.Path) Timed(com.codahale.metrics.annotation.Timed) PUT(javax.ws.rs.PUT)

Example 7 with TopologyEditorMetadata

use of com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata in project streamline by hortonworks.

the class TopologyEditorMetadataResource method getTopologyEditorMetadataByTopologyId.

@GET
@Path("/system/topologyeditormetadata/{id}")
@Timed
public Response getTopologyEditorMetadataByTopologyId(@PathParam("id") Long topologyId, @Context SecurityContext securityContext) {
    SecurityUtil.checkRoleOrPermissions(authorizer, securityContext, Roles.ROLE_TOPOLOGY_USER, Topology.NAMESPACE, topologyId, READ);
    TopologyEditorMetadata result = catalogService.getTopologyEditorMetadata(topologyId);
    if (result != null) {
        return WSUtils.respondEntity(result, OK);
    }
    throw EntityNotFoundException.byId(topologyId.toString());
}
Also used : TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata) Path(javax.ws.rs.Path) Timed(com.codahale.metrics.annotation.Timed) GET(javax.ws.rs.GET)

Example 8 with TopologyEditorMetadata

use of com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata in project streamline by hortonworks.

the class StreamCatalogService method getTopologyEditorMetadata.

public TopologyEditorMetadata getTopologyEditorMetadata(Long topologyId, Long versionId) {
    TopologyEditorMetadata topologyEditorMetadata = new TopologyEditorMetadata();
    topologyEditorMetadata.setTopologyId(topologyId);
    topologyEditorMetadata.setVersionId(versionId);
    return this.dao.get(topologyEditorMetadata.getStorableKey());
}
Also used : TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata)

Example 9 with TopologyEditorMetadata

use of com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata in project streamline by hortonworks.

the class StreamCatalogService method removeTopologyEditorMetadata.

public TopologyEditorMetadata removeTopologyEditorMetadata(Long topologyId, Long versionId) {
    TopologyEditorMetadata topologyEditorMetadata = getTopologyEditorMetadata(topologyId, versionId);
    if (topologyEditorMetadata != null) {
        topologyEditorMetadata = dao.remove(topologyEditorMetadata.getStorableKey());
        topologyEditorMetadata.setTimestamp(updateVersionTimestamp(versionId).getTimestamp());
    }
    return topologyEditorMetadata;
}
Also used : TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata)

Example 10 with TopologyEditorMetadata

use of com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata in project streamline by hortonworks.

the class StreamCatalogService method doImportTopology.

private Topology doImportTopology(Topology newTopology, TopologyData topologyData) throws Exception {
    List<TopologySource> topologySources = topologyData.getSources();
    Map<Long, Long> oldToNewComponentIds = new HashMap<>();
    Map<Long, Long> oldToNewRuleIds = new HashMap<>();
    Map<Long, Long> oldToNewWindowIds = new HashMap<>();
    Map<Long, Long> oldToNewBranchRuleIds = new HashMap<>();
    Map<Long, Long> oldToNewStreamIds = new HashMap<>();
    // import source streams
    for (TopologySource topologySource : topologySources) {
        topologySource.setOutputStreamIds(importOutputStreams(newTopology.getId(), oldToNewStreamIds, topologySource.getOutputStreams()));
        topologySource.setOutputStreams(null);
    }
    // import processor streams
    for (TopologyProcessor topologyProcessor : topologyData.getProcessors()) {
        topologyProcessor.setOutputStreamIds(importOutputStreams(newTopology.getId(), oldToNewStreamIds, topologyProcessor.getOutputStreams()));
        topologyProcessor.setOutputStreams(null);
    }
    // import rules
    for (TopologyRule rule : topologyData.getRules()) {
        Long currentId = rule.getId();
        rule.setId(null);
        TopologyRule addedRule = addRule(newTopology.getId(), rule);
        oldToNewRuleIds.put(currentId, addedRule.getId());
    }
    // import windowed rules
    for (TopologyWindow window : topologyData.getWindows()) {
        Long currentId = window.getId();
        window.setId(null);
        TopologyWindow addedWindow = addWindow(newTopology.getId(), window);
        oldToNewWindowIds.put(currentId, addedWindow.getId());
    }
    // import branch rules
    for (TopologyBranchRule branchRule : topologyData.getBranchRules()) {
        Long currentId = branchRule.getId();
        branchRule.setId(null);
        TopologyBranchRule addedBranchRule = addBranchRule(newTopology.getId(), branchRule);
        oldToNewBranchRuleIds.put(currentId, addedBranchRule.getId());
    }
    // import sources
    for (TopologySource topologySource : topologySources) {
        Long oldComponentId = topologySource.getId();
        topologySource.setId(null);
        topologySource.setTopologyId(newTopology.getId());
        TopologyComponentBundle bundle = getCurrentTopologyComponentBundle(TopologyComponentBundle.TopologyComponentType.SOURCE, topologyData.getBundleIdToType().get(topologySource.getTopologyComponentBundleId().toString()));
        topologySource.setTopologyComponentBundleId(bundle.getId());
        addTopologySource(newTopology.getId(), topologySource);
        oldToNewComponentIds.put(oldComponentId, topologySource.getId());
    }
    // import processors
    for (TopologyProcessor topologyProcessor : topologyData.getProcessors()) {
        Long oldComponentId = topologyProcessor.getId();
        topologyProcessor.setId(null);
        topologyProcessor.setTopologyId(newTopology.getId());
        TopologyComponentBundle bundle;
        String subType = topologyData.getBundleIdToType().get(topologyProcessor.getTopologyComponentBundleId().toString());
        if (TopologyLayoutConstants.JSON_KEY_CUSTOM_PROCESSOR_SUB_TYPE.equals(subType)) {
            QueryParam queryParam = new QueryParam(CustomProcessorInfo.NAME, topologyProcessor.getConfig().get(CustomProcessorInfo.NAME));
            Collection<TopologyComponentBundle> result = listCustomProcessorBundlesWithFilter(Collections.singletonList(queryParam));
            if (result.size() != 1) {
                throw new IllegalStateException("Not able to find topology component bundle for custom processor :" + topologyProcessor.getConfig().get(CustomProcessorInfo.NAME));
            }
            bundle = result.iterator().next();
        } else {
            bundle = getCurrentTopologyComponentBundle(TopologyComponentBundle.TopologyComponentType.PROCESSOR, subType);
        }
        topologyProcessor.setTopologyComponentBundleId(bundle.getId());
        Optional<Object> ruleListObj = topologyProcessor.getConfig().getAnyOptional(RulesProcessor.CONFIG_KEY_RULES);
        ruleListObj.ifPresent(ruleList -> {
            List<Long> ruleIds = new ObjectMapper().convertValue(ruleList, new TypeReference<List<Long>>() {
            });
            List<Long> updatedRuleIds = new ArrayList<>();
            if (ComponentTypes.RULE.equals(bundle.getSubType()) || ComponentTypes.PROJECTION.equals(bundle.getSubType())) {
                ruleIds.forEach(ruleId -> updatedRuleIds.add(oldToNewRuleIds.get(ruleId)));
            } else if (bundle.getSubType().equals(ComponentTypes.BRANCH)) {
                ruleIds.forEach(ruleId -> updatedRuleIds.add(oldToNewBranchRuleIds.get(ruleId)));
            } else if (bundle.getSubType().equals(ComponentTypes.WINDOW)) {
                ruleIds.forEach(ruleId -> updatedRuleIds.add(oldToNewWindowIds.get(ruleId)));
            }
            topologyProcessor.getConfig().setAny(RulesProcessor.CONFIG_KEY_RULES, updatedRuleIds);
        });
        addTopologyProcessor(newTopology.getId(), topologyProcessor);
        oldToNewComponentIds.put(oldComponentId, topologyProcessor.getId());
    }
    // import sinks
    for (TopologySink topologySink : topologyData.getSinks()) {
        topologySink.setTopologyId(newTopology.getId());
        Long currentId = topologySink.getId();
        topologySink.setId(null);
        TopologyComponentBundle bundle = getCurrentTopologyComponentBundle(TopologyComponentBundle.TopologyComponentType.SINK, topologyData.getBundleIdToType().get(topologySink.getTopologyComponentBundleId().toString()));
        topologySink.setTopologyComponentBundleId(bundle.getId());
        if (bundle.getSubType().equals(NOTIFICATION)) {
            updateNotifierJarFileName(topologySink);
        }
        addTopologySink(newTopology.getId(), topologySink);
        oldToNewComponentIds.put(currentId, topologySink.getId());
    }
    // import edges
    for (TopologyEdge topologyEdge : topologyData.getEdges()) {
        List<StreamGrouping> streamGroupings = topologyEdge.getStreamGroupings();
        for (StreamGrouping streamGrouping : streamGroupings) {
            Long newStreamId = oldToNewStreamIds.get(streamGrouping.getStreamId());
            streamGrouping.setStreamId(newStreamId);
        }
        topologyEdge.setId(null);
        topologyEdge.setTopologyId(newTopology.getId());
        topologyEdge.setFromId(oldToNewComponentIds.get(topologyEdge.getFromId()));
        topologyEdge.setToId(oldToNewComponentIds.get(topologyEdge.getToId()));
        addTopologyEdge(newTopology.getId(), topologyEdge);
    }
    // import topology editor metadata
    TopologyEditorMetadata topologyEditorMetadata = topologyData.getTopologyEditorMetadata();
    topologyEditorMetadata.setTopologyId(newTopology.getId());
    if (topologyEditorMetadata.getData() != null) {
        TopologyUIData topologyUIData = new ObjectMapper().readValue(topologyEditorMetadata.getData(), TopologyUIData.class);
        topologyUIData.getSources().forEach(c -> c.setId(oldToNewComponentIds.get(c.getId())));
        topologyUIData.getProcessors().forEach(c -> c.setId(oldToNewComponentIds.get(c.getId())));
        topologyUIData.getSinks().forEach(c -> c.setId(oldToNewComponentIds.get(c.getId())));
        topologyEditorMetadata.setData(new ObjectMapper().writeValueAsString(topologyUIData));
    } else {
        topologyEditorMetadata.setData(StringUtils.EMPTY);
    }
    addTopologyEditorMetadata(newTopology.getId(), topologyData.getTopologyEditorMetadata());
    return newTopology;
}
Also used : Topology(com.hortonworks.streamline.streams.catalog.Topology) TopologyProcessorStreamMap(com.hortonworks.streamline.streams.catalog.TopologyProcessorStreamMap) Rule(com.hortonworks.streamline.streams.layout.component.rule.Rule) MLModelRegistryClient(com.hortonworks.streamline.registries.model.client.MLModelRegistryClient) UDAF(com.hortonworks.streamline.streams.rule.UDAF) Notifier(com.hortonworks.streamline.streams.catalog.Notifier) StringUtils(org.apache.commons.lang.StringUtils) Arrays(java.util.Arrays) Utils(com.hortonworks.streamline.common.util.Utils) QueryParam(com.hortonworks.registries.common.QueryParam) FileStorage(com.hortonworks.registries.common.util.FileStorage) UDAF2(com.hortonworks.streamline.streams.rule.UDAF2) Collections2(com.google.common.collect.Collections2) ComponentUISpecification(com.hortonworks.streamline.common.ComponentUISpecification) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) StreamGrouping(com.hortonworks.streamline.streams.catalog.TopologyEdge.StreamGrouping) ComponentConfigException(com.hortonworks.streamline.common.exception.ComponentConfigException) Matcher(java.util.regex.Matcher) TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata) Map(java.util.Map) RuleParser(com.hortonworks.streamline.streams.catalog.rule.RuleParser) TopologyComponentBundle(com.hortonworks.streamline.streams.catalog.topology.TopologyComponentBundle) WSUtils.versionIdQueryParam(com.hortonworks.streamline.common.util.WSUtils.versionIdQueryParam) TopologyData(com.hortonworks.streamline.streams.catalog.topology.TopologyData) Set(java.util.Set) NOTIFICATION(com.hortonworks.streamline.common.ComponentTypes.NOTIFICATION) CURRENT_VERSION(com.hortonworks.streamline.common.util.WSUtils.CURRENT_VERSION) NoSuchAlgorithmException(java.security.NoSuchAlgorithmException) CustomProcessorInfo(com.hortonworks.streamline.streams.catalog.processor.CustomProcessorInfo) StorableKey(com.hortonworks.registries.storage.StorableKey) TopologyProcessor(com.hortonworks.streamline.streams.catalog.TopologyProcessor) Joiner(com.google.common.base.Joiner) TopologyState(com.hortonworks.streamline.streams.catalog.topology.state.TopologyState) ArrayList(java.util.ArrayList) Lists(com.google.common.collect.Lists) TopologySourceStreamMap(com.hortonworks.streamline.streams.catalog.TopologySourceStreamMap) Projection(com.hortonworks.streamline.streams.catalog.Projection) TopologyOutputComponent(com.hortonworks.streamline.streams.catalog.TopologyOutputComponent) TopologyRule(com.hortonworks.streamline.streams.catalog.TopologyRule) Nullable(javax.annotation.Nullable) TopologyVersion(com.hortonworks.streamline.streams.catalog.TopologyVersion) UDF(com.hortonworks.streamline.streams.catalog.UDF) IOException(java.io.IOException) TopologyTestRunCaseSource(com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource) TopologyWindow(com.hortonworks.streamline.streams.catalog.TopologyWindow) TreeMap(java.util.TreeMap) File(com.hortonworks.streamline.streams.catalog.File) DigestInputStream(java.security.DigestInputStream) Preconditions(com.google.common.base.Preconditions) TopologyDagBuilder(com.hortonworks.streamline.streams.catalog.topology.component.TopologyDagBuilder) BiFunction(java.util.function.BiFunction) UDF4(com.hortonworks.streamline.streams.rule.UDF4) LoggerFactory(org.slf4j.LoggerFactory) UDF3(com.hortonworks.streamline.streams.rule.UDF3) UDF2(com.hortonworks.streamline.streams.rule.UDF2) UDF7(com.hortonworks.streamline.streams.rule.UDF7) UDF6(com.hortonworks.streamline.streams.rule.UDF6) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) UDF5(com.hortonworks.streamline.streams.rule.UDF5) StreamlineEvent(com.hortonworks.streamline.streams.StreamlineEvent) BaseTopologyRule(com.hortonworks.streamline.streams.catalog.BaseTopologyRule) TopologyEdge(com.hortonworks.streamline.streams.catalog.TopologyEdge) WSUtils(com.hortonworks.streamline.common.util.WSUtils) TypeReference(com.fasterxml.jackson.core.type.TypeReference) TopologyStream(com.hortonworks.streamline.streams.catalog.TopologyStream) StorageUtils(com.hortonworks.registries.storage.util.StorageUtils) Function(com.google.common.base.Function) ImmutableSet(com.google.common.collect.ImmutableSet) TopologySink(com.hortonworks.streamline.streams.catalog.TopologySink) FileUtil(com.hortonworks.streamline.common.util.FileUtil) Collection(java.util.Collection) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) TopologySource(com.hortonworks.streamline.streams.catalog.TopologySource) List(java.util.List) Optional(java.util.Optional) Pattern(java.util.regex.Pattern) StorageManager(com.hortonworks.registries.storage.StorageManager) TopologyTestRunCaseSink(com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSink) TopologyTestRunCase(com.hortonworks.streamline.streams.catalog.TopologyTestRunCase) WSUtils.buildEdgesFromQueryParam(com.hortonworks.streamline.common.util.WSUtils.buildEdgesFromQueryParam) TopologyLayoutConstants(com.hortonworks.streamline.streams.layout.TopologyLayoutConstants) MessageDigest(java.security.MessageDigest) TopologyUIData(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata.TopologyUIData) TopologyEditorToolbar(com.hortonworks.streamline.streams.catalog.TopologyEditorToolbar) HashMap(java.util.HashMap) Hex(org.apache.commons.codec.binary.Hex) ComponentTypes(com.hortonworks.streamline.common.ComponentTypes) Schema(com.hortonworks.registries.common.Schema) HashSet(java.util.HashSet) WSUtils.currentVersionQueryParam(com.hortonworks.streamline.common.util.WSUtils.currentVersionQueryParam) ImmutableList(com.google.common.collect.ImmutableList) CustomProcessorRuntime(com.hortonworks.streamline.streams.runtime.CustomProcessorRuntime) WSUtils.buildEdgesToQueryParam(com.hortonworks.streamline.common.util.WSUtils.buildEdgesToQueryParam) TopologyExportVisitor(com.hortonworks.streamline.streams.catalog.topology.component.TopologyExportVisitor) Logger(org.slf4j.Logger) Stream(com.hortonworks.streamline.streams.layout.component.Stream) ProxyUtil(com.hortonworks.streamline.common.util.ProxyUtil) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) FileInputStream(java.io.FileInputStream) StorageException(com.hortonworks.registries.storage.exception.StorageException) TopologyBranchRule(com.hortonworks.streamline.streams.catalog.TopologyBranchRule) TopologyTestRunHistory(com.hortonworks.streamline.streams.catalog.TopologyTestRunHistory) TopologyComponent(com.hortonworks.streamline.streams.catalog.TopologyComponent) FluxComponent(com.hortonworks.streamline.streams.layout.storm.FluxComponent) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Collections(java.util.Collections) InputStream(java.io.InputStream) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) TopologyBranchRule(com.hortonworks.streamline.streams.catalog.TopologyBranchRule) TopologySource(com.hortonworks.streamline.streams.catalog.TopologySource) TopologyRule(com.hortonworks.streamline.streams.catalog.TopologyRule) BaseTopologyRule(com.hortonworks.streamline.streams.catalog.BaseTopologyRule) TopologyEdge(com.hortonworks.streamline.streams.catalog.TopologyEdge) TopologyUIData(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata.TopologyUIData) ArrayList(java.util.ArrayList) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) TopologyProcessor(com.hortonworks.streamline.streams.catalog.TopologyProcessor) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) StreamGrouping(com.hortonworks.streamline.streams.catalog.TopologyEdge.StreamGrouping) TopologyWindow(com.hortonworks.streamline.streams.catalog.TopologyWindow) TopologySink(com.hortonworks.streamline.streams.catalog.TopologySink) TopologyEditorMetadata(com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata) QueryParam(com.hortonworks.registries.common.QueryParam) WSUtils.versionIdQueryParam(com.hortonworks.streamline.common.util.WSUtils.versionIdQueryParam) WSUtils.buildEdgesFromQueryParam(com.hortonworks.streamline.common.util.WSUtils.buildEdgesFromQueryParam) WSUtils.currentVersionQueryParam(com.hortonworks.streamline.common.util.WSUtils.currentVersionQueryParam) WSUtils.buildEdgesToQueryParam(com.hortonworks.streamline.common.util.WSUtils.buildEdgesToQueryParam) TopologyComponentBundle(com.hortonworks.streamline.streams.catalog.topology.TopologyComponentBundle)

Aggregations

TopologyEditorMetadata (com.hortonworks.streamline.streams.catalog.TopologyEditorMetadata)11 Timed (com.codahale.metrics.annotation.Timed)5 Path (javax.ws.rs.Path)5 QueryParam (com.hortonworks.registries.common.QueryParam)3 WSUtils.buildEdgesFromQueryParam (com.hortonworks.streamline.common.util.WSUtils.buildEdgesFromQueryParam)3 WSUtils.buildEdgesToQueryParam (com.hortonworks.streamline.common.util.WSUtils.buildEdgesToQueryParam)3 WSUtils.currentVersionQueryParam (com.hortonworks.streamline.common.util.WSUtils.currentVersionQueryParam)3 WSUtils.versionIdQueryParam (com.hortonworks.streamline.common.util.WSUtils.versionIdQueryParam)3 BaseTopologyRule (com.hortonworks.streamline.streams.catalog.BaseTopologyRule)2 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)1 TypeReference (com.fasterxml.jackson.core.type.TypeReference)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Function (com.google.common.base.Function)1 Joiner (com.google.common.base.Joiner)1 Preconditions (com.google.common.base.Preconditions)1 Collections2 (com.google.common.collect.Collections2)1 ImmutableList (com.google.common.collect.ImmutableList)1 ImmutableSet (com.google.common.collect.ImmutableSet)1 Lists (com.google.common.collect.Lists)1