Search in sources :

Example 1 with TopologyMetricsService

use of com.hortonworks.streamline.streams.metrics.topology.service.TopologyMetricsService in project streamline by hortonworks.

the class StreamsModule method getResources.

@Override
public List<Object> getResources() {
    List<Object> result = new ArrayList<>();
    String catalogRootUrl = (String) config.get(Constants.CONFIG_CATALOG_ROOT_URL);
    // Authorized subject
    final Subject subject = (Subject) config.get(Constants.CONFIG_SUBJECT);
    MLModelRegistryClient modelRegistryClient = new MLModelRegistryClient(catalogRootUrl, subject);
    final StreamCatalogService streamcatalogService = new StreamCatalogService(storageManager, fileStorage, modelRegistryClient);
    final EnvironmentService environmentService = new EnvironmentService(storageManager);
    TagClient tagClient = new TagClient(catalogRootUrl);
    final CatalogService catalogService = new CatalogService(storageManager, fileStorage, tagClient);
    final TopologyActionsService topologyActionsService = new TopologyActionsService(streamcatalogService, environmentService, fileStorage, modelRegistryClient, config, subject, transactionManager);
    final TopologyMetricsService topologyMetricsService = new TopologyMetricsService(environmentService, subject);
    final TopologyLogSearchService topologyLogSearchService = new TopologyLogSearchService(environmentService, subject);
    environmentService.addNamespaceAwareContainer(topologyActionsService);
    environmentService.addNamespaceAwareContainer(topologyMetricsService);
    environmentService.addNamespaceAwareContainer(topologyLogSearchService);
    // authorizer
    final StreamlineAuthorizer authorizer = (StreamlineAuthorizer) config.get(Constants.CONFIG_AUTHORIZER);
    if (authorizer == null) {
        throw new IllegalStateException("Authorizer not set");
    }
    final SecurityCatalogService securityCatalogService = (SecurityCatalogService) config.get(Constants.CONFIG_SECURITY_CATALOG_SERVICE);
    result.addAll(getAuthorizerResources(authorizer, securityCatalogService));
    result.add(new MetricsResource(authorizer, streamcatalogService, topologyMetricsService));
    result.addAll(getClusterRelatedResources(authorizer, environmentService));
    result.add(new FileCatalogResource(authorizer, catalogService));
    result.addAll(getTopologyRelatedResources(authorizer, streamcatalogService, environmentService, topologyActionsService, topologyMetricsService, topologyLogSearchService, securityCatalogService, subject));
    result.add(new UDFCatalogResource(authorizer, streamcatalogService, fileStorage));
    result.addAll(getNotificationsRelatedResources(authorizer, streamcatalogService));
    result.add(new SchemaResource(createSchemaRegistryClient()));
    result.addAll(getServiceMetadataResources(authorizer, environmentService, subject));
    result.add(new NamespaceCatalogResource(authorizer, streamcatalogService, topologyActionsService, environmentService));
    result.add(new SearchCatalogResource(authorizer, streamcatalogService, environmentService, topologyActionsService, topologyMetricsService));
    watchFiles(streamcatalogService);
    setupPlaceholderEntities(streamcatalogService, environmentService);
    return result;
}
Also used : TopologyActionsService(com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService) TagClient(com.hortonworks.streamline.registries.tag.client.TagClient) ArrayList(java.util.ArrayList) SecurityCatalogService(com.hortonworks.streamline.streams.security.service.SecurityCatalogService) CatalogService(com.hortonworks.streamline.streams.catalog.service.CatalogService) StreamCatalogService(com.hortonworks.streamline.streams.catalog.service.StreamCatalogService) MLModelRegistryClient(com.hortonworks.streamline.registries.model.client.MLModelRegistryClient) StreamlineAuthorizer(com.hortonworks.streamline.streams.security.StreamlineAuthorizer) Subject(javax.security.auth.Subject) TopologyMetricsService(com.hortonworks.streamline.streams.metrics.topology.service.TopologyMetricsService) StreamCatalogService(com.hortonworks.streamline.streams.catalog.service.StreamCatalogService) SecurityCatalogService(com.hortonworks.streamline.streams.security.service.SecurityCatalogService) EnvironmentService(com.hortonworks.streamline.streams.cluster.service.EnvironmentService) TopologyLogSearchService(com.hortonworks.streamline.streams.logsearch.topology.service.TopologyLogSearchService)

Example 2 with TopologyMetricsService

use of com.hortonworks.streamline.streams.metrics.topology.service.TopologyMetricsService in project streamline by hortonworks.

the class CatalogResourceUtil method enrichTopology.

static TopologyDashboardResponse enrichTopology(Topology topology, String asUser, Integer latencyTopN, EnvironmentService environmentService, TopologyActionsService actionsService, TopologyMetricsService metricsService, StreamCatalogService catalogService) {
    LOG.debug("[START] enrichTopology - topology id: {}", topology.getId());
    Stopwatch stopwatch = Stopwatch.createStarted();
    try {
        if (latencyTopN == null) {
            latencyTopN = DEFAULT_N_OF_TOP_N_LATENCY;
        }
        TopologyDashboardResponse detailedResponse;
        String namespaceName = null;
        Namespace namespace = environmentService.getNamespace(topology.getNamespaceId());
        if (namespace != null) {
            namespaceName = namespace.getName();
        }
        try {
            String runtimeTopologyId = actionsService.getRuntimeTopologyId(topology, asUser);
            TopologyMetrics.TopologyMetric topologyMetric = metricsService.getTopologyMetric(topology, asUser);
            List<Pair<String, Double>> latenciesTopN = metricsService.getTopNAndOtherComponentsLatency(topology, asUser, latencyTopN);
            detailedResponse = new TopologyDashboardResponse(topology, TopologyRunningStatus.RUNNING, namespaceName);
            detailedResponse.setRuntime(new TopologyRuntimeResponse(runtimeTopologyId, topologyMetric, latenciesTopN));
        } catch (TopologyNotAliveException e) {
            LOG.debug("Topology {} is not alive", topology.getId());
            detailedResponse = new TopologyDashboardResponse(topology, TopologyRunningStatus.NOT_RUNNING, namespaceName);
            catalogService.getTopologyState(topology.getId()).ifPresent(state -> {
                if (TopologyStateFactory.getInstance().getTopologyState(state.getName()) == TopologyStates.TOPOLOGY_STATE_DEPLOYED) {
                    try {
                        LOG.info("Force killing streamline topology since its not alive in the cluster");
                        actionsService.killTopology(topology, asUser);
                    } catch (Exception ex) {
                        LOG.error("Error trying to kill topology", ex);
                    }
                }
            });
        } catch (StormNotReachableException | IOException e) {
            LOG.error("Storm is not reachable or fail to operate", e);
            detailedResponse = new TopologyDashboardResponse(topology, TopologyRunningStatus.UNKNOWN, namespaceName);
        } catch (Exception e) {
            LOG.error("Unhandled exception occurs while operate with Storm", e);
            detailedResponse = new TopologyDashboardResponse(topology, TopologyRunningStatus.UNKNOWN, namespaceName);
        }
        LOG.debug("[END] enrichTopology - topology id: {}, elapsed: {} ms", topology.getId(), stopwatch.elapsed(TimeUnit.MILLISECONDS));
        return detailedResponse;
    } finally {
        stopwatch.stop();
    }
}
Also used : Topology(com.hortonworks.streamline.streams.catalog.Topology) Namespace(com.hortonworks.streamline.streams.cluster.catalog.Namespace) NamespaceServiceClusterMap(com.hortonworks.streamline.streams.cluster.catalog.NamespaceServiceClusterMap) TopologyStateFactory(com.hortonworks.streamline.streams.actions.topology.state.TopologyStateFactory) Logger(org.slf4j.Logger) TopologyMetricsService(com.hortonworks.streamline.streams.metrics.topology.service.TopologyMetricsService) Stopwatch(com.google.common.base.Stopwatch) TopologyActionsService(com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService) Collection(java.util.Collection) LoggerFactory(org.slf4j.LoggerFactory) IOException(java.io.IOException) StormNotReachableException(com.hortonworks.streamline.streams.storm.common.StormNotReachableException) TopologyStates(com.hortonworks.streamline.streams.actions.topology.state.TopologyStates) ArrayList(java.util.ArrayList) EnvironmentService(com.hortonworks.streamline.streams.cluster.service.EnvironmentService) TopologyNotAliveException(com.hortonworks.streamline.streams.exception.TopologyNotAliveException) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) Pair(org.apache.commons.lang3.tuple.Pair) StreamCatalogService(com.hortonworks.streamline.streams.catalog.service.StreamCatalogService) JsonInclude(com.fasterxml.jackson.annotation.JsonInclude) TopologyMetrics(com.hortonworks.streamline.streams.metrics.topology.TopologyMetrics) Stopwatch(com.google.common.base.Stopwatch) TopologyMetrics(com.hortonworks.streamline.streams.metrics.topology.TopologyMetrics) IOException(java.io.IOException) Namespace(com.hortonworks.streamline.streams.cluster.catalog.Namespace) IOException(java.io.IOException) StormNotReachableException(com.hortonworks.streamline.streams.storm.common.StormNotReachableException) TopologyNotAliveException(com.hortonworks.streamline.streams.exception.TopologyNotAliveException) StormNotReachableException(com.hortonworks.streamline.streams.storm.common.StormNotReachableException) TopologyNotAliveException(com.hortonworks.streamline.streams.exception.TopologyNotAliveException) Pair(org.apache.commons.lang3.tuple.Pair)

Aggregations

TopologyActionsService (com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService)2 StreamCatalogService (com.hortonworks.streamline.streams.catalog.service.StreamCatalogService)2 EnvironmentService (com.hortonworks.streamline.streams.cluster.service.EnvironmentService)2 TopologyMetricsService (com.hortonworks.streamline.streams.metrics.topology.service.TopologyMetricsService)2 ArrayList (java.util.ArrayList)2 JsonInclude (com.fasterxml.jackson.annotation.JsonInclude)1 Stopwatch (com.google.common.base.Stopwatch)1 MLModelRegistryClient (com.hortonworks.streamline.registries.model.client.MLModelRegistryClient)1 TagClient (com.hortonworks.streamline.registries.tag.client.TagClient)1 TopologyStateFactory (com.hortonworks.streamline.streams.actions.topology.state.TopologyStateFactory)1 TopologyStates (com.hortonworks.streamline.streams.actions.topology.state.TopologyStates)1 Topology (com.hortonworks.streamline.streams.catalog.Topology)1 CatalogService (com.hortonworks.streamline.streams.catalog.service.CatalogService)1 Namespace (com.hortonworks.streamline.streams.cluster.catalog.Namespace)1 NamespaceServiceClusterMap (com.hortonworks.streamline.streams.cluster.catalog.NamespaceServiceClusterMap)1 TopologyNotAliveException (com.hortonworks.streamline.streams.exception.TopologyNotAliveException)1 TopologyLogSearchService (com.hortonworks.streamline.streams.logsearch.topology.service.TopologyLogSearchService)1 TopologyMetrics (com.hortonworks.streamline.streams.metrics.topology.TopologyMetrics)1 StreamlineAuthorizer (com.hortonworks.streamline.streams.security.StreamlineAuthorizer)1 SecurityCatalogService (com.hortonworks.streamline.streams.security.service.SecurityCatalogService)1