Search in sources :

Example 1 with Service

use of com.logicalclocks.servicediscoverclient.service.Service in project hopsworks by logicalclocks.

the class PushgatewayMonitor method removeActiveApplications.

private void removeActiveApplications(PushgatewayResults pushgatewayResults, List<String> applications) throws ServiceDiscoveryException {
    Set<Map<String, String>> groupsToRemove = new HashSet<>();
    // Get unique group information (labels) containing the application id to remove
    for (String application : applications) {
        groupsToRemove.addAll(pushgatewayResults.getData().stream().flatMap(m -> m.values().stream()).filter(serie -> serie.getMetrics() != null).map(PushgatewaySerie::getMetrics).map(metric -> metric.get(0).getLabels()).filter(labels -> labels.get("job").equalsIgnoreCase(application)).collect(Collectors.toSet()));
    }
    // For each group send a request to pushgateway to remove the group
    Service pushgatewayService = serviceDiscoveryController.getAnyAddressOfServiceWithDNS(ServiceDiscoveryController.HopsworksService.PUSHGATEWAY);
    HttpHost pushgatewayHost = new HttpHost(pushgatewayService.getAddress(), pushgatewayService.getPort());
    String groupPath = "";
    for (Map<String, String> group : groupsToRemove) {
        // Job has to go first
        String job = group.remove("job");
        groupPath = "/metrics/job/" + job + "/" + group.entrySet().stream().map(e -> e.getKey() + "/" + e.getValue()).collect(Collectors.joining("/"));
        HttpDelete httpDelete = new HttpDelete(groupPath);
        try {
            httpClient.execute(pushgatewayHost, httpDelete, new HttpClient.NoBodyResponseHandler<>());
        } catch (IOException e) {
            // Keep iterating if there is an issue with a group
            LOGGER.log(Level.SEVERE, "Error deleting group: " + groupPath, e);
        }
    }
}
Also used : YarnClient(org.apache.hadoop.yarn.client.api.YarnClient) YarnClientService(io.hops.hopsworks.common.yarn.YarnClientService) ArrayList(java.util.ArrayList) Level(java.util.logging.Level) HashSet(java.util.HashSet) HttpDelete(org.apache.http.client.methods.HttpDelete) DependsOn(javax.ejb.DependsOn) Timer(javax.ejb.Timer) Service(com.logicalclocks.servicediscoverclient.service.Service) ServiceDiscoveryController(io.hops.hopsworks.common.hosts.ServiceDiscoveryController) YarnException(org.apache.hadoop.yarn.exceptions.YarnException) Map(java.util.Map) Schedule(javax.ejb.Schedule) EJB(javax.ejb.EJB) Set(java.util.Set) IOException(java.io.IOException) Logger(java.util.logging.Logger) ServiceDiscoveryException(com.logicalclocks.servicediscoverclient.exceptions.ServiceDiscoveryException) Collectors(java.util.stream.Collectors) YarnClientWrapper(io.hops.hopsworks.common.yarn.YarnClientWrapper) List(java.util.List) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) HttpClient(io.hops.hopsworks.common.proxies.client.HttpClient) HttpGet(org.apache.http.client.methods.HttpGet) Singleton(javax.ejb.Singleton) FinalApplicationStatus(org.apache.hadoop.yarn.api.records.FinalApplicationStatus) Pattern(java.util.regex.Pattern) HttpHost(org.apache.http.HttpHost) HttpDelete(org.apache.http.client.methods.HttpDelete) YarnClientService(io.hops.hopsworks.common.yarn.YarnClientService) Service(com.logicalclocks.servicediscoverclient.service.Service) IOException(java.io.IOException) HttpHost(org.apache.http.HttpHost) HttpClient(io.hops.hopsworks.common.proxies.client.HttpClient) Map(java.util.Map) HashSet(java.util.HashSet)

Example 2 with Service

use of com.logicalclocks.servicediscoverclient.service.Service in project hopsworks by logicalclocks.

the class PushgatewayMonitor method scrapeMetrics.

private PushgatewayResults scrapeMetrics() throws ServiceDiscoveryException, IOException {
    Service pushgatewayService = serviceDiscoveryController.getAnyAddressOfServiceWithDNS(ServiceDiscoveryController.HopsworksService.PUSHGATEWAY);
    HttpHost pushgatewayHost = new HttpHost(pushgatewayService.getAddress(), pushgatewayService.getPort());
    return httpClient.execute(pushgatewayHost, new HttpGet(METRICS_ENDPOINT), new HttpClient.ObjectResponseHandler<>(PushgatewayResults.class, httpClient.getObjectMapper()));
}
Also used : HttpHost(org.apache.http.HttpHost) HttpGet(org.apache.http.client.methods.HttpGet) HttpClient(io.hops.hopsworks.common.proxies.client.HttpClient) YarnClientService(io.hops.hopsworks.common.yarn.YarnClientService) Service(com.logicalclocks.servicediscoverclient.service.Service)

Example 3 with Service

use of com.logicalclocks.servicediscoverclient.service.Service in project hopsworks by logicalclocks.

the class ServiceDiscoveryController method getAnyAddressOfServiceWithDNSSRVOnly.

@Lock(LockType.READ)
public Service getAnyAddressOfServiceWithDNSSRVOnly(HopsworksService serviceName) throws ServiceDiscoveryException {
    ServiceQuery serviceQuery = ServiceQuery.of(constructServiceFQDN(serviceName), Collections.emptySet());
    DnsResolver client = (DnsResolver) getClient(Type.DNS);
    Optional<Service> serviceOpt = client.getServiceSRVOnly(serviceQuery).findAny();
    return serviceOpt.orElseThrow(() -> new ServiceNotFoundException("Could not find service with: " + serviceQuery));
}
Also used : DnsResolver(com.logicalclocks.servicediscoverclient.resolvers.DnsResolver) ServiceNotFoundException(com.logicalclocks.servicediscoverclient.exceptions.ServiceNotFoundException) Service(com.logicalclocks.servicediscoverclient.service.Service) ServiceQuery(com.logicalclocks.servicediscoverclient.service.ServiceQuery) Lock(javax.ejb.Lock)

Example 4 with Service

use of com.logicalclocks.servicediscoverclient.service.Service in project hopsworks by logicalclocks.

the class Settings method getAlertManagerAddress.

public static URI getAlertManagerAddress(String serviceFQDN, String scheme) throws ServiceDiscoveryException {
    serviceFQDN = Strings.isNullOrEmpty(serviceFQDN) ? DEFAULT_ALERTMANAGER_FQDN : serviceFQDN;
    if (alertManagerAddresses.get(serviceFQDN) == null) {
        Optional<Service> optionalService = getAlertManagerService(serviceFQDN);
        Service service = optionalService.orElseThrow(() -> new ServiceDiscoveryException("Service not found."));
        UriBuilder uriBuilder = UriBuilder.fromPath("").scheme(Strings.isNullOrEmpty(scheme) ? "http" : scheme).host(service.getName());
        if (service.getPort() != null && service.getPort() > 0) {
            uriBuilder.port(service.getPort());
        }
        URI alertManagerAddress = uriBuilder.build();
        alertManagerAddresses.put(serviceFQDN, alertManagerAddress);
    }
    return alertManagerAddresses.get(serviceFQDN);
}
Also used : Service(com.logicalclocks.servicediscoverclient.service.Service) ServiceDiscoveryException(com.logicalclocks.servicediscoverclient.exceptions.ServiceDiscoveryException) UriBuilder(javax.ws.rs.core.UriBuilder) URI(java.net.URI)

Example 5 with Service

use of com.logicalclocks.servicediscoverclient.service.Service in project hopsworks by logicalclocks.

the class ClusterUtilisationService method metrics.

@GET
@Path("/metrics")
@Produces(MediaType.APPLICATION_JSON)
public Response metrics() throws ServiceException {
    Service rm = null;
    try {
        rm = serviceDiscoveryController.getAnyAddressOfServiceWithDNS(ServiceDiscoveryController.HopsworksService.HTTPS_RESOURCEMANAGER);
    } catch (ServiceDiscoveryException e) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.SERVICE_DISCOVERY_ERROR, Level.FINE);
    }
    HttpHost rmHost = new HttpHost(rm.getAddress(), rm.getPort(), "https");
    HttpGet getRequest = new HttpGet(METRICS_ENDPOINT);
    // defined as string as we don't really need to look inside it
    String response = null;
    try {
        response = httpClient.execute(rmHost, getRequest, new HttpClient.StringResponseHandler());
    } catch (IOException e) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.RM_METRICS_ERROR, Level.FINE);
    }
    JSONObject jsonObject = new JSONObject(response);
    jsonObject.put("deploying", hostsFacade.countUnregistered());
    return Response.ok().entity(jsonObject.toString()).build();
}
Also used : ServiceException(io.hops.hopsworks.exceptions.ServiceException) JSONObject(org.json.JSONObject) HttpHost(org.apache.http.HttpHost) HttpGet(org.apache.http.client.methods.HttpGet) Service(com.logicalclocks.servicediscoverclient.service.Service) ServiceDiscoveryException(com.logicalclocks.servicediscoverclient.exceptions.ServiceDiscoveryException) IOException(java.io.IOException) Path(javax.ws.rs.Path) Produces(javax.ws.rs.Produces) GET(javax.ws.rs.GET)

Aggregations

Service (com.logicalclocks.servicediscoverclient.service.Service)12 ServiceDiscoveryException (com.logicalclocks.servicediscoverclient.exceptions.ServiceDiscoveryException)7 IOException (java.io.IOException)5 HttpHost (org.apache.http.HttpHost)4 ServiceException (io.hops.hopsworks.exceptions.ServiceException)3 HttpGet (org.apache.http.client.methods.HttpGet)3 ServiceNotFoundException (com.logicalclocks.servicediscoverclient.exceptions.ServiceNotFoundException)2 ServiceQuery (com.logicalclocks.servicediscoverclient.service.ServiceQuery)2 TemplateException (freemarker.template.TemplateException)2 HttpClient (io.hops.hopsworks.common.proxies.client.HttpClient)2 YarnClientService (io.hops.hopsworks.common.yarn.YarnClientService)2 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 Lock (javax.ejb.Lock)2 UriBuilder (javax.ws.rs.core.UriBuilder)2 Builder (com.logicalclocks.servicediscoverclient.Builder)1 ServiceDiscoveryClient (com.logicalclocks.servicediscoverclient.ServiceDiscoveryClient)1 DnsResolver (com.logicalclocks.servicediscoverclient.resolvers.DnsResolver)1 FeaturestoreHopsfsConnectorDTO (io.hops.hopsworks.common.featurestore.storageconnectors.hopsfs.FeaturestoreHopsfsConnectorDTO)1 ServiceDiscoveryController (io.hops.hopsworks.common.hosts.ServiceDiscoveryController)1