Search in sources :

Example 1 with ServiceException

use of io.hops.hopsworks.exceptions.ServiceException in project hopsworks by logicalclocks.

the class ElasticFeaturestoreBuilder method build.

public ElasticFeaturestoreDTO build(ElasticFeaturestoreRequest req, Integer projectId) throws ElasticException, ServiceException, GenericException {
    checkRequest(req);
    Project project = projectFacade.find(projectId);
    Map<FeaturestoreDocType, Set<Integer>> searchProjects = datasetAccessCtrl.featurestoreSearchContext(project, req.getDocType());
    Map<FeaturestoreDocType, SearchResponse> response = elasticCtrl.featurestoreSearch(req.getTerm(), searchProjects, req.getFrom(), req.getSize());
    DatasetAccessController.DatasetAccessCtrl localProjectOnly = (datasetDetails, projectsCollector) -> projectsCollector.addAccessProject(project);
    ElasticFeaturestoreDTO result = parseResult(response, localProjectOnly);
    result.setFeaturegroupsFrom(req.getFrom());
    result.setTrainingdatasetsFrom(req.getFrom());
    result.setFeaturesFrom(req.getFrom());
    return result;
}
Also used : HopsworksJAXBContext(io.hops.hopsworks.common.util.HopsworksJAXBContext) ProjectFacade(io.hops.hopsworks.common.dao.project.ProjectFacade) FeaturestoreXAttrsConstants(io.hops.hopsworks.common.featurestore.xattr.dto.FeaturestoreXAttrsConstants) Function(java.util.function.Function) Project(io.hops.hopsworks.persistence.entity.project.Project) ArrayList(java.util.ArrayList) Level(java.util.logging.Level) Inject(javax.inject.Inject) TransactionAttributeType(javax.ejb.TransactionAttributeType) Triplet(org.javatuples.Triplet) Text(org.elasticsearch.common.text.Text) TransactionAttribute(javax.ejb.TransactionAttribute) Map(java.util.Map) ElasticException(io.hops.hopsworks.exceptions.ElasticException) SearchResponse(org.elasticsearch.action.search.SearchResponse) HighlightField(org.elasticsearch.search.fetch.subphase.highlight.HighlightField) BiConsumer(java.util.function.BiConsumer) EJB(javax.ejb.EJB) SearchHit(org.elasticsearch.search.SearchHit) Stateless(javax.ejb.Stateless) Set(java.util.Set) ElasticController(io.hops.hopsworks.common.elastic.ElasticController) RESTCodes(io.hops.hopsworks.restutils.RESTCodes) ServiceException(io.hops.hopsworks.exceptions.ServiceException) ElasticFeaturestoreHit(io.hops.hopsworks.common.elastic.ElasticFeaturestoreHit) GenericException(io.hops.hopsworks.exceptions.GenericException) FeaturestoreDocType(io.hops.hopsworks.common.elastic.FeaturestoreDocType) Users(io.hops.hopsworks.persistence.entity.user.Users) Project(io.hops.hopsworks.persistence.entity.project.Project) Set(java.util.Set) FeaturestoreDocType(io.hops.hopsworks.common.elastic.FeaturestoreDocType) SearchResponse(org.elasticsearch.action.search.SearchResponse)

Example 2 with ServiceException

use of io.hops.hopsworks.exceptions.ServiceException in project hopsworks by logicalclocks.

the class JupyterService method getGitStatusOfJupyterRepo.

@GET
@Path("/git/status")
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_OWNER, AllowedProjectRoles.DATA_SCIENTIST })
@JWTRequired(acceptedTokens = { Audience.API }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response getGitStatusOfJupyterRepo(@Context SecurityContext sc) throws ProjectException, ServiceException {
    Users user = jWTHelper.getUserPrincipal(sc);
    String projectUser = hdfsUsersController.getHdfsUserName(project, user);
    JupyterProject jupyterProject = jupyterFacade.findByUser(projectUser);
    if (jupyterProject == null) {
        throw new ProjectException(RESTCodes.ProjectErrorCode.JUPYTER_SERVER_NOT_FOUND, Level.FINE, "Could not found Jupyter server", "Could not found Jupyter server for Hopsworks user: " + projectUser);
    }
    if (!jupyterManager.ping(jupyterProject)) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_SERVERS_NOT_RUNNING, Level.FINE, "Jupyter server is not running", "Jupyter server for Hopsworks user: " + projectUser + " is not running");
    }
    JupyterSettings jupyterSettings = jupyterSettingsFacade.findByProjectUser(project, user.getEmail());
    RepositoryStatus status = NullJupyterNbVCSController.EMPTY_REPOSITORY_STATUS;
    if (jupyterSettings.isGitBackend()) {
        status = jupyterNbVCSController.status(jupyterProject, jupyterSettings);
    }
    return Response.ok(status).build();
}
Also used : ProjectException(io.hops.hopsworks.exceptions.ProjectException) ServiceException(io.hops.hopsworks.exceptions.ServiceException) JupyterSettings(io.hops.hopsworks.persistence.entity.jupyter.JupyterSettings) RepositoryStatus(io.hops.hopsworks.common.jupyter.RepositoryStatus) JupyterProject(io.hops.hopsworks.persistence.entity.jupyter.JupyterProject) HdfsUsers(io.hops.hopsworks.persistence.entity.hdfs.user.HdfsUsers) Users(io.hops.hopsworks.persistence.entity.user.Users) Path(javax.ws.rs.Path) Produces(javax.ws.rs.Produces) GET(javax.ws.rs.GET) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 3 with ServiceException

use of io.hops.hopsworks.exceptions.ServiceException in project hopsworks by logicalclocks.

the class JupyterService method startNotebookServer.

@POST
@Path("/start")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_OWNER, AllowedProjectRoles.DATA_SCIENTIST })
@JWTRequired(acceptedTokens = { Audience.API }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response startNotebookServer(JupyterSettings jupyterSettings, @Context HttpServletRequest req, @Context SecurityContext sc, @Context UriInfo uriInfo) throws ProjectException, HopsSecurityException, ServiceException, GenericException, JobException {
    Users hopsworksUser = jWTHelper.getUserPrincipal(sc);
    String hdfsUser = hdfsUsersController.getHdfsUserName(project, hopsworksUser);
    // from in the front-end
    if (jupyterSettings.getUsers() == null) {
        jupyterSettings.setUsers(hopsworksUser);
    }
    if (project.getPaymentType().equals(PaymentType.PREPAID)) {
        YarnProjectsQuota projectQuota = yarnProjectsQuotaFacade.findByProjectName(project.getName());
        if (projectQuota == null || projectQuota.getQuotaRemaining() <= 0) {
            throw new ProjectException(RESTCodes.ProjectErrorCode.PROJECT_QUOTA_ERROR, Level.FINE);
        }
    }
    if (project.getPythonEnvironment() == null) {
        throw new ProjectException(RESTCodes.ProjectErrorCode.ANACONDA_NOT_ENABLED, Level.FINE);
    }
    if (jupyterSettings.getMode() == null) {
        // set default mode for jupyter if mode is null
        jupyterSettings.setMode(JupyterMode.JUPYTER_LAB);
    }
    // Jupyter Git works only for JupyterLab
    if (jupyterSettings.isGitBackend() && jupyterSettings.getMode().equals(JupyterMode.JUPYTER_CLASSIC)) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_START_ERROR, Level.FINE, "Git support available only in JupyterLab");
    }
    // Do not allow auto push on shutdown if api key is missing
    GitConfig gitConfig = jupyterSettings.getGitConfig();
    if (jupyterSettings.isGitBackend() && gitConfig.getShutdownAutoPush() && Strings.isNullOrEmpty(gitConfig.getApiKeyName())) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_START_ERROR, Level.FINE, "Auto push not supported if api key is not configured.");
    }
    // Verify that API token has got write access on the repo if ShutdownAutoPush is enabled
    if (jupyterSettings.isGitBackend() && gitConfig.getShutdownAutoPush() && !jupyterNbVCSController.hasWriteAccess(hopsworksUser, gitConfig.getApiKeyName(), gitConfig.getRemoteGitURL(), gitConfig.getGitBackend())) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_START_ERROR, Level.FINE, "API token " + gitConfig.getApiKeyName() + " does not have write access on " + gitConfig.getRemoteGitURL());
    }
    JupyterProject jp = jupyterFacade.findByUser(hdfsUser);
    if (jp == null) {
        HdfsUsers user = hdfsUsersFacade.findByName(hdfsUser);
        String configSecret = DigestUtils.sha256Hex(Integer.toString(ThreadLocalRandom.current().nextInt()));
        JupyterDTO dto = null;
        DistributedFileSystemOps dfso = dfsService.getDfsOps();
        String allowOriginHost = uriInfo.getBaseUri().getHost();
        int allowOriginPort = uriInfo.getBaseUri().getPort();
        String allowOriginPortStr = allowOriginPort != -1 ? ":" + allowOriginPort : "";
        String allowOrigin = settings.getJupyterOriginScheme() + "://" + allowOriginHost + allowOriginPortStr;
        try {
            jupyterSettingsFacade.update(jupyterSettings);
            // Inspect dependencies
            sparkController.inspectDependencies(project, hopsworksUser, (SparkJobConfiguration) jupyterSettings.getJobConfig());
            dto = jupyterManager.startJupyterServer(project, configSecret, hdfsUser, hopsworksUser, jupyterSettings, allowOrigin);
            jupyterJWTManager.materializeJWT(hopsworksUser, project, jupyterSettings, dto.getCid(), dto.getPort(), JUPYTER_JWT_AUD);
            HopsUtils.materializeCertificatesForUserCustomDir(project.getName(), user.getUsername(), settings.getHdfsTmpCertDir(), dfso, certificateMaterializer, settings, dto.getCertificatesDir());
            jupyterManager.waitForStartup(project, hopsworksUser);
        } catch (ServiceException | TimeoutException ex) {
            if (dto != null) {
                jupyterController.shutdownQuietly(project, hdfsUser, hopsworksUser, configSecret, dto.getCid(), dto.getPort());
            }
            throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_START_ERROR, Level.SEVERE, ex.getMessage(), null, ex);
        } catch (IOException ex) {
            if (dto != null) {
                jupyterController.shutdownQuietly(project, hdfsUser, hopsworksUser, configSecret, dto.getCid(), dto.getPort());
            }
            throw new HopsSecurityException(RESTCodes.SecurityErrorCode.CERT_MATERIALIZATION_ERROR, Level.SEVERE, ex.getMessage(), null, ex);
        } finally {
            if (dfso != null) {
                dfsService.closeDfsClient(dfso);
            }
        }
        String externalIp = Ip.getHost(req.getRequestURL().toString());
        try {
            Date expirationDate = new Date();
            Calendar cal = Calendar.getInstance();
            cal.setTime(expirationDate);
            cal.add(Calendar.HOUR_OF_DAY, jupyterSettings.getShutdownLevel());
            expirationDate = cal.getTime();
            jp = jupyterFacade.saveServer(externalIp, project, configSecret, dto.getPort(), user.getId(), dto.getToken(), dto.getCid(), expirationDate, jupyterSettings.isNoLimit());
            // set minutes left until notebook server is killed
            Duration durationLeft = Duration.between(new Date().toInstant(), jp.getExpires().toInstant());
            jp.setMinutesUntilExpiration(durationLeft.toMinutes());
        } catch (Exception e) {
            LOGGER.log(Level.SEVERE, "Failed to save Jupyter notebook settings", e);
            jupyterController.shutdownQuietly(project, hdfsUser, hopsworksUser, configSecret, dto.getCid(), dto.getPort());
        }
        if (jp == null) {
            throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_SAVE_SETTINGS_ERROR, Level.SEVERE);
        }
        if (jupyterSettings.isGitBackend()) {
            try {
                // Init is idempotent, calling it on an already initialized repo won't affect it
                jupyterNbVCSController.init(jp, jupyterSettings);
                if (jupyterSettings.getGitConfig().getStartupAutoPull()) {
                    jupyterNbVCSController.pull(jp, jupyterSettings);
                }
            } catch (ServiceException ex) {
                jupyterController.shutdownQuietly(project, hdfsUser, hopsworksUser, configSecret, dto.getCid(), dto.getPort());
                throw ex;
            }
        }
    } else {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_SERVER_ALREADY_RUNNING, Level.FINE);
    }
    return noCacheResponse.getNoCacheResponseBuilder(Response.Status.OK).entity(jp).build();
}
Also used : DistributedFileSystemOps(io.hops.hopsworks.common.hdfs.DistributedFileSystemOps) Calendar(java.util.Calendar) JupyterProject(io.hops.hopsworks.persistence.entity.jupyter.JupyterProject) Duration(java.time.Duration) HdfsUsers(io.hops.hopsworks.persistence.entity.hdfs.user.HdfsUsers) Users(io.hops.hopsworks.persistence.entity.user.Users) IOException(java.io.IOException) HdfsUsers(io.hops.hopsworks.persistence.entity.hdfs.user.HdfsUsers) Date(java.util.Date) TimeoutException(java.util.concurrent.TimeoutException) ProjectException(io.hops.hopsworks.exceptions.ProjectException) JobException(io.hops.hopsworks.exceptions.JobException) GenericException(io.hops.hopsworks.exceptions.GenericException) HopsSecurityException(io.hops.hopsworks.exceptions.HopsSecurityException) ElasticException(io.hops.hopsworks.exceptions.ElasticException) IOException(java.io.IOException) ServiceException(io.hops.hopsworks.exceptions.ServiceException) HopsSecurityException(io.hops.hopsworks.exceptions.HopsSecurityException) ProjectException(io.hops.hopsworks.exceptions.ProjectException) ServiceException(io.hops.hopsworks.exceptions.ServiceException) GitConfig(io.hops.hopsworks.persistence.entity.jupyter.config.GitConfig) YarnProjectsQuota(io.hops.hopsworks.persistence.entity.jobs.quota.YarnProjectsQuota) JupyterDTO(io.hops.hopsworks.common.dao.jupyter.config.JupyterDTO) TimeoutException(java.util.concurrent.TimeoutException) Path(javax.ws.rs.Path) POST(javax.ws.rs.POST) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 4 with ServiceException

use of io.hops.hopsworks.exceptions.ServiceException in project hopsworks by logicalclocks.

the class JupyterService method isRunning.

@GET
@Path("/running")
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_OWNER, AllowedProjectRoles.DATA_SCIENTIST })
@JWTRequired(acceptedTokens = { Audience.API }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response isRunning(@Context HttpServletRequest req, @Context SecurityContext sc) throws ServiceException {
    String hdfsUser = getHdfsUser(sc);
    JupyterProject jp = jupyterFacade.findByUser(hdfsUser);
    if (jp == null) {
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_SERVERS_NOT_FOUND, Level.FINE);
    }
    // Check to make sure the jupyter notebook server is running
    boolean running = jupyterManager.ping(jp);
    // we should remove the DB entry (and restart the notebook server).
    if (!running) {
        jupyterFacade.remove(hdfsUser, jp.getPort());
        throw new ServiceException(RESTCodes.ServiceErrorCode.JUPYTER_SERVERS_NOT_RUNNING, Level.FINE);
    }
    // set minutes left until notebook server is killed
    Duration durationLeft = Duration.between(new Date().toInstant(), jp.getExpires().toInstant());
    jp.setMinutesUntilExpiration(durationLeft.toMinutes());
    return noCacheResponse.getNoCacheResponseBuilder(Response.Status.OK).entity(jp).build();
}
Also used : ServiceException(io.hops.hopsworks.exceptions.ServiceException) JupyterProject(io.hops.hopsworks.persistence.entity.jupyter.JupyterProject) Duration(java.time.Duration) Date(java.util.Date) Path(javax.ws.rs.Path) Produces(javax.ws.rs.Produces) GET(javax.ws.rs.GET) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 5 with ServiceException

use of io.hops.hopsworks.exceptions.ServiceException in project hopsworks by logicalclocks.

the class RemoteCommandExecutor method executeInternal.

private RemoteCommandResult executeInternal(RemoteCommand command) throws ServiceException {
    Channel channel = null;
    Session session = null;
    try {
        LOGGER.log(Level.FINE, "Executing remote command: " + command);
        JSch jsch = new JSch();
        jsch.addIdentity(command.getIdentity().toString());
        session = jsch.getSession(command.getUser(), command.getHost(), command.getPort());
        session.setConfig(command.getSSHConfig());
        session.setConfig("StrictHostKeyChecking", "no");
        session.connect(command.getConnectTimeout());
        channel = session.openChannel("exec");
        ((ChannelExec) channel).setCommand(command.getCommand());
        String stdout = waitForCommandToFinish(channel, command);
        int exitCode = channel.getExitStatus();
        return new RemoteCommandResult(stdout, exitCode);
    } catch (JSchException | IOException ex) {
        LOGGER.log(Level.WARNING, "Error executing remote command " + command, ex);
        throw new ServiceException(RESTCodes.ServiceErrorCode.ERROR_EXECUTING_REMOTE_COMMAND, Level.WARNING, "Error while executing remote command", "Error executing remote command: " + command, ex);
    } catch (RemoteCommandTimeoutException ex) {
        return new RemoteCommandResult("Command time-out: " + ex.getMessage(), 20);
    } finally {
        if (channel != null) {
            channel.disconnect();
        }
        if (session != null) {
            session.disconnect();
        }
    }
}
Also used : JSchException(com.jcraft.jsch.JSchException) ServiceException(io.hops.hopsworks.exceptions.ServiceException) Channel(com.jcraft.jsch.Channel) IOException(java.io.IOException) JSch(com.jcraft.jsch.JSch) ChannelExec(com.jcraft.jsch.ChannelExec) Session(com.jcraft.jsch.Session)

Aggregations

ServiceException (io.hops.hopsworks.exceptions.ServiceException)53 IOException (java.io.IOException)22 ServiceDiscoveryException (com.logicalclocks.servicediscoverclient.exceptions.ServiceDiscoveryException)16 Users (io.hops.hopsworks.persistence.entity.user.Users)12 DistributedFileSystemOps (io.hops.hopsworks.common.hdfs.DistributedFileSystemOps)10 Path (javax.ws.rs.Path)9 Produces (javax.ws.rs.Produces)9 GenericException (io.hops.hopsworks.exceptions.GenericException)8 ProjectException (io.hops.hopsworks.exceptions.ProjectException)8 ElasticException (io.hops.hopsworks.exceptions.ElasticException)7 JupyterProject (io.hops.hopsworks.persistence.entity.jupyter.JupyterProject)7 Project (io.hops.hopsworks.persistence.entity.project.Project)7 TransactionAttribute (javax.ejb.TransactionAttribute)7 AllowedProjectRoles (io.hops.hopsworks.api.filter.AllowedProjectRoles)6 JobException (io.hops.hopsworks.exceptions.JobException)6 UserException (io.hops.hopsworks.exceptions.UserException)6 JWTRequired (io.hops.hopsworks.jwt.annotation.JWTRequired)6 ProcessDescriptor (io.hops.hopsworks.common.util.ProcessDescriptor)4 ProcessResult (io.hops.hopsworks.common.util.ProcessResult)4 HdfsUsers (io.hops.hopsworks.persistence.entity.hdfs.user.HdfsUsers)4