Search in sources :

Example 1 with LivyServerNotReachableException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException in project kylo by Teradata.

the class SparkLivyProcessManager method startLivySession.

private Session startLivySession(SparkLivyProcess sparkLivyProcess) {
    // it was determined we needed a
    sparkLivyProcess.newSession();
    JerseyRestClient jerseyClient = getClient(sparkLivyProcess);
    Map<String, String> sparkProps = livyProperties.getSparkProperties();
    SessionsPost.Builder builder = new SessionsPost.Builder().kind(// "shared" most likely
    livyProperties.getLivySessionKind().toString()).conf(// "spark.driver.extraJavaOptions", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8990"
    sparkProps);
    logger.debug("LivyProperties={}", livyProperties);
    if (livyProperties.getProxyUser()) {
        String user = processCache.inverse().get(sparkLivyProcess);
        if (user != null) {
            builder.proxyUser(user);
        }
    }
    SessionsPost sessionsPost = builder.build();
    logger.info("sessionsPost={}", sessionsPost);
    Session currentSession;
    try {
        currentSession = livyClient.postSessions(jerseyClient, sessionsPost);
        if (currentSession == null) {
            logger.error("Server not reachable", new LivyServerNotReachableException("Empty result from LivyClient.postSessions"));
            throw new LivyUserException("livy.server_not_found");
        }
    } catch (LivyException le) {
        throw le;
    } catch (Exception e) {
        // resets the latch
        sparkLivyProcess.startFailed();
        this.processStopped(sparkLivyProcess);
        // NOTE: you can get "javax.ws.rs.ProcessingException: java.io.IOException: Error writing to server" on Ubuntu see: https://stackoverflow.com/a/39718929/154461
        throw new LivyException(e);
    }
    sparkLivyProcess.setSessionId(currentSession.getId());
    // notifies listeners
    this.processStarted(sparkLivyProcess);
    // begin monitoring this session if configured to do so..
    heartbeatMonitor.monitorSession(sparkLivyProcess);
    return currentSession;
}
Also used : SessionsPost(com.thinkbiganalytics.kylo.spark.model.SessionsPost) LivyException(com.thinkbiganalytics.kylo.spark.exceptions.LivyException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) JerseyRestClient(com.thinkbiganalytics.rest.JerseyRestClient) LivyException(com.thinkbiganalytics.kylo.spark.exceptions.LivyException) LivyServerNotReachableException(com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) LivyInvalidSessionException(com.thinkbiganalytics.kylo.spark.exceptions.LivyInvalidSessionException) Session(com.thinkbiganalytics.kylo.spark.model.Session) LivyServerNotReachableException(com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException)

Example 2 with LivyServerNotReachableException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException in project kylo by Teradata.

the class SparkLivyProcessManager method getLivySession.

private Optional<Session> getLivySession(SparkLivyProcess sparkLivyProcess) {
    JerseyRestClient jerseyClient = getClient(sparkLivyProcess);
    SessionsGetResponse sessions = livyClient.getSessions(jerseyClient);
    if (sessions == null) {
        logger.error("Server not reachable", new LivyServerNotReachableException("Empty result from LivyClient.getSessions"));
        throw new LivyUserException("livy.server_not_found");
    }
    Optional<Session> optSession = sessions.getSessionWithId(sparkLivyProcess.getSessionId());
    if (!optSession.isPresent()) {
        // current client not found... let's make a new one
        clearClientState(sparkLivyProcess);
    }
    return optSession;
}
Also used : SessionsGetResponse(com.thinkbiganalytics.kylo.spark.model.SessionsGetResponse) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) JerseyRestClient(com.thinkbiganalytics.rest.JerseyRestClient) LivyServerNotReachableException(com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException) Session(com.thinkbiganalytics.kylo.spark.model.Session)

Aggregations

LivyServerNotReachableException (com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException)2 LivyUserException (com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)2 Session (com.thinkbiganalytics.kylo.spark.model.Session)2 JerseyRestClient (com.thinkbiganalytics.rest.JerseyRestClient)2 LivyException (com.thinkbiganalytics.kylo.spark.exceptions.LivyException)1 LivyInvalidSessionException (com.thinkbiganalytics.kylo.spark.exceptions.LivyInvalidSessionException)1 SessionsGetResponse (com.thinkbiganalytics.kylo.spark.model.SessionsGetResponse)1 SessionsPost (com.thinkbiganalytics.kylo.spark.model.SessionsPost)1