Search in sources :

Example 1 with LivyCodeException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException in project kylo by Teradata.

the class LivyRestModelTransformer method toTransformResponse.

public static TransformResponse toTransformResponse(Statement statement, String transformId) {
    TransformResponse transformResponse = prepTransformResponse(statement, transformId);
    if (transformResponse.getStatus() == TransformResponse.Status.SUCCESS) {
        String code = statement.getCode().trim();
        if (code.endsWith("dfResultsAsJson")) {
            transformResponse.setResults(toTransformQueryResultWithSchema(transformResponse, statement.getOutput()));
        } else if (code.endsWith("dfProf")) {
            List<OutputRow> rows = toTransformResponseProfileStats(statement.getOutput());
            transformResponse.setProfile(toTransformResponseProfileStats(statement.getOutput()));
            transformResponse.setActualCols(1);
            Integer actualRows = rows.stream().filter(metric -> metric.getMetricType().equals(MetricType.TOTAL_COUNT.toString())).map(metric -> Integer.valueOf(metric.getMetricValue())).findFirst().orElse(1);
            transformResponse.setActualRows(actualRows);
            transformResponse.setResults(emptyResult());
        } else if (code.endsWith("transformAsStr")) {
            /* expects that 'statement' contains a payload of TransformResponse in JSON format */
            TransformResponse tr = serializeStatementOutputResponse(checkCodeWasWellFormed(statement.getOutput()), TransformResponse.class);
            statementIdCache.put(tr.getTable(), statement.getId());
            return tr;
        } else {
            logger.error("Exception Processing Result: ", new LivyCodeException("Unsupported result type requested of Livy.  Results not recognized"));
            throw new LivyUserException("livy.unsupported_result_type");
        }
    // end if
    }
    return transformResponse;
}
Also used : XLogger(org.slf4j.ext.XLogger) LivyServerStatus(com.thinkbiganalytics.kylo.spark.client.model.enums.LivyServerStatus) XLoggerFactory(org.slf4j.ext.XLoggerFactory) StatementOutputResponse(com.thinkbiganalytics.kylo.spark.model.StatementOutputResponse) SparkJobResult(com.thinkbiganalytics.kylo.spark.rest.model.job.SparkJobResult) StringUtils(org.apache.commons.lang3.StringUtils) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) SparkJobResponse(com.thinkbiganalytics.kylo.spark.rest.model.job.SparkJobResponse) ObjectMapperSerializer(com.thinkbiganalytics.json.ObjectMapperSerializer) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) Lists(com.google.common.collect.Lists) MetricType(com.thinkbiganalytics.spark.dataprofiler.model.MetricType) StatementState(com.thinkbiganalytics.kylo.spark.model.enums.StatementState) Map(java.util.Map) JsonNode(com.fasterxml.jackson.databind.JsonNode) URI(java.net.URI) Nonnull(javax.annotation.Nonnull) SessionState(com.thinkbiganalytics.kylo.spark.model.enums.SessionState) LivySessionStatus(com.thinkbiganalytics.kylo.spark.client.model.enums.LivySessionStatus) Iterator(java.util.Iterator) DefaultQueryResultColumn(com.thinkbiganalytics.discovery.model.DefaultQueryResultColumn) Statement(com.thinkbiganalytics.kylo.spark.model.Statement) StatementOutputStatus(com.thinkbiganalytics.kylo.spark.model.enums.StatementOutputStatus) OutputRow(com.thinkbiganalytics.spark.dataprofiler.output.OutputRow) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) SparkLivySaveException(com.thinkbiganalytics.kylo.spark.livy.SparkLivySaveException) IOException(java.io.IOException) DataSources(com.thinkbiganalytics.spark.rest.model.DataSources) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) SaveResponse(com.thinkbiganalytics.spark.rest.model.SaveResponse) LivyServer(com.thinkbiganalytics.kylo.spark.client.model.LivyServer) LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) QueryResultColumn(com.thinkbiganalytics.discovery.schema.QueryResultColumn) ServerStatusResponse(com.thinkbiganalytics.spark.rest.model.ServerStatusResponse) WebApplicationException(javax.ws.rs.WebApplicationException) CacheBuilder(com.google.common.cache.CacheBuilder) LivyDeserializationException(com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException) Cache(com.google.common.cache.Cache) TransformQueryResult(com.thinkbiganalytics.spark.rest.model.TransformQueryResult) LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) List(java.util.List) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)

Example 2 with LivyCodeException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException in project kylo by Teradata.

the class LivyRestModelTransformer method checkCodeWasWellFormed.

private static StatementOutputResponse checkCodeWasWellFormed(StatementOutputResponse statementOutputResponse) {
    if (statementOutputResponse != null && statementOutputResponse.getStatus() != StatementOutputStatus.ok) {
        String msg = String.format("Malformed code sent to Livy.  ErrorType='%s', Error='%s', Traceback='%s'", statementOutputResponse.getEname(), statementOutputResponse.getEvalue(), statementOutputResponse.getTraceback());
        logger.error("Exception Processing Query: ", new LivyCodeException(msg));
        throw new LivyUserException("livy.syntax");
    }
    return statementOutputResponse;
}
Also used : LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)

Example 3 with LivyCodeException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException in project kylo by Teradata.

the class DefaultLivyClient method pollStatement.

@Override
public Statement pollStatement(JerseyRestClient jerseyClient, SparkLivyProcess sparkLivyProcess, Integer stmtId, Long wait) {
    logger.entry(jerseyClient, sparkLivyProcess, stmtId, wait);
    long stopPolling = Long.MAX_VALUE;
    long startMillis = System.currentTimeMillis();
    if (wait != null) {
        // Limit the amount of time we will poll for a statement to complete.
        stopPolling = startMillis + livyProperties.getPollingLimit();
    }
    Statement statement;
    int pollCount = 1;
    do {
        statement = getStatement(jerseyClient, sparkLivyProcess, stmtId);
        if (statement.getState().equals(StatementState.error)) {
            // TODO: what about cancelled? or cancelling?
            logger.error("Unexpected error encountered while processing a statement", new LivyCodeException(statement.toString()));
            throw logger.throwing(new LivyUserException("livy.unexpected_error"));
        }
        if (System.currentTimeMillis() > stopPolling || statement.getState().equals(StatementState.available)) {
            break;
        }
        logger.trace("Statement was not ready, polling now with attempt '{}'", pollCount++);
        // statement not ready, wait for some time...
        try {
            Thread.sleep(livyProperties.getPollingInterval());
        } catch (InterruptedException e) {
            logger.error("Thread interrupted while polling Livy", e);
        }
    } while (true);
    logger.debug("exit DefaultLivyClient poll statement in '{}' millis, after '{}' attempts ", System.currentTimeMillis() - startMillis, pollCount);
    return logger.exit(statement);
}
Also used : Statement(com.thinkbiganalytics.kylo.spark.model.Statement) LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)

Aggregations

LivyCodeException (com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException)3 LivyUserException (com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)3 Statement (com.thinkbiganalytics.kylo.spark.model.Statement)2 JsonNode (com.fasterxml.jackson.databind.JsonNode)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 ArrayNode (com.fasterxml.jackson.databind.node.ArrayNode)1 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)1 Cache (com.google.common.cache.Cache)1 CacheBuilder (com.google.common.cache.CacheBuilder)1 Lists (com.google.common.collect.Lists)1 DefaultQueryResultColumn (com.thinkbiganalytics.discovery.model.DefaultQueryResultColumn)1 QueryResultColumn (com.thinkbiganalytics.discovery.schema.QueryResultColumn)1 ObjectMapperSerializer (com.thinkbiganalytics.json.ObjectMapperSerializer)1 LivyServer (com.thinkbiganalytics.kylo.spark.client.model.LivyServer)1 LivyServerStatus (com.thinkbiganalytics.kylo.spark.client.model.enums.LivyServerStatus)1 LivySessionStatus (com.thinkbiganalytics.kylo.spark.client.model.enums.LivySessionStatus)1 LivyDeserializationException (com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException)1 SparkLivySaveException (com.thinkbiganalytics.kylo.spark.livy.SparkLivySaveException)1 StatementOutputResponse (com.thinkbiganalytics.kylo.spark.model.StatementOutputResponse)1 SessionState (com.thinkbiganalytics.kylo.spark.model.enums.SessionState)1