Search in sources :

Example 1 with LivyUserException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException in project kylo by Teradata.

the class LivyRestModelTransformer method toTransformResponse.

public static TransformResponse toTransformResponse(Statement statement, String transformId) {
    TransformResponse transformResponse = prepTransformResponse(statement, transformId);
    if (transformResponse.getStatus() == TransformResponse.Status.SUCCESS) {
        String code = statement.getCode().trim();
        if (code.endsWith("dfResultsAsJson")) {
            transformResponse.setResults(toTransformQueryResultWithSchema(transformResponse, statement.getOutput()));
        } else if (code.endsWith("dfProf")) {
            List<OutputRow> rows = toTransformResponseProfileStats(statement.getOutput());
            transformResponse.setProfile(toTransformResponseProfileStats(statement.getOutput()));
            transformResponse.setActualCols(1);
            Integer actualRows = rows.stream().filter(metric -> metric.getMetricType().equals(MetricType.TOTAL_COUNT.toString())).map(metric -> Integer.valueOf(metric.getMetricValue())).findFirst().orElse(1);
            transformResponse.setActualRows(actualRows);
            transformResponse.setResults(emptyResult());
        } else if (code.endsWith("transformAsStr")) {
            /* expects that 'statement' contains a payload of TransformResponse in JSON format */
            TransformResponse tr = serializeStatementOutputResponse(checkCodeWasWellFormed(statement.getOutput()), TransformResponse.class);
            statementIdCache.put(tr.getTable(), statement.getId());
            return tr;
        } else {
            logger.error("Exception Processing Result: ", new LivyCodeException("Unsupported result type requested of Livy.  Results not recognized"));
            throw new LivyUserException("livy.unsupported_result_type");
        }
    // end if
    }
    return transformResponse;
}
Also used : XLogger(org.slf4j.ext.XLogger) LivyServerStatus(com.thinkbiganalytics.kylo.spark.client.model.enums.LivyServerStatus) XLoggerFactory(org.slf4j.ext.XLoggerFactory) StatementOutputResponse(com.thinkbiganalytics.kylo.spark.model.StatementOutputResponse) SparkJobResult(com.thinkbiganalytics.kylo.spark.rest.model.job.SparkJobResult) StringUtils(org.apache.commons.lang3.StringUtils) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) SparkJobResponse(com.thinkbiganalytics.kylo.spark.rest.model.job.SparkJobResponse) ObjectMapperSerializer(com.thinkbiganalytics.json.ObjectMapperSerializer) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) Lists(com.google.common.collect.Lists) MetricType(com.thinkbiganalytics.spark.dataprofiler.model.MetricType) StatementState(com.thinkbiganalytics.kylo.spark.model.enums.StatementState) Map(java.util.Map) JsonNode(com.fasterxml.jackson.databind.JsonNode) URI(java.net.URI) Nonnull(javax.annotation.Nonnull) SessionState(com.thinkbiganalytics.kylo.spark.model.enums.SessionState) LivySessionStatus(com.thinkbiganalytics.kylo.spark.client.model.enums.LivySessionStatus) Iterator(java.util.Iterator) DefaultQueryResultColumn(com.thinkbiganalytics.discovery.model.DefaultQueryResultColumn) Statement(com.thinkbiganalytics.kylo.spark.model.Statement) StatementOutputStatus(com.thinkbiganalytics.kylo.spark.model.enums.StatementOutputStatus) OutputRow(com.thinkbiganalytics.spark.dataprofiler.output.OutputRow) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) SparkLivySaveException(com.thinkbiganalytics.kylo.spark.livy.SparkLivySaveException) IOException(java.io.IOException) DataSources(com.thinkbiganalytics.spark.rest.model.DataSources) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) SaveResponse(com.thinkbiganalytics.spark.rest.model.SaveResponse) LivyServer(com.thinkbiganalytics.kylo.spark.client.model.LivyServer) LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) QueryResultColumn(com.thinkbiganalytics.discovery.schema.QueryResultColumn) ServerStatusResponse(com.thinkbiganalytics.spark.rest.model.ServerStatusResponse) WebApplicationException(javax.ws.rs.WebApplicationException) CacheBuilder(com.google.common.cache.CacheBuilder) LivyDeserializationException(com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException) Cache(com.google.common.cache.Cache) TransformQueryResult(com.thinkbiganalytics.spark.rest.model.TransformQueryResult) LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) List(java.util.List) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)

Example 2 with LivyUserException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException in project kylo by Teradata.

the class LivyRestModelTransformer method convertDataFrameDataType.

private static String convertDataFrameDataType(JsonNode dataType) {
    if (dataType.isObject()) {
        String type = dataType.get("type").asText();
        if (type.equals("udt")) {
            if (dataType.get("class").asText().equals("org.apache.spark.mllib.linalg.VectorUDT")) {
                // TODO: null check
                ArrayNode fields = (ArrayNode) dataType.get("sqlType").get("fields");
                Iterator<JsonNode> fieldsIter = fields.elements();
                while (fieldsIter.hasNext()) {
                    ObjectNode fieldDescriptors = (ObjectNode) fieldsIter.next();
                    if (fieldDescriptors.get("name").asText().equals("values")) {
                        ObjectNode fdObj = (ObjectNode) fieldDescriptors.get("type");
                        return new StringBuilder(fdObj.get("type").asText()).append("<").append(fdObj.get("elementType").asText()).append(">").toString();
                    }
                }
                return "Unknown UDT";
            } else {
                if (dataType.get("class") != null) {
                    logger.error("UDT error encountered", new LivyDeserializationException("don't know how to deserialize UDT types for class = " + dataType.get("class").asText()));
                } else {
                    logger.error("UDT error encountered", new LivyDeserializationException("don't know how to deserialize UDT type of unspecified class"));
                }
                // end if
                throw new LivyUserException("livy.unexpected_error");
            }
        // end if
        } else if (type.equals("map")) {
            return new StringBuilder(dataType.get("type").asText()).append("<").append(dataType.get("keyType").asText()).append(",").append(dataType.get("valueType").asText()).append(">").toString();
        } else if (type.equals("struct")) {
            ArrayNode fields = (ArrayNode) dataType.get("fields");
            Iterator<JsonNode> nodes = fields.elements();
            StringBuilder sb = new StringBuilder("struct<");
            // assumes min of 1 field in struct
            while (nodes.hasNext()) {
                ObjectNode node = (ObjectNode) nodes.next();
                String sfName = node.get("name").asText();
                String sfType = node.get("type").asText();
                sb.append(sfName).append(":").append(sfType).append(",");
            }
            sb.deleteCharAt(sb.length() - 1);
            return sb.toString();
        } else {
            // can there be other types?
            return "Unknown Type";
        }
    // end if
    } else {
        return dataType.asText();
    }
// end if
}
Also used : ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) JsonNode(com.fasterxml.jackson.databind.JsonNode) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) LivyDeserializationException(com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException)

Example 3 with LivyUserException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException in project kylo by Teradata.

the class LivyRestModelTransformer method checkCodeWasWellFormed.

private static StatementOutputResponse checkCodeWasWellFormed(StatementOutputResponse statementOutputResponse) {
    if (statementOutputResponse != null && statementOutputResponse.getStatus() != StatementOutputStatus.ok) {
        String msg = String.format("Malformed code sent to Livy.  ErrorType='%s', Error='%s', Traceback='%s'", statementOutputResponse.getEname(), statementOutputResponse.getEvalue(), statementOutputResponse.getTraceback());
        logger.error("Exception Processing Query: ", new LivyCodeException(msg));
        throw new LivyUserException("livy.syntax");
    }
    return statementOutputResponse;
}
Also used : LivyCodeException(com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)

Example 4 with LivyUserException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException in project kylo by Teradata.

the class LivyRestModelTransformer method serializeStatementOutputResponse.

private static <T extends Object> T serializeStatementOutputResponse(StatementOutputResponse sor, Class<T> clazz) {
    String errMsg = String.format("Unable to deserialize JSON returned from Livy into class '%s'", clazz.getSimpleName());
    JsonNode data = sor.getData();
    if (data != null) {
        JsonNode json = data.get("application/json");
        String jsonString = json.asText();
        try {
            return mapper.readValue(jsonString, clazz);
        } catch (IOException e) {
            logger.error("Deserialization error occured", new LivyDeserializationException(errMsg));
        }
    // end try/catch
    } else {
        logger.error("Deserialization error occured", new LivyDeserializationException(errMsg));
    }
    throw new LivyUserException("livy.unexpected_error");
}
Also used : JsonNode(com.fasterxml.jackson.databind.JsonNode) IOException(java.io.IOException) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) LivyDeserializationException(com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException)

Example 5 with LivyUserException

use of com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException in project kylo by Teradata.

the class SparkLivyRestClient method getTransformResult.

@Nonnull
@Override
public Optional<TransformResponse> getTransformResult(@Nonnull SparkShellProcess process, @Nonnull String transformId) {
    logger.entry(process, transformId);
    Validate.isInstanceOf(SparkLivyProcess.class, process, "SparkLivyRestClient.getTransformResult called on non Livy Process");
    SparkLivyProcess sparkLivyProcess = (SparkLivyProcess) process;
    JerseyRestClient client = sparkLivyProcessManager.getClient(process);
    Integer stmtId = sparkLivyProcessManager.getStatementId(transformId);
    Statement statement = livyClient.getStatement(client, sparkLivyProcess, stmtId);
    sparkLivyProcessManager.setStatementId(transformId, statement.getId());
    TransformResponse response = LivyRestModelTransformer.toTransformResponse(statement, transformId);
    if (statement.getState() == StatementState.available && response.getStatus() == TransformResponse.Status.PENDING) {
        // TODO:: change this so that if transformId is a livyQueryID it knows what to do. similar to saveResponse.  A good first stab at this is at KYLO-2639-withLivyId
        // The result came back from Livy, but the transform result still needs to be fetched
        String script = scriptGenerator.script("getTransform", ScalaScriptUtils.scalaStr(transformId));
        statement = submitCode(client, script, process);
        // associate transformId to this new query and get results on subsequent call
        sparkLivyProcessManager.setStatementId(transformId, statement.getId());
    } else if (response.getStatus() == TransformResponse.Status.ERROR) {
        logger.error(String.format("Unexpected error found in transform response:\n%s", response.getMessage()));
        throw new LivyUserException("livy.transform_error");
    }
    // end if
    return logger.exit(Optional.of(response));
}
Also used : Statement(com.thinkbiganalytics.kylo.spark.model.Statement) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) LivyUserException(com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException) JerseyRestClient(com.thinkbiganalytics.rest.JerseyRestClient) Nonnull(javax.annotation.Nonnull)

Aggregations

LivyUserException (com.thinkbiganalytics.kylo.spark.exceptions.LivyUserException)13 JerseyRestClient (com.thinkbiganalytics.rest.JerseyRestClient)7 Statement (com.thinkbiganalytics.kylo.spark.model.Statement)6 Nonnull (javax.annotation.Nonnull)5 JsonNode (com.fasterxml.jackson.databind.JsonNode)4 LivyCodeException (com.thinkbiganalytics.kylo.spark.exceptions.LivyCodeException)4 LivyDeserializationException (com.thinkbiganalytics.kylo.spark.exceptions.LivyDeserializationException)4 ArrayNode (com.fasterxml.jackson.databind.node.ArrayNode)3 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)3 Session (com.thinkbiganalytics.kylo.spark.model.Session)3 TransformResponse (com.thinkbiganalytics.spark.rest.model.TransformResponse)3 IOException (java.io.IOException)3 DefaultQueryResultColumn (com.thinkbiganalytics.discovery.model.DefaultQueryResultColumn)2 QueryResultColumn (com.thinkbiganalytics.discovery.schema.QueryResultColumn)2 LivyServerNotReachableException (com.thinkbiganalytics.kylo.spark.exceptions.LivyServerNotReachableException)2 SparkLivySaveException (com.thinkbiganalytics.kylo.spark.livy.SparkLivySaveException)2 SparkJobResponse (com.thinkbiganalytics.kylo.spark.rest.model.job.SparkJobResponse)2 SaveResponse (com.thinkbiganalytics.spark.rest.model.SaveResponse)2 ServerStatusResponse (com.thinkbiganalytics.spark.rest.model.ServerStatusResponse)2 TransformQueryResult (com.thinkbiganalytics.spark.rest.model.TransformQueryResult)2