Search in sources :

Example 1 with KyloCatalogReadRequest

use of com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest in project kylo by Teradata.

the class SparkShellProxyController method preview.

@POST
@Path("/preview")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@ApiOperation("Returns the dataset preview")
@ApiResponses({ @ApiResponse(code = 200, message = "Returns the status of the file-metadata job.", response = PreviewDataSetTransformResponse.class), @ApiResponse(code = 400, message = "The requested data source does not exist.", response = RestResponseStatus.class), @ApiResponse(code = 500, message = "There was a problem processing the data.", response = RestResponseStatus.class) })
public Response preview(PreviewDataSetRequest previewRequest) {
    DataSource catalogDataSource = fetchCatalogDataSource(previewRequest.getDataSource().getId());
    previewRequest.setDataSource(catalogDataSource);
    if (previewRequest.isFilePreview() && previewRequest.getSchemaParser() == null) {
        // set it to a text preview
        previewRequest.setSchemaParser(getTextSchemaParserDescriptor());
    }
    KyloCatalogReadRequest request = KyloCatalogReaderUtil.toKyloCatalogRequest(previewRequest);
    final SparkShellProcess process = getSparkShellProcess();
    return getTransformResponse(() -> {
        PreviewDataSetTransformResponse response = null;
        boolean fallbackToTextParser = previewRequest.isFallbackToTextOnError();
        try {
            TransformResponse transformResponse = restClient.kyloCatalogTransform(process, request);
            response = new PreviewDataSetTransformResponse(transformResponse, previewRequest.getSchemaParser());
        } catch (Exception e) {
            // should we attempt to re preview the data as plain text
            if (fallbackToTextParser && previewRequest.getSchemaParser() != null && !"text".equalsIgnoreCase(previewRequest.getSchemaParser().getSparkFormat())) {
                previewRequest.setSchemaParser(getTextSchemaParserDescriptor());
                KyloCatalogReadRequest request2 = KyloCatalogReaderUtil.toKyloCatalogRequest(previewRequest);
                TransformResponse transformResponse = restClient.kyloCatalogTransform(process, request2);
                response = new PreviewDataSetTransformResponse(transformResponse, previewRequest.getSchemaParser());
            } else {
                throw e;
            }
            throw e;
        }
        return response;
    });
}
Also used : SparkShellProcess(com.thinkbiganalytics.spark.shell.SparkShellProcess) PreviewDataSetTransformResponse(com.thinkbiganalytics.spark.rest.model.PreviewDataSetTransformResponse) KyloCatalogReadRequest(com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest) PreviewDataSetTransformResponse(com.thinkbiganalytics.spark.rest.model.PreviewDataSetTransformResponse) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) ModifiedTransformResponse(com.thinkbiganalytics.spark.rest.model.ModifiedTransformResponse) SparkException(com.thinkbiganalytics.kylo.spark.SparkException) WebApplicationException(javax.ws.rs.WebApplicationException) NoSuchMessageException(org.springframework.context.NoSuchMessageException) TimeoutException(java.util.concurrent.TimeoutException) BadRequestException(javax.ws.rs.BadRequestException) NotFoundException(javax.ws.rs.NotFoundException) SparkShellSaveException(com.thinkbiganalytics.spark.shell.SparkShellSaveException) SparkShellTransformException(com.thinkbiganalytics.spark.shell.SparkShellTransformException) DataSource(com.thinkbiganalytics.kylo.catalog.rest.model.DataSource) Path(javax.ws.rs.Path) POST(javax.ws.rs.POST) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) ApiOperation(io.swagger.annotations.ApiOperation) ApiResponses(io.swagger.annotations.ApiResponses)

Example 2 with KyloCatalogReadRequest

use of com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest in project kylo by Teradata.

the class KyloCatalogReaderUtil method toKyloCatalogRequest.

public static KyloCatalogReadRequest toKyloCatalogRequest(PreviewDataSetRequest previewRequest) {
    DataSource dataSource = previewRequest.getDataSource();
    Connector connector = dataSource.getConnector();
    // merge template
    DataSetTemplate dataSetTemplate = DataSourceUtil.mergeTemplates(dataSource);
    // get data out of the dataset template
    List<String> jars = dataSetTemplate.getJars();
    List<String> paths = dataSetTemplate.getPaths();
    List<String> files = dataSetTemplate.getFiles();
    String format = dataSetTemplate.getFormat();
    Map<String, String> options = dataSetTemplate.getOptions();
    if (options == null) {
        options = new HashMap<>();
    }
    // parse the SchemaParser if it exists and add options and update the format
    if (previewRequest.getSchemaParser() != null) {
        SchemaParserDescriptor schemaParser = previewRequest.getSchemaParser();
        Map<String, String> sparkOptions = schemaParser.getProperties().stream().collect(Collectors.toMap(p -> p.getAdditionalProperties().stream().filter(labelValue -> "spark.option".equalsIgnoreCase(labelValue.getLabel())).map(labelValue -> labelValue.getValue()).findFirst().orElse(""), p -> p.getValue()));
        // remove any options that produced an empty key
        sparkOptions.remove("");
        // supplied options by the schema parse take precedence over the template options
        options.putAll(sparkOptions);
        format = schemaParser.getSparkFormat();
    }
    // add in additional preview options
    if (previewRequest.getProperties() != null && !previewRequest.getProperties().isEmpty()) {
        options.putAll(previewRequest.getProperties());
    }
    KyloCatalogReadRequest request = new KyloCatalogReadRequest();
    request.setFiles(files);
    request.setJars(jars);
    request.setFormat(format);
    request.setOptions(options);
    if (previewRequest.getPreviewItem() != null && previewRequest.isAddPreviewItemToPath()) {
        request.addPath(previewRequest.getPreviewItem());
    }
    PageSpec pageSpec = previewRequest.getPageSpec();
    if (pageSpec == null) {
        pageSpec = new PageSpec();
    }
    request.setPageSpec(pageSpec);
    return request;
}
Also used : DataSetTemplate(com.thinkbiganalytics.kylo.catalog.rest.model.DataSetTemplate) List(java.util.List) Connector(com.thinkbiganalytics.kylo.catalog.rest.model.Connector) PreviewDataSetRequest(com.thinkbiganalytics.spark.rest.model.PreviewDataSetRequest) DataSourceUtil(com.thinkbiganalytics.kylo.catalog.datasource.DataSourceUtil) DataSource(com.thinkbiganalytics.kylo.catalog.rest.model.DataSource) Map(java.util.Map) SchemaParserDescriptor(com.thinkbiganalytics.discovery.model.SchemaParserDescriptor) KyloCatalogReadRequest(com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest) PageSpec(com.thinkbiganalytics.spark.rest.model.PageSpec) HashMap(java.util.HashMap) Collectors(java.util.stream.Collectors) Connector(com.thinkbiganalytics.kylo.catalog.rest.model.Connector) DataSetTemplate(com.thinkbiganalytics.kylo.catalog.rest.model.DataSetTemplate) KyloCatalogReadRequest(com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest) SchemaParserDescriptor(com.thinkbiganalytics.discovery.model.SchemaParserDescriptor) PageSpec(com.thinkbiganalytics.spark.rest.model.PageSpec) DataSource(com.thinkbiganalytics.kylo.catalog.rest.model.DataSource)

Aggregations

DataSource (com.thinkbiganalytics.kylo.catalog.rest.model.DataSource)2 KyloCatalogReadRequest (com.thinkbiganalytics.spark.rest.model.KyloCatalogReadRequest)2 SchemaParserDescriptor (com.thinkbiganalytics.discovery.model.SchemaParserDescriptor)1 DataSourceUtil (com.thinkbiganalytics.kylo.catalog.datasource.DataSourceUtil)1 Connector (com.thinkbiganalytics.kylo.catalog.rest.model.Connector)1 DataSetTemplate (com.thinkbiganalytics.kylo.catalog.rest.model.DataSetTemplate)1 SparkException (com.thinkbiganalytics.kylo.spark.SparkException)1 ModifiedTransformResponse (com.thinkbiganalytics.spark.rest.model.ModifiedTransformResponse)1 PageSpec (com.thinkbiganalytics.spark.rest.model.PageSpec)1 PreviewDataSetRequest (com.thinkbiganalytics.spark.rest.model.PreviewDataSetRequest)1 PreviewDataSetTransformResponse (com.thinkbiganalytics.spark.rest.model.PreviewDataSetTransformResponse)1 TransformResponse (com.thinkbiganalytics.spark.rest.model.TransformResponse)1 SparkShellProcess (com.thinkbiganalytics.spark.shell.SparkShellProcess)1 SparkShellSaveException (com.thinkbiganalytics.spark.shell.SparkShellSaveException)1 SparkShellTransformException (com.thinkbiganalytics.spark.shell.SparkShellTransformException)1 ApiOperation (io.swagger.annotations.ApiOperation)1 ApiResponses (io.swagger.annotations.ApiResponses)1 HashMap (java.util.HashMap)1 List (java.util.List)1 Map (java.util.Map)1