Examples with ShellTransformStage com.thinkbiganalytics.spark.metadata.ShellTransformStage used on opensource projects

Example 1 with ShellTransformStage

use of com.thinkbiganalytics.spark.metadata.ShellTransformStage in project kylo by Teradata.

the class TransformService method execute.

/**
 * Executes the specified transformation and returns the name of the Hive table containing the results.
 *
 * @param request the transformation request
 * @return the Hive table containing the results
 * @throws IllegalStateException if this service is not running
 * @throws ScriptException       if the script cannot be executed
 */
@Nonnull
public TransformResponse execute(@Nonnull final TransformRequest request) throws ScriptException {
    log.entry(request);
    // Handle async request
    if (request.isAsync()) {
        return cacheTransform(request);
    }
    // Execute script
    final DataSet dataSet = createShellTask(request);
    final StructType schema = dataSet.schema();
    TransformResponse response = submitTransformJob(new ShellTransformStage(dataSet), getPolicies(request));
    // Build response
    if (response.getStatus() != TransformResponse.Status.SUCCESS) {
        final String table = response.getTable();
        final TransformQueryResult partialResult = new TransformQueryResult();
        partialResult.setColumns(Arrays.<QueryResultColumn>asList(new QueryResultRowTransform(schema, table).columns()));
        response = new TransformResponse();
        response.setProgress(0.0);
        response.setResults(partialResult);
        response.setStatus(TransformResponse.Status.PENDING);
        response.setTable(table);
    }
    return log.exit(response);
}

Example 2 with ShellTransformStage

use of com.thinkbiganalytics.spark.metadata.ShellTransformStage in project kylo by Teradata.

the class TransformService method saveShell.

/**
 * Executes and saves a Spark shell request.
 */
@Nonnull
public SaveResponse saveShell(@Nonnull final String id, @Nonnull final SaveRequest save) throws ScriptException {
    log.entry(id, save);
    final DataSet dataSet = createShellTask(getTransformRequest(id));
    final SaveResponse response = submitSaveJob(createSaveTask(save, new ShellTransformStage(dataSet)));
    return log.exit(response);
}