Examples with SparkClassLoader co.cask.cdap.app.runtime.spark.SparkClassLoader used on opensource projects

Example 1 with SparkClassLoader

use of co.cask.cdap.app.runtime.spark.SparkClassLoader in project cdap by caskdata.

the class AbstractSparkSubmitter method submit.

/**
   * Submits the Spark job using {@link SparkSubmit}.
   *
   * @param runtimeContext context representing the Spark program
   * @param args arguments for the {@link SparkSubmit#main(String[])} method.
   */
private void submit(SparkRuntimeContext runtimeContext, String[] args) {
    Cancellable cancellable = SparkRuntimeUtils.setContextClassLoader(new SparkClassLoader(runtimeContext));
    try {
        LOG.debug("Calling SparkSubmit for {} {}: {}", runtimeContext.getProgram().getId(), runtimeContext.getRunId(), Arrays.toString(args));
        // Explicitly set the SPARK_SUBMIT property as it is no longer set on the System properties by the SparkSubmit
        // after the class rewrite. This property only control logging of a warning when submitting the Spark job,
        // hence it's harmless to just leave it there.
        System.setProperty("SPARK_SUBMIT", "true");
        SparkSubmit.main(args);
        LOG.debug("SparkSubmit returned for {} {}", runtimeContext.getProgram().getId(), runtimeContext.getRunId());
    } finally {
        cancellable.cancel();
    }
}