Search in sources :

Example 1 with DefaultAppConfigurer

use of io.cdap.cdap.app.DefaultAppConfigurer in project cdap by caskdata.

the class InMemoryConfigurator method createResponse.

private <T extends Config> ConfigResponse createResponse(Application<T> app, ClassLoader artifactClassLoader) throws Exception {
    // This Gson cannot be static since it is used to deserialize user class.
    // Gson will keep a static map to class, hence will leak the classloader
    Gson gson = new GsonBuilder().registerTypeAdapterFactory(new CaseInsensitiveEnumTypeAdapterFactory()).create();
    // Now, we call configure, which returns application specification.
    DefaultAppConfigurer configurer;
    File tempDir = DirUtils.createTempDir(baseUnpackDir);
    try (PluginInstantiator pluginInstantiator = new PluginInstantiator(cConf, app.getClass().getClassLoader(), tempDir)) {
        RuntimeConfigurer runtimeConfigurer = runtimeInfo != null ? new DefaultAppRuntimeConfigurer(appNamespace.getId(), remoteClientFactory, runtimeInfo.getUserArguments(), runtimeInfo.getExistingAppSpec()) : null;
        configurer = new DefaultAppConfigurer(appNamespace, artifactId, app, configString, pluginFinder, pluginInstantiator, runtimeConfigurer, runtimeInfo, featureFlagsProvider);
        T appConfig;
        Type configType = Artifacts.getConfigType(app.getClass());
        if (configString.isEmpty()) {
            // noinspection unchecked
            appConfig = ((Class<T>) configType).newInstance();
        } else {
            try {
                appConfig = gson.fromJson(configString, configType);
            } catch (JsonSyntaxException e) {
                throw new IllegalArgumentException("Invalid JSON configuration was provided. Please check the syntax.", e);
            }
        }
        try {
            ClassLoader oldClassLoader = ClassLoaders.setContextClassLoader(new CombineClassLoader(null, app.getClass().getClassLoader(), getClass().getClassLoader()));
            try {
                app.configure(configurer, new DefaultApplicationContext<>(appConfig));
            } finally {
                ClassLoaders.setContextClassLoader(oldClassLoader);
            }
        } catch (Throwable t) {
            Throwable rootCause = Throwables.getRootCause(t);
            if (rootCause instanceof ClassNotFoundException) {
                // Heuristic to provide better error message
                String missingClass = rootCause.getMessage();
                // If the missing class has "spark" in the name, try to see if Spark is available
                if (missingClass.startsWith("org.apache.spark.") || missingClass.startsWith("io.cdap.cdap.api.spark.")) {
                    // Try to load the SparkContext class, which should be available if Spark is available in the platform
                    try {
                        artifactClassLoader.loadClass("org.apache.spark.SparkContext");
                    } catch (ClassNotFoundException e) {
                        // Spark is not available, it is most likely caused by missing Spark in the platform
                        throw new IllegalStateException("Missing Spark related class " + missingClass + ". It may be caused by unavailability of Spark. " + "Please verify environment variable " + Constants.SPARK_HOME + " is set correctly", t);
                    }
                    // Spark is available, can be caused by incompatible Spark version
                    throw new InvalidArtifactException("Missing Spark related class " + missingClass + ". Configured to use Spark located at " + System.getenv(Constants.SPARK_HOME) + ", which may be incompatible with the one required by the application", t);
                }
                // then the missing class is most likely due to some missing library in the artifact jar
                throw new InvalidArtifactException("Missing class " + missingClass + ". It may be caused by missing dependency jar(s) in the artifact jar.", t);
            }
            throw t;
        }
    } finally {
        try {
            DirUtils.deleteDirectoryContents(tempDir);
        } catch (IOException e) {
            LOG.warn("Exception raised when deleting directory {}", tempDir, e);
        }
    }
    ApplicationSpecification specification = configurer.createSpecification(applicationName, applicationVersion);
    AppSpecInfo appSpecInfo = new AppSpecInfo(specification, configurer.getSystemTables(), configurer.getMetadata());
    return new DefaultConfigResponse(0, appSpecInfo);
}
Also used : ApplicationSpecification(io.cdap.cdap.api.app.ApplicationSpecification) RuntimeConfigurer(io.cdap.cdap.api.app.RuntimeConfigurer) DefaultAppRuntimeConfigurer(io.cdap.cdap.app.DefaultAppRuntimeConfigurer) AppSpecInfo(io.cdap.cdap.internal.app.deploy.pipeline.AppSpecInfo) GsonBuilder(com.google.gson.GsonBuilder) DefaultAppConfigurer(io.cdap.cdap.app.DefaultAppConfigurer) Gson(com.google.gson.Gson) IOException(java.io.IOException) CombineClassLoader(io.cdap.cdap.common.lang.CombineClassLoader) Type(java.lang.reflect.Type) CaseInsensitiveEnumTypeAdapterFactory(io.cdap.cdap.common.io.CaseInsensitiveEnumTypeAdapterFactory) JsonSyntaxException(com.google.gson.JsonSyntaxException) DefaultAppRuntimeConfigurer(io.cdap.cdap.app.DefaultAppRuntimeConfigurer) CloseableClassLoader(io.cdap.cdap.api.artifact.CloseableClassLoader) CombineClassLoader(io.cdap.cdap.common.lang.CombineClassLoader) PluginInstantiator(io.cdap.cdap.internal.app.runtime.plugin.PluginInstantiator) File(java.io.File) InvalidArtifactException(io.cdap.cdap.common.InvalidArtifactException)

Example 2 with DefaultAppConfigurer

use of io.cdap.cdap.app.DefaultAppConfigurer in project cdap by caskdata.

the class Specifications method from.

public static ApplicationSpecification from(Application app) {
    DefaultAppConfigurer appConfigurer = new DefaultAppConfigurer(Id.Namespace.fromEntityId(DefaultId.NAMESPACE), Id.Artifact.fromEntityId(DefaultId.ARTIFACT), app);
    app.configure(appConfigurer, new DefaultApplicationContext());
    return appConfigurer.createSpecification(null);
}
Also used : DefaultApplicationContext(io.cdap.cdap.app.DefaultApplicationContext) DefaultAppConfigurer(io.cdap.cdap.app.DefaultAppConfigurer)

Example 3 with DefaultAppConfigurer

use of io.cdap.cdap.app.DefaultAppConfigurer in project cdap by caskdata.

the class DistributedWorkflowProgramRunnerTest method createWorkflowProgram.

/**
 * Creates a workflow {@link Program}.
 */
private Program createWorkflowProgram(CConfiguration cConf, ProgramRunner programRunner, String workflowName) throws IOException {
    Location appJarLocation = AppJarHelper.createDeploymentJar(new LocalLocationFactory(TEMP_FOLDER.newFolder()), DistributedWorkflowTestApp.class);
    ArtifactId artifactId = NamespaceId.DEFAULT.artifact("test", "1.0.0");
    DistributedWorkflowTestApp app = new DistributedWorkflowTestApp();
    DefaultAppConfigurer configurer = new DefaultAppConfigurer(Id.Namespace.DEFAULT, Id.Artifact.fromEntityId(artifactId), app);
    app.configure(configurer, new DefaultApplicationContext<>());
    ApplicationSpecification appSpec = configurer.createSpecification(null);
    ProgramId programId = NamespaceId.DEFAULT.app(appSpec.getName()).program(ProgramType.WORKFLOW, workflowName);
    return Programs.create(cConf, programRunner, new ProgramDescriptor(programId, appSpec), appJarLocation, TEMP_FOLDER.newFolder());
}
Also used : ApplicationSpecification(io.cdap.cdap.api.app.ApplicationSpecification) ArtifactId(io.cdap.cdap.proto.id.ArtifactId) DefaultAppConfigurer(io.cdap.cdap.app.DefaultAppConfigurer) ProgramDescriptor(io.cdap.cdap.app.program.ProgramDescriptor) ProgramId(io.cdap.cdap.proto.id.ProgramId) LocalLocationFactory(org.apache.twill.filesystem.LocalLocationFactory) Location(org.apache.twill.filesystem.Location)

Example 4 with DefaultAppConfigurer

use of io.cdap.cdap.app.DefaultAppConfigurer in project cdap by caskdata.

the class DistributedProgramRunnerTxTimeoutTest method setup.

@BeforeClass
public static void setup() {
    Application app = new AppWithAllProgramTypes();
    DefaultAppConfigurer configurer = new DefaultAppConfigurer(Id.Namespace.DEFAULT, new Id.Artifact(Id.Namespace.DEFAULT, "artifact", new ArtifactVersion("0.1")), app);
    app.configure(configurer, () -> null);
    appSpec = configurer.createSpecification("app", "1.0");
    // System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(appSpec));
    cConf.setInt(TxConstants.Manager.CFG_TX_MAX_TIMEOUT, 60);
    serviceRunner = new DistributedServiceProgramRunner(cConf, yConf, null, ClusterMode.ON_PREMISE, null);
    workerRunner = new DistributedWorkerProgramRunner(cConf, yConf, null, ClusterMode.ON_PREMISE, null);
    mapreduceRunner = new DistributedMapReduceProgramRunner(cConf, yConf, null, ClusterMode.ON_PREMISE, null);
    sparkRunner = new DistributedSparkProgramRunner(SparkCompat.SPARK2_2_11, cConf, yConf, null, null, ClusterMode.ON_PREMISE, null);
    workflowRunner = new DistributedWorkflowProgramRunner(cConf, yConf, null, ClusterMode.ON_PREMISE, null, null);
}
Also used : ArtifactVersion(io.cdap.cdap.api.artifact.ArtifactVersion) DefaultAppConfigurer(io.cdap.cdap.app.DefaultAppConfigurer) ProgramId(io.cdap.cdap.proto.id.ProgramId) Id(io.cdap.cdap.common.id.Id) AbstractApplication(io.cdap.cdap.api.app.AbstractApplication) Application(io.cdap.cdap.api.app.Application) DistributedSparkProgramRunner(io.cdap.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner) BeforeClass(org.junit.BeforeClass)

Aggregations

DefaultAppConfigurer (io.cdap.cdap.app.DefaultAppConfigurer)4 ApplicationSpecification (io.cdap.cdap.api.app.ApplicationSpecification)2 ProgramId (io.cdap.cdap.proto.id.ProgramId)2 Gson (com.google.gson.Gson)1 GsonBuilder (com.google.gson.GsonBuilder)1 JsonSyntaxException (com.google.gson.JsonSyntaxException)1 AbstractApplication (io.cdap.cdap.api.app.AbstractApplication)1 Application (io.cdap.cdap.api.app.Application)1 RuntimeConfigurer (io.cdap.cdap.api.app.RuntimeConfigurer)1 ArtifactVersion (io.cdap.cdap.api.artifact.ArtifactVersion)1 CloseableClassLoader (io.cdap.cdap.api.artifact.CloseableClassLoader)1 DefaultAppRuntimeConfigurer (io.cdap.cdap.app.DefaultAppRuntimeConfigurer)1 DefaultApplicationContext (io.cdap.cdap.app.DefaultApplicationContext)1 ProgramDescriptor (io.cdap.cdap.app.program.ProgramDescriptor)1 DistributedSparkProgramRunner (io.cdap.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner)1 InvalidArtifactException (io.cdap.cdap.common.InvalidArtifactException)1 Id (io.cdap.cdap.common.id.Id)1 CaseInsensitiveEnumTypeAdapterFactory (io.cdap.cdap.common.io.CaseInsensitiveEnumTypeAdapterFactory)1 CombineClassLoader (io.cdap.cdap.common.lang.CombineClassLoader)1 AppSpecInfo (io.cdap.cdap.internal.app.deploy.pipeline.AppSpecInfo)1