Search in sources :

Example 1 with LocalJobLauncher

use of org.apache.gobblin.runtime.local.LocalJobLauncher in project incubator-gobblin by apache.

the class TestJobLauncherExecutionDriver method testConstructor.

@Test
public void testConstructor() throws IOException, InterruptedException {
    File tmpTestDir = Files.createTempDir();
    try {
        File localJobRootDir = new File(tmpTestDir, "localJobRoot");
        Assert.assertTrue(localJobRootDir.mkdir());
        final Logger log = LoggerFactory.getLogger(getClass().getSimpleName() + ".testConstructor");
        Config jobConf1 = ConfigFactory.empty().withValue(ConfigurationKeys.JOB_NAME_KEY, ConfigValueFactory.fromAnyRef("myJob")).withValue(ConfigurationKeys.JOB_GROUP_KEY, ConfigValueFactory.fromAnyRef("myGroup")).withValue(ConfigurationKeys.JOB_DESCRIPTION_KEY, ConfigValueFactory.fromAnyRef("Awesome job")).withValue(ConfigurationKeys.STATE_STORE_ROOT_DIR_KEY, ConfigValueFactory.fromAnyRef(localJobRootDir.getPath())).withValue(ConfigurationKeys.SOURCE_CLASS_KEY, ConfigValueFactory.fromAnyRef(TestingSource.class.getName())).withValue(ConfigurationKeys.JOB_LOCK_ENABLED_KEY, ConfigValueFactory.fromAnyRef(false));
        JobSpec jobSpec1 = JobSpec.builder().withConfig(jobConf1).build();
        JobLauncherExecutionDriver.Launcher launcher = new JobLauncherExecutionDriver.Launcher().withJobLauncherType(JobLauncherFactory.JobLauncherType.LOCAL).withLog(log);
        JobLauncherExecutionDriver jled = (JobLauncherExecutionDriver) launcher.launchJob(jobSpec1);
        Assert.assertTrue(jled.getLegacyLauncher() instanceof LocalJobLauncher);
        JobExecution jex1 = jled.getJobExecution();
        Assert.assertEquals(jex1.getJobSpecURI(), jobSpec1.getUri());
        Assert.assertEquals(jex1.getJobSpecVersion(), jobSpec1.getVersion());
        Thread.sleep(5000);
        File mrJobRootDir = new File(tmpTestDir, "mrJobRoot");
        Assert.assertTrue(mrJobRootDir.mkdir());
        Config jobConf2 = ConfigFactory.empty().withValue(ConfigurationKeys.JOB_NAME_KEY, ConfigValueFactory.fromAnyRef("myJob2")).withValue(ConfigurationKeys.JOB_GROUP_KEY, ConfigValueFactory.fromAnyRef("myGroup")).withValue(ConfigurationKeys.JOB_DESCRIPTION_KEY, ConfigValueFactory.fromAnyRef("Awesome job")).withValue(ConfigurationKeys.STATE_STORE_ROOT_DIR_KEY, ConfigValueFactory.fromAnyRef(mrJobRootDir.getPath())).withValue(ConfigurationKeys.MR_JOB_ROOT_DIR_KEY, ConfigValueFactory.fromAnyRef(mrJobRootDir.getPath())).withValue(ConfigurationKeys.SOURCE_CLASS_KEY, ConfigValueFactory.fromAnyRef(TestingSource.class.getName())).withValue(ConfigurationKeys.JOB_LOCK_ENABLED_KEY, ConfigValueFactory.fromAnyRef(false));
        JobSpec jobSpec2 = JobSpec.builder().withConfig(jobConf2).build();
        jled = (JobLauncherExecutionDriver) launcher.withJobLauncherType(JobLauncherFactory.JobLauncherType.MAPREDUCE).launchJob(jobSpec2);
        Assert.assertTrue(jled.getLegacyLauncher() instanceof MRJobLauncher);
        JobExecution jex2 = jled.getJobExecution();
        Assert.assertEquals(jex2.getJobSpecURI(), jobSpec2.getUri());
        Assert.assertEquals(jex2.getJobSpecVersion(), jobSpec2.getVersion());
        Assert.assertTrue(jex2.getLaunchTimeMillis() >= jex1.getLaunchTimeMillis());
    } finally {
        FileUtils.deleteDirectory(tmpTestDir);
    }
}
Also used : JobExecution(org.apache.gobblin.runtime.api.JobExecution) Config(com.typesafe.config.Config) MRJobLauncher(org.apache.gobblin.runtime.mapreduce.MRJobLauncher) LocalJobLauncher(org.apache.gobblin.runtime.local.LocalJobLauncher) MRJobLauncher(org.apache.gobblin.runtime.mapreduce.MRJobLauncher) TestingSource(org.apache.gobblin.util.test.TestingSource) JobSpec(org.apache.gobblin.runtime.api.JobSpec) Logger(org.slf4j.Logger) File(java.io.File) LocalJobLauncher(org.apache.gobblin.runtime.local.LocalJobLauncher) Test(org.testng.annotations.Test)

Example 2 with LocalJobLauncher

use of org.apache.gobblin.runtime.local.LocalJobLauncher in project incubator-gobblin by apache.

the class GobblinLocalJobLauncherUtils method invokeLocalJobLauncher.

public static void invokeLocalJobLauncher(Properties properties) throws Exception {
    try (ApplicationLauncher applicationLauncher = new ServiceBasedAppLauncher(properties, properties.getProperty(ServiceBasedAppLauncher.APP_NAME, "CliLocalJob-" + UUID.randomUUID()))) {
        applicationLauncher.start();
        try (LocalJobLauncher localJobLauncher = new LocalJobLauncher(properties)) {
            localJobLauncher.launchJob(null);
        }
        applicationLauncher.stop();
    }
}
Also used : ServiceBasedAppLauncher(org.apache.gobblin.runtime.app.ServiceBasedAppLauncher) ApplicationLauncher(org.apache.gobblin.runtime.app.ApplicationLauncher) LocalJobLauncher(org.apache.gobblin.runtime.local.LocalJobLauncher)

Aggregations

LocalJobLauncher (org.apache.gobblin.runtime.local.LocalJobLauncher)2 Config (com.typesafe.config.Config)1 File (java.io.File)1 JobExecution (org.apache.gobblin.runtime.api.JobExecution)1 JobSpec (org.apache.gobblin.runtime.api.JobSpec)1 ApplicationLauncher (org.apache.gobblin.runtime.app.ApplicationLauncher)1 ServiceBasedAppLauncher (org.apache.gobblin.runtime.app.ServiceBasedAppLauncher)1 MRJobLauncher (org.apache.gobblin.runtime.mapreduce.MRJobLauncher)1 TestingSource (org.apache.gobblin.util.test.TestingSource)1 Logger (org.slf4j.Logger)1 Test (org.testng.annotations.Test)1