Search in sources :

Example 21 with JobExecutionResult

use of org.apache.gobblin.runtime.api.JobExecutionResult in project incubator-gobblin by apache.

the class MRCompactionTaskTest method testDedup.

@Test
public void testDedup() throws Exception {
    File basePath = Files.createTempDir();
    basePath.deleteOnExit();
    File jobDir = new File(basePath, "Identity/MemberAccount/minutely/2017/04/03/10/20_30/run_2017-04-03-10-20");
    Assert.assertTrue(jobDir.mkdirs());
    GenericRecord r1 = createRandomRecord();
    GenericRecord r2 = createRandomRecord();
    writeFileWithContent(jobDir, "file1", r1, 20);
    writeFileWithContent(jobDir, "file2", r2, 18);
    EmbeddedGobblin embeddedGobblin = createEmbeddedGobblin("dedup", basePath.getAbsolutePath().toString());
    JobExecutionResult result = embeddedGobblin.run();
    Assert.assertTrue(result.isSuccessful());
}
Also used : JobExecutionResult(org.apache.gobblin.runtime.api.JobExecutionResult) EmbeddedGobblin(org.apache.gobblin.runtime.embedded.EmbeddedGobblin) GenericRecord(org.apache.avro.generic.GenericRecord) File(java.io.File) Test(org.testng.annotations.Test)

Aggregations

JobExecutionResult (org.apache.gobblin.runtime.api.JobExecutionResult)21 Test (org.testng.annotations.Test)18 EmbeddedGobblin (org.apache.gobblin.runtime.embedded.EmbeddedGobblin)12 File (java.io.File)9 JobExecutionDriver (org.apache.gobblin.runtime.api.JobExecutionDriver)8 GenericRecord (org.apache.avro.generic.GenericRecord)7 JobSpec (org.apache.gobblin.runtime.api.JobSpec)5 TestingEventBusAsserter (org.apache.gobblin.writer.test.TestingEventBusAsserter)5 ArrayList (java.util.ArrayList)4 ArrayBlockingQueue (java.util.concurrent.ArrayBlockingQueue)4 JobLifecycleListener (org.apache.gobblin.runtime.api.JobLifecycleListener)4 ResolvedJobSpec (org.apache.gobblin.runtime.job_spec.ResolvedJobSpec)4 DefaultJobLifecycleListenerImpl (org.apache.gobblin.runtime.std.DefaultJobLifecycleListenerImpl)4 FilteredJobLifecycleListener (org.apache.gobblin.runtime.std.FilteredJobLifecycleListener)4 EventBus (com.google.common.eventbus.EventBus)3 StandardGobblinInstanceDriver (org.apache.gobblin.runtime.instance.StandardGobblinInstanceDriver)2 Path (org.apache.hadoop.fs.Path)2 Config (com.typesafe.config.Config)1 IOException (java.io.IOException)1 URI (java.net.URI)1