Search in sources :

Example 6 with SpecExecutor

use of org.apache.gobblin.runtime.api.SpecExecutor in project incubator-gobblin by apache.

the class IdentityFlowToJobSpecCompilerTest method testCompilerWithTemplateCatalog.

@Test
public void testCompilerWithTemplateCatalog() {
    FlowSpec flowSpec = initFlowSpec();
    // Run compiler on flowSpec
    Map<Spec, SpecExecutor> specExecutorMapping = this.compilerWithTemplateCalague.compileFlow(flowSpec);
    // Assert pre-requisites
    Assert.assertNotNull(specExecutorMapping, "Expected non null mapping.");
    Assert.assertTrue(specExecutorMapping.size() == 1, "Exepected 1 executor for FlowSpec.");
    // Assert FlowSpec compilation
    Spec spec = specExecutorMapping.keySet().iterator().next();
    Assert.assertTrue(spec instanceof JobSpec, "Expected JobSpec compiled from FlowSpec.");
    // Assert JobSpec properties
    JobSpec jobSpec = (JobSpec) spec;
    Assert.assertEquals(jobSpec.getConfig().getString("testProperty1"), "testValue1");
    Assert.assertEquals(jobSpec.getConfig().getString("testProperty2"), "test.Value1");
    Assert.assertEquals(jobSpec.getConfig().getString("testProperty3"), "100");
    Assert.assertEquals(jobSpec.getConfig().getString(ServiceConfigKeys.FLOW_SOURCE_IDENTIFIER_KEY), TEST_SOURCE_NAME);
    Assert.assertFalse(jobSpec.getConfig().hasPath(ConfigurationKeys.JOB_SCHEDULE_KEY));
    Assert.assertEquals(jobSpec.getConfig().getString(ConfigurationKeys.JOB_NAME_KEY), TEST_FLOW_NAME);
    Assert.assertEquals(jobSpec.getConfig().getString(ConfigurationKeys.JOB_GROUP_KEY), TEST_FLOW_GROUP);
    Assert.assertEquals(jobSpec.getConfig().getString(ConfigurationKeys.FLOW_NAME_KEY), TEST_FLOW_NAME);
    Assert.assertEquals(jobSpec.getConfig().getString(ConfigurationKeys.FLOW_GROUP_KEY), TEST_FLOW_GROUP);
    Assert.assertTrue(jobSpec.getConfig().hasPath(ConfigurationKeys.FLOW_EXECUTION_ID_KEY));
}
Also used : FlowSpec(org.apache.gobblin.runtime.api.FlowSpec) SpecExecutor(org.apache.gobblin.runtime.api.SpecExecutor) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) JobSpec(org.apache.gobblin.runtime.api.JobSpec) TopologySpec(org.apache.gobblin.runtime.api.TopologySpec) JobSpec(org.apache.gobblin.runtime.api.JobSpec) Spec(org.apache.gobblin.runtime.api.Spec) FlowSpec(org.apache.gobblin.runtime.api.FlowSpec) Test(org.testng.annotations.Test)

Example 7 with SpecExecutor

use of org.apache.gobblin.runtime.api.SpecExecutor in project incubator-gobblin by apache.

the class IdentityFlowToJobSpecCompilerTest method initTopologySpec.

private TopologySpec initTopologySpec() {
    Properties properties = new Properties();
    properties.put("specStore.fs.dir", TOPOLOGY_SPEC_STORE_DIR);
    properties.put("specExecInstance.capabilities", TEST_SOURCE_NAME + ":" + TEST_SINK_NAME);
    Config config = ConfigUtils.propertiesToConfig(properties);
    SpecExecutor specExecutorInstance = new InMemorySpecExecutor(config);
    TopologySpec.Builder topologySpecBuilder = TopologySpec.builder(computeTopologySpecURI(SPEC_STORE_PARENT_DIR, TOPOLOGY_SPEC_STORE_DIR)).withConfig(config).withDescription(SPEC_DESCRIPTION).withVersion(SPEC_VERSION).withSpecExecutor(specExecutorInstance);
    return topologySpecBuilder.build();
}
Also used : Config(com.typesafe.config.Config) SpecExecutor(org.apache.gobblin.runtime.api.SpecExecutor) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) Properties(java.util.Properties) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) TopologySpec(org.apache.gobblin.runtime.api.TopologySpec)

Example 8 with SpecExecutor

use of org.apache.gobblin.runtime.api.SpecExecutor in project incubator-gobblin by apache.

the class MultiHopsFlowToJobSpecCompilerTest method initTopologySpec.

// The topology is: Src - A - B - Dest
private TopologySpec initTopologySpec(String storeDir, String... args) {
    Properties properties = new Properties();
    properties.put("specStore.fs.dir", storeDir);
    String capabilitiesString = "";
    for (int i = 0; i < args.length - 1; i++) {
        capabilitiesString = capabilitiesString + (args[i] + ":" + args[i + 1] + ",");
    }
    Assert.assertEquals(capabilitiesString.charAt(capabilitiesString.length() - 1), ',');
    capabilitiesString = capabilitiesString.substring(0, capabilitiesString.length() - 1);
    properties.put("specExecInstance.capabilities", capabilitiesString);
    properties.put("executorAttrs", new Properties());
    Config config = ConfigUtils.propertiesToConfig(properties);
    SpecExecutor specExecutorInstance = new InMemorySpecExecutor(config);
    TopologySpec.Builder topologySpecBuilder = TopologySpec.builder(IdentityFlowToJobSpecCompilerTest.computeTopologySpecURI(SPEC_STORE_PARENT_DIR, storeDir)).withConfig(config).withDescription(SPEC_DESCRIPTION).withVersion(SPEC_VERSION).withSpecExecutor(specExecutorInstance);
    return topologySpecBuilder.build();
}
Also used : Config(com.typesafe.config.Config) SpecExecutor(org.apache.gobblin.runtime.api.SpecExecutor) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) Properties(java.util.Properties) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) TopologySpec(org.apache.gobblin.runtime.api.TopologySpec)

Example 9 with SpecExecutor

use of org.apache.gobblin.runtime.api.SpecExecutor in project incubator-gobblin by apache.

the class OrchestratorTest method deleteFlowSpec.

@Test(dependsOnMethods = "createFlowSpec")
public void deleteFlowSpec() throws Exception {
    // Since only 1 Flow has been added in previous test it should be available
    IdentityFlowToJobSpecCompiler specCompiler = (IdentityFlowToJobSpecCompiler) this.orchestrator.getSpecCompiler();
    SpecExecutor sei = specCompiler.getTopologySpecMap().values().iterator().next().getSpecExecutor();
    // List Current Specs
    Collection<Spec> specs = flowCatalog.getSpecs();
    logger.info("[Before Delete] Number of specs: " + specs.size());
    int i = 0;
    for (Spec spec : specs) {
        FlowSpec flowSpec = (FlowSpec) spec;
        logger.info("[Before Delete] Spec " + i++ + ": " + gson.toJson(flowSpec));
    }
    // Make sure FlowCatalog has the previously added Flow
    Assert.assertTrue(specs.size() == 1, "Spec store should contain 1 Flow that was added in last test");
    // Orchestrator is a no-op listener for any new FlowSpecs, so no FlowSpecs should be around
    int specsInSEI = ((List) (sei.getProducer().get().listSpecs().get())).size();
    Assert.assertTrue(specsInSEI == 0, "SpecProducer should contain 0 " + "Spec after addition because Orchestrator is a no-op listener for any new FlowSpecs");
    // Remove the flow
    this.flowCatalog.remove(flowSpec.getUri());
    // List Specs after adding
    specs = flowCatalog.getSpecs();
    logger.info("[After Delete] Number of specs: " + specs.size());
    i = 0;
    for (Spec spec : specs) {
        flowSpec = (FlowSpec) spec;
        logger.info("[After Delete] Spec " + i++ + ": " + gson.toJson(flowSpec));
    }
    // Make sure FlowCatalog has the Flow removed
    Assert.assertTrue(specs.size() == 0, "Spec store should not contain Spec after deletion");
    // Make sure FlowCatalog Listener knows about the deletion
    specsInSEI = ((List) (sei.getProducer().get().listSpecs().get())).size();
    Assert.assertTrue(specsInSEI == 0, "SpecProducer should not contain " + "Spec after deletion");
}
Also used : FlowSpec(org.apache.gobblin.runtime.api.FlowSpec) SpecExecutor(org.apache.gobblin.runtime.api.SpecExecutor) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) List(java.util.List) IdentityFlowToJobSpecCompiler(org.apache.gobblin.service.modules.flow.IdentityFlowToJobSpecCompiler) TopologySpec(org.apache.gobblin.runtime.api.TopologySpec) Spec(org.apache.gobblin.runtime.api.Spec) FlowSpec(org.apache.gobblin.runtime.api.FlowSpec) Test(org.testng.annotations.Test)

Example 10 with SpecExecutor

use of org.apache.gobblin.runtime.api.SpecExecutor in project incubator-gobblin by apache.

the class OrchestratorTest method initTopologySpec.

private TopologySpec initTopologySpec() {
    Properties properties = new Properties();
    properties.put("specStore.fs.dir", TOPOLOGY_SPEC_STORE_DIR);
    properties.put("specExecInstance.capabilities", "source:destination");
    Config config = ConfigUtils.propertiesToConfig(properties);
    SpecExecutor specExecutorInstance = new InMemorySpecExecutor(config);
    TopologySpec.Builder topologySpecBuilder = TopologySpec.builder(computeTopologySpecURI(SPEC_STORE_PARENT_DIR, TOPOLOGY_SPEC_STORE_DIR)).withConfig(config).withDescription(SPEC_DESCRIPTION).withVersion(SPEC_VERSION).withSpecExecutor(specExecutorInstance);
    return topologySpecBuilder.build();
}
Also used : Config(com.typesafe.config.Config) SpecExecutor(org.apache.gobblin.runtime.api.SpecExecutor) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) Properties(java.util.Properties) InMemorySpecExecutor(org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor) TopologySpec(org.apache.gobblin.runtime.api.TopologySpec)

Aggregations

SpecExecutor (org.apache.gobblin.runtime.api.SpecExecutor)15 TopologySpec (org.apache.gobblin.runtime.api.TopologySpec)14 InMemorySpecExecutor (org.apache.gobblin.runtime.spec_executorInstance.InMemorySpecExecutor)11 FlowSpec (org.apache.gobblin.runtime.api.FlowSpec)10 Spec (org.apache.gobblin.runtime.api.Spec)9 Config (com.typesafe.config.Config)6 Properties (java.util.Properties)5 JobSpec (org.apache.gobblin.runtime.api.JobSpec)5 Test (org.testng.annotations.Test)5 InvocationTargetException (java.lang.reflect.InvocationTargetException)3 Map (java.util.Map)3 List (java.util.List)2 ServiceNode (org.apache.gobblin.runtime.api.ServiceNode)2 SpecProducer (org.apache.gobblin.runtime.api.SpecProducer)2 IdentityFlowToJobSpecCompiler (org.apache.gobblin.service.modules.flow.IdentityFlowToJobSpecCompiler)2 URI (java.net.URI)1 URISyntaxException (java.net.URISyntaxException)1 HashMap (java.util.HashMap)1 ExecutionException (java.util.concurrent.ExecutionException)1 ResolvedJobSpec (org.apache.gobblin.runtime.job_spec.ResolvedJobSpec)1