Search in sources :

Example 56 with LogicalPlan

use of com.datatorrent.stram.plan.logical.LogicalPlan in project apex-core by apache.

the class ModuleAppTest method validateTestApplication.

@Test
public void validateTestApplication() {
    Configuration conf = new Configuration(false);
    LogicalPlanConfiguration lpc = new LogicalPlanConfiguration(conf);
    LogicalPlan dag = new LogicalPlan();
    lpc.prepareDAG(dag, new Application(), "TestApp");
    Assert.assertEquals(2, dag.getAllModules().size(), 2);
    Assert.assertEquals(5, dag.getAllOperators().size());
    Assert.assertEquals(4, dag.getAllStreams().size());
    dag.validate();
}
Also used : LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) Configuration(org.apache.hadoop.conf.Configuration) LogicalPlan(com.datatorrent.stram.plan.logical.LogicalPlan) StreamingApplication(com.datatorrent.api.StreamingApplication) Test(org.junit.Test)

Example 57 with LogicalPlan

use of com.datatorrent.stram.plan.logical.LogicalPlan in project apex-core by apache.

the class TestModuleExpansion method testLoadFromPropertiesFile.

@Test
public void testLoadFromPropertiesFile() throws IOException {
    Properties props = new Properties();
    String resourcePath = "/testModuleTopology.properties";
    InputStream is = this.getClass().getResourceAsStream(resourcePath);
    if (is == null) {
        throw new RuntimeException("Could not load " + resourcePath);
    }
    props.load(is);
    LogicalPlanConfiguration pb = new LogicalPlanConfiguration(new Configuration(false)).addFromProperties(props, null);
    LogicalPlan dag = new LogicalPlan();
    pb.populateDAG(dag);
    pb.prepareDAG(dag, null, "testApplication");
    dag.validate();
    validateTopLevelOperators(dag);
    validateTopLevelStreams(dag);
    validatePublicMethods(dag);
}
Also used : LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) Configuration(org.apache.hadoop.conf.Configuration) InputStream(java.io.InputStream) LogicalPlan(com.datatorrent.stram.plan.logical.LogicalPlan) Properties(java.util.Properties) Test(org.junit.Test)

Example 58 with LogicalPlan

use of com.datatorrent.stram.plan.logical.LogicalPlan in project apex-core by apache.

the class TestModuleExpansion method testLoadFromJson.

@Test
public void testLoadFromJson() throws Exception {
    String resourcePath = "/testModuleTopology.json";
    InputStream is = this.getClass().getResourceAsStream(resourcePath);
    if (is == null) {
        throw new RuntimeException("Could not load " + resourcePath);
    }
    StringWriter writer = new StringWriter();
    IOUtils.copy(is, writer);
    JSONObject json = new JSONObject(writer.toString());
    Configuration conf = new Configuration(false);
    conf.set(StreamingApplication.APEX_PREFIX + "operator.operator3.prop.myStringProperty", "o3StringFromConf");
    LogicalPlanConfiguration planConf = new LogicalPlanConfiguration(conf);
    LogicalPlan dag = planConf.createFromJson(json, "testLoadFromJson");
    planConf.prepareDAG(dag, null, "testApplication");
    dag.validate();
    validateTopLevelOperators(dag);
    validateTopLevelStreams(dag);
    validatePublicMethods(dag);
}
Also used : LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) StringWriter(java.io.StringWriter) JSONObject(org.codehaus.jettison.json.JSONObject) LogicalPlanConfiguration(com.datatorrent.stram.plan.logical.LogicalPlanConfiguration) Configuration(org.apache.hadoop.conf.Configuration) InputStream(java.io.InputStream) LogicalPlan(com.datatorrent.stram.plan.logical.LogicalPlan) Test(org.junit.Test)

Example 59 with LogicalPlan

use of com.datatorrent.stram.plan.logical.LogicalPlan in project apex-core by apache.

the class PluginTests method testDispatch.

@Test
public void testDispatch() throws InterruptedException {
    DebugPlugin debugPlugin = new DebugPlugin();
    StaticPluginLocator<? extends DAGExecutionPlugin> locator = new StaticPluginLocator<>(debugPlugin);
    ApexPluginDispatcher pluginManager = new DefaultApexPluginDispatcher(locator, new StramTestSupport.TestAppContext(new Attribute.AttributeMap.DefaultAttributeMap()), null, null);
    pluginManager.init(new Configuration());
    pluginManager.dispatch(new DAGExecutionEvent.StramExecutionEvent(new StramEvent(StramEvent.LogLevel.DEBUG) {

        @Override
        public String getType() {
            return "TestEvent";
        }
    }));
    pluginManager.dispatch(new DAGExecutionEvent.CommitExecutionEvent(1234));
    pluginManager.dispatch(new DAGExecutionEvent.HeartbeatExecutionEvent(new StreamingContainerUmbilicalProtocol.ContainerHeartbeat()));
    LogicalPlan plan = new LogicalPlan();
    pluginManager.dispatch(new ApexPluginDispatcher.DAGChangeEvent(plan));
    debugPlugin.waitForEventDelivery(10);
    pluginManager.stop();
    Assert.assertEquals(1, debugPlugin.getEventCount());
    Assert.assertEquals(1, debugPlugin.getHeartbeatCount());
    Assert.assertEquals(1, debugPlugin.getCommitCount());
    Assert.assertEquals(plan, debugPlugin.getLogicalPlan());
}
Also used : Configuration(org.apache.hadoop.conf.Configuration) Attribute(com.datatorrent.api.Attribute) StramEvent(com.datatorrent.stram.api.StramEvent) StaticPluginLocator(org.apache.apex.engine.plugin.loaders.StaticPluginLocator) StramTestSupport(com.datatorrent.stram.support.StramTestSupport) LogicalPlan(com.datatorrent.stram.plan.logical.LogicalPlan) DAGExecutionEvent(org.apache.apex.engine.api.plugin.DAGExecutionEvent) Test(org.junit.Test)

Example 60 with LogicalPlan

use of com.datatorrent.stram.plan.logical.LogicalPlan in project apex-core by apache.

the class StreamingContainerManager method getCheckpointGroups.

protected Map<OperatorMeta, Set<OperatorMeta>> getCheckpointGroups() {
    if (this.checkpointGroups == null) {
        this.checkpointGroups = new HashMap<>();
        LogicalPlan dag = this.plan.getLogicalPlan();
        dag.resetNIndex();
        LogicalPlan.ValidationContext vc = new LogicalPlan.ValidationContext();
        for (OperatorMeta om : dag.getRootOperators()) {
            this.plan.getLogicalPlan().findStronglyConnected(om, vc);
        }
        for (Set<OperatorMeta> checkpointGroup : vc.stronglyConnected) {
            for (OperatorMeta om : checkpointGroup) {
                this.checkpointGroups.put(om, checkpointGroup);
            }
        }
    }
    return checkpointGroups;
}
Also used : OperatorMeta(com.datatorrent.stram.plan.logical.LogicalPlan.OperatorMeta) LogicalPlan(com.datatorrent.stram.plan.logical.LogicalPlan)

Aggregations

LogicalPlan (com.datatorrent.stram.plan.logical.LogicalPlan)99 Test (org.junit.Test)84 GenericTestOperator (com.datatorrent.stram.engine.GenericTestOperator)40 TestPlanContext (com.datatorrent.stram.plan.TestPlanContext)29 PartitioningTest (com.datatorrent.stram.PartitioningTest)27 File (java.io.File)23 OperatorMeta (com.datatorrent.stram.plan.logical.LogicalPlan.OperatorMeta)22 StramLocalCluster (com.datatorrent.stram.StramLocalCluster)19 Checkpoint (com.datatorrent.stram.api.Checkpoint)17 MemoryStorageAgent (com.datatorrent.stram.support.StramTestSupport.MemoryStorageAgent)16 AsyncFSStorageAgent (com.datatorrent.common.util.AsyncFSStorageAgent)15 StatsListener (com.datatorrent.api.StatsListener)13 StramTestSupport (com.datatorrent.stram.support.StramTestSupport)13 Configuration (org.apache.hadoop.conf.Configuration)13 LogicalPlanConfiguration (com.datatorrent.stram.plan.logical.LogicalPlanConfiguration)11 NodeReport (org.apache.hadoop.yarn.api.records.NodeReport)10 ContainerStartRequest (com.datatorrent.stram.StreamingContainerAgent.ContainerStartRequest)9 PTOperator (com.datatorrent.stram.plan.physical.PTOperator)9 ArrayList (java.util.ArrayList)9 ConstraintViolationException (javax.validation.ConstraintViolationException)9