Search in sources :

Example 11 with TopologyDag

use of com.hortonworks.streamline.streams.layout.component.TopologyDag in project streamline by hortonworks.

the class TestTopologyDagCreatingVisitorTest method visitSink_connectedFromProcessor.

@Test
public void visitSink_connectedFromProcessor() throws Exception {
    StreamlineProcessor originProcessor = TopologyTestHelper.createStreamlineProcessor("1");
    StreamlineSink originSink = TopologyTestHelper.createStreamlineSink("2");
    TopologyDag originTopologyDag = new TopologyDag();
    originTopologyDag.add(originProcessor);
    originTopologyDag.add(originSink);
    originTopologyDag.addEdge(new Edge("e1", originProcessor, originSink, "default", Stream.Grouping.SHUFFLE));
    TestRunProcessor testProcessor = createTestRunProcessor(originProcessor);
    TestRunSink testSink = createTestRunSink(originSink);
    TestTopologyDagCreatingVisitor visitor = new TestTopologyDagCreatingVisitor(originTopologyDag, Collections.emptyMap(), Collections.singletonMap(originProcessor.getName(), testProcessor), Collections.emptyMap(), Collections.singletonMap(originSink.getName(), testSink));
    visitor.visit(originProcessor);
    visitor.visit(originSink);
    TopologyDag testTopologyDag = visitor.getTestTopologyDag();
    List<OutputComponent> testProcessors = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunProcessor && o.getName().equals(originProcessor.getName()))).collect(toList());
    List<InputComponent> testSinks = testTopologyDag.getInputComponents().stream().filter(o -> (o instanceof TestRunSink && o.getName().equals(originSink.getName()))).collect(toList());
    assertEquals(1, testProcessors.size());
    assertEquals(1, testSinks.size());
    TestRunProcessor testRunProcessor = (TestRunProcessor) testProcessors.get(0);
    assertEquals(originProcessor.getId(), testRunProcessor.getId());
    TestRunSink testRunSink = (TestRunSink) testSinks.get(0);
    assertEquals(originSink.getId(), testRunSink.getId());
    assertEquals(testSink.getOutputFilePath(), testRunSink.getOutputFilePath());
    assertEquals(1, testTopologyDag.getEdgesFrom(testProcessor).size());
    assertEquals(1, testTopologyDag.getEdgesTo(testRunSink).size());
    assertTrue(testTopologyDag.getEdgesFrom(testRunProcessor).get(0) == testTopologyDag.getEdgesTo(testRunSink).get(0));
}
Also used : OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) Stream(com.hortonworks.streamline.streams.layout.component.Stream) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) HashMap(java.util.HashMap) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) TopologyTestHelper(com.hortonworks.streamline.streams.actions.utils.TopologyTestHelper) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Map(java.util.Map) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) Optional(java.util.Optional) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) Assert.fail(org.junit.Assert.fail) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) Test(org.junit.Test)

Example 12 with TopologyDag

use of com.hortonworks.streamline.streams.layout.component.TopologyDag in project streamline by hortonworks.

the class TestTopologyDagCreatingVisitorTest method visitRulesProcessor_connectedFromSource.

@Test
public void visitRulesProcessor_connectedFromSource() throws Exception {
    StreamlineSource originSource = TopologyTestHelper.createStreamlineSource("1");
    RulesProcessor rulesProcessor = TopologyTestHelper.createRulesProcessor("2");
    TopologyDag originTopologyDag = new TopologyDag();
    originTopologyDag.add(originSource);
    originTopologyDag.add(rulesProcessor);
    originTopologyDag.addEdge(new Edge("e1", originSource, rulesProcessor, "default", Stream.Grouping.SHUFFLE));
    TestRunSource testSource = createTestRunSource(originSource);
    TestRunRulesProcessor testRulesProcessor = createTestRunRulesProcessor(rulesProcessor);
    TestTopologyDagCreatingVisitor visitor = new TestTopologyDagCreatingVisitor(originTopologyDag, Collections.singletonMap(originSource.getName(), testSource), Collections.emptyMap(), Collections.singletonMap(rulesProcessor.getName(), testRulesProcessor), Collections.emptyMap());
    visitor.visit(originSource);
    visitor.visit(rulesProcessor);
    TopologyDag testTopologyDag = visitor.getTestTopologyDag();
    List<OutputComponent> testRulesProcessors = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunRulesProcessor && o.getName().equals(rulesProcessor.getName()))).collect(toList());
    List<OutputComponent> testSources = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunSource && o.getName().equals(originSource.getName()))).collect(toList());
    TestRunRulesProcessor testRunRulesProcessor = (TestRunRulesProcessor) testRulesProcessors.get(0);
    TestRunSource testRunSource = (TestRunSource) testSources.get(0);
    assertEquals(1, testTopologyDag.getEdgesFrom(testRunSource).size());
    assertEquals(1, testTopologyDag.getEdgesTo(testRunRulesProcessor).size());
    assertTrue(testTopologyDag.getEdgesFrom(testRunSource).get(0) == testTopologyDag.getEdgesTo(testRunRulesProcessor).get(0));
}
Also used : OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) Stream(com.hortonworks.streamline.streams.layout.component.Stream) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) HashMap(java.util.HashMap) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) TopologyTestHelper(com.hortonworks.streamline.streams.actions.utils.TopologyTestHelper) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Map(java.util.Map) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) Optional(java.util.Optional) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) Assert.fail(org.junit.Assert.fail) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) Test(org.junit.Test)

Example 13 with TopologyDag

use of com.hortonworks.streamline.streams.layout.component.TopologyDag in project streamline by hortonworks.

the class TestTopologyDagCreatingVisitorTest method visitSink_connectedFromSource.

@Test
public void visitSink_connectedFromSource() throws Exception {
    StreamlineSource originSource = TopologyTestHelper.createStreamlineSource("1");
    StreamlineSink originSink = TopologyTestHelper.createStreamlineSink("2");
    TopologyDag originTopologyDag = new TopologyDag();
    originTopologyDag.add(originSource);
    originTopologyDag.add(originSink);
    originTopologyDag.addEdge(new Edge("e1", originSource, originSink, "default", Stream.Grouping.SHUFFLE));
    TestRunSource testSource = createTestRunSource(originSource);
    TestRunSink testSink = createTestRunSink(originSink);
    TestTopologyDagCreatingVisitor visitor = new TestTopologyDagCreatingVisitor(originTopologyDag, Collections.singletonMap(originSource.getName(), testSource), Collections.emptyMap(), Collections.emptyMap(), Collections.singletonMap(originSink.getName(), testSink));
    visitor.visit(originSource);
    visitor.visit(originSink);
    TopologyDag testTopologyDag = visitor.getTestTopologyDag();
    List<OutputComponent> testSources = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunSource && o.getName().equals(originSource.getName()))).collect(toList());
    List<InputComponent> testSinks = testTopologyDag.getInputComponents().stream().filter(o -> (o instanceof TestRunSink && o.getName().equals(originSink.getName()))).collect(toList());
    assertEquals(1, testSinks.size());
    TestRunSource testRunSource = (TestRunSource) testSources.get(0);
    TestRunSink testRunSink = (TestRunSink) testSinks.get(0);
    assertEquals(originSink.getId(), testRunSink.getId());
    assertEquals(testSink.getOutputFilePath(), testRunSink.getOutputFilePath());
    assertEquals(1, testTopologyDag.getEdgesFrom(testRunSource).size());
    assertEquals(1, testTopologyDag.getEdgesTo(testRunSink).size());
    assertTrue(testTopologyDag.getEdgesFrom(testRunSource).get(0) == testTopologyDag.getEdgesTo(testRunSink).get(0));
}
Also used : OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) Stream(com.hortonworks.streamline.streams.layout.component.Stream) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) HashMap(java.util.HashMap) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) TopologyTestHelper(com.hortonworks.streamline.streams.actions.utils.TopologyTestHelper) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Map(java.util.Map) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) Optional(java.util.Optional) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) Assert.fail(org.junit.Assert.fail) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) Test(org.junit.Test)

Example 14 with TopologyDag

use of com.hortonworks.streamline.streams.layout.component.TopologyDag in project streamline by hortonworks.

the class TestTopologyDagCreatingVisitorTest method visitRulesProcessor_connectedFromProcessor.

@Test
public void visitRulesProcessor_connectedFromProcessor() throws Exception {
    StreamlineProcessor originProcessor = TopologyTestHelper.createStreamlineProcessor("1");
    RulesProcessor rulesProcessor = TopologyTestHelper.createRulesProcessor("2");
    TopologyDag originTopologyDag = new TopologyDag();
    originTopologyDag.add(originProcessor);
    originTopologyDag.add(rulesProcessor);
    originTopologyDag.addEdge(new Edge("e1", originProcessor, rulesProcessor, "default", Stream.Grouping.SHUFFLE));
    TestRunProcessor testProcessor = createTestRunProcessor(originProcessor);
    TestRunRulesProcessor testRulesProcessor = createTestRunRulesProcessor(rulesProcessor);
    TestTopologyDagCreatingVisitor visitor = new TestTopologyDagCreatingVisitor(originTopologyDag, Collections.emptyMap(), Collections.singletonMap(originProcessor.getName(), testProcessor), Collections.singletonMap(rulesProcessor.getName(), testRulesProcessor), Collections.emptyMap());
    visitor.visit(originProcessor);
    visitor.visit(rulesProcessor);
    TopologyDag testTopologyDag = visitor.getTestTopologyDag();
    List<OutputComponent> testProcessors = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunProcessor)).collect(toList());
    List<OutputComponent> testRulesProcessors = testTopologyDag.getOutputComponents().stream().filter(o -> (o instanceof TestRunRulesProcessor)).collect(toList());
    Optional<OutputComponent> testRunProcessorOptional = testProcessors.stream().filter(o -> o.getName().equals(originProcessor.getName())).findAny();
    Optional<OutputComponent> testRunRuleProcessorOptional = testRulesProcessors.stream().filter(o -> o.getName().equals(rulesProcessor.getName())).findAny();
    assertTrue(testRunProcessorOptional.isPresent());
    assertTrue(testRunRuleProcessorOptional.isPresent());
    TestRunProcessor testRunProcessor = (TestRunProcessor) testRunProcessorOptional.get();
    TestRunRulesProcessor testRunRuleProcessor = (TestRunRulesProcessor) testRunRuleProcessorOptional.get();
    assertEquals(1, testTopologyDag.getEdgesFrom(testRunProcessor).size());
    assertEquals(1, testTopologyDag.getEdgesTo(testRunRuleProcessor).size());
    assertTrue(testTopologyDag.getEdgesFrom(testRunProcessor).get(0) == testTopologyDag.getEdgesTo(testRunRuleProcessor).get(0));
}
Also used : OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) Stream(com.hortonworks.streamline.streams.layout.component.Stream) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) HashMap(java.util.HashMap) InputComponent(com.hortonworks.streamline.streams.layout.component.InputComponent) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) TestRunSink(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) TopologyTestHelper(com.hortonworks.streamline.streams.actions.utils.TopologyTestHelper) TestRunSource(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource) Map(java.util.Map) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) Optional(java.util.Optional) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) StreamlineSource(com.hortonworks.streamline.streams.layout.component.StreamlineSource) Assert.fail(org.junit.Assert.fail) StreamlineSink(com.hortonworks.streamline.streams.layout.component.StreamlineSink) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) StreamlineProcessor(com.hortonworks.streamline.streams.layout.component.StreamlineProcessor) RulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) TestRunProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor) OutputComponent(com.hortonworks.streamline.streams.layout.component.OutputComponent) TestRunRulesProcessor(com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor) Edge(com.hortonworks.streamline.streams.layout.component.Edge) TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag) Test(org.junit.Test)

Example 15 with TopologyDag

use of com.hortonworks.streamline.streams.layout.component.TopologyDag in project streamline by hortonworks.

the class TopologyDagBuilder method getDag.

public TopologyDag getDag(Topology topology) {
    try {
        TopologyDag dag = new TopologyDag();
        addSources(dag, topology);
        addProcessors(dag, topology);
        addSinks(dag, topology);
        addEdges(dag, topology);
        return dag;
    } catch (Exception ex) {
        throw new RuntimeException(ex);
    }
}
Also used : TopologyDag(com.hortonworks.streamline.streams.layout.component.TopologyDag)

Aggregations

TopologyDag (com.hortonworks.streamline.streams.layout.component.TopologyDag)15 StreamlineSink (com.hortonworks.streamline.streams.layout.component.StreamlineSink)9 StreamlineSource (com.hortonworks.streamline.streams.layout.component.StreamlineSource)9 Test (org.junit.Test)9 Edge (com.hortonworks.streamline.streams.layout.component.Edge)8 StreamlineProcessor (com.hortonworks.streamline.streams.layout.component.StreamlineProcessor)8 HashMap (java.util.HashMap)8 Map (java.util.Map)8 TopologyTestHelper (com.hortonworks.streamline.streams.actions.utils.TopologyTestHelper)7 InputComponent (com.hortonworks.streamline.streams.layout.component.InputComponent)7 OutputComponent (com.hortonworks.streamline.streams.layout.component.OutputComponent)7 Stream (com.hortonworks.streamline.streams.layout.component.Stream)7 RulesProcessor (com.hortonworks.streamline.streams.layout.component.impl.RulesProcessor)7 TestRunProcessor (com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunProcessor)7 TestRunRulesProcessor (com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunRulesProcessor)7 TestRunSink (com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSink)7 TestRunSource (com.hortonworks.streamline.streams.layout.component.impl.testing.TestRunSource)7 Collections (java.util.Collections)7 List (java.util.List)7 Optional (java.util.Optional)7