Search in sources :

Example 1 with TopologyBuilder

use of org.apache.heron.api.topology.TopologyBuilder in project heron by twitter.

the class AbstractTestTopology method submit.

public final void submit(Config userConf) throws AlreadyAliveException, InvalidTopologyException {
    Config conf = buildConfig(new BasicConfig());
    if (userConf != null) {
        conf.putAll(userConf);
    }
    if (this.httpServerResultsUrl == null) {
        TopologyBuilder builder = new TopologyBuilder();
        HeronSubmitter.submitTopology(topologyName, conf, buildTopology(builder).createTopology());
    } else {
        TopologyTestTopologyBuilder builder = new TopologyTestTopologyBuilder(httpServerResultsUrl);
        conf.setTopologyReliabilityMode(Config.TopologyReliabilityMode.EFFECTIVELY_ONCE);
        conf.setTopologyStatefulCheckpointIntervalSecs(CHECKPOINT_INTERVAL);
        HeronSubmitter.submitTopology(topologyName, conf, buildStatefulTopology(builder).createTopology());
    }
}
Also used : TopologyTestTopologyBuilder(org.apache.heron.integration_topology_test.core.TopologyTestTopologyBuilder) TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) Config(org.apache.heron.api.Config) TopologyTestTopologyBuilder(org.apache.heron.integration_topology_test.core.TopologyTestTopologyBuilder)

Example 2 with TopologyBuilder

use of org.apache.heron.api.topology.TopologyBuilder in project heron by twitter.

the class StreamletImplTest method testStreamletNameIfDuplicateNameIsSet.

@Test(expected = RuntimeException.class)
public void testStreamletNameIfDuplicateNameIsSet() {
    // create SupplierStreamlet
    Streamlet<String> baseStreamlet = builder.newSource(() -> "This is test content");
    SupplierStreamlet<String> supplierStreamlet = (SupplierStreamlet<String>) baseStreamlet;
    // set duplicate streamlet name and expect thrown exception
    supplierStreamlet.map((content) -> content.toUpperCase()).setName("MyMapStreamlet").map((content) -> content + "_test_suffix").setName("MyMapStreamlet");
    // build SupplierStreamlet
    assertFalse(supplierStreamlet.isBuilt());
    TopologyBuilder topologyBuilder = new TopologyBuilder();
    Set<String> stageNames = new HashSet<>();
    supplierStreamlet.build(topologyBuilder, stageNames);
}
Also used : Arrays(java.util.Arrays) SpoutStreamlet(org.apache.heron.streamlet.impl.streamlets.SpoutStreamlet) WindowConfig(org.apache.heron.streamlet.WindowConfig) ShuffleStreamGrouping(org.apache.heron.api.grouping.ShuffleStreamGrouping) Builder(org.apache.heron.streamlet.Builder) CountByKeyStreamlet(org.apache.heron.streamlet.impl.streamlets.CountByKeyStreamlet) JoinStreamlet(org.apache.heron.streamlet.impl.streamlets.JoinStreamlet) Map(java.util.Map) Utils(org.apache.heron.api.utils.Utils) Assert.fail(org.junit.Assert.fail) StreamletReducers(org.apache.heron.streamlet.StreamletReducers) TestSpout(org.apache.heron.resource.TestSpout) SupplierStreamlet(org.apache.heron.streamlet.impl.streamlets.SupplierStreamlet) TestBasicBolt(org.apache.heron.resource.TestBasicBolt) ReduceByKeyAndWindowStreamlet(org.apache.heron.streamlet.impl.streamlets.ReduceByKeyAndWindowStreamlet) IStreamletWindowOperator(org.apache.heron.streamlet.IStreamletWindowOperator) KeyByStreamlet(org.apache.heron.streamlet.impl.streamlets.KeyByStreamlet) SourceStreamlet(org.apache.heron.streamlet.impl.streamlets.SourceStreamlet) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) Collection(java.util.Collection) Set(java.util.Set) Context(org.apache.heron.streamlet.Context) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) SerializableTransformer(org.apache.heron.streamlet.SerializableTransformer) TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) KVStreamlet(org.apache.heron.streamlet.KVStreamlet) SerializablePredicate(org.apache.heron.streamlet.SerializablePredicate) CountByKeyAndWindowStreamlet(org.apache.heron.streamlet.impl.streamlets.CountByKeyAndWindowStreamlet) HashMap(java.util.HashMap) Function(java.util.function.Function) ConsumerStreamlet(org.apache.heron.streamlet.impl.streamlets.ConsumerStreamlet) HashSet(java.util.HashSet) UnionStreamlet(org.apache.heron.streamlet.impl.streamlets.UnionStreamlet) KVStreamletShadow(org.apache.heron.streamlet.impl.streamlets.KVStreamletShadow) FlatMapStreamlet(org.apache.heron.streamlet.impl.streamlets.FlatMapStreamlet) ReduceByKeyStreamlet(org.apache.heron.streamlet.impl.streamlets.ReduceByKeyStreamlet) MapStreamlet(org.apache.heron.streamlet.impl.streamlets.MapStreamlet) ByteAmount(org.apache.heron.common.basics.ByteAmount) Source(org.apache.heron.streamlet.Source) TransformStreamlet(org.apache.heron.streamlet.impl.streamlets.TransformStreamlet) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) TestBolt(org.apache.heron.resource.TestBolt) Config(org.apache.heron.streamlet.Config) Streamlet(org.apache.heron.streamlet.Streamlet) Consumer(java.util.function.Consumer) IStreamletRichOperator(org.apache.heron.streamlet.IStreamletRichOperator) CustomStreamlet(org.apache.heron.streamlet.impl.streamlets.CustomStreamlet) IStreamletBasicOperator(org.apache.heron.streamlet.IStreamletBasicOperator) TestWindowBolt(org.apache.heron.resource.TestWindowBolt) SerializableConsumer(org.apache.heron.streamlet.SerializableConsumer) GeneralReduceByKeyStreamlet(org.apache.heron.streamlet.impl.streamlets.GeneralReduceByKeyStreamlet) FilterStreamlet(org.apache.heron.streamlet.impl.streamlets.FilterStreamlet) Assert.assertEquals(org.junit.Assert.assertEquals) TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) SupplierStreamlet(org.apache.heron.streamlet.impl.streamlets.SupplierStreamlet) HashSet(java.util.HashSet) Test(org.junit.Test)

Example 3 with TopologyBuilder

use of org.apache.heron.api.topology.TopologyBuilder in project heron by twitter.

the class HeronSubmitterTest method createTopologyBuilderWithMinimumSetup.

private TopologyBuilder createTopologyBuilderWithMinimumSetup() {
    TopologyBuilder builder = new TopologyBuilder();
    int spouts = 2;
    int bolts = 2;
    builder.setSpout("word", new TestSpout2(), spouts);
    builder.setBolt("exclaim1", new TestBolt(), bolts).shuffleGrouping("word");
    return builder;
}
Also used : TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder)

Example 4 with TopologyBuilder

use of org.apache.heron.api.topology.TopologyBuilder in project heron by twitter.

the class HeronSubmitterTest method testTopologySubmissionWhenTmpDirectoryIsSetAsInvalidPath.

@Test(expected = TopologySubmissionException.class)
@PrepareForTest(HeronSubmitter.class)
public void testTopologySubmissionWhenTmpDirectoryIsSetAsInvalidPath() throws AlreadyAliveException, InvalidTopologyException {
    TopologyBuilder builder = createTopologyBuilderWithMinimumSetup();
    Config conf = new Config();
    Map<String, String> map = new HashMap();
    map.put("cmdline.topologydefn.tmpdirectory", "invalid_path");
    PowerMockito.spy(HeronSubmitter.class);
    Mockito.when(HeronSubmitter.getHeronCmdOptions()).thenReturn(map);
    HeronSubmitter.submitTopology("test", conf, builder.createTopology());
}
Also used : TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) HashMap(java.util.HashMap) Test(org.junit.Test) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest)

Example 5 with TopologyBuilder

use of org.apache.heron.api.topology.TopologyBuilder in project heron by twitter.

the class HeronSubmitterTest method testTopologySubmissionWhenTmpDirectoryIsNotSet.

@Test(expected = TopologySubmissionException.class)
public void testTopologySubmissionWhenTmpDirectoryIsNotSet() throws AlreadyAliveException, InvalidTopologyException {
    TopologyBuilder builder = createTopologyBuilderWithMinimumSetup();
    Config conf = new Config();
    HeronSubmitter.submitTopology("test", conf, builder.createTopology());
}
Also used : TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) Test(org.junit.Test) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest)

Aggregations

TopologyBuilder (org.apache.heron.api.topology.TopologyBuilder)41 Config (org.apache.heron.api.Config)19 Test (org.junit.Test)15 HashMap (java.util.HashMap)8 Fields (org.apache.heron.api.tuple.Fields)6 Simulator (org.apache.heron.simulator.Simulator)6 TestWordSpout (org.apache.heron.examples.api.spout.TestWordSpout)5 PrepareForTest (org.powermock.core.classloader.annotations.PrepareForTest)5 Duration (java.time.Duration)4 HashSet (java.util.HashSet)4 Map (java.util.Map)4 Arrays (java.util.Arrays)3 Collection (java.util.Collection)3 List (java.util.List)3 Set (java.util.Set)3 Consumer (java.util.function.Consumer)3 Function (java.util.function.Function)3 HeronTopology (org.apache.heron.api.HeronTopology)3 ShuffleStreamGrouping (org.apache.heron.api.grouping.ShuffleStreamGrouping)3 Utils (org.apache.heron.api.utils.Utils)3