Examples with Runner org.apache.heron.streamlet.Runner used on opensource projects

Example 11 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class WindowedWordCountTopology method main.

public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    processingGraphBuilder.newSource(() -> StreamletUtils.randomFromList(SENTENCES)).setName("random-sentences-source").flatMap(sentence -> Arrays.asList(sentence.toLowerCase().split("\\s+"))).setName("flatten-into-individual-words").reduceByKeyAndWindow(// The key extractor (the word is left unchanged)
    word -> word, // Value extractor (the value is always 1)
    word -> 1, WindowConfig.TumblingCountWindow(50), StreamletReducers::sum).setName("reduce-operation").consume(kv -> {
        String logMessage = String.format("(word: %s, count: %d)", kv.getKey().getKey(), kv.getValue());
        LOG.info(logMessage);
    });
    // The topology's parallelism (the number of containers across which the topology's
    // processing instance will be split) can be defined via the second command-line
    // argument (or else the default of 2 will be used).
    int topologyParallelism = StreamletUtils.getParallelism(args, 2);
    Config config = Config.newBuilder().setNumContainers(topologyParallelism).build();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 12 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class ComponentConfigTopology method main.

public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    processingGraphBuilder.newSource(() -> StreamletUtils.randomFromList(SENTENCES)).setName("random-sentences-source").flatMap(sentence -> Arrays.asList(sentence.toLowerCase().split("\\s+"))).setName("flatten-into-individual-words").consume(w -> {
        String logMessage = String.format("(word: %s)", w);
        LOG.info(logMessage);
    }).setName("consumer");
    // The topology's parallelism (the number of containers across which the topology's
    // processing instance will be split) can be defined via the second command-line
    // argument (or else the default of 2 will be used).
    int topologyParallelism = StreamletUtils.getParallelism(args, 2);
    Config config = Config.newBuilder().setNumContainers(topologyParallelism).setPerContainerCpu(1).build();
    config.getHeronConfig().setComponentCpu("random-sentences-source", 0.3);
    config.getHeronConfig().setComponentRam("random-sentences-source", ByteAmount.fromMegabytes(300));
    config.getHeronConfig().setComponentCpu("flatten-into-individual-words", 0.3);
    config.getHeronConfig().setComponentRam("flatten-into-individual-words", ByteAmount.fromMegabytes(300));
    config.getHeronConfig().setComponentCpu("consumer", 0.2);
    config.getHeronConfig().setComponentRam("consumer", ByteAmount.fromMegabytes(200));
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}