Examples with Runner org.apache.heron.streamlet.Runner used on opensource projects

Example 1 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class SmartWatchTopology method main.

public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    processingGraphBuilder.newSource(SmartWatchReading::new).setName("incoming-watch-readings").reduceByKeyAndWindow(// Key extractor
    reading -> reading.getJoggerId(), // Value extractor
    reading -> reading.getFeetRun(), // The time window (1 minute of clock time)
    WindowConfig.TumblingTimeWindow(Duration.ofSeconds(10)), // The reduce function (produces a cumulative sum)
    (cumulative, incoming) -> cumulative + incoming).setName("reduce-to-total-distance-per-jogger").map(keyWindow -> {
        // The per-key result of the previous reduce step
        long totalFeetRun = keyWindow.getValue();
        // The amount of time elapsed
        long startTime = keyWindow.getKey().getWindow().getStartTime();
        long endTime = keyWindow.getKey().getWindow().getEndTime();
        // Cast to float to use as denominator
        long timeLengthMillis = endTime - startTime;
        // The feet-per-minute calculation
        float feetPerMinute = totalFeetRun / (float) (timeLengthMillis / 1000);
        // Reduce to two decimal places
        String paceString = new DecimalFormat("#.##").format(feetPerMinute);
        // Return a per-jogger average pace
        return new KeyValue<>(keyWindow.getKey().getKey(), paceString);
    }).setName("calculate-average-speed").consume(kv -> {
        String logMessage = String.format("(runner: %s, avgFeetPerMinute: %s)", kv.getKey(), kv.getValue());
        LOG.info(logMessage);
    });
    Config config = Config.defaultConfig();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 2 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class TransformsTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder builder = Builder.newBuilder();
    /**
     * The processing graph consists of a supplier streamlet that emits
     * random integers between 1 and 100. From there, a series of transformers
     * is applied. At the end of the graph, the original value is ultimately
     * unchanged.
     */
    builder.newSource(() -> ThreadLocalRandom.current().nextInt(100)).transform(new DoNothingTransformer<>()).transform(new IncrementTransformer(10)).transform(new IncrementTransformer(-7)).transform(new DoNothingTransformer<>()).transform(new IncrementTransformer(-3)).log();
    Config config = Config.defaultConfig();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, builder);
}

Example 3 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class WireRequestsTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder builder = Builder.newBuilder();
    // Requests from the "quiet" bank branch (high throttling).
    Streamlet<WireRequest> quietBranch = builder.newSource(() -> new WireRequest(20)).setNumPartitions(1).setName("quiet-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("quiet-branch-check-balance");
    // Requests from the "medium" bank branch (medium throttling).
    Streamlet<WireRequest> mediumBranch = builder.newSource(() -> new WireRequest(10)).setNumPartitions(2).setName("medium-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("medium-branch-check-balance");
    // Requests from the "busy" bank branch (low throttling).
    Streamlet<WireRequest> busyBranch = builder.newSource(() -> new WireRequest(5)).setNumPartitions(4).setName("busy-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("busy-branch-check-balance");
    // Here, the streamlets for the three bank branches are united into one. The fraud
    // detection filter then operates on that unified streamlet.
    quietBranch.union(mediumBranch).setNumPartitions(2).setName("union-1").union(busyBranch).setName("union-2").setNumPartitions(4).filter(WireRequestsTopology::fraudDetect).setName("all-branches-fraud-detect").log();
    Config config = Config.newBuilder().setDeliverySemantics(Config.DeliverySemantics.EFFECTIVELY_ONCE).setNumContainers(2).build();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, builder);
}

Example 4 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class SimplePulsarSourceTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    /**
     * A Pulsar source is constructed for a specific Pulsar installation, topic, and
     * subsecription.
     */
    Source<String> pulsarSource = new PulsarSource(// Pulsar connection URL
    "pulsar://localhost:6650", // Pulsar topic
    "persistent://sample/nomad/ns1/heron-pulsar-test-topic", // Subscription name for the Pulsar topic
    "subscription-1");
    /**
     * In this processing graph, the source streamlet consists of messages on a
     * Pulsar topic. Those messages are simply logged without any processing logic
     * applied to them.
     */
    processingGraphBuilder.newSource(pulsarSource).setName("incoming-pulsar-messages").consume(s -> LOG.info(String.format("Message received from Pulsar: \"%s\"", s)));
    Config config = Config.defaultConfig();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 5 with Runner

use of org.apache.heron.streamlet.Runner in project heron by twitter.

the class FilesystemSinkTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    // Creates a temporary file to write output into.
    File file = File.createTempFile("filesystem-sink-example", ".tmp");
    LOG.info(String.format("Ready to write to file %s", file.getAbsolutePath()));
    processingGraphBuilder.newSource(() -> {
        // This applies a "brake" that makes the processing graph write
        // to the temporary file at a reasonable, readable pace.
        StreamletUtils.sleep(500);
        return ThreadLocalRandom.current().nextInt(100);
    }).setName("incoming-integers").toSink(new FilesystemSink<>(file));
    // The topology's parallelism (the number of containers across which the topology's
    // processing instance will be split) can be defined via the second command-line
    // argument (or else the default of 2 will be used).
    int topologyParallelism = StreamletUtils.getParallelism(args, 2);
    Config config = Config.newBuilder().setNumContainers(topologyParallelism).build();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}