Examples with Builder com.twitter.heron.streamlet.Builder used on opensource projects

Example 6 with Builder

use of com.twitter.heron.streamlet.Builder in project incubator-heron by apache.

the class FormattedOutputTopology method main.

public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    processingGraphBuilder.newSource(SensorReading::new).filter(reading -> reading.getHumidity() < .9 && reading.getTemperature() < 90).consume(reading -> LOG.info(String.format("Reading from device %s: (temp: %f, humidity: %f)", reading.getDeviceId(), reading.getTemperature(), reading.getHumidity())));
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    Config config = Config.defaultConfig();
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 7 with Builder

use of com.twitter.heron.streamlet.Builder in project incubator-heron by apache.

the class ImpressionsAndClicksTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    // A KVStreamlet is produced. Each element is a KeyValue object where the key
    // is the impression ID and the user ID is the value.
    Streamlet<AdImpression> impressions = processingGraphBuilder.newSource(AdImpression::new);
    // A KVStreamlet is produced. Each element is a KeyValue object where the key
    // is the ad ID and the user ID is the value.
    Streamlet<AdClick> clicks = processingGraphBuilder.newSource(AdClick::new);
    /**
     * Here, the impressions KVStreamlet is joined to the clicks KVStreamlet.
     */
    impressions.join(// The other streamlet that's being joined to
    clicks, // Key extractor for the impressions streamlet
    impression -> impression.getUserId(), // Key extractor for the clicks streamlet
    click -> click.getUserId(), // Window configuration for the join operation
    WindowConfig.TumblingCountWindow(25), // Join type (inner join means that all elements from both streams will be included)
    JoinType.INNER, // if the ad IDs match between the elements (or a value of 0 if they don't).
    (user1, user2) -> (user1.getAdId().equals(user2.getAdId())) ? 1 : 0).reduceByKeyAndWindow(// Key extractor for the reduce operation
    kv -> String.format("user-%s", kv.getKey().getKey()), // Value extractor for the reduce operation
    kv -> kv.getValue(), // Window configuration for the reduce operation
    WindowConfig.TumblingCountWindow(50), // A running cumulative total is calculated for each key
    (cumulative, incoming) -> cumulative + incoming).consume(kw -> {
        LOG.info(String.format("(user: %s, clicks: %d)", kw.getKey().getKey(), kw.getValue()));
    });
    Config config = Config.defaultConfig();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 8 with Builder

use of com.twitter.heron.streamlet.Builder in project incubator-heron by apache.

the class RepartitionTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder processingGraphBuilder = Builder.newBuilder();
    Streamlet<Integer> randomIntegers = processingGraphBuilder.newSource(() -> {
        // Random integers are emitted every 50 milliseconds
        StreamletUtils.sleep(50);
        return ThreadLocalRandom.current().nextInt(100);
    }).setNumPartitions(2).setName("random-integer-source");
    randomIntegers.repartition(8, RepartitionTopology::repartitionStreamlet).setName("repartition-incoming-values").repartition(2).setName("reduce-partitions-for-logging-operation").log();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    Config config = Config.defaultConfig();
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, processingGraphBuilder);
}

Example 9 with Builder

use of com.twitter.heron.streamlet.Builder in project incubator-heron by apache.

the class TransformsTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder builder = Builder.newBuilder();
    /**
     * The processing graph consists of a supplier streamlet that emits
     * random integers between 1 and 100. From there, a series of transformers
     * is applied. At the end of the graph, the original value is ultimately
     * unchanged.
     */
    builder.newSource(() -> ThreadLocalRandom.current().nextInt(100)).transform(new DoNothingTransformer<>()).transform(new IncrementTransformer(10)).transform(new IncrementTransformer(-7)).transform(new DoNothingTransformer<>()).transform(new IncrementTransformer(-3)).log();
    Config config = Config.defaultConfig();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, builder);
}

Example 10 with Builder

use of com.twitter.heron.streamlet.Builder in project incubator-heron by apache.

the class WireRequestsTopology method main.

/**
 * All Heron topologies require a main function that defines the topology's behavior
 * at runtime
 */
public static void main(String[] args) throws Exception {
    Builder builder = Builder.newBuilder();
    // Requests from the "quiet" bank branch (high throttling).
    Streamlet<WireRequest> quietBranch = builder.newSource(() -> new WireRequest(20)).setNumPartitions(1).setName("quiet-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("quiet-branch-check-balance");
    // Requests from the "medium" bank branch (medium throttling).
    Streamlet<WireRequest> mediumBranch = builder.newSource(() -> new WireRequest(10)).setNumPartitions(2).setName("medium-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("medium-branch-check-balance");
    // Requests from the "busy" bank branch (low throttling).
    Streamlet<WireRequest> busyBranch = builder.newSource(() -> new WireRequest(5)).setNumPartitions(4).setName("busy-branch-requests").filter(WireRequestsTopology::checkRequestAmount).setName("busy-branch-check-balance");
    // Here, the streamlets for the three bank branches are united into one. The fraud
    // detection filter then operates on that unified streamlet.
    quietBranch.union(mediumBranch).setNumPartitions(2).setName("union-1").union(busyBranch).setName("union-2").setNumPartitions(4).filter(WireRequestsTopology::fraudDetect).setName("all-branches-fraud-detect").log();
    Config config = Config.newBuilder().setDeliverySemantics(Config.DeliverySemantics.EFFECTIVELY_ONCE).setNumContainers(2).build();
    // Fetches the topology name from the first command-line argument
    String topologyName = StreamletUtils.getTopologyName(args);
    // Finally, the processing graph and configuration are passed to the Runner, which converts
    // the graph into a Heron topology that can be run in a Heron cluster.
    new Runner().run(topologyName, config, builder);
}