Search in sources :

Example 1 with BoltFileSink

use of org.apache.flink.storm.util.BoltFileSink in project flink by apache.

the class SplitBoltTopology method buildTopology.

public static TopologyBuilder buildTopology() {
    final TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(spoutId, new RandomSpout(false, seed));
    builder.setBolt(boltId, new SplitBolt()).shuffleGrouping(spoutId);
    builder.setBolt(evenVerifierId, new VerifyAndEnrichBolt(true)).shuffleGrouping(boltId, SplitBolt.EVEN_STREAM);
    builder.setBolt(oddVerifierId, new VerifyAndEnrichBolt(false)).shuffleGrouping(boltId, SplitBolt.ODD_STREAM);
    // emit result
    if (outputPath != null) {
        // read the text file from given input path
        final String[] tokens = outputPath.split(":");
        final String outputFile = tokens[tokens.length - 1];
        builder.setBolt(sinkId, new BoltFileSink(outputFile, formatter)).shuffleGrouping(evenVerifierId).shuffleGrouping(oddVerifierId);
    } else {
        builder.setBolt(sinkId, new BoltPrintSink(formatter), 4).shuffleGrouping(evenVerifierId).shuffleGrouping(oddVerifierId);
    }
    return builder;
}
Also used : RandomSpout(org.apache.flink.storm.split.operators.RandomSpout) BoltFileSink(org.apache.flink.storm.util.BoltFileSink) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) VerifyAndEnrichBolt(org.apache.flink.storm.split.operators.VerifyAndEnrichBolt) BoltPrintSink(org.apache.flink.storm.util.BoltPrintSink)

Example 2 with BoltFileSink

use of org.apache.flink.storm.util.BoltFileSink in project flink by apache.

the class SplitSpoutTopology method buildTopology.

public static TopologyBuilder buildTopology() {
    final TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(spoutId, new RandomSpout(true, seed));
    builder.setBolt(evenVerifierId, new VerifyAndEnrichBolt(true)).shuffleGrouping(spoutId, RandomSpout.EVEN_STREAM);
    builder.setBolt(oddVerifierId, new VerifyAndEnrichBolt(false)).shuffleGrouping(spoutId, RandomSpout.ODD_STREAM);
    // emit result
    if (outputPath != null) {
        // read the text file from given input path
        final String[] tokens = outputPath.split(":");
        final String outputFile = tokens[tokens.length - 1];
        builder.setBolt(sinkId, new BoltFileSink(outputFile, formatter)).shuffleGrouping(evenVerifierId).shuffleGrouping(oddVerifierId);
    } else {
        builder.setBolt(sinkId, new BoltPrintSink(formatter), 4).shuffleGrouping(evenVerifierId).shuffleGrouping(oddVerifierId);
    }
    return builder;
}
Also used : RandomSpout(org.apache.flink.storm.split.operators.RandomSpout) BoltFileSink(org.apache.flink.storm.util.BoltFileSink) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) VerifyAndEnrichBolt(org.apache.flink.storm.split.operators.VerifyAndEnrichBolt) BoltPrintSink(org.apache.flink.storm.util.BoltPrintSink)

Example 3 with BoltFileSink

use of org.apache.flink.storm.util.BoltFileSink in project flink by apache.

the class StormFieldsGroupingITCase method testProgram.

@Override
protected void testProgram() throws Exception {
    final String[] tokens = this.resultPath.split(":");
    final String outputFile = tokens[tokens.length - 1];
    final TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(spoutId, new FiniteRandomSpout(0, 10, 2));
    builder.setBolt(boltId, new TaskIdBolt(), 2).fieldsGrouping(spoutId, FiniteRandomSpout.STREAM_PREFIX + 0, new Fields("number"));
    builder.setBolt(sinkId, new BoltFileSink(outputFile)).shuffleGrouping(boltId);
    final FlinkLocalCluster cluster = FlinkLocalCluster.getLocalCluster();
    Config conf = new Config();
    // only required to stabilize integration test
    conf.put(FlinkLocalCluster.SUBMIT_BLOCKING, true);
    cluster.submitTopology(topologyId, conf, FlinkTopology.createTopology(builder));
    cluster.shutdown();
}
Also used : BoltFileSink(org.apache.flink.storm.util.BoltFileSink) Fields(org.apache.storm.tuple.Fields) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) FlinkLocalCluster(org.apache.flink.storm.api.FlinkLocalCluster) Config(org.apache.storm.Config) FiniteRandomSpout(org.apache.flink.storm.tests.operators.FiniteRandomSpout) TaskIdBolt(org.apache.flink.storm.tests.operators.TaskIdBolt)

Example 4 with BoltFileSink

use of org.apache.flink.storm.util.BoltFileSink in project flink by apache.

the class ExclamationTopology method buildTopology.

public static TopologyBuilder buildTopology() {
    final TopologyBuilder builder = new TopologyBuilder();
    // get input data
    if (fileInputOutput) {
        // read the text file from given input path
        final String[] tokens = textPath.split(":");
        final String inputFile = tokens[tokens.length - 1];
        builder.setSpout(spoutId, new FiniteFileSpout(inputFile));
    } else {
        builder.setSpout(spoutId, new FiniteInMemorySpout(WordCountData.WORDS));
    }
    builder.setBolt(firstBoltId, new ExclamationBolt(), 3).shuffleGrouping(spoutId);
    builder.setBolt(secondBoltId, new ExclamationBolt(), 2).shuffleGrouping(firstBoltId);
    // emit result
    if (fileInputOutput) {
        // read the text file from given input path
        final String[] tokens = outputPath.split(":");
        final String outputFile = tokens[tokens.length - 1];
        builder.setBolt(sinkId, new BoltFileSink(outputFile, formatter)).shuffleGrouping(secondBoltId);
    } else {
        builder.setBolt(sinkId, new BoltPrintSink(formatter), 4).shuffleGrouping(secondBoltId);
    }
    return builder;
}
Also used : FiniteInMemorySpout(org.apache.flink.storm.util.FiniteInMemorySpout) ExclamationBolt(org.apache.flink.storm.exclamation.operators.ExclamationBolt) BoltFileSink(org.apache.flink.storm.util.BoltFileSink) TopologyBuilder(org.apache.storm.topology.TopologyBuilder) FiniteFileSpout(org.apache.flink.storm.util.FiniteFileSpout) BoltPrintSink(org.apache.flink.storm.util.BoltPrintSink)

Example 5 with BoltFileSink

use of org.apache.flink.storm.util.BoltFileSink in project flink by apache.

the class SingleJoinExample method main.

public static void main(String[] args) throws Exception {
    final FeederSpout genderSpout = new FeederSpout(new Fields("id", "gender", "hobbies"));
    final FeederSpout ageSpout = new FeederSpout(new Fields("id", "age"));
    Config conf = new Config();
    TopologyBuilder builder = new TopologyBuilder();
    //  only required to stabilize integration test
    conf.put(FlinkLocalCluster.SUBMIT_BLOCKING, true);
    final NullTerminatingSpout finalGenderSpout = new NullTerminatingSpout(genderSpout);
    final NullTerminatingSpout finalAgeSpout = new NullTerminatingSpout(ageSpout);
    builder.setSpout("gender", finalGenderSpout);
    builder.setSpout("age", finalAgeSpout);
    builder.setBolt("join", new SingleJoinBolt(new Fields("gender", "age"))).fieldsGrouping("gender", new Fields("id")).fieldsGrouping("age", new Fields("id"));
    // emit result
    if (args.length > 0) {
        // read the text file from given input path
        builder.setBolt("fileOutput", new BoltFileSink(args[0], new TupleOutputFormatter())).shuffleGrouping("join");
    } else {
        builder.setBolt("print", new PrinterBolt()).shuffleGrouping("join");
    }
    String[] hobbies = new String[] { "reading", "biking", "travelling", "watching tv" };
    for (int i = 0; i < 10; i++) {
        String gender;
        if (i % 2 == 0) {
            gender = "male";
        } else {
            gender = "female";
        }
        genderSpout.feed(new Values(i, gender, hobbies[i % hobbies.length]));
    }
    for (int i = 9; i >= 0; i--) {
        ageSpout.feed(new Values(i, i + 20));
    }
    final FlinkLocalCluster cluster = FlinkLocalCluster.getLocalCluster();
    cluster.submitTopology("joinTopology", conf, FlinkTopology.createTopology(builder));
    cluster.shutdown();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) Config(org.apache.storm.Config) TupleOutputFormatter(org.apache.flink.storm.util.TupleOutputFormatter) Values(org.apache.storm.tuple.Values) NullTerminatingSpout(org.apache.flink.storm.util.NullTerminatingSpout) BoltFileSink(org.apache.flink.storm.util.BoltFileSink) Fields(org.apache.storm.tuple.Fields) FlinkLocalCluster(org.apache.flink.storm.api.FlinkLocalCluster) SingleJoinBolt(org.apache.storm.starter.bolt.SingleJoinBolt) FeederSpout(org.apache.storm.testing.FeederSpout) PrinterBolt(org.apache.storm.starter.bolt.PrinterBolt)

Aggregations

BoltFileSink (org.apache.flink.storm.util.BoltFileSink)7 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)7 BoltPrintSink (org.apache.flink.storm.util.BoltPrintSink)4 FlinkLocalCluster (org.apache.flink.storm.api.FlinkLocalCluster)3 Config (org.apache.storm.Config)3 Fields (org.apache.storm.tuple.Fields)3 RandomSpout (org.apache.flink.storm.split.operators.RandomSpout)2 VerifyAndEnrichBolt (org.apache.flink.storm.split.operators.VerifyAndEnrichBolt)2 FiniteRandomSpout (org.apache.flink.storm.tests.operators.FiniteRandomSpout)2 NullTerminatingSpout (org.apache.flink.storm.util.NullTerminatingSpout)2 ExclamationBolt (org.apache.flink.storm.exclamation.operators.ExclamationBolt)1 MergerBolt (org.apache.flink.storm.tests.operators.MergerBolt)1 TaskIdBolt (org.apache.flink.storm.tests.operators.TaskIdBolt)1 FiniteFileSpout (org.apache.flink.storm.util.FiniteFileSpout)1 FiniteInMemorySpout (org.apache.flink.storm.util.FiniteInMemorySpout)1 TupleOutputFormatter (org.apache.flink.storm.util.TupleOutputFormatter)1 BoltCounter (org.apache.flink.storm.wordcount.operators.BoltCounter)1 BoltCounterByName (org.apache.flink.storm.wordcount.operators.BoltCounterByName)1 BoltTokenizer (org.apache.flink.storm.wordcount.operators.BoltTokenizer)1 BoltTokenizerByName (org.apache.flink.storm.wordcount.operators.BoltTokenizerByName)1