Search in sources :

Example 1 with HdfsBolt

use of org.apache.storm.hdfs.bolt.HdfsBolt in project storm by apache.

the class KafkaHdfsTopo method getTopology.

public static StormTopology getTopology(Map config) {
    final int spoutNum = getInt(config, SPOUT_NUM, DEFAULT_SPOUT_NUM);
    final int boltNum = getInt(config, BOLT_NUM, DEFAULT_BOLT_NUM);
    final int hdfsBatch = getInt(config, HDFS_BATCH, DEFAULT_HDFS_BATCH);
    // 1 -  Setup Kafka Spout   --------
    String zkConnString = getStr(config, ZOOKEEPER_URI);
    String topicName = getStr(config, KAFKA_TOPIC);
    BrokerHosts brokerHosts = new ZkHosts(zkConnString);
    SpoutConfig spoutConfig = new SpoutConfig(brokerHosts, topicName, "/" + topicName, UUID.randomUUID().toString());
    spoutConfig.scheme = new StringMultiSchemeWithTopic();
    spoutConfig.ignoreZkOffsets = true;
    KafkaSpout spout = new KafkaSpout(spoutConfig);
    // 2 -  Setup HFS Bolt   --------
    String Hdfs_url = getStr(config, HDFS_URI);
    RecordFormat format = new LineWriter("str");
    SyncPolicy syncPolicy = new CountSyncPolicy(hdfsBatch);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, FileSizeRotationPolicy.Units.GB);
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath(getStr(config, HDFS_PATH));
    // Instantiate the HdfsBolt
    HdfsBolt bolt = new HdfsBolt().withFsUrl(Hdfs_url).withFileNameFormat(fileNameFormat).withRecordFormat(format).withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy);
    // 3 - Setup Topology  --------
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(SPOUT_ID, spout, spoutNum);
    builder.setBolt(BOLT_ID, bolt, boltNum).localOrShuffleGrouping(SPOUT_ID);
    return builder.createTopology();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) SpoutConfig(org.apache.storm.kafka.SpoutConfig) ZkHosts(org.apache.storm.kafka.ZkHosts) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) StringMultiSchemeWithTopic(org.apache.storm.kafka.StringMultiSchemeWithTopic) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) BrokerHosts(org.apache.storm.kafka.BrokerHosts) HdfsBolt(org.apache.storm.hdfs.bolt.HdfsBolt) KafkaSpout(org.apache.storm.kafka.KafkaSpout) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)

Example 2 with HdfsBolt

use of org.apache.storm.hdfs.bolt.HdfsBolt in project storm by apache.

the class StrGenSpoutHdfsBoltTopo method getTopology.

public static StormTopology getTopology(Map topoConf) {
    final int hdfsBatch = Helper.getInt(topoConf, HDFS_BATCH, DEFAULT_HDFS_BATCH);
    // 1 -  Setup StringGen Spout   --------
    StringGenSpout spout = new StringGenSpout(100).withFieldName("str");
    // 2 -  Setup HFS Bolt   --------
    String Hdfs_url = Helper.getStr(topoConf, HDFS_URI);
    RecordFormat format = new LineWriter("str");
    SyncPolicy syncPolicy = new CountSyncPolicy(hdfsBatch);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, FileSizeRotationPolicy.Units.GB);
    final int spoutNum = Helper.getInt(topoConf, SPOUT_NUM, DEFAULT_SPOUT_NUM);
    final int boltNum = Helper.getInt(topoConf, BOLT_NUM, DEFAULT_BOLT_NUM);
    // Use default, Storm-generated file names
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath(Helper.getStr(topoConf, HDFS_PATH));
    // Instantiate the HdfsBolt
    HdfsBolt bolt = new HdfsBolt().withFsUrl(Hdfs_url).withFileNameFormat(fileNameFormat).withRecordFormat(format).withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy);
    // 3 - Setup Topology  --------
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(SPOUT_ID, spout, spoutNum);
    builder.setBolt(BOLT_ID, bolt, boltNum).localOrShuffleGrouping(SPOUT_ID);
    return builder.createTopology();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) HdfsBolt(org.apache.storm.hdfs.bolt.HdfsBolt) StringGenSpout(org.apache.storm.perf.spout.StringGenSpout) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)

Example 3 with HdfsBolt

use of org.apache.storm.hdfs.bolt.HdfsBolt in project storm by apache.

the class KafkaClientHdfsTopo method getTopology.

static StormTopology getTopology(Map<String, Object> config) {
    final int spoutNum = getInt(config, SPOUT_NUM, DEFAULT_SPOUT_NUM);
    final int boltNum = getInt(config, BOLT_NUM, DEFAULT_BOLT_NUM);
    final int hdfsBatch = getInt(config, HDFS_BATCH, DEFAULT_HDFS_BATCH);
    // 1 -  Setup Kafka Spout   --------
    String bootstrapHosts = getStr(config, KAFKA_BOOTSTRAP_HOSTS);
    String topicName = getStr(config, KAFKA_TOPIC);
    KafkaSpoutConfig<String, String> spoutConfig = KafkaSpoutConfig.builder(bootstrapHosts, topicName).setFirstPollOffsetStrategy(FirstPollOffsetStrategy.EARLIEST).build();
    KafkaSpout<String, String> spout = new KafkaSpout<>(spoutConfig);
    // 2 -  Setup HFS Bolt   --------
    String hdfsUrls = getStr(config, HDFS_URI);
    RecordFormat format = new LineWriter("value");
    SyncPolicy syncPolicy = new CountSyncPolicy(hdfsBatch);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, FileSizeRotationPolicy.Units.GB);
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath(getStr(config, HDFS_PATH));
    // Instantiate the HdfsBolt
    HdfsBolt bolt = new HdfsBolt().withFsUrl(hdfsUrls).withFileNameFormat(fileNameFormat).withRecordFormat(format).withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy);
    // 3 - Setup Topology  --------
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(SPOUT_ID, spout, spoutNum);
    builder.setBolt(BOLT_ID, bolt, boltNum).localOrShuffleGrouping(SPOUT_ID);
    return builder.createTopology();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) HdfsBolt(org.apache.storm.hdfs.bolt.HdfsBolt) KafkaSpout(org.apache.storm.kafka.spout.KafkaSpout) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)

Example 4 with HdfsBolt

use of org.apache.storm.hdfs.bolt.HdfsBolt in project storm by apache.

the class StrGenSpoutHdfsBoltTopo method getTopology.

static StormTopology getTopology(Map<String, Object> topoConf) {
    final int hdfsBatch = Helper.getInt(topoConf, HDFS_BATCH, DEFAULT_HDFS_BATCH);
    // 1 -  Setup StringGen Spout   --------
    StringGenSpout spout = new StringGenSpout(100).withFieldName("str");
    // 2 -  Setup HFS Bolt   --------
    String hdfsUrl = Helper.getStr(topoConf, HDFS_URI);
    RecordFormat format = new LineWriter("str");
    SyncPolicy syncPolicy = new CountSyncPolicy(hdfsBatch);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, FileSizeRotationPolicy.Units.GB);
    final int spoutNum = Helper.getInt(topoConf, SPOUT_NUM, DEFAULT_SPOUT_NUM);
    final int boltNum = Helper.getInt(topoConf, BOLT_NUM, DEFAULT_BOLT_NUM);
    // Use default, Storm-generated file names
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath(Helper.getStr(topoConf, HDFS_PATH));
    // Instantiate the HdfsBolt
    HdfsBolt bolt = new HdfsBolt().withFsUrl(hdfsUrl).withFileNameFormat(fileNameFormat).withRecordFormat(format).withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy);
    // 3 - Setup Topology  --------
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(SPOUT_ID, spout, spoutNum);
    builder.setBolt(BOLT_ID, bolt, boltNum).localOrShuffleGrouping(SPOUT_ID);
    return builder.createTopology();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) HdfsBolt(org.apache.storm.hdfs.bolt.HdfsBolt) StringGenSpout(org.apache.storm.perf.spout.StringGenSpout) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)

Aggregations

HdfsBolt (org.apache.storm.hdfs.bolt.HdfsBolt)4 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)4 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)4 RecordFormat (org.apache.storm.hdfs.bolt.format.RecordFormat)4 FileRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy)4 FileSizeRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)4 CountSyncPolicy (org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)4 SyncPolicy (org.apache.storm.hdfs.bolt.sync.SyncPolicy)4 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)4 StringGenSpout (org.apache.storm.perf.spout.StringGenSpout)2 BrokerHosts (org.apache.storm.kafka.BrokerHosts)1 KafkaSpout (org.apache.storm.kafka.KafkaSpout)1 SpoutConfig (org.apache.storm.kafka.SpoutConfig)1 StringMultiSchemeWithTopic (org.apache.storm.kafka.StringMultiSchemeWithTopic)1 ZkHosts (org.apache.storm.kafka.ZkHosts)1 KafkaSpout (org.apache.storm.kafka.spout.KafkaSpout)1