Search in sources :

Example 1 with FileSizeRotationPolicy

use of org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy in project storm by apache.

the class TestSequenceFileBolt method makeSeqBolt.

private SequenceFileBolt makeSeqBolt(String nameNodeAddr, int countSync, float rotationSizeMB) {
    SyncPolicy fieldsSyncPolicy = new CountSyncPolicy(countSync);
    FileRotationPolicy fieldsRotationPolicy = new FileSizeRotationPolicy(rotationSizeMB, FileSizeRotationPolicy.Units.MB);
    FileNameFormat fieldsFileNameFormat = new DefaultFileNameFormat().withPath(testRoot);
    SequenceFormat seqFormat = new DefaultSequenceFormat("key", "value");
    return new SequenceFileBolt().withFsUrl(nameNodeAddr).withFileNameFormat(fieldsFileNameFormat).withRotationPolicy(fieldsRotationPolicy).withSequenceFormat(seqFormat).withSyncPolicy(fieldsSyncPolicy);
}
Also used : DefaultSequenceFormat(org.apache.storm.hdfs.bolt.format.DefaultSequenceFormat) SequenceFormat(org.apache.storm.hdfs.bolt.format.SequenceFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) DefaultSequenceFormat(org.apache.storm.hdfs.bolt.format.DefaultSequenceFormat)

Example 2 with FileSizeRotationPolicy

use of org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy in project storm by apache.

the class TestHdfsBolt method makeHdfsBolt.

private HdfsBolt makeHdfsBolt(String nameNodeAddr, int countSync, float rotationSizeMB) {
    RecordFormat fieldsFormat = new DelimitedRecordFormat().withFieldDelimiter("|");
    SyncPolicy fieldsSyncPolicy = new CountSyncPolicy(countSync);
    FileRotationPolicy fieldsRotationPolicy = new FileSizeRotationPolicy(rotationSizeMB, FileSizeRotationPolicy.Units.MB);
    FileNameFormat fieldsFileNameFormat = new DefaultFileNameFormat().withPath(testRoot);
    return new HdfsBolt().withFsUrl(nameNodeAddr).withFileNameFormat(fieldsFileNameFormat).withRecordFormat(fieldsFormat).withRotationPolicy(fieldsRotationPolicy).withSyncPolicy(fieldsSyncPolicy);
}
Also used : DelimitedRecordFormat(org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) DelimitedRecordFormat(org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)

Example 3 with FileSizeRotationPolicy

use of org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy in project storm by apache.

the class TestHdfsBolt method testCleanupDoesNotThrowExceptionWhenRotationPolicyIsNotTimed.

@Test
public void testCleanupDoesNotThrowExceptionWhenRotationPolicyIsNotTimed() {
    // STORM-3372: Rotation policy other than TimedRotationPolicy causes NPE on cleanup
    FileRotationPolicy fieldsRotationPolicy = new FileSizeRotationPolicy(10_000, FileSizeRotationPolicy.Units.MB);
    HdfsBolt bolt = makeHdfsBolt(hdfsURI, 10, 10000f).withRotationPolicy(fieldsRotationPolicy);
    bolt.prepare(new Config(), topologyContext, collector);
    bolt.cleanup();
}
Also used : Config(org.apache.storm.Config) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) Test(org.junit.Test)

Example 4 with FileSizeRotationPolicy

use of org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy in project metron by apache.

the class SourceHandlerTest method testRotateOutputFile.

@Test
public void testRotateOutputFile() throws IOException {
    SourceHandler handler = new SourceHandler(rotActions, // Don't actually care about the rotation
    new FileSizeRotationPolicy(10000, Units.MB), new CountSyncPolicy(1), testFormat, callback);
    handler.rotateOutputFile();
    // Function should ensure rotation actions and callback are called.
    verify(rotAction1).execute(any(), any());
    verify(rotAction2).execute(any(), any());
    verify(callback).removeKey();
}
Also used : CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) Test(org.junit.jupiter.api.Test)

Example 5 with FileSizeRotationPolicy

use of org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy in project storm by apache.

the class KafkaHdfsTopo method getTopology.

public static StormTopology getTopology(Map config) {
    final int spoutNum = getInt(config, SPOUT_NUM, DEFAULT_SPOUT_NUM);
    final int boltNum = getInt(config, BOLT_NUM, DEFAULT_BOLT_NUM);
    final int hdfsBatch = getInt(config, HDFS_BATCH, DEFAULT_HDFS_BATCH);
    // 1 -  Setup Kafka Spout   --------
    String zkConnString = getStr(config, ZOOKEEPER_URI);
    String topicName = getStr(config, KAFKA_TOPIC);
    BrokerHosts brokerHosts = new ZkHosts(zkConnString);
    SpoutConfig spoutConfig = new SpoutConfig(brokerHosts, topicName, "/" + topicName, UUID.randomUUID().toString());
    spoutConfig.scheme = new StringMultiSchemeWithTopic();
    spoutConfig.ignoreZkOffsets = true;
    KafkaSpout spout = new KafkaSpout(spoutConfig);
    // 2 -  Setup HFS Bolt   --------
    String Hdfs_url = getStr(config, HDFS_URI);
    RecordFormat format = new LineWriter("str");
    SyncPolicy syncPolicy = new CountSyncPolicy(hdfsBatch);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, FileSizeRotationPolicy.Units.GB);
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath(getStr(config, HDFS_PATH));
    // Instantiate the HdfsBolt
    HdfsBolt bolt = new HdfsBolt().withFsUrl(Hdfs_url).withFileNameFormat(fileNameFormat).withRecordFormat(format).withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy);
    // 3 - Setup Topology  --------
    TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout(SPOUT_ID, spout, spoutNum);
    builder.setBolt(BOLT_ID, bolt, boltNum).localOrShuffleGrouping(SPOUT_ID);
    return builder.createTopology();
}
Also used : TopologyBuilder(org.apache.storm.topology.TopologyBuilder) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) SpoutConfig(org.apache.storm.kafka.SpoutConfig) ZkHosts(org.apache.storm.kafka.ZkHosts) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) StringMultiSchemeWithTopic(org.apache.storm.kafka.StringMultiSchemeWithTopic) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) BrokerHosts(org.apache.storm.kafka.BrokerHosts) HdfsBolt(org.apache.storm.hdfs.bolt.HdfsBolt) KafkaSpout(org.apache.storm.kafka.KafkaSpout) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)

Aggregations

FileSizeRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)10 FileRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy)9 CountSyncPolicy (org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)9 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)8 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)8 SyncPolicy (org.apache.storm.hdfs.bolt.sync.SyncPolicy)8 RecordFormat (org.apache.storm.hdfs.bolt.format.RecordFormat)5 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)5 HdfsBolt (org.apache.storm.hdfs.bolt.HdfsBolt)4 Config (org.apache.storm.Config)2 DefaultSequenceFormat (org.apache.storm.hdfs.bolt.format.DefaultSequenceFormat)2 StringGenSpout (org.apache.storm.perf.spout.StringGenSpout)2 FileInputStream (java.io.FileInputStream)1 InputStream (java.io.InputStream)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)1 DelimitedRecordFormat (org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat)1 SequenceFormat (org.apache.storm.hdfs.bolt.format.SequenceFormat)1 MoveFileAction (org.apache.storm.hdfs.common.rotation.MoveFileAction)1