Search in sources :

Example 1 with CountSyncPolicy

use of org.apache.storm.hdfs.bolt.sync.CountSyncPolicy in project metron by apache.

the class HdfsWriter method init.

@Override
public void init(Map stormConfig, TopologyContext topologyContext, WriterConfiguration configurations) {
    this.stormConfig = stormConfig;
    this.stellarProcessor = new StellarProcessor();
    this.fileNameFormat.prepare(stormConfig, topologyContext);
    if (syncPolicy != null) {
        // if the user has specified the sync policy, we don't want to override their wishes.
        syncPolicyCreator = new ClonedSyncPolicyCreator(syncPolicy);
    } else {
        // if the user has not, then we want to have the sync policy depend on the batch size.
        syncPolicyCreator = (source, config) -> new CountSyncPolicy(config == null ? 1 : config.getBatchSize(source));
    }
}
Also used : StellarProcessor(org.apache.metron.stellar.common.StellarProcessor) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)

Example 2 with CountSyncPolicy

use of org.apache.storm.hdfs.bolt.sync.CountSyncPolicy in project storm by apache.

the class TestSequenceFileBolt method makeSeqBolt.

private SequenceFileBolt makeSeqBolt(String nameNodeAddr, int countSync, float rotationSizeMB) {
    SyncPolicy fieldsSyncPolicy = new CountSyncPolicy(countSync);
    FileRotationPolicy fieldsRotationPolicy = new FileSizeRotationPolicy(rotationSizeMB, FileSizeRotationPolicy.Units.MB);
    FileNameFormat fieldsFileNameFormat = new DefaultFileNameFormat().withPath(testRoot);
    SequenceFormat seqFormat = new DefaultSequenceFormat("key", "value");
    return new SequenceFileBolt().withFsUrl(nameNodeAddr).withFileNameFormat(fieldsFileNameFormat).withRotationPolicy(fieldsRotationPolicy).withSequenceFormat(seqFormat).withSyncPolicy(fieldsSyncPolicy);
}
Also used : DefaultSequenceFormat(org.apache.storm.hdfs.bolt.format.DefaultSequenceFormat) SequenceFormat(org.apache.storm.hdfs.bolt.format.SequenceFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) DefaultSequenceFormat(org.apache.storm.hdfs.bolt.format.DefaultSequenceFormat)

Example 3 with CountSyncPolicy

use of org.apache.storm.hdfs.bolt.sync.CountSyncPolicy in project storm by apache.

the class TestHdfsBolt method makeHdfsBolt.

private HdfsBolt makeHdfsBolt(String nameNodeAddr, int countSync, float rotationSizeMB) {
    RecordFormat fieldsFormat = new DelimitedRecordFormat().withFieldDelimiter("|");
    SyncPolicy fieldsSyncPolicy = new CountSyncPolicy(countSync);
    FileRotationPolicy fieldsRotationPolicy = new FileSizeRotationPolicy(rotationSizeMB, FileSizeRotationPolicy.Units.MB);
    FileNameFormat fieldsFileNameFormat = new DefaultFileNameFormat().withPath(testRoot);
    return new HdfsBolt().withFsUrl(nameNodeAddr).withFileNameFormat(fieldsFileNameFormat).withRecordFormat(fieldsFormat).withRotationPolicy(fieldsRotationPolicy).withSyncPolicy(fieldsSyncPolicy);
}
Also used : DelimitedRecordFormat(org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat) RecordFormat(org.apache.storm.hdfs.bolt.format.RecordFormat) DelimitedRecordFormat(org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) FileRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)

Example 4 with CountSyncPolicy

use of org.apache.storm.hdfs.bolt.sync.CountSyncPolicy in project metron by apache.

the class SourceHandlerTest method testRotateOutputFile.

@Test
public void testRotateOutputFile() throws IOException {
    SourceHandler handler = new SourceHandler(rotActions, // Don't actually care about the rotation
    new FileSizeRotationPolicy(10000, Units.MB), new CountSyncPolicy(1), testFormat, callback);
    handler.rotateOutputFile();
    // Function should ensure rotation actions and callback are called.
    verify(rotAction1).execute(any(), any());
    verify(rotAction2).execute(any(), any());
    verify(callback).removeKey();
}
Also used : CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) FileSizeRotationPolicy(org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy) Test(org.junit.jupiter.api.Test)

Example 5 with CountSyncPolicy

use of org.apache.storm.hdfs.bolt.sync.CountSyncPolicy in project metron by apache.

the class ClonedSyncPolicyCreatorTest method testClonedPolicy.

@Test
public void testClonedPolicy() {
    CountSyncPolicy basePolicy = new CountSyncPolicy(5);
    ClonedSyncPolicyCreator creator = new ClonedSyncPolicyCreator(basePolicy);
    // ensure cloned policy continues to work and adheres to the contract: mark on 5th call.
    SyncPolicy clonedPolicy = creator.create("blah", null);
    for (int i = 0; i < 4; ++i) {
        assertFalse(clonedPolicy.mark(null, i));
    }
    assertTrue(clonedPolicy.mark(null, 5));
    // reclone policy and ensure it adheres to the original contract.
    clonedPolicy = creator.create("blah", null);
    assertFalse(clonedPolicy.mark(null, 0));
}
Also used : CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) CountSyncPolicy(org.apache.storm.hdfs.bolt.sync.CountSyncPolicy) SyncPolicy(org.apache.storm.hdfs.bolt.sync.SyncPolicy) Test(org.junit.jupiter.api.Test)

Aggregations

CountSyncPolicy (org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)15 SyncPolicy (org.apache.storm.hdfs.bolt.sync.SyncPolicy)10 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)9 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)9 FileRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy)9 FileSizeRotationPolicy (org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy)9 RecordFormat (org.apache.storm.hdfs.bolt.format.RecordFormat)6 TopologyBuilder (org.apache.storm.topology.TopologyBuilder)6 HdfsBolt (org.apache.storm.hdfs.bolt.HdfsBolt)4 Test (org.junit.jupiter.api.Test)3 File (java.io.File)2 FileInputStream (java.io.FileInputStream)2 InputStream (java.io.InputStream)2 HashMap (java.util.HashMap)2 Map (java.util.Map)2 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)2 IndexingWriterConfiguration (org.apache.metron.common.configuration.writer.IndexingWriterConfiguration)2 WriterConfiguration (org.apache.metron.common.configuration.writer.WriterConfiguration)2 StellarProcessor (org.apache.metron.stellar.common.StellarProcessor)2 Config (org.apache.storm.Config)2