Search in sources :

Example 6 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathStringReturned.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathStringReturned() {
    IndexingConfigurations indexingConfig = new IndexingConfigurations();
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    message.put("test.key", "test.value");
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "TO_UPPER(FORMAT(MAP_GET('key', {'key': 'AbC%s'}), test.key))", message);
    writer.close();
    Assert.assertEquals("ABCTEST.VALUE", result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 7 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class HdfsWriterTest method testWriteSingleFileWithNull.

@Test
@SuppressWarnings("unchecked")
public void testWriteSingleFileWithNull() throws Exception {
    String function = "FORMAT('test-%s/%s', test.key, test.key)";
    WriterConfiguration config = buildWriterConfiguration(function);
    FileNameFormat format = new DefaultFileNameFormat().withPath(folder.toString()).withExtension(".json").withPrefix("prefix-");
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(format);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    // These two messages will be routed to the same folder, because test.key is the same
    JSONObject message = new JSONObject();
    message.put("test.key2", "test.value2");
    ArrayList<JSONObject> messages = new ArrayList<>();
    messages.add(message);
    ArrayList<Tuple> tuples = new ArrayList<>();
    writer.write(SENSOR_NAME, config, tuples, messages);
    writer.close();
    ArrayList<String> expected = new ArrayList<>();
    expected.add(message.toJSONString());
    Collections.sort(expected);
    File outputFolder = new File(folder.getAbsolutePath() + "/test-null/null/");
    Assert.assertTrue(outputFolder.exists() && outputFolder.isDirectory());
    Assert.assertEquals(1, outputFolder.listFiles().length);
    for (File file : outputFolder.listFiles()) {
        List<String> lines = Files.readAllLines(file.toPath());
        Collections.sort(lines);
        Assert.assertEquals(expected, lines);
    }
}
Also used : ArrayList(java.util.ArrayList) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) FileNameFormat(org.apache.storm.hdfs.bolt.format.FileNameFormat) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) DefaultFileNameFormat(org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat) JSONObject(org.json.simple.JSONObject) File(java.io.File) Tuple(org.apache.storm.tuple.Tuple) Test(org.junit.Test)

Example 8 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathEmptyString.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathEmptyString() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "", message);
    writer.close();
    Assert.assertEquals(SENSOR_NAME, result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 9 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetSourceHandlerOpenFilesOverMax.

@Test(expected = IllegalStateException.class)
public void testGetSourceHandlerOpenFilesOverMax() throws IOException {
    int maxFiles = 2;
    IndexingConfigurations indexingConfig = new IndexingConfigurations();
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat).withMaxOpenFiles(maxFiles);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    for (int i = 0; i < maxFiles + 1; i++) {
        writer.getSourceHandler(SENSOR_NAME, Integer.toString(i), null);
    }
}
Also used : IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 10 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathNonString.

@Test(expected = IllegalArgumentException.class)
public void testGetHdfsPathNonString() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    writer.getHdfsPathExtension(SENSOR_NAME, "{'key':'value'}", message);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Aggregations

WriterConfiguration (org.apache.metron.common.configuration.writer.WriterConfiguration)32 Test (org.junit.Test)29 JSONObject (org.json.simple.JSONObject)26 IndexingWriterConfiguration (org.apache.metron.common.configuration.writer.IndexingWriterConfiguration)21 IndexingConfigurations (org.apache.metron.common.configuration.IndexingConfigurations)13 SimpleHbaseEnrichmentWriter (org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter)9 File (java.io.File)6 ArrayList (java.util.ArrayList)6 Tuple (org.apache.storm.tuple.Tuple)6 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)4 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)4 LookupKV (org.apache.metron.enrichment.lookup.LookupKV)3 WriterToBulkWriter (org.apache.metron.writer.WriterToBulkWriter)3 ParserWriterConfiguration (org.apache.metron.common.configuration.writer.ParserWriterConfiguration)2 CountSyncPolicy (org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)2