Search in sources :

Example 16 with IndexingWriterConfiguration

use of org.apache.metron.common.configuration.writer.IndexingWriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathMultipleFunctions.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathMultipleFunctions() {
    IndexingConfigurations indexingConfig = new IndexingConfigurations();
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    message.put("test.key", "test.value");
    message.put("test.key.2", "test.value.2");
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "FORMAT('%s', test.key)", message);
    Assert.assertEquals("test.value", result);
    result = writer.getHdfsPathExtension(SENSOR_NAME, "FORMAT('%s/%s', test.key, test.key.2)", message);
    Assert.assertEquals("test.value/test.value.2", result);
    result = writer.getHdfsPathExtension(SENSOR_NAME, "FORMAT('%s', test.key)", message);
    writer.close();
    Assert.assertEquals("test.value", result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 17 with IndexingWriterConfiguration

use of org.apache.metron.common.configuration.writer.IndexingWriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathNull.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathNull() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, null, message);
    writer.close();
    Assert.assertEquals(SENSOR_NAME, result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 18 with IndexingWriterConfiguration

use of org.apache.metron.common.configuration.writer.IndexingWriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetSourceHandlerOpenFilesMax.

@Test
public void testGetSourceHandlerOpenFilesMax() throws IOException {
    int maxFiles = 2;
    IndexingConfigurations indexingConfig = new IndexingConfigurations();
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat).withMaxOpenFiles(maxFiles);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    for (int i = 0; i < maxFiles; i++) {
        writer.getSourceHandler(SENSOR_NAME, Integer.toString(i), null);
    }
}
Also used : IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 19 with IndexingWriterConfiguration

use of org.apache.metron.common.configuration.writer.IndexingWriterConfiguration in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathDirectVariable.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathDirectVariable() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    message.put("test.key", "test.value");
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "test.key", message);
    writer.close();
    Assert.assertEquals("test.value", result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 20 with IndexingWriterConfiguration

use of org.apache.metron.common.configuration.writer.IndexingWriterConfiguration in project metron by apache.

the class HdfsWriterTest method buildWriterConfiguration.

protected WriterConfiguration buildWriterConfiguration(String function) {
    IndexingConfigurations indexingConfig = new IndexingConfigurations();
    Map<String, Object> sensorIndexingConfig = new HashMap<>();
    Map<String, Object> writerIndexingConfig = new HashMap<>();
    writerIndexingConfig.put(IndexingConfigurations.OUTPUT_PATH_FUNCTION_CONF, function);
    sensorIndexingConfig.put(WRITER_NAME, writerIndexingConfig);
    indexingConfig.updateSensorIndexingConfig(SENSOR_NAME, sensorIndexingConfig);
    return new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
}
Also used : HashMap(java.util.HashMap) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations)

Aggregations

IndexingWriterConfiguration (org.apache.metron.common.configuration.writer.IndexingWriterConfiguration)22 IndexingConfigurations (org.apache.metron.common.configuration.IndexingConfigurations)19 Test (org.junit.Test)18 WriterConfiguration (org.apache.metron.common.configuration.writer.WriterConfiguration)16 JSONObject (org.json.simple.JSONObject)14 WriterToBulkWriter (org.apache.metron.writer.WriterToBulkWriter)3 ArrayList (java.util.ArrayList)2 File (java.io.File)1 FileInputStream (java.io.FileInputStream)1 HashMap (java.util.HashMap)1 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)1 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)1 Tuple (org.apache.storm.tuple.Tuple)1