Search in sources :

Example 11 with IndexingConfigurations

use of org.apache.metron.common.configuration.IndexingConfigurations in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathNonString.

@Test(expected = IllegalArgumentException.class)
public void testGetHdfsPathNonString() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    writer.getHdfsPathExtension(SENSOR_NAME, "{'key':'value'}", message);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 12 with IndexingConfigurations

use of org.apache.metron.common.configuration.IndexingConfigurations in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathFormatConstant.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathFormatConstant() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "FORMAT('/test/folder/')", message);
    writer.close();
    Assert.assertEquals("/test/folder/", result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 13 with IndexingConfigurations

use of org.apache.metron.common.configuration.IndexingConfigurations in project metron by apache.

the class HdfsWriterTest method testGetHdfsPathConstant.

@Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathConstant() {
    WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
    HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
    writer.init(new HashMap<String, String>(), createTopologyContext(), config);
    JSONObject message = new JSONObject();
    Object result = writer.getHdfsPathExtension(SENSOR_NAME, "'new'", message);
    writer.close();
    Assert.assertEquals("new", result);
}
Also used : JSONObject(org.json.simple.JSONObject) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) IndexingWriterConfiguration(org.apache.metron.common.configuration.writer.IndexingWriterConfiguration) JSONObject(org.json.simple.JSONObject) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 14 with IndexingConfigurations

use of org.apache.metron.common.configuration.IndexingConfigurations in project metron by apache.

the class SensorIndexingConfigServiceImplTest method getAllIndicesWithOnlyParsers.

@Test
public void getAllIndicesWithOnlyParsers() throws RestException {
    ParserConfigurations parserConfiguration = mock(ParserConfigurations.class);
    when(parserConfiguration.getTypes()).thenReturn(ImmutableList.of("bro", "snort"));
    IndexingConfigurations indexingConfiguration = mock(IndexingConfigurations.class);
    when(indexingConfiguration.getTypes()).thenReturn(Collections.emptyList());
    when(indexingConfiguration.getIndex(eq("bro"), eq("elasticsearch"))).thenReturn(null);
    when(indexingConfiguration.getIndex(eq("snort"), eq("elasticsearch"))).thenReturn(null);
    when(indexingConfiguration.isEnabled(eq("snort"), eq("elasticsearch"))).thenReturn(true);
    when(indexingConfiguration.isEnabled(eq("bro"), eq("elasticsearch"))).thenReturn(true);
    when(cache.get(eq(ParserConfigurations.class))).thenReturn(parserConfiguration);
    when(cache.get(eq(IndexingConfigurations.class))).thenReturn(indexingConfiguration);
    List<String> indices = new ArrayList<String>();
    Iterables.addAll(indices, sensorIndexingConfigService.getAllIndices("elasticsearch"));
    Assert.assertEquals(2, indices.size());
    Assert.assertTrue(indices.contains("bro"));
    Assert.assertTrue(indices.contains("snort"));
}
Also used : ParserConfigurations(org.apache.metron.common.configuration.ParserConfigurations) ArrayList(java.util.ArrayList) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Example 15 with IndexingConfigurations

use of org.apache.metron.common.configuration.IndexingConfigurations in project metron by apache.

the class SensorIndexingConfigServiceImplTest method getAllIndicesWithOnlyIndexing.

@Test
public void getAllIndicesWithOnlyIndexing() throws RestException {
    ParserConfigurations parserConfiguration = mock(ParserConfigurations.class);
    when(parserConfiguration.getTypes()).thenReturn(Collections.emptyList());
    IndexingConfigurations indexingConfiguration = mock(IndexingConfigurations.class);
    // rename bro, include snort by default configs, and disable yaf
    when(indexingConfiguration.getTypes()).thenReturn(ImmutableList.of("bro", "snort", "yaf"));
    when(indexingConfiguration.getIndex(eq("bro"), eq("elasticsearch"))).thenReturn("renamed_bro");
    when(indexingConfiguration.getIndex(eq("snort"), eq("elasticsearch"))).thenReturn(null);
    when(indexingConfiguration.isEnabled(eq("snort"), eq("elasticsearch"))).thenReturn(true);
    when(indexingConfiguration.isEnabled(eq("bro"), eq("elasticsearch"))).thenReturn(true);
    when(indexingConfiguration.isEnabled(eq("yaf"), eq("elasticsearch"))).thenReturn(false);
    when(cache.get(eq(ParserConfigurations.class))).thenReturn(parserConfiguration);
    when(cache.get(eq(IndexingConfigurations.class))).thenReturn(indexingConfiguration);
    List<String> indices = new ArrayList<String>();
    Iterables.addAll(indices, sensorIndexingConfigService.getAllIndices("elasticsearch"));
    Assert.assertEquals(2, indices.size());
    Assert.assertTrue(indices.contains("renamed_bro"));
    Assert.assertTrue(indices.contains("snort"));
}
Also used : ParserConfigurations(org.apache.metron.common.configuration.ParserConfigurations) ArrayList(java.util.ArrayList) IndexingConfigurations(org.apache.metron.common.configuration.IndexingConfigurations) Test(org.junit.Test)

Aggregations

IndexingConfigurations (org.apache.metron.common.configuration.IndexingConfigurations)27 Test (org.junit.Test)25 IndexingWriterConfiguration (org.apache.metron.common.configuration.writer.IndexingWriterConfiguration)19 WriterConfiguration (org.apache.metron.common.configuration.writer.WriterConfiguration)13 JSONObject (org.json.simple.JSONObject)12 ArrayList (java.util.ArrayList)7 ParserConfigurations (org.apache.metron.common.configuration.ParserConfigurations)5 HashMap (java.util.HashMap)2 File (java.io.File)1 FileInputStream (java.io.FileInputStream)1 HashSet (java.util.HashSet)1 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)1 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)1 Tuple (org.apache.storm.tuple.Tuple)1