Search in sources :

Example 31 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class KafkaWriterTest method testHappyPathGlobalConfig.

@Test
public void testHappyPathGlobalConfig() throws Exception {
    KafkaWriter writer = new KafkaWriter();
    WriterConfiguration configuration = createConfiguration(new HashMap<String, Object>() {

        {
            put("kafka.brokerUrl", "localhost:6667");
            put("kafka.topic", SENSOR_TYPE);
            put("kafka.producerConfigs", ImmutableMap.of("key1", 1, "key2", "value2"));
        }
    });
    writer.configure(SENSOR_TYPE, configuration);
    Map<String, Object> producerConfigs = writer.createProducerConfigs();
    Assert.assertEquals(producerConfigs.get("bootstrap.servers"), "localhost:6667");
    Assert.assertEquals(producerConfigs.get("key.serializer"), "org.apache.kafka.common.serialization.StringSerializer");
    Assert.assertEquals(producerConfigs.get("value.serializer"), "org.apache.kafka.common.serialization.StringSerializer");
    Assert.assertEquals(producerConfigs.get("request.required.acks"), 1);
    Assert.assertEquals(producerConfigs.get("key1"), 1);
    Assert.assertEquals(producerConfigs.get("key2"), "value2");
}
Also used : WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) ParserWriterConfiguration(org.apache.metron.common.configuration.writer.ParserWriterConfiguration) Test(org.junit.Test)

Example 32 with WriterConfiguration

use of org.apache.metron.common.configuration.writer.WriterConfiguration in project metron by apache.

the class KafkaWriterTest method testHappyPathGlobalConfigWithPrefix.

@Test
public void testHappyPathGlobalConfigWithPrefix() throws Exception {
    KafkaWriter writer = new KafkaWriter();
    writer.withConfigPrefix("prefix");
    WriterConfiguration configuration = createConfiguration(new HashMap<String, Object>() {

        {
            put("prefix.kafka.brokerUrl", "localhost:6667");
            put("prefix.kafka.topic", SENSOR_TYPE);
            put("prefix.kafka.producerConfigs", ImmutableMap.of("key1", 1, "key2", "value2"));
        }
    });
    writer.configure(SENSOR_TYPE, configuration);
    Map<String, Object> producerConfigs = writer.createProducerConfigs();
    Assert.assertEquals(producerConfigs.get("bootstrap.servers"), "localhost:6667");
    Assert.assertEquals(producerConfigs.get("key.serializer"), "org.apache.kafka.common.serialization.StringSerializer");
    Assert.assertEquals(producerConfigs.get("value.serializer"), "org.apache.kafka.common.serialization.StringSerializer");
    Assert.assertEquals(producerConfigs.get("request.required.acks"), 1);
    Assert.assertEquals(producerConfigs.get("key1"), 1);
    Assert.assertEquals(producerConfigs.get("key2"), "value2");
}
Also used : WriterConfiguration(org.apache.metron.common.configuration.writer.WriterConfiguration) ParserWriterConfiguration(org.apache.metron.common.configuration.writer.ParserWriterConfiguration) Test(org.junit.Test)

Aggregations

WriterConfiguration (org.apache.metron.common.configuration.writer.WriterConfiguration)32 Test (org.junit.Test)29 JSONObject (org.json.simple.JSONObject)26 IndexingWriterConfiguration (org.apache.metron.common.configuration.writer.IndexingWriterConfiguration)21 IndexingConfigurations (org.apache.metron.common.configuration.IndexingConfigurations)13 SimpleHbaseEnrichmentWriter (org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter)9 File (java.io.File)6 ArrayList (java.util.ArrayList)6 Tuple (org.apache.storm.tuple.Tuple)6 DefaultFileNameFormat (org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat)4 FileNameFormat (org.apache.storm.hdfs.bolt.format.FileNameFormat)4 LookupKV (org.apache.metron.enrichment.lookup.LookupKV)3 WriterToBulkWriter (org.apache.metron.writer.WriterToBulkWriter)3 ParserWriterConfiguration (org.apache.metron.common.configuration.writer.ParserWriterConfiguration)2 CountSyncPolicy (org.apache.storm.hdfs.bolt.sync.CountSyncPolicy)2