Examples with ParserWriterConfiguration org.apache.metron.common.configuration.writer.ParserWriterConfiguration used on opensource projects

Example 1 with ParserWriterConfiguration

use of org.apache.metron.common.configuration.writer.ParserWriterConfiguration in project metron by apache.

the class ParserWriterConfigurationTest method testDefaultIndex.

@Test
public void testDefaultIndex() {
    ParserWriterConfiguration config = new ParserWriterConfiguration(new ParserConfigurations());
    Assert.assertEquals("foo", config.getIndex("foo"));
}

Example 2 with ParserWriterConfiguration

use of org.apache.metron.common.configuration.writer.ParserWriterConfiguration in project metron by apache.

the class ParserTopologyBuilder method createErrorBolt.

/**
 * Create a bolt that handles error messages.
 *
 * @param zookeeperUrl    Kafka zookeeper URL
 * @param brokerUrl    Kafka Broker URL
 * @param sensorType   Type of sensor that is being consumed.
 * @param securityProtocol   Security protocol used (if any)
 * @param configs
 * @param parserConfig
 * @return A Storm bolt that handles error messages.
 */
private static WriterBolt createErrorBolt(String zookeeperUrl, Optional<String> brokerUrl, String sensorType, Optional<String> securityProtocol, ParserConfigurations configs, SensorParserConfig parserConfig) {
    // create writer - if not configured uses a sensible default
    AbstractWriter writer = parserConfig.getErrorWriterClassName() == null ? createKafkaWriter(brokerUrl, zookeeperUrl, securityProtocol).withTopic((String) configs.getGlobalConfig().get("parser.error.topic")).withConfigPrefix("error") : ReflectionUtils.createInstance(parserConfig.getWriterClassName());
    writer.configure(sensorType, new ParserWriterConfiguration(configs));
    // create a writer handler
    WriterHandler writerHandler = createWriterHandler(writer);
    return new WriterBolt(writerHandler, configs, sensorType).withErrorType(Constants.ErrorType.PARSER_ERROR);
}

Example 3 with ParserWriterConfiguration

use of org.apache.metron.common.configuration.writer.ParserWriterConfiguration in project metron by apache.

the class ParserWriterConfigurationTest method testDefaultBatchSize.

@Test
public void testDefaultBatchSize() {
    ParserWriterConfiguration config = new ParserWriterConfiguration(new ParserConfigurations());
    Assert.assertEquals(1, config.getBatchSize("foo"));
}

Example 4 with ParserWriterConfiguration

use of org.apache.metron.common.configuration.writer.ParserWriterConfiguration in project metron by apache.

the class KafkaWriterTest method createConfiguration.

public WriterConfiguration createConfiguration(final Map<String, Object> parserConfig) {
    ParserConfigurations configurations = new ParserConfigurations();
    configurations.updateSensorParserConfig(SENSOR_TYPE, new SensorParserConfig() {

        {
            setParserConfig(parserConfig);
        }
    });
    return new ParserWriterConfiguration(configurations);
}

Example 5 with ParserWriterConfiguration

use of org.apache.metron.common.configuration.writer.ParserWriterConfiguration in project metron by apache.

the class ParserTopologyBuilder method createParserBolt.

/**
 * Create a bolt that parses input from a sensor.
 *
 * @param zookeeperUrl Zookeeper URL
 * @param brokerUrl    Kafka Broker URL
 * @param sensorType   Type of sensor that is being consumed.
 * @param configs
 * @param parserConfig
 * @return A Storm bolt that parses input from a sensor
 */
private static ParserBolt createParserBolt(String zookeeperUrl, Optional<String> brokerUrl, String sensorType, Optional<String> securityProtocol, ParserConfigurations configs, SensorParserConfig parserConfig, Optional<String> outputTopic) {
    // create message parser
    MessageParser<JSONObject> parser = ReflectionUtils.createInstance(parserConfig.getParserClassName());
    parser.configure(parserConfig.getParserConfig());
    // create writer - if not configured uses a sensible default
    AbstractWriter writer = parserConfig.getWriterClassName() == null ? createKafkaWriter(brokerUrl, zookeeperUrl, securityProtocol).withTopic(outputTopic.orElse(Constants.ENRICHMENT_TOPIC)) : ReflectionUtils.createInstance(parserConfig.getWriterClassName());
    writer.configure(sensorType, new ParserWriterConfiguration(configs));
    // create a writer handler
    WriterHandler writerHandler = createWriterHandler(writer);
    return new ParserBolt(zookeeperUrl, sensorType, parser, writerHandler);
}