Search in sources :

Example 1 with PipelineModel

use of com.amazon.dataprepper.model.configuration.PipelineModel in project data-prepper by opensearch-project.

the class LogstashMapper method mapPipeline.

public PipelineModel mapPipeline(LogstashConfiguration logstashConfiguration) {
    List<PluginModel> sourcePluginModels = mapPluginSection(logstashConfiguration, LogstashPluginType.INPUT);
    PluginModel sourcePlugin = null;
    if (sourcePluginModels.size() > 1)
        throw new LogstashMappingException("More than 1 source plugins are not supported");
    else if (sourcePluginModels.size() == 1)
        sourcePlugin = sourcePluginModels.get(0);
    List<PluginModel> prepperPluginModels = mapPluginSection(logstashConfiguration, LogstashPluginType.FILTER);
    List<PluginModel> sinkPluginModels = mapPluginSection(logstashConfiguration, LogstashPluginType.OUTPUT);
    return new PipelineModel(sourcePlugin, prepperPluginModels, sinkPluginModels, null, null);
}
Also used : PluginModel(com.amazon.dataprepper.model.configuration.PluginModel) LogstashMappingException(org.opensearch.dataprepper.logstash.exception.LogstashMappingException) PipelineModel(com.amazon.dataprepper.model.configuration.PipelineModel)

Example 2 with PipelineModel

use of com.amazon.dataprepper.model.configuration.PipelineModel in project data-prepper by opensearch-project.

the class LogstashMapperTest method mapPipeline_with_no_plugins_should_return_pipeline_model_without_plugins_Test.

@Test
void mapPipeline_with_no_plugins_should_return_pipeline_model_without_plugins_Test() {
    LogstashConfiguration logstashConfiguration = mock(LogstashConfiguration.class);
    when(logstashConfiguration.getPluginSection(LogstashPluginType.INPUT)).thenReturn(Collections.emptyList());
    when(logstashConfiguration.getPluginSection(LogstashPluginType.FILTER)).thenReturn(Collections.emptyList());
    when(logstashConfiguration.getPluginSection(LogstashPluginType.OUTPUT)).thenReturn(Collections.emptyList());
    PipelineModel actualPipelineModel = logstashMapper.mapPipeline(logstashConfiguration);
    assertThat(actualPipelineModel.getSource(), equalTo(null));
    assertThat(actualPipelineModel.getProcessors(), equalTo(Collections.emptyList()));
    assertThat(actualPipelineModel.getSinks(), equalTo(Collections.emptyList()));
    assertThat(actualPipelineModel.getReadBatchDelay(), equalTo(null));
    assertThat(actualPipelineModel.getWorkers(), equalTo(null));
}
Also used : LogstashConfiguration(org.opensearch.dataprepper.logstash.model.LogstashConfiguration) PipelineModel(com.amazon.dataprepper.model.configuration.PipelineModel) Test(org.junit.jupiter.api.Test)

Example 3 with PipelineModel

use of com.amazon.dataprepper.model.configuration.PipelineModel in project data-prepper by opensearch-project.

the class LogstashConfigConverter method convertLogstashConfigurationToPipeline.

public String convertLogstashConfigurationToPipeline(String logstashConfigurationPath, String outputDirectory) throws IOException {
    final Path configurationFilePath = Paths.get(logstashConfigurationPath);
    final String logstashConfigAsString = new String(Files.readAllBytes(configurationFilePath));
    LogstashLexer lexer = new LogstashLexer(CharStreams.fromString(logstashConfigAsString));
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    LogstashParser parser = new LogstashParser(tokens);
    final ParseTree tree = parser.config();
    ModelConvertingLogstashVisitor visitor = new ModelConvertingLogstashVisitor();
    LogstashConfiguration logstashConfiguration = (LogstashConfiguration) visitor.visit(tree);
    LogstashMapper logstashMapper = new LogstashMapper();
    PipelineModel pipelineModel = logstashMapper.mapPipeline(logstashConfiguration);
    PipelinesDataFlowModel pipelinesDataFlowModel = new PipelinesDataFlowModel(Collections.singletonMap("logstash-converted-pipeline", pipelineModel));
    ObjectMapper mapper = new ObjectMapper(YAMLFactory.builder().disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER).enable(YAMLGenerator.Feature.INDENT_ARRAYS_WITH_INDICATOR).enable(YAMLGenerator.Feature.USE_PLATFORM_LINE_BREAKS).disable(YAMLGenerator.Feature.SPLIT_LINES).enable(YAMLGenerator.Feature.LITERAL_BLOCK_STYLE).build());
    final String confFileName = configurationFilePath.getFileName().toString();
    final String yamlFileName = confFileName.substring(0, confFileName.lastIndexOf(".conf"));
    final Path yamlFilePath = Paths.get(outputDirectory, yamlFileName + ".yaml");
    mapper.writeValue(new File(String.valueOf(yamlFilePath)), pipelinesDataFlowModel);
    return String.valueOf(yamlFilePath);
}
Also used : Path(java.nio.file.Path) LogstashConfiguration(org.opensearch.dataprepper.logstash.model.LogstashConfiguration) LogstashMapper(org.opensearch.dataprepper.logstash.mapping.LogstashMapper) CommonTokenStream(org.antlr.v4.runtime.CommonTokenStream) PipelinesDataFlowModel(com.amazon.dataprepper.model.configuration.PipelinesDataFlowModel) PipelineModel(com.amazon.dataprepper.model.configuration.PipelineModel) ModelConvertingLogstashVisitor(org.opensearch.dataprepper.logstash.parser.ModelConvertingLogstashVisitor) File(java.io.File) ParseTree(org.antlr.v4.runtime.tree.ParseTree) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper)

Example 4 with PipelineModel

use of com.amazon.dataprepper.model.configuration.PipelineModel in project data-prepper by opensearch-project.

the class LogstashMapperTest method mapPipeline_returns_pipeline_model.

@Test
void mapPipeline_returns_pipeline_model() {
    LogstashConfiguration logstashConfiguration = mock(LogstashConfiguration.class);
    when(logstashConfiguration.getPluginSection(LogstashPluginType.INPUT)).thenReturn(Collections.singletonList(TestDataProvider.pluginData()));
    PipelineModel actualPipelineModel = logstashMapper.mapPipeline(logstashConfiguration);
    PipelineModel expectedPipelineModel = new PipelineModel(TestDataProvider.samplePluginModel(), null, null, null, null);
    assertThat(actualPipelineModel.getSource().getPluginName(), equalTo(expectedPipelineModel.getSource().getPluginName()));
    assertThat(actualPipelineModel.getSource().getPluginSettings(), equalTo(expectedPipelineModel.getSource().getPluginSettings()));
    assertThat(actualPipelineModel.getReadBatchDelay(), equalTo(expectedPipelineModel.getReadBatchDelay()));
    assertThat(actualPipelineModel.getWorkers(), equalTo(expectedPipelineModel.getWorkers()));
}
Also used : LogstashConfiguration(org.opensearch.dataprepper.logstash.model.LogstashConfiguration) PipelineModel(com.amazon.dataprepper.model.configuration.PipelineModel) Test(org.junit.jupiter.api.Test)

Aggregations

PipelineModel (com.amazon.dataprepper.model.configuration.PipelineModel)4 LogstashConfiguration (org.opensearch.dataprepper.logstash.model.LogstashConfiguration)3 Test (org.junit.jupiter.api.Test)2 PipelinesDataFlowModel (com.amazon.dataprepper.model.configuration.PipelinesDataFlowModel)1 PluginModel (com.amazon.dataprepper.model.configuration.PluginModel)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 File (java.io.File)1 Path (java.nio.file.Path)1 CommonTokenStream (org.antlr.v4.runtime.CommonTokenStream)1 ParseTree (org.antlr.v4.runtime.tree.ParseTree)1 LogstashMappingException (org.opensearch.dataprepper.logstash.exception.LogstashMappingException)1 LogstashMapper (org.opensearch.dataprepper.logstash.mapping.LogstashMapper)1 ModelConvertingLogstashVisitor (org.opensearch.dataprepper.logstash.parser.ModelConvertingLogstashVisitor)1