Search in sources :

Example 1 with IndexManager

use of com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager in project data-prepper by opensearch-project.

the class OpenSearchSink method initialize.

public void initialize() throws IOException {
    LOG.info("Initializing OpenSearch sink");
    restHighLevelClient = openSearchSinkConfig.getConnectionConfiguration().createClient();
    indexManager = indexManagerFactory.getIndexManager(indexType, restHighLevelClient, openSearchSinkConfig);
    final String dlqFile = openSearchSinkConfig.getRetryConfiguration().getDlqFile();
    if (dlqFile != null) {
        dlqWriter = Files.newBufferedWriter(Paths.get(dlqFile), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
    }
    indexManager.setupIndex();
    OpenSearchTransport transport = new RestClientTransport(restHighLevelClient.getLowLevelClient(), new PreSerializedJsonpMapper());
    openSearchClient = new OpenSearchClient(transport);
    bulkRequestSupplier = () -> new JavaClientAccumulatingBulkRequest(new BulkRequest.Builder().index(indexManager.getIndexAlias()));
    bulkRetryStrategy = new BulkRetryStrategy(bulkRequest -> openSearchClient.bulk(bulkRequest.getRequest()), this::logFailure, pluginMetrics, bulkRequestSupplier);
    LOG.info("Initialized OpenSearch sink");
    objectMapper = new ObjectMapper();
}
Also used : IndexManager(com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager) LoggerFactory(org.slf4j.LoggerFactory) ByteSizeUnit(org.opensearch.common.unit.ByteSizeUnit) PreSerializedJsonpMapper(com.amazon.dataprepper.plugins.sink.opensearch.bulk.PreSerializedJsonpMapper) IndexType(com.amazon.dataprepper.plugins.sink.opensearch.index.IndexType) Supplier(java.util.function.Supplier) AbstractSink(com.amazon.dataprepper.model.sink.AbstractSink) BulkRequest(org.opensearch.client.opensearch.core.BulkRequest) BulkOperation(org.opensearch.client.opensearch.core.bulk.BulkOperation) XContentParser(org.opensearch.common.xcontent.XContentParser) Timer(io.micrometer.core.instrument.Timer) IndexManagerFactory(com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManagerFactory) RestClientTransport(org.opensearch.client.transport.rest_client.RestClientTransport) BulkOperationWriter(com.amazon.dataprepper.plugins.sink.opensearch.bulk.BulkOperationWriter) Map(java.util.Map) XContentFactory(org.opensearch.common.xcontent.XContentFactory) Record(com.amazon.dataprepper.model.record.Record) RestHighLevelClient(org.opensearch.client.RestHighLevelClient) Counter(io.micrometer.core.instrument.Counter) PluginSetting(com.amazon.dataprepper.model.configuration.PluginSetting) AccumulatingBulkRequest(com.amazon.dataprepper.plugins.sink.opensearch.bulk.AccumulatingBulkRequest) JavaClientAccumulatingBulkRequest(com.amazon.dataprepper.plugins.sink.opensearch.bulk.JavaClientAccumulatingBulkRequest) Logger(org.slf4j.Logger) Sink(com.amazon.dataprepper.model.sink.Sink) Files(java.nio.file.Files) BufferedWriter(java.io.BufferedWriter) IndexOperation(org.opensearch.client.opensearch.core.bulk.IndexOperation) Collection(java.util.Collection) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) LoggingDeprecationHandler(org.opensearch.common.xcontent.LoggingDeprecationHandler) StandardOpenOption(java.nio.file.StandardOpenOption) SerializedJson(com.amazon.dataprepper.plugins.sink.opensearch.bulk.SerializedJson) OpenSearchClient(org.opensearch.client.opensearch.OpenSearchClient) IOException(java.io.IOException) OpenSearchTransport(org.opensearch.client.transport.OpenSearchTransport) Event(com.amazon.dataprepper.model.event.Event) DistributionSummary(io.micrometer.core.instrument.DistributionSummary) Paths(java.nio.file.Paths) NamedXContentRegistry(org.opensearch.common.xcontent.NamedXContentRegistry) XContentType(org.opensearch.common.xcontent.XContentType) DataPrepperPlugin(com.amazon.dataprepper.model.annotations.DataPrepperPlugin) RestClientTransport(org.opensearch.client.transport.rest_client.RestClientTransport) JavaClientAccumulatingBulkRequest(com.amazon.dataprepper.plugins.sink.opensearch.bulk.JavaClientAccumulatingBulkRequest) BulkRequest(org.opensearch.client.opensearch.core.BulkRequest) AccumulatingBulkRequest(com.amazon.dataprepper.plugins.sink.opensearch.bulk.AccumulatingBulkRequest) JavaClientAccumulatingBulkRequest(com.amazon.dataprepper.plugins.sink.opensearch.bulk.JavaClientAccumulatingBulkRequest) PreSerializedJsonpMapper(com.amazon.dataprepper.plugins.sink.opensearch.bulk.PreSerializedJsonpMapper) OpenSearchClient(org.opensearch.client.opensearch.OpenSearchClient) OpenSearchTransport(org.opensearch.client.transport.OpenSearchTransport) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper)

Aggregations

DataPrepperPlugin (com.amazon.dataprepper.model.annotations.DataPrepperPlugin)1 PluginSetting (com.amazon.dataprepper.model.configuration.PluginSetting)1 Event (com.amazon.dataprepper.model.event.Event)1 Record (com.amazon.dataprepper.model.record.Record)1 AbstractSink (com.amazon.dataprepper.model.sink.AbstractSink)1 Sink (com.amazon.dataprepper.model.sink.Sink)1 AccumulatingBulkRequest (com.amazon.dataprepper.plugins.sink.opensearch.bulk.AccumulatingBulkRequest)1 BulkOperationWriter (com.amazon.dataprepper.plugins.sink.opensearch.bulk.BulkOperationWriter)1 JavaClientAccumulatingBulkRequest (com.amazon.dataprepper.plugins.sink.opensearch.bulk.JavaClientAccumulatingBulkRequest)1 PreSerializedJsonpMapper (com.amazon.dataprepper.plugins.sink.opensearch.bulk.PreSerializedJsonpMapper)1 SerializedJson (com.amazon.dataprepper.plugins.sink.opensearch.bulk.SerializedJson)1 IndexManager (com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager)1 IndexManagerFactory (com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManagerFactory)1 IndexType (com.amazon.dataprepper.plugins.sink.opensearch.index.IndexType)1 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 Counter (io.micrometer.core.instrument.Counter)1 DistributionSummary (io.micrometer.core.instrument.DistributionSummary)1 Timer (io.micrometer.core.instrument.Timer)1 BufferedWriter (java.io.BufferedWriter)1 IOException (java.io.IOException)1