Search in sources :

Example 1 with ElasticsearchSinkFunction

use of org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction in project flink by apache.

the class ElasticsearchSinkExample method main.

public static void main(String[] args) throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStream<String> source = env.generateSequence(0, 20).map(new MapFunction<Long, String>() {

        @Override
        public String map(Long value) throws Exception {
            return "message #" + value;
        }
    });
    Map<String, String> userConfig = new HashMap<>();
    userConfig.put("cluster.name", "elasticsearch");
    // This instructs the sink to emit after every element, otherwise they would be buffered
    userConfig.put(ElasticsearchSink.CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, "1");
    List<TransportAddress> transports = new ArrayList<>();
    transports.add(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9300));
    source.addSink(new ElasticsearchSink<>(userConfig, transports, new ElasticsearchSinkFunction<String>() {

        @Override
        public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
        }
    }));
    env.execute("Elasticsearch Sink Example");
}
Also used : HashMap(java.util.HashMap) InetSocketTransportAddress(org.elasticsearch.common.transport.InetSocketTransportAddress) TransportAddress(org.elasticsearch.common.transport.TransportAddress) ArrayList(java.util.ArrayList) RequestIndexer(org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer) InetSocketTransportAddress(org.elasticsearch.common.transport.InetSocketTransportAddress) ElasticsearchSinkFunction(org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext)

Example 2 with ElasticsearchSinkFunction

use of org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction in project flink by apache.

the class ElasticsearchSinkExample method main.

public static void main(String[] args) throws Exception {
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStream<String> source = env.generateSequence(0, 20).map(new MapFunction<Long, String>() {

        @Override
        public String map(Long value) throws Exception {
            return "message #" + value;
        }
    });
    Map<String, String> userConfig = new HashMap<>();
    userConfig.put("cluster.name", "elasticsearch");
    // This instructs the sink to emit after every element, otherwise they would be buffered
    userConfig.put(ElasticsearchSink.CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, "1");
    List<InetSocketAddress> transports = new ArrayList<>();
    transports.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300));
    source.addSink(new ElasticsearchSink<>(userConfig, transports, new ElasticsearchSinkFunction<String>() {

        @Override
        public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
        }
    }));
    env.execute("Elasticsearch Sink Example");
}
Also used : HashMap(java.util.HashMap) InetSocketAddress(java.net.InetSocketAddress) ArrayList(java.util.ArrayList) RequestIndexer(org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer) ElasticsearchSinkFunction(org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext)

Aggregations

ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 RuntimeContext (org.apache.flink.api.common.functions.RuntimeContext)2 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)2 ElasticsearchSinkFunction (org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction)2 RequestIndexer (org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer)2 InetSocketAddress (java.net.InetSocketAddress)1 InetSocketTransportAddress (org.elasticsearch.common.transport.InetSocketTransportAddress)1 TransportAddress (org.elasticsearch.common.transport.TransportAddress)1