Search in sources :

Example 46 with TridentTuple

use of org.apache.storm.trident.tuple.TridentTuple in project storm by apache.

the class TridentKafkaState method updateState.

public void updateState(List<TridentTuple> tuples, TridentCollector collector) {
    String topic = null;
    try {
        List<Future<RecordMetadata>> futures = new ArrayList<>(tuples.size());
        for (TridentTuple tuple : tuples) {
            topic = topicSelector.getTopic(tuple);
            if (topic != null) {
                Future<RecordMetadata> result = producer.send(new ProducerRecord(topic, mapper.getKeyFromTuple(tuple), mapper.getMessageFromTuple(tuple)));
                futures.add(result);
            } else {
                LOG.warn("skipping key = " + mapper.getKeyFromTuple(tuple) + ", topic selector returned null.");
            }
        }
        List<ExecutionException> exceptions = new ArrayList<>(futures.size());
        for (Future<RecordMetadata> future : futures) {
            try {
                future.get();
            } catch (ExecutionException e) {
                exceptions.add(e);
            }
        }
        if (exceptions.size() > 0) {
            String errorMsg = "Could not retrieve result for messages " + tuples + " from topic = " + topic + " because of the following exceptions: \n";
            for (ExecutionException exception : exceptions) {
                errorMsg = errorMsg + exception.getMessage() + "\n";
            }
            LOG.error(errorMsg);
            throw new FailedException(errorMsg);
        }
    } catch (Exception ex) {
        String errorMsg = "Could not send messages " + tuples + " to topic = " + topic;
        LOG.warn(errorMsg, ex);
        throw new FailedException(errorMsg, ex);
    }
}
Also used : RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) FailedException(org.apache.storm.topology.FailedException) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) ArrayList(java.util.ArrayList) Future(java.util.concurrent.Future) ExecutionException(java.util.concurrent.ExecutionException) FailedException(org.apache.storm.topology.FailedException) ExecutionException(java.util.concurrent.ExecutionException) TridentTuple(org.apache.storm.trident.tuple.TridentTuple)

Aggregations

TridentTuple (org.apache.storm.trident.tuple.TridentTuple)46 ArrayList (java.util.ArrayList)18 FailedException (org.apache.storm.topology.FailedException)11 List (java.util.List)10 Values (org.apache.storm.tuple.Values)8 ISqlTridentDataSource (org.apache.storm.sql.runtime.ISqlTridentDataSource)6 Test (org.junit.Test)6 HashMap (java.util.HashMap)5 TridentTopology (org.apache.storm.trident.TridentTopology)5 Consumer (org.apache.storm.trident.operation.Consumer)5 StateUpdater (org.apache.storm.trident.state.StateUpdater)5 Stream (org.apache.storm.trident.Stream)4 Fields (org.apache.storm.tuple.Fields)4 Map (java.util.Map)3 FixedBatchSpout (org.apache.storm.trident.testing.FixedBatchSpout)3 BatchStatement (com.datastax.driver.core.BatchStatement)2 Statement (com.datastax.driver.core.Statement)2 IOException (java.io.IOException)2 Future (java.util.concurrent.Future)2 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)2