Search in sources :

Example 41 with Transaction

use of org.apache.flume.Transaction in project rocketmq-externals by apache.

the class RocketMQSourceTest method testEvent.

@Test
public void testEvent() throws EventDeliveryException, MQBrokerException, MQClientException, InterruptedException, UnsupportedEncodingException {
    // publish test message
    DefaultMQProducer producer = new DefaultMQProducer(producerGroup);
    producer.setNamesrvAddr(nameServer);
    String sendMsg = "\"Hello Flume\"" + "," + DateFormatUtils.format(new Date(), "yyyy-MM-DD hh:mm:ss");
    try {
        producer.start();
        Message msg = new Message(TOPIC_DEFAULT, tag, sendMsg.getBytes("UTF-8"));
        SendResult sendResult = producer.send(msg);
        log.info("publish message : {}, sendResult:{}", sendMsg, sendResult);
    } catch (Exception e) {
        throw new MQClientException("Failed to publish messages", e);
    } finally {
        producer.shutdown();
    }
    // start source
    Context context = new Context();
    context.put(NAME_SERVER_CONFIG, nameServer);
    context.put(TAG_CONFIG, tag);
    Channel channel = new MemoryChannel();
    Configurables.configure(channel, context);
    List<Channel> channels = new ArrayList<>();
    channels.add(channel);
    ChannelSelector channelSelector = new ReplicatingChannelSelector();
    channelSelector.setChannels(channels);
    ChannelProcessor channelProcessor = new ChannelProcessor(channelSelector);
    RocketMQSource source = new RocketMQSource();
    source.setChannelProcessor(channelProcessor);
    Configurables.configure(source, context);
    source.start();
    PollableSource.Status status = source.process();
    if (status == PollableSource.Status.BACKOFF) {
        fail("Error");
    }
    /*
        wait for processQueueTable init
         */
    Thread.sleep(1000);
    source.stop();
    /*
        mock flume sink
         */
    Transaction transaction = channel.getTransaction();
    transaction.begin();
    Event event = channel.take();
    if (event == null) {
        transaction.commit();
        fail("Error");
    }
    byte[] body = event.getBody();
    String receiveMsg = new String(body, "UTF-8");
    log.info("receive message : {}", receiveMsg);
    assertEquals(sendMsg, receiveMsg);
}
Also used : Context(org.apache.flume.Context) MemoryChannel(org.apache.flume.channel.MemoryChannel) Message(org.apache.rocketmq.common.message.Message) MemoryChannel(org.apache.flume.channel.MemoryChannel) Channel(org.apache.flume.Channel) ArrayList(java.util.ArrayList) ChannelProcessor(org.apache.flume.channel.ChannelProcessor) DefaultMQProducer(org.apache.rocketmq.client.producer.DefaultMQProducer) Date(java.util.Date) MQClientException(org.apache.rocketmq.client.exception.MQClientException) EventDeliveryException(org.apache.flume.EventDeliveryException) MQBrokerException(org.apache.rocketmq.client.exception.MQBrokerException) UnsupportedEncodingException(java.io.UnsupportedEncodingException) PollableSource(org.apache.flume.PollableSource) ReplicatingChannelSelector(org.apache.flume.channel.ReplicatingChannelSelector) Transaction(org.apache.flume.Transaction) SendResult(org.apache.rocketmq.client.producer.SendResult) Event(org.apache.flume.Event) ReplicatingChannelSelector(org.apache.flume.channel.ReplicatingChannelSelector) ChannelSelector(org.apache.flume.ChannelSelector) MQClientException(org.apache.rocketmq.client.exception.MQClientException) Test(org.junit.Test)

Aggregations

Transaction (org.apache.flume.Transaction)41 Event (org.apache.flume.Event)38 Test (org.junit.Test)34 Channel (org.apache.flume.Channel)29 MemoryChannel (org.apache.flume.channel.MemoryChannel)26 PhoenixSink (org.apache.phoenix.flume.sink.PhoenixSink)22 Context (org.apache.flume.Context)11 EventDeliveryException (org.apache.flume.EventDeliveryException)9 Connection (java.sql.Connection)5 ResultSet (java.sql.ResultSet)5 Properties (java.util.Properties)5 SQLException (java.sql.SQLException)4 ChannelException (org.apache.flume.ChannelException)4 ArrayList (java.util.ArrayList)3 Date (java.util.Date)3 Sink (org.apache.flume.Sink)3 IOException (java.io.IOException)2 UnsupportedEncodingException (java.io.UnsupportedEncodingException)2 CountDownLatch (java.util.concurrent.CountDownLatch)2 FlumeException (org.apache.flume.FlumeException)2