Search in sources :

Example 11 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class TestHiveBolt method testJsonWriter.

@Test
public void testJsonWriter() throws Exception {
    // json record doesn't need columns to be in the same order
    // as table in hive.
    JsonRecordHiveMapper mapper = new JsonRecordHiveMapper().withColumnFields(new Fields(colNames1)).withPartitionFields(new Fields(partNames));
    HiveOptions hiveOptions = new HiveOptions(metaStoreURI, dbName, tblName, mapper).withTxnsPerBatch(2).withBatchSize(1);
    bolt = new HiveBolt(hiveOptions);
    bolt.prepare(config, null, collector);
    Tuple tuple1 = generateTestTuple(1, "SJC", "Sunnyvale", "CA");
    //Tuple tuple2 = generateTestTuple(2,"SFO","San Jose","CA");
    bolt.execute(tuple1);
    verify(collector).ack(tuple1);
    //bolt.execute(tuple2);
    //verify(collector).ack(tuple2);
    checkDataWritten(tblName, dbName, "1,SJC,Sunnyvale,CA");
    bolt.cleanup();
}
Also used : JsonRecordHiveMapper(org.apache.storm.hive.bolt.mapper.JsonRecordHiveMapper) Fields(org.apache.storm.tuple.Fields) HiveOptions(org.apache.storm.hive.common.HiveOptions) Tuple(org.apache.storm.tuple.Tuple) Test(org.junit.Test)

Example 12 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class KafkaBoltTest method executeWithBrokerDown.

@Test
public void executeWithBrokerDown() throws Exception {
    broker.shutdown();
    String message = "value-234";
    Tuple tuple = generateTestTuple(message);
    bolt.execute(tuple);
    verify(collector).fail(tuple);
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 13 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class KafkaBoltTest method executeWithKey.

@Test
public void executeWithKey() throws Exception {
    String message = "value-123";
    String key = "key-123";
    Tuple tuple = generateTestTuple(key, message);
    bolt.execute(tuple);
    verify(collector).ack(tuple);
    verifyMessage(key, message);
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 14 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class KafkaBoltTest method mockTickTuple.

private Tuple mockTickTuple() {
    Tuple tuple = mock(Tuple.class);
    when(tuple.getSourceComponent()).thenReturn(Constants.SYSTEM_COMPONENT_ID);
    when(tuple.getSourceStreamId()).thenReturn(Constants.SYSTEM_TICK_STREAM_ID);
    // Sanity check
    assertTrue(TupleUtils.isTick(tuple));
    return tuple;
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 15 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class KafkaBoltTest method shouldNotAcknowledgeTickTuples.

@Test
public void shouldNotAcknowledgeTickTuples() throws Exception {
    // Given
    Tuple tickTuple = mockTickTuple();
    // When
    bolt.execute(tickTuple);
    // Then
    verify(collector, never()).ack(tickTuple);
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Aggregations

Tuple (org.apache.storm.tuple.Tuple)85 Test (org.junit.Test)30 Fields (org.apache.storm.tuple.Fields)13 OutputCollector (org.apache.storm.task.OutputCollector)11 Values (org.apache.storm.tuple.Values)11 ArrayList (java.util.ArrayList)10 HiveOptions (org.apache.storm.hive.common.HiveOptions)10 TupleWindow (org.apache.storm.windowing.TupleWindow)9 HashMap (java.util.HashMap)7 Test (org.testng.annotations.Test)7 GlobalStreamId (org.apache.storm.generated.GlobalStreamId)6 DelimitedRecordHiveMapper (org.apache.storm.hive.bolt.mapper.DelimitedRecordHiveMapper)6 HashSet (java.util.HashSet)5 JsonRecordHiveMapper (org.apache.storm.hive.bolt.mapper.JsonRecordHiveMapper)5 TopologyContext (org.apache.storm.task.TopologyContext)5 TupleImpl (org.apache.storm.tuple.TupleImpl)5 BasicOutputCollector (org.apache.storm.topology.BasicOutputCollector)4 Map (java.util.Map)3 Callback (org.apache.kafka.clients.producer.Callback)3 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)3