Search in sources :

Example 26 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class TestJoinBolt method testThreeStreamMixedJoin.

@Test
public void testThreeStreamMixedJoin() throws Exception {
    ArrayList<Tuple> userStream = makeStream("users", userFields, users);
    ArrayList<Tuple> storesStream = makeStream("stores", storeFields, stores);
    ArrayList<Tuple> cityStream = makeStream("cities", cityFields, cities);
    TupleWindow window = makeTupleWindow(userStream, cityStream, storesStream);
    JoinBolt bolt = new JoinBolt(JoinBolt.Selector.STREAM, "users", userFields[2]).join("stores", "city", "users").leftJoin("cities", "cityName", "users").select("name,storeName,city,country");
    MockCollector collector = new MockCollector();
    bolt.prepare(null, null, collector);
    bolt.execute(window);
    printResults(collector);
    // stores.length+1 as 2 users in Bengaluru
    Assert.assertEquals(stores.length + 1, collector.actualResults.size());
}
Also used : TupleWindow(org.apache.storm.windowing.TupleWindow) Tuple(org.apache.storm.tuple.Tuple) Test(org.junit.Test)

Example 27 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class ShellBolt method handleEmit.

private void handleEmit(ShellMsg shellMsg) throws InterruptedException {
    List<Tuple> anchors = new ArrayList<>();
    List<String> recvAnchors = shellMsg.getAnchors();
    if (recvAnchors != null) {
        for (String anchor : recvAnchors) {
            Tuple t = _inputs.get(anchor);
            if (t == null) {
                throw new RuntimeException("Anchored onto " + anchor + " after ack/fail");
            }
            anchors.add(t);
        }
    }
    if (shellMsg.getTask() == 0) {
        List<Integer> outtasks = _collector.emit(shellMsg.getStream(), anchors, shellMsg.getTuple());
        if (shellMsg.areTaskIdsNeeded()) {
            _pendingWrites.putTaskIds(outtasks);
        }
    } else {
        _collector.emitDirect((int) shellMsg.getTask(), shellMsg.getStream(), anchors, shellMsg.getTuple());
    }
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 28 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class ShellBolt method handleFail.

private void handleFail(Object id) {
    Tuple failed = _inputs.remove(id);
    if (failed == null) {
        throw new RuntimeException("Failed a non-existent or already acked/failed id: " + id);
    }
    _collector.fail(failed);
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 29 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class ShellBolt method handleAck.

private void handleAck(Object id) {
    Tuple acked = _inputs.remove(id);
    if (acked == null) {
        throw new RuntimeException("Acked a non-existent or already acked/failed id: " + id);
    }
    _collector.ack(acked);
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Example 30 with Tuple

use of org.apache.storm.tuple.Tuple in project storm by apache.

the class BatchHelper method fail.

public void fail(Exception e) {
    collector.reportError(e);
    for (Tuple t : tupleBatch) {
        collector.fail(t);
    }
    tupleBatch.clear();
    forceFlush = false;
}
Also used : Tuple(org.apache.storm.tuple.Tuple)

Aggregations

Tuple (org.apache.storm.tuple.Tuple)85 Test (org.junit.Test)30 Fields (org.apache.storm.tuple.Fields)13 OutputCollector (org.apache.storm.task.OutputCollector)11 Values (org.apache.storm.tuple.Values)11 ArrayList (java.util.ArrayList)10 HiveOptions (org.apache.storm.hive.common.HiveOptions)10 TupleWindow (org.apache.storm.windowing.TupleWindow)9 HashMap (java.util.HashMap)7 Test (org.testng.annotations.Test)7 GlobalStreamId (org.apache.storm.generated.GlobalStreamId)6 DelimitedRecordHiveMapper (org.apache.storm.hive.bolt.mapper.DelimitedRecordHiveMapper)6 HashSet (java.util.HashSet)5 JsonRecordHiveMapper (org.apache.storm.hive.bolt.mapper.JsonRecordHiveMapper)5 TopologyContext (org.apache.storm.task.TopologyContext)5 TupleImpl (org.apache.storm.tuple.TupleImpl)5 BasicOutputCollector (org.apache.storm.topology.BasicOutputCollector)4 Map (java.util.Map)3 Callback (org.apache.kafka.clients.producer.Callback)3 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)3