Search in sources :

Example 6 with TupleWindow

use of org.apache.heron.api.windowing.TupleWindow in project heron by twitter.

the class WindowedBoltExecutorTest method testExecuteWithTs.

@Test
public void testExecuteWithTs() throws Exception {
    long[] timestamps = { 603, 605, 607, 618, 626, 636 };
    for (long ts : timestamps) {
        executor.execute(getTuple("s1", new Fields("ts"), new Values(ts)));
    }
    // Thread.sleep(120);
    executor.waterMarkEventGenerator.run();
    // System.out.println(testWindowedBolt.tupleWindows);
    assertEquals(3, testWindowedBolt.tupleWindows.size());
    TupleWindow first = testWindowedBolt.tupleWindows.get(0);
    assertArrayEquals(new long[] { 603, 605, 607 }, new long[] { (long) first.get().get(0).getValue(0), (long) first.get().get(1).getValue(0), (long) first.get().get(2).getValue(0) });
    TupleWindow second = testWindowedBolt.tupleWindows.get(1);
    assertArrayEquals(new long[] { 603, 605, 607, 618 }, new long[] { (long) second.get().get(0).getValue(0), (long) second.get().get(1).getValue(0), (long) second.get().get(2).getValue(0), (long) second.get().get(3).getValue(0) });
    TupleWindow third = testWindowedBolt.tupleWindows.get(2);
    assertArrayEquals(new long[] { 618, 626 }, new long[] { (long) third.get().get(0).getValue(0), (long) third.get().get(1).getValue(0) });
}
Also used : Fields(org.apache.heron.api.tuple.Fields) Values(org.apache.heron.api.tuple.Values) TupleWindow(org.apache.heron.api.windowing.TupleWindow) Test(org.junit.Test)

Example 7 with TupleWindow

use of org.apache.heron.api.windowing.TupleWindow in project heron by twitter.

the class ReduceByKeyAndWindowOperator method execute.

@SuppressWarnings("unchecked")
@Override
public void execute(TupleWindow inputWindow) {
    Map<K, T> reduceMap = new HashMap<>();
    Map<K, Integer> windowCountMap = new HashMap<>();
    for (Tuple tuple : inputWindow.get()) {
        R tup = (R) tuple.getValue(0);
        addMap(reduceMap, windowCountMap, tup);
    }
    long startWindow;
    long endWindow;
    if (inputWindow.getStartTimestamp() == null) {
        startWindow = 0;
    } else {
        startWindow = inputWindow.getStartTimestamp();
    }
    if (inputWindow.getEndTimestamp() == null) {
        endWindow = 0;
    } else {
        endWindow = inputWindow.getEndTimestamp();
    }
    for (K key : reduceMap.keySet()) {
        Window window = new Window(startWindow, endWindow, windowCountMap.get(key));
        KeyedWindow<K> keyedWindow = new KeyedWindow<>(key, window);
        collector.emit(new Values(new KeyValue<>(keyedWindow, reduceMap.get(key))));
    }
}
Also used : Window(org.apache.heron.streamlet.Window) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) TupleWindow(org.apache.heron.api.windowing.TupleWindow) KeyValue(org.apache.heron.streamlet.KeyValue) HashMap(java.util.HashMap) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) Values(org.apache.heron.api.tuple.Values) Tuple(org.apache.heron.api.tuple.Tuple)

Example 8 with TupleWindow

use of org.apache.heron.api.windowing.TupleWindow in project heron by twitter.

the class GeneralReduceByKeyAndWindowOperatorTest method getTupleWindow.

private TupleWindow getTupleWindow(int nkeys, int count) {
    TopologyAPI.StreamId componentStreamId = TopologyAPI.StreamId.newBuilder().setComponentName("sourceComponent").setId("default").build();
    List<Tuple> tuples = new LinkedList<>();
    for (int i = 0; i < nkeys; i++) {
        for (int j = 0; j < count; ++j) {
            Tuple tuple = getTuple(componentStreamId, new Fields("a"), new Values(new KeyValue<>(String.valueOf(i), j)));
            tuples.add(tuple);
        }
    }
    TupleWindow tupleWindow = new TupleWindowImpl(tuples, new LinkedList<>(), new LinkedList<>(), startTime, endTime);
    return tupleWindow;
}
Also used : Fields(org.apache.heron.api.tuple.Fields) KeyValue(org.apache.heron.streamlet.KeyValue) TupleWindowImpl(org.apache.heron.api.windowing.TupleWindowImpl) Values(org.apache.heron.api.tuple.Values) TupleWindow(org.apache.heron.api.windowing.TupleWindow) Tuple(org.apache.heron.api.tuple.Tuple) LinkedList(java.util.LinkedList) TopologyAPI(org.apache.heron.api.generated.TopologyAPI)

Example 9 with TupleWindow

use of org.apache.heron.api.windowing.TupleWindow in project heron by twitter.

the class GeneralReduceByKeyAndWindowOperatorTest method testReduceByWindowOperator.

@Test
@SuppressWarnings({ "rawtypes", "unchecked" })
public void testReduceByWindowOperator() {
    GeneralReduceByKeyAndWindowOperator<KeyValue<String, Integer>, String, Integer> reduceOperator = getReduceByWindowOperator(12);
    TupleWindow tupleWindow = getTupleWindow(3, 5);
    HashMap<String, Integer> expectedResults = new HashMap<>();
    expectedResults.put("0", 22);
    expectedResults.put("1", 22);
    expectedResults.put("2", 22);
    reduceOperator.execute(tupleWindow);
    Assert.assertEquals(3, emittedTuples.size());
    for (Object object : emittedTuples) {
        KeyValue<KeyedWindow<String>, Integer> tuple = (KeyValue<KeyedWindow<String>, Integer>) object;
        KeyedWindow<String> window = tuple.getKey();
        String key = window.getKey();
        Assert.assertEquals(5, window.getWindow().getCount());
        Assert.assertEquals(startTime, window.getWindow().getStartTime());
        Assert.assertEquals(endTime, window.getWindow().getEndTime());
        Assert.assertEquals(expectedResults.get(key), tuple.getValue());
    }
}
Also used : KeyValue(org.apache.heron.streamlet.KeyValue) HashMap(java.util.HashMap) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) TupleWindow(org.apache.heron.api.windowing.TupleWindow) Test(org.junit.Test)

Example 10 with TupleWindow

use of org.apache.heron.api.windowing.TupleWindow in project heron by twitter.

the class JoinOperatorTest method testOuterRightJoinOperator.

@Test
@SuppressWarnings({ "rawtypes", "unchecked" })
public void testOuterRightJoinOperator() {
    JoinOperator<String, KeyValue<String, String>, KeyValue<String, String>, String> joinOperator = getJoinOperator(JoinType.OUTER_RIGHT);
    TupleWindow tupleWindow = getTupleWindow();
    Set<String> expectedResultsK1 = new HashSet<>();
    expectedResultsK1.add("01");
    expectedResultsK1.add("03");
    expectedResultsK1.add("21");
    expectedResultsK1.add("23");
    expectedResultsK1.add("41");
    expectedResultsK1.add("43");
    Set<String> expectedResultsK2 = new HashSet<>();
    expectedResultsK2.add("null8");
    expectedResultsK2.add("null9");
    expectedResultsK2.add("null10");
    expectedResultsK2.add("null11");
    joinOperator.execute(tupleWindow);
    Assert.assertEquals(10, emittedTuples.size());
    for (Object object : emittedTuples) {
        KeyValue<KeyedWindow<String>, String> tuple = (KeyValue<KeyedWindow<String>, String>) object;
        KeyedWindow<String> keyedWindow = tuple.getKey();
        switch(keyedWindow.getKey()) {
            case "key1":
                Assert.assertTrue(expectedResultsK1.contains(tuple.getValue()));
                expectedResultsK1.remove(tuple.getValue());
                break;
            case "key2":
                Assert.assertTrue(expectedResultsK2.contains(tuple.getValue()));
                expectedResultsK2.remove(tuple.getValue());
                break;
            case "key3":
                Assert.assertTrue(expectedResultsK2.contains(tuple.getValue()));
                expectedResultsK2.remove(tuple.getValue());
                break;
            default:
                Assert.fail();
        }
        Assert.assertEquals(12, keyedWindow.getWindow().getCount());
        Assert.assertEquals(startTime, keyedWindow.getWindow().getStartTime());
        Assert.assertEquals(endTime, keyedWindow.getWindow().getEndTime());
    }
    Assert.assertEquals(0, expectedResultsK1.size());
    Assert.assertEquals(0, expectedResultsK2.size());
}
Also used : KeyValue(org.apache.heron.streamlet.KeyValue) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) TupleWindow(org.apache.heron.api.windowing.TupleWindow) HashSet(java.util.HashSet) Test(org.junit.Test)

Aggregations

TupleWindow (org.apache.heron.api.windowing.TupleWindow)13 KeyValue (org.apache.heron.streamlet.KeyValue)10 KeyedWindow (org.apache.heron.streamlet.KeyedWindow)9 Test (org.junit.Test)7 Values (org.apache.heron.api.tuple.Values)6 Tuple (org.apache.heron.api.tuple.Tuple)5 HashMap (java.util.HashMap)4 HashSet (java.util.HashSet)4 Fields (org.apache.heron.api.tuple.Fields)4 LinkedList (java.util.LinkedList)3 TopologyAPI (org.apache.heron.api.generated.TopologyAPI)3 TupleWindowImpl (org.apache.heron.api.windowing.TupleWindowImpl)3 Window (org.apache.heron.streamlet.Window)3