Examples with Traverser com.hazelcast.jet.Traverser used on opensource projects

Example 16 with Traverser

use of com.hazelcast.jet.Traverser in project hazelcast-jet by hazelcast.

the class TransformPipe method buildDAG.

@Override
public Vertex buildDAG(DAG dag) {
    Vertex previous = upstream.buildDAG(dag);
    // the lambda below must not capture `this`, therefore the instance variable
    // must first be loaded into a local variable
    DistributedFunction<Traverser<E_IN>, Traverser<E_OUT>> transformer = this.transformer;
    Vertex transform = dag.newVertex(uniqueVertexName("transform"), () -> new TransformP<>(transformer));
    if (upstream.isOrdered()) {
        transform.localParallelism(1);
    }
    dag.edge(between(previous, transform));
    return transform;
}

Example 17 with Traverser

use of com.hazelcast.jet.Traverser in project hazelcast-jet by hazelcast.

the class AppendableTraverserTest method test_flatMapperUsage.

@Test
public void test_flatMapperUsage() {
    // an instance of AppendableTraverser is repeatedly returned
    // from a flatMap function
    Traverser tt = Traverser.over(10, 20).flatMap(item -> {
        assertTrue(t.isEmpty());
        t.append(item);
        t.append(item + 1);
        return t;
    });
    assertEquals(10, tt.next());
    assertEquals(11, tt.next());
    assertEquals(20, tt.next());
    assertEquals(21, tt.next());
    assertNull(tt.next());
}

Example 18 with Traverser

use of com.hazelcast.jet.Traverser in project hazelcast-jet by hazelcast.

the class StreamKafkaP method saveToSnapshot.

@Override
public boolean saveToSnapshot() {
    if (!emitFromTraverser(traverser)) {
        return false;
    }
    if (snapshotTraverser == null) {
        Stream<Entry<BroadcastKey<TopicPartition>, long[]>> snapshotStream = offsets.entrySet().stream().flatMap(entry -> IntStream.range(0, entry.getValue().length).filter(partition -> entry.getValue()[partition] >= 0).mapToObj(partition -> {
            TopicPartition key = new TopicPartition(entry.getKey(), partition);
            long offset = entry.getValue()[partition];
            long watermark = watermarkSourceUtil.getWatermark(currentAssignment.get(key));
            return entry(broadcastKey(key), new long[] { offset, watermark });
        }));
        snapshotTraverser = traverseStream(snapshotStream).onFirstNull(() -> snapshotTraverser = null);
    }
    return emitFromTraverserToSnapshot(snapshotTraverser);
}