Examples with BroadcastEntry com.hazelcast.jet.impl.execution.BroadcastEntry used on opensource projects

Example 1 with BroadcastEntry

use of com.hazelcast.jet.impl.execution.BroadcastEntry in project hazelcast-jet by hazelcast.

the class MasterContext method rewriteDagWithSnapshotRestore.

private void rewriteDagWithSnapshotRestore(DAG dag, long snapshotId) {
    logger.info(jobIdString() + ": restoring state from snapshotId=" + snapshotId);
    for (Vertex vertex : dag) {
        // items with keys of type BroadcastKey need to be broadcast to all processors
        DistributedFunction<Entry<Object, Object>, ?> projection = (Entry<Object, Object> e) -> (e.getKey() instanceof BroadcastKey) ? new BroadcastEntry<>(e) : e;
        // We add the vertex even in case when the map is empty: this ensures, that
        // Processor.finishSnapshotRestore() method is always called on all vertices in
        // a job which is restored from a snapshot.
        String mapName = snapshotDataMapName(jobId, snapshotId, vertex.getName());
        Vertex readSnapshotVertex = dag.newVertex("__snapshot_read." + vertex.getName(), readMapP(mapName));
        // We need a separate mapping vertex and can't use readMapP's projectionFn:
        // the projection will cause key/value deserialization on partition thread, which doesn't have job's
        // class loader. If the key/value uses a custom object, it will fail. For example, StreamKafkaP uses
        // TopicPartition as the key or a custom processor can use custom key.
        Vertex mapSnapshotVertex = dag.newVertex("__snapshot_map_." + vertex.getName(), mapP(projection));
        readSnapshotVertex.localParallelism(vertex.getLocalParallelism());
        mapSnapshotVertex.localParallelism(vertex.getLocalParallelism());
        int destOrdinal = dag.getInboundEdges(vertex.getName()).size();
        dag.edge(between(readSnapshotVertex, mapSnapshotVertex).isolated()).edge(new SnapshotRestoreEdge(mapSnapshotVertex, vertex, destOrdinal));
    }
}

Example 2 with BroadcastEntry

use of com.hazelcast.jet.impl.execution.BroadcastEntry in project hazelcast by hazelcast.

the class ExplodeSnapshotP method traverser.

/* We can't close the BufferObjectDataInput cleanly. We close it when the returned traverser is fully iterated,
    but the caller might not fully iterate it and we have no opportunity to close it.
    On the other hand, the returned object doesn't hold any resources, so relying on the GC is sufficient.
    See #19799 */
@SuppressWarnings("squid:S2095")
private Traverser<Object> traverser(byte[] data) {
    BufferObjectDataInput in = serializationService.createObjectDataInput(data);
    return () -> uncheckCall(() -> {
        Object key = in.readObject();
        if (key == SnapshotDataValueTerminator.INSTANCE) {
            return null;
        }
        Object value = in.readObject();
        return key instanceof BroadcastKey ? new BroadcastEntry(key, value) : entry(key, value);
    });
}