Examples with StreamPool org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.StreamPool used on opensource projects

Example 1 with StreamPool

use of org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.StreamPool in project beam by apache.

the class StreamingDataflowWorker method streamingCommitLoop.

private void streamingCommitLoop() {
    StreamPool<CommitWorkStream> streamPool = new StreamPool<>(NUM_COMMIT_STREAMS, COMMIT_STREAM_TIMEOUT, windmillServer::commitWorkStream);
    Commit initialCommit = null;
    while (running.get()) {
        if (initialCommit == null) {
            try {
                initialCommit = commitQueue.take();
            } catch (InterruptedException e) {
                continue;
            }
        }
        // We initialize the commit stream only after we have a commit to make sure it is fresh.
        CommitWorkStream commitStream = streamPool.getStream();
        if (!addCommitToStream(initialCommit, commitStream)) {
            throw new AssertionError("Initial commit on flushed stream should always be accepted.");
        }
        // Batch additional commits to the stream and possibly make an un-batched commit the next
        // initial commit.
        initialCommit = batchCommitsToStream(commitStream);
        commitStream.flush();
        streamPool.releaseStream(commitStream);
    }
}