Examples with ProcessingQueue io.confluent.ksql.physical.scalablepush.ProcessingQueue used on opensource projects

Example 1 with ProcessingQueue

use of io.confluent.ksql.physical.scalablepush.ProcessingQueue in project ksql by confluentinc.

the class ScalablePushConsumer method handleRow.

private boolean handleRow(final Object key, final GenericRow value, final long timestamp) {
    // We don't currently handle null in either field
    if ((key == null && !logicalSchema.key().isEmpty()) || value == null) {
        return false;
    }
    numRowsReceived.incrementAndGet();
    for (ProcessingQueue queue : processingQueues.values()) {
        try {
            // The physical operators may modify the keys and values, so we make a copy to ensure
            // that there's no cross-query interference.
            final QueryRow row = RowUtil.createRow(key, value, timestamp, windowed, logicalSchema);
            queue.offer(row);
            afterOfferedRow(queue);
        } catch (final Throwable t) {
            LOG.error("Error while offering row", t);
        }
    }
    return false;
}

Example 2 with ProcessingQueue

use of io.confluent.ksql.physical.scalablepush.ProcessingQueue in project ksql by confluentinc.

the class PeekStreamOperatorTest method shouldGetRowsFromOperator.

@Test
public void shouldGetRowsFromOperator() {
    // Given:
    final PeekStreamOperator locator = new PeekStreamOperator(registry, dataSourceNode, QUERY_ID, Optional.empty());
    locator.setNewRowCallback(newRowCallback);
    // When:
    locator.open();
    // Then:
    verify(registry, times(1)).register(processingQueueCaptor.capture(), eq(Optional.empty()));
    final ProcessingQueue processingQueue = processingQueueCaptor.getValue();
    processingQueue.offer(row1);
    processingQueue.offer(row2);
    assertThat(locator.next(), is(row1));
    assertThat(locator.next(), is(row2));
    assertThat(locator.next(), nullValue());
    verify(newRowCallback, times(2)).run();
    locator.close();
    verify(registry, times(1)).unregister(processingQueue);
}

Example 3 with ProcessingQueue

use of io.confluent.ksql.physical.scalablepush.ProcessingQueue in project ksql by confluentinc.

the class ScalablePushConsumer method handleProgressToken.

private void handleProgressToken(final PushOffsetVector startOffsetVector, final PushOffsetVector endOffsetVector) {
    final PushOffsetRange range = new PushOffsetRange(Optional.of(startOffsetVector), endOffsetVector);
    for (ProcessingQueue queue : processingQueues.values()) {
        final QueryRow row = OffsetsRow.of(clock.millis(), range);
        queue.offer(row);
    }
}

Example 4 with ProcessingQueue

use of io.confluent.ksql.physical.scalablepush.ProcessingQueue in project ksql by confluentinc.

the class ScalablePushConsumer method closeAsync.

/**
 * Closes async, avoiding blocking the caller. Can be closed by another thread.
 */
public void closeAsync() {
    closed = true;
    for (final ProcessingQueue processingQueue : processingQueues.values()) {
        processingQueue.close();
    }
    consumer.wakeup();
}