Examples with BatchIterator io.crate.data.BatchIterator used on opensource projects

Example 1 with BatchIterator

use of io.crate.data.BatchIterator in project crate by crate.

the class IndexWriterProjectorUnitTest method testNullPKValue.

@Test
public void testNullPKValue() throws Throwable {
    InputCollectExpression sourceInput = new InputCollectExpression(0);
    List<CollectExpression<Row, ?>> collectExpressions = Collections.<CollectExpression<Row, ?>>singletonList(sourceInput);
    final IndexWriterProjector indexWriter = new IndexWriterProjector(clusterService, TestingHelpers.getFunctions(), new IndexNameExpressionResolver(Settings.EMPTY), Settings.EMPTY, mock(TransportBulkCreateIndicesAction.class), mock(BulkRequestExecutor.class), () -> "foo", mock(BulkRetryCoordinatorPool.class, Answers.RETURNS_DEEP_STUBS.get()), rawSourceReference, ImmutableList.of(ID_IDENT), Arrays.<Symbol>asList(new InputColumn(1)), null, null, sourceInput, collectExpressions, 20, null, null, false, false, UUID.randomUUID());
    RowN rowN = new RowN(new Object[] { new BytesRef("{\"y\": \"x\"}"), null });
    BatchIterator batchIterator = RowsBatchIterator.newInstance(Collections.singletonList(rowN), rowN.numColumns());
    batchIterator = indexWriter.apply(batchIterator);
    TestingBatchConsumer testingBatchConsumer = new TestingBatchConsumer();
    testingBatchConsumer.accept(batchIterator, null);
    expectedException.expect(IllegalArgumentException.class);
    expectedException.expectMessage("A primary key value must not be NULL");
    testingBatchConsumer.getResult();
}

Example 2 with BatchIterator

use of io.crate.data.BatchIterator in project crate by crate.

the class OrderedLuceneBatchIteratorBenchmark method measureLoadAndConsumeOrderedLuceneBatchIterator.

@Benchmark
public void measureLoadAndConsumeOrderedLuceneBatchIterator(Blackhole blackhole) {
    BatchIterator it = OrderedLuceneBatchIteratorFactory.newInstance(Collections.singletonList(createOrderedCollector(indexSearcher, columnName)), 1, OrderingByPosition.rowOrdering(new int[] { 0 }, reverseFlags, nullsFirst), MoreExecutors.directExecutor(), false);
    while (!it.allLoaded()) {
        it.loadNextBatch().toCompletableFuture().join();
    }
    Input<?> input = it.rowData().get(0);
    while (it.moveNext()) {
        blackhole.consume(input.value());
    }
}

Example 3 with BatchIterator

use of io.crate.data.BatchIterator in project crate by crate.

the class BatchConsumerToResultReceiver method resume.

public void resume() {
    assert activeIt != null : "resume must only be called if suspended() returned true and activeIt is not null";
    BatchIterator iterator = this.activeIt;
    this.activeIt = null;
    consumeIt(iterator);
}

Example 4 with BatchIterator

use of io.crate.data.BatchIterator in project crate by crate.

the class FileReadingIteratorTest method testIteratorContract.

@Test
public void testIteratorContract() throws Exception {
    String fileUri = tempFilePath.toUri().toString();
    Supplier<BatchIterator> batchIteratorSupplier = () -> createBatchIterator(Collections.singletonList(fileUri), null);
    byte[] firstLine = "{\"name\": \"Arthur\", \"id\": 4, \"details\": {\"age\": 38}}".getBytes(StandardCharsets.UTF_8);
    byte[] secondLine = "{\"id\": 5, \"name\": \"Trillian\", \"details\": {\"age\": 33}}".getBytes(StandardCharsets.UTF_8);
    List<Object[]> expectedResult = Arrays.asList(new Object[] { new BytesRef(firstLine) }, new Object[] { new BytesRef(secondLine) });
    BatchIteratorTester tester = new BatchIteratorTester(batchIteratorSupplier);
    tester.verifyResultAndEdgeCaseBehaviour(expectedResult);
}

Example 5 with BatchIterator

use of io.crate.data.BatchIterator in project crate by crate.

the class NodeStatsCollectSource method getCollector.

@Override
public CrateCollector getCollector(CollectPhase phase, BatchConsumer consumer, JobCollectContext jobCollectContext) {
    RoutedCollectPhase collectPhase = (RoutedCollectPhase) phase;
    if (collectPhase.whereClause().noMatch()) {
        return RowsCollector.empty(consumer);
    }
    Collection<DiscoveryNode> nodes = nodeIds(collectPhase.whereClause(), Lists.newArrayList(clusterService.state().getNodes().iterator()), functions);
    if (nodes.isEmpty()) {
        return RowsCollector.empty(consumer);
    }
    BatchIterator nodeStatsIterator = NodeStatsIterator.newInstance(nodeStatsAction, collectPhase, nodes, inputFactory);
    return BatchIteratorCollectorBridge.newInstance(nodeStatsIterator, consumer);
}