Examples with DocumentQueue org.icij.extract.queue.DocumentQueue used on opensource projects

Example 1 with DocumentQueue

use of org.icij.extract.queue.DocumentQueue in project datashare by ICIJ.

the class PipelineTaskTest method test_pipeline_task_conditional_transfer_to_output_queue.

@Test
public void test_pipeline_task_conditional_transfer_to_output_queue() throws Exception {
    task.queue.put(get("/path/to/doc1"));
    task.queue.put(get("/path/to/doc2"));
    task.queue.put(POISON);
    task.transferToOutputQueue(p -> p.toString().contains("1"));
    assertThat(task.queue.isEmpty()).isTrue();
    DocumentQueue outputQueue = docCollectionFactory.createQueue(options, task.getOutputQueueName());
    assertThat(outputQueue.size()).isEqualTo(2);
    assertThat(outputQueue.poll().toString()).isEqualTo("/path/to/doc1");
    assertThat(outputQueue.poll().toString()).isEqualTo(POISON.toString());
}

Example 2 with DocumentQueue

use of org.icij.extract.queue.DocumentQueue in project datashare by ICIJ.

the class PipelineTask method transferToOutputQueue.

protected long transferToOutputQueue(Predicate<Path> filter) throws Exception {
    long originalSize = queue.size();
    try (DocumentQueue outputQueue = factory.createQueue(propertiesProvider, getOutputQueueName())) {
        Path path;
        while (!(path = queue.take()).equals(POISON)) {
            if (filter.test(path)) {
                outputQueue.add(path);
            }
        }
        outputQueue.add(POISON);
        return originalSize - outputQueue.size();
    }
}

Example 3 with DocumentQueue

use of org.icij.extract.queue.DocumentQueue in project datashare by ICIJ.

the class PipelineTaskTest method test_pipeline_task_transfer_to_output_queue.

@Test
public void test_pipeline_task_transfer_to_output_queue() throws Exception {
    task.queue.put(get("/path/to/doc1"));
    task.queue.put(get("/path/to/doc2"));
    task.queue.put(POISON);
    task.transferToOutputQueue();
    assertThat(task.queue.isEmpty()).isTrue();
    DocumentQueue outputQueue = docCollectionFactory.createQueue(options, task.getOutputQueueName());
    assertThat(outputQueue.size()).isEqualTo(3);
    assertThat(outputQueue.poll().toString()).isEqualTo("/path/to/doc1");
    assertThat(outputQueue.poll().toString()).isEqualTo("/path/to/doc2");
    assertThat(outputQueue.poll().toString()).isEqualTo(POISON.toString());
}

Example 4 with DocumentQueue

use of org.icij.extract.queue.DocumentQueue in project datashare by ICIJ.

the class StatusResourceTest method test_get_queue_with_io_exception.

@Test
public void test_get_queue_with_io_exception() {
    DocumentQueue mockQueue = mock(DocumentQueue.class);
    when(mockQueue.size()).thenThrow(new RuntimeException("test"));
    when(documentCollectionFactory.createQueue(any(), eq(new PropertiesProvider().get(PropertiesProvider.QUEUE_NAME_OPTION).orElse("extract:queue")))).thenReturn(mockQueue);
    configure(routes -> routes.add(new StatusResource(new PropertiesProvider(), repository, indexer, dataBus, documentCollectionFactory)));
    get("/api/status").should().respond(200).contain("\"document_queue_status\":false");
}