Search in sources :

Example 1 with AvroRecordBuffer

use of io.divolte.server.AvroRecordBuffer in project divolte-collector by divolte.

the class FileFlusherTest method shouldAttemptReconnectAfterHeartbeatRollCreateFailure.

@Test
public void shouldAttemptReconnectAfterHeartbeatRollCreateFailure() throws IOException, InterruptedException {
    final FileStrategyConfiguration fileStrategyConfiguration = setupConfiguration("50 milliseconds", "1 hour", "2");
    // Mocks
    final FileManager manager = mock(FileManager.class);
    final DivolteFile file = mock(DivolteFile.class);
    final InOrder calls = inOrder(manager, file);
    final Item<AvroRecordBuffer> item = itemFromAvroRecordBuffer(newAvroRecordBuffer());
    when(manager.createFile(anyString())).thenReturn(// Flusher construction succeeds
    file).thenThrow(// Second file creation fails
    new IOException("file create")).thenReturn(// Third file creation succeeds
    file);
    final FileFlusher flusher = new FileFlusher(fileStrategyConfiguration, manager, 50L);
    assertEquals(CONTINUE, flusher.process(item));
    assertEquals(CONTINUE, flusher.process(item));
    calls.verify(file, times(2)).append(item.payload);
    calls.verify(file).sync();
    Thread.sleep(100);
    // Rolling the file fails at this point (due to closeAndPublish failure)
    assertEquals(PAUSE, flusher.heartbeat());
    calls.verify(file).closeAndPublish();
    // File rolled; expecting creation of a new file, which fails
    calls.verify(manager).createFile(anyString());
    Thread.sleep(100);
    assertEquals(CONTINUE, flusher.heartbeat());
    // Reconnect attempt
    calls.verify(manager).createFile(anyString());
    calls.verifyNoMoreInteractions();
}
Also used : InOrder(org.mockito.InOrder) FileStrategyConfiguration(io.divolte.server.config.FileStrategyConfiguration) AvroRecordBuffer(io.divolte.server.AvroRecordBuffer) DivolteFile(io.divolte.server.filesinks.FileManager.DivolteFile) IOException(java.io.IOException) Test(org.junit.Test)

Example 2 with AvroRecordBuffer

use of io.divolte.server.AvroRecordBuffer in project divolte-collector by divolte.

the class FileFlusherTest method shouldAttemptReconnectMoreThanOnceAfterProcessFailure.

@Test
public void shouldAttemptReconnectMoreThanOnceAfterProcessFailure() throws IOException, InterruptedException {
    final FileStrategyConfiguration fileStrategyConfiguration = setupConfiguration("1 hour", "1 hour", "200");
    // Mocks
    final FileManager manager = mock(FileManager.class);
    final DivolteFile file = mock(DivolteFile.class);
    final InOrder calls = inOrder(manager, file);
    final Item<AvroRecordBuffer> item = itemFromAvroRecordBuffer(newAvroRecordBuffer());
    when(manager.createFile(anyString())).thenReturn(// Flusher construction succeeds
    file).thenThrow(// Second file creation fails
    new IOException("create file")).thenReturn(// Third creation succeeds
    file);
    doThrow(new IOException("append")).when(file).append(// first append fails
    item.payload);
    final FileFlusher flusher = new FileFlusher(fileStrategyConfiguration, manager, 50L);
    assertEquals(PAUSE, flusher.process(item));
    calls.verify(file).append(item.payload);
    calls.verify(file).discard();
    Thread.sleep(100);
    // Fail recovery on first attempt
    assertEquals(PAUSE, flusher.heartbeat());
    calls.verify(manager).createFile(anyString());
    Thread.sleep(100);
    // Succeed recovery on second attempt
    assertEquals(CONTINUE, flusher.heartbeat());
    calls.verify(manager).createFile(anyString());
    calls.verifyNoMoreInteractions();
}
Also used : InOrder(org.mockito.InOrder) FileStrategyConfiguration(io.divolte.server.config.FileStrategyConfiguration) AvroRecordBuffer(io.divolte.server.AvroRecordBuffer) DivolteFile(io.divolte.server.filesinks.FileManager.DivolteFile) IOException(java.io.IOException) Test(org.junit.Test)

Example 3 with AvroRecordBuffer

use of io.divolte.server.AvroRecordBuffer in project divolte-collector by divolte.

the class FileFlusherTest method shouldAttemptReconnectAfterProcessFailure.

@Test
public void shouldAttemptReconnectAfterProcessFailure() throws IOException, InterruptedException {
    final FileStrategyConfiguration fileStrategyConfiguration = setupConfiguration("1 hour", "1 hour", "200");
    // Mocks
    final FileManager manager = mock(FileManager.class);
    final DivolteFile file = mock(DivolteFile.class);
    final InOrder calls = inOrder(manager, file);
    final Item<AvroRecordBuffer> item = itemFromAvroRecordBuffer(newAvroRecordBuffer());
    // Expect new file creation on file flusher construction
    when(manager.createFile(anyString())).thenReturn(file);
    final FileFlusher flusher = new FileFlusher(fileStrategyConfiguration, manager, 50L);
    // throw exception on first record
    doThrow(new IOException("append")).when(file).append(item.payload);
    assertEquals(PAUSE, flusher.process(item));
    calls.verify(file).append(item.payload);
    calls.verify(file).discard();
    Thread.sleep(100);
    // Recover on first attempt, since creating a new file works
    assertEquals(CONTINUE, flusher.heartbeat());
    calls.verify(manager).createFile(anyString());
    calls.verifyNoMoreInteractions();
}
Also used : InOrder(org.mockito.InOrder) FileStrategyConfiguration(io.divolte.server.config.FileStrategyConfiguration) AvroRecordBuffer(io.divolte.server.AvroRecordBuffer) DivolteFile(io.divolte.server.filesinks.FileManager.DivolteFile) IOException(java.io.IOException) Test(org.junit.Test)

Example 4 with AvroRecordBuffer

use of io.divolte.server.AvroRecordBuffer in project divolte-collector by divolte.

the class FileFlusherTest method shouldSyncAndRollFileTimeBased.

@Test
public void shouldSyncAndRollFileTimeBased() throws IOException, InterruptedException {
    final FileStrategyConfiguration fileStrategyConfiguration = setupConfiguration("300 milliseconds", "50 milliseconds", "200");
    // Mocks
    final FileManager manager = mock(FileManager.class);
    final DivolteFile file = mock(DivolteFile.class);
    final InOrder calls = inOrder(manager, file);
    // Expect new file creation on file flusher construction
    when(manager.createFile(anyString())).thenReturn(file);
    final FileFlusher flusher = new FileFlusher(fileStrategyConfiguration, manager, 1L);
    final Item<AvroRecordBuffer> item = itemFromAvroRecordBuffer(newAvroRecordBuffer());
    assertEquals(CONTINUE, flusher.process(item));
    calls.verify(file).append(item.payload);
    Thread.sleep(100);
    assertEquals(CONTINUE, flusher.heartbeat());
    calls.verify(file).sync();
    Thread.sleep(400);
    assertEquals(CONTINUE, flusher.process(item));
    calls.verify(file).append(item.payload);
    calls.verify(file).closeAndPublish();
    calls.verify(manager).createFile(anyString());
    calls.verifyNoMoreInteractions();
}
Also used : InOrder(org.mockito.InOrder) FileStrategyConfiguration(io.divolte.server.config.FileStrategyConfiguration) AvroRecordBuffer(io.divolte.server.AvroRecordBuffer) DivolteFile(io.divolte.server.filesinks.FileManager.DivolteFile) Test(org.junit.Test)

Example 5 with AvroRecordBuffer

use of io.divolte.server.AvroRecordBuffer in project divolte-collector by divolte.

the class TopicFlusher method process.

@Override
public final ProcessingDirective process(final Item<AvroRecordBuffer> item) {
    final AvroRecordBuffer record = item.payload;
    logger.debug("Processing individual event: {}", record);
    return flush(ImmutableList.of(buildRecord(record)));
}
Also used : AvroRecordBuffer(io.divolte.server.AvroRecordBuffer)

Aggregations

AvroRecordBuffer (io.divolte.server.AvroRecordBuffer)13 FileStrategyConfiguration (io.divolte.server.config.FileStrategyConfiguration)11 DivolteFile (io.divolte.server.filesinks.FileManager.DivolteFile)11 Test (org.junit.Test)11 InOrder (org.mockito.InOrder)11 IOException (java.io.IOException)7 ImmutableList (com.google.common.collect.ImmutableList)1 DivolteIdentifier (io.divolte.server.DivolteIdentifier)1 ArrayList (java.util.ArrayList)1 ExecutionException (java.util.concurrent.ExecutionException)1 Future (java.util.concurrent.Future)1 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)1 RecordMetadata (org.apache.kafka.clients.producer.RecordMetadata)1 RetriableException (org.apache.kafka.common.errors.RetriableException)1