Examples with SequentialBasedBatchAccumulator org.apache.gobblin.writer.SequentialBasedBatchAccumulator used on opensource projects

Example 6 with SequentialBasedBatchAccumulator

use of org.apache.gobblin.writer.SequentialBasedBatchAccumulator in project incubator-gobblin by apache.

the class EventhubAccumulatorTest method testClose.

@Test
public void testClose() throws IOException, InterruptedException {
    SequentialBasedBatchAccumulator accumulator = new EventhubBatchAccumulator(64, 3000, 5);
    StringBuffer buffer = new StringBuffer();
    for (int i = 0; i < 40; ++i) {
        buffer.append('a');
    }
    String record1 = buffer.toString() + "1";
    String record2 = buffer.toString() + "2";
    String record3 = buffer.toString() + "3";
    String record4 = buffer.toString() + "4";
    String record5 = buffer.toString() + "5";
    accumulator.append(record1, WriteCallback.EMPTY);
    accumulator.append(record2, WriteCallback.EMPTY);
    accumulator.append(record3, WriteCallback.EMPTY);
    accumulator.append(record4, WriteCallback.EMPTY);
    accumulator.append(record5, WriteCallback.EMPTY);
    (new Thread(new CloseAccumulatorThread(accumulator))).start();
    Thread.sleep(1000);
    Assert.assertEquals(accumulator.getNextAvailableBatch().getRecords().get(0), record1);
    Assert.assertEquals(accumulator.getNextAvailableBatch().getRecords().get(0), record2);
    Assert.assertEquals(accumulator.getNextAvailableBatch().getRecords().get(0), record3);
    Assert.assertEquals(accumulator.getNextAvailableBatch().getRecords().get(0), record4);
    Assert.assertEquals(accumulator.getNextAvailableBatch().getRecords().get(0), record5);
}

Example 7 with SequentialBasedBatchAccumulator

use of org.apache.gobblin.writer.SequentialBasedBatchAccumulator in project incubator-gobblin by apache.

the class EventhubAccumulatorTest method testCloseAfterAwait.

@Test
public void testCloseAfterAwait() throws IOException, InterruptedException {
    SequentialBasedBatchAccumulator accumulator = new EventhubBatchAccumulator(64, 1000, 5);
    (new Thread(new CloseAccumulatorThread(accumulator))).start();
    // this thread should be blocked and waked up by spawned thread
    Assert.assertNull(accumulator.getNextAvailableBatch());
}

Example 8 with SequentialBasedBatchAccumulator

use of org.apache.gobblin.writer.SequentialBasedBatchAccumulator in project incubator-gobblin by apache.

the class EventhubAccumulatorTest method testCloseBeforeAwait.

@Test
public void testCloseBeforeAwait() throws IOException, InterruptedException {
    SequentialBasedBatchAccumulator accumulator = new EventhubBatchAccumulator(64, 1000, 5);
    (new Thread(new CloseAccumulatorThread(accumulator))).start();
    Thread.sleep(1000);
    Assert.assertNull(accumulator.getNextAvailableBatch());
}

Example 9 with SequentialBasedBatchAccumulator

use of org.apache.gobblin.writer.SequentialBasedBatchAccumulator in project incubator-gobblin by apache.

the class EventhubAccumulatorTest method testExpiredBatch.

@Test
public void testExpiredBatch() throws IOException, InterruptedException {
    SequentialBasedBatchAccumulator accumulator = new EventhubBatchAccumulator(64, 3000, 5);
    String record = "1";
    accumulator.append(record, WriteCallback.EMPTY);
    Assert.assertNull(accumulator.getNextAvailableBatch());
    Thread.sleep(3000);
    Assert.assertNotNull(accumulator.getNextAvailableBatch());
}