Examples with TestDataSetConsumer org.deeplearning4j.util.TestDataSetConsumer used on opensource projects

Example 1 with TestDataSetConsumer

use of org.deeplearning4j.util.TestDataSetConsumer in project deeplearning4j by deeplearning4j.

the class MultipleEpochsIteratorTest method testMEDIWithLoad2.

@Test
public void testMEDIWithLoad2() throws Exception {
    ExistingDataSetIterator iter = new ExistingDataSetIterator(new IterableWithoutException(100));
    MultipleEpochsIterator iterator = new MultipleEpochsIterator(10, iter, 24);
    TestDataSetConsumer consumer = new TestDataSetConsumer(iterator, 2);
    long num1 = 0;
    for (; num1 < 150; num1++) {
        consumer.consumeOnce(iterator.next(), true);
    }
    iterator.reset();
    long num2 = consumer.consumeWhileHasNext(true);
    assertEquals((10 * 100) + 150, num1 + num2);
}

Example 2 with TestDataSetConsumer

use of org.deeplearning4j.util.TestDataSetConsumer in project deeplearning4j by deeplearning4j.

the class MultipleEpochsIteratorTest method testMEDIWithLoad3.

@Test
public void testMEDIWithLoad3() throws Exception {
    ExistingDataSetIterator iter = new ExistingDataSetIterator(new IterableWithoutException(10000));
    MultipleEpochsIterator iterator = new MultipleEpochsIterator(iter, 24, 136);
    TestDataSetConsumer consumer = new TestDataSetConsumer(iterator, 2);
    long num1 = 0;
    while (iterator.hasNext()) {
        consumer.consumeOnce(iterator.next(), true);
        num1++;
    }
    assertEquals(136, num1);
}

Example 3 with TestDataSetConsumer

use of org.deeplearning4j.util.TestDataSetConsumer in project deeplearning4j by deeplearning4j.

the class AsyncDataSetIteratorTest method hasNextWithResetAndLoad.

@Test
public void hasNextWithResetAndLoad() throws Exception {
    for (int iter = 0; iter < ITERATIONS; iter++) {
        for (int prefetchSize = 2; prefetchSize <= 8; prefetchSize++) {
            AsyncDataSetIterator iterator = new AsyncDataSetIterator(backIterator, prefetchSize);
            TestDataSetConsumer consumer = new TestDataSetConsumer(EXECUTION_SMALL);
            int cnt = 0;
            while (iterator.hasNext()) {
                DataSet ds = iterator.next();
                consumer.consumeOnce(ds, false);
                cnt++;
                if (cnt == TEST_SIZE / 2)
                    iterator.reset();
            }
            assertEquals(TEST_SIZE + (TEST_SIZE / 2), cnt);
            iterator.shutdown();
        }
    }
}

Example 4 with TestDataSetConsumer

use of org.deeplearning4j.util.TestDataSetConsumer in project deeplearning4j by deeplearning4j.

the class AsyncDataSetIteratorTest method testWithException.

@Test(expected = ArrayIndexOutOfBoundsException.class)
public void testWithException() {
    ExistingDataSetIterator crashingIterator = new ExistingDataSetIterator(new IterableWithException(100));
    AsyncDataSetIterator iterator = new AsyncDataSetIterator(crashingIterator, 8);
    TestDataSetConsumer consumer = new TestDataSetConsumer(iterator, EXECUTION_SMALL);
    consumer.consumeWhileHasNext(true);
    iterator.shutdown();
}

Example 5 with TestDataSetConsumer

use of org.deeplearning4j.util.TestDataSetConsumer in project deeplearning4j by deeplearning4j.

the class AsyncDataSetIteratorTest method testWithLoad.

@Test
public void testWithLoad() {
    for (int iter = 0; iter < ITERATIONS; iter++) {
        AsyncDataSetIterator iterator = new AsyncDataSetIterator(backIterator, 8);
        TestDataSetConsumer consumer = new TestDataSetConsumer(iterator, EXECUTION_TIME);
        consumer.consumeWhileHasNext(true);
        assertEquals(TEST_SIZE, consumer.getCount());
        iterator.shutdown();
    }
}