Examples with Firehose org.apache.druid.data.input.Firehose used on opensource projects

Example 6 with Firehose

use of org.apache.druid.data.input.Firehose in project druid by druid-io.

the class PrefetchableTextFilesFirehoseFactoryTest method testWithoutCache.

@Test
public void testWithoutCache() throws IOException {
    final TestPrefetchableTextFilesFirehoseFactory factory = TestPrefetchableTextFilesFirehoseFactory.with(TEST_DIR, 0, 2048);
    final List<Row> rows = new ArrayList<>();
    final File firehoseTmpDir = createFirehoseTmpDir("testWithoutCache");
    try (Firehose firehose = factory.connect(PARSER, firehoseTmpDir)) {
        while (firehose.hasMore()) {
            rows.add(firehose.nextRow());
        }
    }
    Assert.assertEquals(0, factory.getCacheManager().getTotalCachedBytes());
    assertResult(rows);
    assertNumRemainingCacheFiles(firehoseTmpDir, 0);
}

Example 7 with Firehose

use of org.apache.druid.data.input.Firehose in project druid by druid-io.

the class PrefetchableTextFilesFirehoseFactoryTest method testWithoutCacheAndFetchAgainstConnectionReset.

@Test
public void testWithoutCacheAndFetchAgainstConnectionReset() throws IOException {
    final TestPrefetchableTextFilesFirehoseFactory factory = TestPrefetchableTextFilesFirehoseFactory.withConnectionResets(TEST_DIR, 0, 0, 2);
    final List<Row> rows = new ArrayList<>();
    final File firehoseTmpDir = createFirehoseTmpDir("testWithoutCacheAndFetch");
    try (Firehose firehose = factory.connect(PARSER, firehoseTmpDir)) {
        while (firehose.hasMore()) {
            rows.add(firehose.nextRow());
        }
    }
    Assert.assertEquals(0, factory.getCacheManager().getTotalCachedBytes());
    assertResult(rows);
    assertNumRemainingCacheFiles(firehoseTmpDir, 0);
}

Example 8 with Firehose

use of org.apache.druid.data.input.Firehose in project druid by druid-io.

the class PrefetchableTextFilesFirehoseFactoryTest method testWithLargeCacheAndSmallFetch.

@Test
public void testWithLargeCacheAndSmallFetch() throws IOException {
    final TestPrefetchableTextFilesFirehoseFactory factory = TestPrefetchableTextFilesFirehoseFactory.with(TEST_DIR, 2048, 1024);
    final List<Row> rows = new ArrayList<>();
    final File firehoseTmpDir = createFirehoseTmpDir("testWithLargeCacheAndSmallFetch");
    try (Firehose firehose = factory.connect(PARSER, firehoseTmpDir)) {
        while (firehose.hasMore()) {
            rows.add(firehose.nextRow());
        }
    }
    assertResult(rows);
    assertNumRemainingCacheFiles(firehoseTmpDir, 2);
}

Example 9 with Firehose

use of org.apache.druid.data.input.Firehose in project druid by druid-io.

the class PrefetchableTextFilesFirehoseFactoryTest method testMaxRetry.

@Test
public void testMaxRetry() throws IOException {
    expectedException.expect(RuntimeException.class);
    expectedException.expectCause(CoreMatchers.instanceOf(ExecutionException.class));
    expectedException.expectMessage("Exception for retry test");
    final TestPrefetchableTextFilesFirehoseFactory factory = TestPrefetchableTextFilesFirehoseFactory.withOpenExceptions(TEST_DIR, 5);
    try (Firehose firehose = factory.connect(PARSER, createFirehoseTmpDir("testMaxRetry"))) {
        while (firehose.hasMore()) {
            firehose.nextRow();
        }
    }
}

Example 10 with Firehose

use of org.apache.druid.data.input.Firehose in project druid by druid-io.

the class PrefetchableTextFilesFirehoseFactoryTest method testRetry.

@Test
public void testRetry() throws IOException {
    final TestPrefetchableTextFilesFirehoseFactory factory = TestPrefetchableTextFilesFirehoseFactory.withOpenExceptions(TEST_DIR, 1);
    final List<Row> rows = new ArrayList<>();
    final File firehoseTmpDir = createFirehoseTmpDir("testRetry");
    try (Firehose firehose = factory.connect(PARSER, firehoseTmpDir)) {
        while (firehose.hasMore()) {
            rows.add(firehose.nextRow());
        }
    }
    assertResult(rows);
    assertNumRemainingCacheFiles(firehoseTmpDir, 2);
}