Search in sources :

Example 6 with StreamEventWriter

use of org.apache.druid.testing.utils.StreamEventWriter in project druid by druid-io.

the class AbstractStreamIndexingTest method doTestIndexDataWithAutoscaler.

protected void doTestIndexDataWithAutoscaler(@Nullable Boolean transactionEnabled) throws Exception {
    final GeneratedTestConfig generatedTestConfig = new GeneratedTestConfig(INPUT_FORMAT, getResourceAsString(JSON_INPUT_FORMAT_PATH));
    try (final Closeable closer = createResourceCloser(generatedTestConfig);
        final StreamEventWriter streamEventWriter = createStreamEventWriter(config, transactionEnabled)) {
        final String taskSpec = generatedTestConfig.getStreamIngestionPropsTransform().apply(getResourceAsString(SUPERVISOR_WITH_AUTOSCALER_SPEC_TEMPLATE_PATH));
        LOG.info("supervisorSpec: [%s]\n", taskSpec);
        // Start supervisor
        generatedTestConfig.setSupervisorId(indexer.submitSupervisor(taskSpec));
        LOG.info("Submitted supervisor");
        // Start generating half of the data
        int secondsToGenerateRemaining = TOTAL_NUMBER_OF_SECOND;
        int secondsToGenerateFirstRound = TOTAL_NUMBER_OF_SECOND / 2;
        secondsToGenerateRemaining = secondsToGenerateRemaining - secondsToGenerateFirstRound;
        final StreamGenerator streamGenerator = new WikipediaStreamEventStreamGenerator(new JsonEventSerializer(jsonMapper), EVENTS_PER_SECOND, CYCLE_PADDING_MS);
        long numWritten = streamGenerator.run(generatedTestConfig.getStreamName(), streamEventWriter, secondsToGenerateFirstRound, FIRST_EVENT_TIME);
        // Verify supervisor is healthy before suspension
        ITRetryUtil.retryUntil(() -> SupervisorStateManager.BasicState.RUNNING.equals(indexer.getSupervisorStatus(generatedTestConfig.getSupervisorId())), true, 10000, 30, "Waiting for supervisor to be healthy");
        // wait for autoScaling task numbers from 1 to 2.
        ITRetryUtil.retryUntil(() -> indexer.getRunningTasks().size() == 2, true, 10000, 50, "waiting for autoScaling task numbers from 1 to 2");
        // Start generating remainning half of the data
        numWritten += streamGenerator.run(generatedTestConfig.getStreamName(), streamEventWriter, secondsToGenerateRemaining, FIRST_EVENT_TIME.plusSeconds(secondsToGenerateFirstRound));
        // Verify that supervisor can catch up with the stream
        verifyIngestedData(generatedTestConfig, numWritten);
    }
}
Also used : JsonEventSerializer(org.apache.druid.testing.utils.JsonEventSerializer) WikipediaStreamEventStreamGenerator(org.apache.druid.testing.utils.WikipediaStreamEventStreamGenerator) Closeable(java.io.Closeable) StreamEventWriter(org.apache.druid.testing.utils.StreamEventWriter) StreamGenerator(org.apache.druid.testing.utils.StreamGenerator) WikipediaStreamEventStreamGenerator(org.apache.druid.testing.utils.WikipediaStreamEventStreamGenerator)

Aggregations

Closeable (java.io.Closeable)6 StreamEventWriter (org.apache.druid.testing.utils.StreamEventWriter)6 JsonEventSerializer (org.apache.druid.testing.utils.JsonEventSerializer)5 StreamGenerator (org.apache.druid.testing.utils.StreamGenerator)5 WikipediaStreamEventStreamGenerator (org.apache.druid.testing.utils.WikipediaStreamEventStreamGenerator)5 IOException (java.io.IOException)1 RowIngestionMetersTotals (org.apache.druid.segment.incremental.RowIngestionMetersTotals)1 TaskResponseObject (org.apache.druid.testing.clients.TaskResponseObject)1 EventSerializer (org.apache.druid.testing.utils.EventSerializer)1 AbstractKafkaIndexingServiceTest (org.apache.druid.tests.indexer.AbstractKafkaIndexingServiceTest)1 AbstractStreamIndexingTest (org.apache.druid.tests.indexer.AbstractStreamIndexingTest)1 Interval (org.joda.time.Interval)1 Test (org.testng.annotations.Test)1