Examples with MessageOutput org.graylog2.plugin.outputs.MessageOutput used on opensource projects

Example 6 with MessageOutput

use of org.graylog2.plugin.outputs.MessageOutput in project graylog2-server by Graylog2.

the class OutputBufferProcessor method onEvent.

/**
     * Each message will be written to one or more outputs.
     * <p>
     * The default output is always being used for every message, but optionally the message can be routed to additional
     * outputs, currently based on the stream outputs that are configured in the system.
     * </p>
     * <p>
     * The stream outputs are time limited so one bad output does not impact throughput too much. Essentially this means
     * that the work of writing to the outputs is performed, but the writer threads will not wait forever for stream
     * outputs to finish their work. <b>This might lead to increased memory usage!</b>
     * </p>
     * <p>
     * The default output, however, is allowed to block and is not subject to time limiting. This is important because it
     * can exert back pressure on the processing pipeline this way, making sure we don't run into excessive heap usage.
     * </p>
     *
     * @param event the message to write to outputs
     * @throws Exception
     */
@Override
public void onEvent(MessageEvent event) throws Exception {
    incomingMessages.mark();
    final Message msg = event.getMessage();
    if (msg == null) {
        LOG.debug("Skipping null message.");
        return;
    }
    LOG.debug("Processing message <{}> from OutputBuffer.", msg.getId());
    final Set<MessageOutput> messageOutputs = outputRouter.getStreamOutputsForMessage(msg);
    msg.recordCounter(serverStatus, "matched-outputs", messageOutputs.size());
    final Future<?> defaultOutputCompletion = processMessage(msg, defaultMessageOutput);
    final CountDownLatch streamOutputsDoneSignal = new CountDownLatch(messageOutputs.size());
    for (final MessageOutput output : messageOutputs) {
        processMessage(msg, output, streamOutputsDoneSignal);
    }
    // Wait until all writer threads for stream outputs have finished or timeout is reached.
    if (!streamOutputsDoneSignal.await(configuration.getOutputModuleTimeout(), TimeUnit.MILLISECONDS)) {
        LOG.warn("Timeout reached. Not waiting any longer for stream output writer threads to complete.");
    }
    // this exerts the back pressure to the system
    if (defaultOutputCompletion != null) {
        Uninterruptibles.getUninterruptibly(defaultOutputCompletion);
    } else {
        LOG.error("The default output future was null, this is a bug!");
    }
    if (msg.hasRecordings()) {
        LOG.debug("Message event trace: {}", msg.recordingsAsString());
    }
    outputThroughput.inc();
    LOG.debug("Wrote message <{}> to all outputs. Finished handling.", msg.getId());
    event.clearMessages();
}

Example 7 with MessageOutput

use of org.graylog2.plugin.outputs.MessageOutput in project graylog2-server by Graylog2.

the class OutputSetupService method shutDownRunningOutputs.

private void shutDownRunningOutputs() {
    for (MessageOutput output : outputRegistry.getMessageOutputs()) {
        try {
            // TODO: change to debug
            LOG.info("Stopping output {}", output.getClass().getName());
            output.stop();
        } catch (Exception e) {
            LOG.error("Error stopping output", e);
        }
    }
}

Example 8 with MessageOutput

use of org.graylog2.plugin.outputs.MessageOutput in project graylog2-server by Graylog2.

the class OutputRegistry method removeOutput.

public void removeOutput(Output output) {
    final MessageOutput messageOutput = runningMessageOutputs.getIfPresent(output.getId());
    if (messageOutput != null) {
        messageOutput.stop();
    }
    runningMessageOutputs.invalidate(output.getId());
    faultCounters.invalidate(output.getId());
}

Example 9 with MessageOutput

use of org.graylog2.plugin.outputs.MessageOutput in project graylog2-server by Graylog2.

the class BatchedElasticSearchOutputFlushThread method doRun.

@Override
public void doRun() {
    LOG.debug("Checking for outputs to flush ...");
    for (MessageOutput output : outputRegistry.getMessageOutputs()) {
        if (output instanceof BlockingBatchedESOutput) {
            try {
                LOG.debug("Flushing output <{}>", output);
                ((BlockingBatchedESOutput) output).forceFlushIfTimedout();
            } catch (Exception e) {
                LOG.error("Caught exception while trying to flush output: {}", e);
            }
        }
    }
}

Example 10 with MessageOutput

use of org.graylog2.plugin.outputs.MessageOutput in project graylog2-server by Graylog2.

the class OutputRegistryTest method testLaunchNewOutput.

@Test
public void testLaunchNewOutput() throws Exception {
    final String outputId = "foobar";
    final Stream stream = mock(Stream.class);
    when(messageOutputFactory.fromStreamOutput(eq(output), eq(stream), any(Configuration.class))).thenReturn(messageOutput);
    when(outputService.load(eq(outputId))).thenReturn(output);
    final OutputRegistry outputRegistry = new OutputRegistry(null, outputService, messageOutputFactory, null, null, FAULT_COUNT_THRESHOLD, FAULT_PENALTY_SECONDS);
    assertEquals(outputRegistry.getRunningMessageOutputs().size(), 0);
    MessageOutput result = outputRegistry.getOutputForIdAndStream(outputId, stream);
    assertSame(result, messageOutput);
    assertNotNull(outputRegistry.getRunningMessageOutputs());
    assertEquals(outputRegistry.getRunningMessageOutputs().size(), 1);
}