Examples with RawMessage org.graylog2.plugin.journal.RawMessage used on opensource projects

Example 56 with RawMessage

use of org.graylog2.plugin.journal.RawMessage in project graylog2-server by Graylog2.

the class MessageResource method parse.

@POST
@Path("/parse")
@Timed
@Consumes(MediaType.APPLICATION_JSON)
@ApiOperation(value = "Parse a raw message")
@ApiResponses(value = { @ApiResponse(code = 404, message = "Specified codec does not exist."), @ApiResponse(code = 400, message = "Could not decode message.") })
@NoAuditEvent("only used to parse a test message")
public ResultMessage parse(@ApiParam(name = "JSON body", required = true) MessageParseRequest request) {
    Codec codec;
    try {
        final Configuration configuration = new Configuration(request.configuration());
        codec = codecFactory.create(request.codec(), configuration);
    } catch (IllegalArgumentException e) {
        throw new NotFoundException(e);
    }
    final ResolvableInetSocketAddress remoteAddress = ResolvableInetSocketAddress.wrap(new InetSocketAddress(request.remoteAddress(), 1234));
    final RawMessage rawMessage = new RawMessage(0, new UUID(), Tools.nowUTC(), remoteAddress, request.message().getBytes(StandardCharsets.UTF_8));
    final Message message = decodeMessage(codec, remoteAddress, rawMessage);
    return ResultMessage.createFromMessage(message);
}

Example 57 with RawMessage

use of org.graylog2.plugin.journal.RawMessage in project graylog2-server by Graylog2.

the class LocalKafkaMessageQueueReader method run.

@Override
protected void run() throws Exception {
    try {
        requestedReadCount = metricRegistry.register(name(this.getClass(), "requestedReadCount"), new HdrHistogram(processBuffer.getRingBufferSize() + 1, 3));
    } catch (IllegalArgumentException e) {
        log.warn("Metric already exists", e);
        throw e;
    }
    while (isRunning()) {
        // TODO interfere with reading if we are not 100% certain we should be reading, see #listenForLifecycleChanges
        if (!shouldBeReading()) {
            Uninterruptibles.sleepUninterruptibly(100, MILLISECONDS);
            // don't read immediately, but check if we should be shutting down.
            continue;
        }
        // approximate count to read from the journal to backfill the processing chain
        final long remainingCapacity = processBuffer.getRemainingCapacity();
        requestedReadCount.update(remainingCapacity);
        final List<Journal.JournalReadEntry> encodedRawMessages = journal.read(remainingCapacity);
        if (encodedRawMessages.isEmpty()) {
            log.debug("No messages to read from Journal, waiting until the writer adds more messages.");
            // block until something is written to the journal again
            try {
                readBlocked.inc();
                journalFilled.acquire();
            } catch (InterruptedException ignored) {
                // this can happen when we are blocked but the system wants to shut down. We don't have to do anything in that case.
                continue;
            }
            log.debug("Messages have been written to Journal, continuing to read.");
            // we don't care how many messages were inserted in the meantime, we'll read all of them eventually
            journalFilled.drainPermits();
        } else {
            readMessages.mark(encodedRawMessages.size());
            readerMetrics.readMessages().mark(encodedRawMessages.size());
            log.debug("Processing {} messages from journal.", encodedRawMessages.size());
            for (final Journal.JournalReadEntry encodedRawMessage : encodedRawMessages) {
                final RawMessage rawMessage = RawMessage.decode(encodedRawMessage.getPayload(), encodedRawMessage.getOffset());
                readerMetrics.readBytes().mark(encodedRawMessage.getPayload().length);
                if (rawMessage == null) {
                    // never insert null objects into the ringbuffer, as that is useless
                    log.error("Found null raw message!");
                    journal.markJournalOffsetCommitted(encodedRawMessage.getOffset());
                    continue;
                }
                processBuffer.insertBlocking(rawMessage);
            }
        }
    }
    log.info("Stopping.");
}

Example 58 with RawMessage

use of org.graylog2.plugin.journal.RawMessage in project graylog2-server by Graylog2.

the class DirectMessageHandler method onEvent.

@Override
public void onEvent(RawMessageEvent event) throws Exception {
    final RawMessage rawMessage = event.getRawMessage();
    processBuffer.insertBlocking(rawMessage);
    if (rawMessage != null) {
        processingStatusRecorder.updateIngestReceiveTime(rawMessage.getTimestamp());
    }
    // clear out for gc and to avoid promoting the raw message event to a tenured gen
    event.clear();
}

Example 59 with RawMessage

use of org.graylog2.plugin.journal.RawMessage in project graylog2-server by Graylog2.

the class ProcessBuffer method insertBlocking.

public void insertBlocking(@Nonnull RawMessage rawMessage) {
    final long sequence = ringBuffer.next();
    final MessageEvent event = ringBuffer.get(sequence);
    event.setRaw(rawMessage);
    ringBuffer.publish(sequence);
    afterInsert(1);
}

Example 60 with RawMessage

use of org.graylog2.plugin.journal.RawMessage in project graylog2-server by Graylog2.

the class BeatsCodecTest method messageFromJson.

private RawMessage messageFromJson(String resourceName) throws IOException {
    final URL resource = Resources.getResource(this.getClass(), resourceName);
    final byte[] json = Resources.toByteArray(resource);
    return new RawMessage(json);
}