Examples with IngestService com.bakdata.quick.ingest.service.IngestService used on opensource projects

Example 1 with IngestService

use of com.bakdata.quick.ingest.service.IngestService in project quick by bakdata.

the class IngestController method convertIngestData.

/**
 * Processes raw input data for ingesting.
 *
 * <p>
 * Two steps are necessary to ingest the raw payload:
 * <ul>
 *    <li> parse it
 *    <li> check for existing keys if the topic is immutable
 * </ul>
 *
 * <p>
 * This method handles both steps and then forwards all non-existing {@link KeyValuePair} to the
 * {@link IngestService}.
 * Further, it handles merging the error messages. The method returns an error for existing keys as well as possible
 * errors in the ingest service.
 *
 * @param topic   the topic to ingest to
 * @param payload the raw payload
 * @param data    the topic information for the topic
 * @param <K>     the key type of the topic
 * @param <V>     the value type of the topic
 * @return merged completable possibly containing errors from existing keys or the ingest service
 */
private <K, V> Completable convertIngestData(final String topic, final String payload, final QuickTopicData<K, V> data) {
    final Single<List<KeyValuePair<K, V>>> list = Single.fromCallable(() -> this.parser.parseInputData(payload, data));
    return list.flatMap(pairs -> this.filter.prepareIngest(data, pairs)).flatMapCompletable(pairs -> {
        final Completable existingError = createErrorsForExistingKeys(topic, pairs);
        final Completable ingest = this.ingestService.sendData(topic, pairs.getDataToIngest());
        return Completable.mergeArrayDelayError(existingError, ingest);
    });
}