Examples with AddElementsFromKafka uk.gov.gchq.gaffer.operation.impl.add.AddElementsFromKafka used on opensource projects

Example 1 with AddElementsFromKafka

use of uk.gov.gchq.gaffer.operation.impl.add.AddElementsFromKafka in project gaffer-doc by gchq.

the class AddElementsFromKafkaExample method addElementsFromKafka.

public void addElementsFromKafka() {
    // ---------------------------------------------------------
    final AddElementsFromKafka op = new AddElementsFromKafka.Builder().bootstrapServers("hostname1:8080,hostname2:8080").groupId("groupId1").topic("topic1").generator(ElementGenerator.class).parallelism(1).validate(true).skipInvalidElements(false).build();
    // ---------------------------------------------------------
    showExample(op, null);
}

Example 2 with AddElementsFromKafka

use of uk.gov.gchq.gaffer.operation.impl.add.AddElementsFromKafka in project Gaffer by gchq.

the class AddElementsFromKafkaHandlerIT method shouldAddElements.

protected <T> void shouldAddElements(final Class<T> consumeAs, final Class<? extends Function<Iterable<? extends T>, Iterable<? extends Element>>> elementGenerator, final Class<? extends Serializer> valueSerialiser) throws Exception {
    // Given
    final Graph graph = createGraph();
    final boolean validate = true;
    final boolean skipInvalid = false;
    final AddElementsFromKafka op = new AddElementsFromKafka.Builder().generator(consumeAs, elementGenerator).parallelism(1).validate(validate).skipInvalidElements(skipInvalid).topic(TOPIC).bootstrapServers(bootstrapServers).groupId(GROUP_ID).build();
    // When
    new Thread(() -> {
        try {
            graph.execute(op, new User());
        } catch (final OperationException e) {
            throw new RuntimeException(e);
        }
    }).start();
    waitForOperationToStart();
    producer = new KafkaProducer<>(producerProps(valueSerialiser));
    for (final String dataValue : DATA_VALUES) {
        producer.send(new ProducerRecord(TOPIC, convert(consumeAs, dataValue))).get();
    }
    waitForElements(consumeAs, elementGenerator);
    // Then
    verifyElements(consumeAs, testFileSink, elementGenerator);
}