Examples with SystemStream org.apache.samza.system.SystemStream used on opensource projects

Example 11 with SystemStream

use of org.apache.samza.system.SystemStream in project samza by apache.

the class IdentityStreamTask method process.

@Override
public void process(IncomingMessageEnvelope incomingMessageEnvelope, MessageCollector messageCollector, TaskCoordinator taskCoordinator) throws Exception {
    messageCollector.send(new OutgoingMessageEnvelope(new SystemStream(outputSystem, outputTopic), incomingMessageEnvelope.getMessage()));
    processedMessageCount++;
    if (processedMessageCount == expectedMessageCount) {
        endLatch.countDown();
    }
}

Example 12 with SystemStream

use of org.apache.samza.system.SystemStream in project samza by apache.

the class Checker method window.

@Override
public void window(MessageCollector collector, TaskCoordinator coordinator) {
    String currentEpoch = this.store.get(CURRENT_EPOCH);
    logger.info("Checking if epoch " + currentEpoch + " is complete.");
    int count = 0;
    KeyValueIterator<String, String> iter = this.store.all();
    while (iter.hasNext()) {
        Entry<String, String> entry = iter.next();
        String foundEpoch = entry.getValue();
        if (foundEpoch.equals(currentEpoch)) {
            count += 1;
        } else {
            logger.info("####### Found a different epoch! - " + foundEpoch + " Current epoch is " + currentEpoch);
        }
    }
    iter.close();
    if (count == expectedKeys + 1) {
        logger.info("Epoch " + currentEpoch + " is complete.");
        int nextEpoch = Integer.parseInt(currentEpoch) + 1;
        for (int i = 0; i < numPartitions; i++) {
            logger.info("Emitting next epoch - " + Integer.toString(i) + " -> " + Integer.toString(nextEpoch));
            collector.send(new OutgoingMessageEnvelope(new SystemStream("kafka", "epoch"), Integer.toString(i), Integer.toString(nextEpoch)));
        }
        this.store.put(CURRENT_EPOCH, Integer.toString(nextEpoch));
    } else if (count > expectedKeys + 1) {
        throw new IllegalStateException("Got " + count + " keys, which is more than the expected " + (expectedKeys + 1));
    } else {
        logger.info("Only found " + count + " valid keys, try again later.");
    }
}

Example 13 with SystemStream

use of org.apache.samza.system.SystemStream in project samza by apache.

the class Joiner method process.

@Override
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {
    String key = (String) envelope.getKey();
    String value = (String) envelope.getMessage();
    String[] pieces = value.split("-");
    int epoch = Integer.parseInt(pieces[0]);
    int partition = Integer.parseInt(pieces[1].split(" ")[1]);
    Partitions partitions = loadPartitions(epoch, key);
    logger.info("Joiner got epoch = " + epoch + ", partition = " + partition + ", parts = " + partitions);
    if (partitions.epoch < epoch) {
        // we are in a new era
        if (partitions.partitions.size() != expected)
            throw new IllegalArgumentException("Should have " + expected + " partitions when new epoch starts.");
        logger.info("Reseting epoch to " + epoch);
        this.store.delete(key);
        partitions.epoch = epoch;
        partitions.partitions.clear();
        partitions.partitions.add(partition);
    } else if (partitions.epoch > epoch) {
        logger.info("Ignoring message for epoch " + epoch);
    } else {
        partitions.partitions.add(partition);
        if (partitions.partitions.size() == expected) {
            logger.info("Completed: " + key + " -> " + Integer.toString(epoch));
            collector.send(new OutgoingMessageEnvelope(new SystemStream("kafka", "completed-keys"), key, Integer.toString(epoch)));
        }
    }
    this.store.put(key, partitions.toString());
    logger.info("Join store in Task " + this.taskName + " " + key + " -> " + partitions.toString());
}

Example 14 with SystemStream

use of org.apache.samza.system.SystemStream in project samza by apache.

the class Log4jSystemConfig method getStreamSerdeName.

public String getStreamSerdeName(String systemName, String streamName) {
    StreamConfig streamConfig = new StreamConfig(this);
    scala.Option<String> option = streamConfig.getStreamMsgSerde(new SystemStream(systemName, streamName));
    return option.isEmpty() ? null : option.get();
}

Example 15 with SystemStream

use of org.apache.samza.system.SystemStream in project samza by apache.

the class DefaultChooserConfig method getBootstrapStreams.

/**
   * @return  the set of SystemStreams which were configured as bootstrap streams.
   */
public Set<SystemStream> getBootstrapStreams() {
    Set<SystemStream> bootstrapInputs = new HashSet<>();
    Set<SystemStream> allInputs = taskConfigJava.getAllInputStreams();
    for (SystemStream systemStream : allInputs) {
        if (streamConfig.getBootstrapEnabled(systemStream)) {
            bootstrapInputs.add(systemStream);
        }
    }
    return Collections.unmodifiableSet(bootstrapInputs);
}