Search in sources :

Example 1 with InMemorySystemProducer

use of org.apache.samza.system.inmemory.InMemorySystemProducer in project samza by apache.

the class TestRunner method initializeInMemoryInputStream.

/**
 * Creates an in memory stream with {@link InMemorySystemFactory} and feeds its partition with stream of messages
 * @param partitionData key of the map represents partitionId and value represents messages in the partition
 * @param descriptor describes a stream to initialize with the in memory system
 */
private <StreamMessageType> void initializeInMemoryInputStream(InMemoryInputDescriptor<?> descriptor, Map<Integer, Iterable<StreamMessageType>> partitionData) {
    String systemName = descriptor.getSystemName();
    String streamName = (String) descriptor.getPhysicalName().orElse(descriptor.getStreamId());
    if (this.app instanceof LegacyTaskApplication) {
        // for legacy applications that only specify task.class.
        if (configs.containsKey(TaskConfig.INPUT_STREAMS)) {
            configs.put(TaskConfig.INPUT_STREAMS, configs.get(TaskConfig.INPUT_STREAMS).concat("," + systemName + "." + streamName));
        } else {
            configs.put(TaskConfig.INPUT_STREAMS, systemName + "." + streamName);
        }
    }
    InMemorySystemDescriptor imsd = (InMemorySystemDescriptor) descriptor.getSystemDescriptor();
    imsd.withInMemoryScope(this.inMemoryScope);
    addConfig(descriptor.toConfig());
    addConfig(descriptor.getSystemDescriptor().toConfig());
    addSerdeConfigs(descriptor);
    StreamSpec spec = new StreamSpec(descriptor.getStreamId(), streamName, systemName, partitionData.size());
    SystemFactory factory = new InMemorySystemFactory();
    Config config = new MapConfig(descriptor.toConfig(), descriptor.getSystemDescriptor().toConfig());
    factory.getAdmin(systemName, config).createStream(spec);
    InMemorySystemProducer producer = (InMemorySystemProducer) factory.getProducer(systemName, config, null);
    SystemStream sysStream = new SystemStream(systemName, streamName);
    partitionData.forEach((partitionId, partition) -> {
        partition.forEach(e -> {
            Object key = e instanceof KV ? ((KV) e).getKey() : null;
            Object value = e instanceof KV ? ((KV) e).getValue() : e;
            if (value instanceof IncomingMessageEnvelope) {
                producer.send((IncomingMessageEnvelope) value);
            } else {
                producer.send(systemName, new OutgoingMessageEnvelope(sysStream, Integer.valueOf(partitionId), key, value));
            }
        });
        producer.send(systemName, new OutgoingMessageEnvelope(sysStream, Integer.valueOf(partitionId), null, new EndOfStreamMessage(null)));
    });
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) InMemorySystemFactory(org.apache.samza.system.inmemory.InMemorySystemFactory) SystemFactory(org.apache.samza.system.SystemFactory) MapConfig(org.apache.samza.config.MapConfig) InMemorySystemConfig(org.apache.samza.config.InMemorySystemConfig) JobCoordinatorConfig(org.apache.samza.config.JobCoordinatorConfig) Config(org.apache.samza.config.Config) JobConfig(org.apache.samza.config.JobConfig) ClusterManagerConfig(org.apache.samza.config.ClusterManagerConfig) StreamConfig(org.apache.samza.config.StreamConfig) ApplicationConfig(org.apache.samza.config.ApplicationConfig) TaskConfig(org.apache.samza.config.TaskConfig) SystemStream(org.apache.samza.system.SystemStream) IncomingMessageEnvelope(org.apache.samza.system.IncomingMessageEnvelope) LegacyTaskApplication(org.apache.samza.application.LegacyTaskApplication) KV(org.apache.samza.operators.KV) InMemorySystemDescriptor(org.apache.samza.test.framework.system.descriptors.InMemorySystemDescriptor) EndOfStreamMessage(org.apache.samza.system.EndOfStreamMessage) MapConfig(org.apache.samza.config.MapConfig) OutgoingMessageEnvelope(org.apache.samza.system.OutgoingMessageEnvelope) InMemorySystemFactory(org.apache.samza.system.inmemory.InMemorySystemFactory) InMemorySystemProducer(org.apache.samza.system.inmemory.InMemorySystemProducer)

Aggregations

LegacyTaskApplication (org.apache.samza.application.LegacyTaskApplication)1 ApplicationConfig (org.apache.samza.config.ApplicationConfig)1 ClusterManagerConfig (org.apache.samza.config.ClusterManagerConfig)1 Config (org.apache.samza.config.Config)1 InMemorySystemConfig (org.apache.samza.config.InMemorySystemConfig)1 JobConfig (org.apache.samza.config.JobConfig)1 JobCoordinatorConfig (org.apache.samza.config.JobCoordinatorConfig)1 MapConfig (org.apache.samza.config.MapConfig)1 StreamConfig (org.apache.samza.config.StreamConfig)1 TaskConfig (org.apache.samza.config.TaskConfig)1 KV (org.apache.samza.operators.KV)1 EndOfStreamMessage (org.apache.samza.system.EndOfStreamMessage)1 IncomingMessageEnvelope (org.apache.samza.system.IncomingMessageEnvelope)1 OutgoingMessageEnvelope (org.apache.samza.system.OutgoingMessageEnvelope)1 StreamSpec (org.apache.samza.system.StreamSpec)1 SystemFactory (org.apache.samza.system.SystemFactory)1 SystemStream (org.apache.samza.system.SystemStream)1 InMemorySystemFactory (org.apache.samza.system.inmemory.InMemorySystemFactory)1 InMemorySystemProducer (org.apache.samza.system.inmemory.InMemorySystemProducer)1 InMemorySystemDescriptor (org.apache.samza.test.framework.system.descriptors.InMemorySystemDescriptor)1