Search in sources :

Example 6 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamSamzaPropertiesOmitted.

// The samza properties (which are invalid for the underlying system) should be filtered out.
@Test
public void testGetStreamSamzaPropertiesOmitted() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME, StreamConfig.SYSTEM, TEST_SYSTEM, "systemProperty1", "systemValue1", "systemProperty2", "systemValue2", "systemProperty3", "systemValue3");
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    Map<String, String> properties = spec.getConfig();
    assertEquals(3, properties.size());
    assertNull(properties.get(String.format(StreamConfig.PHYSICAL_NAME_FOR_STREAM_ID, STREAM_ID)));
    assertNull(properties.get(String.format(StreamConfig.SYSTEM_FOR_STREAM_ID, STREAM_ID)));
    assertNull(spec.get(String.format(StreamConfig.PHYSICAL_NAME_FOR_STREAM_ID, STREAM_ID)));
    assertNull(spec.get(String.format(StreamConfig.SYSTEM_FOR_STREAM_ID, STREAM_ID)));
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 7 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamWithPhysicalNameInConfig.

// The physical name should be pulled from the StreamConfig.PHYSICAL_NAME property value.
@Test
public void testGetStreamWithPhysicalNameInConfig() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME, StreamConfig.SYSTEM, TEST_SYSTEM);
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    assertEquals(TEST_PHYSICAL_NAME, spec.getPhysicalName());
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 8 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamSystemNameArgInvalid.

// Special characters are NOT allowed for system name, because it's used as an identifier in the config.
@Test(expected = IllegalArgumentException.class)
public void testGetStreamSystemNameArgInvalid() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME, StreamConfig.SYSTEM, TEST_SYSTEM_INVALID);
    StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
}
Also used : JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 9 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamPhysicalNameArgNull.

// Null is allowed for the physical name
@Test
public void testGetStreamPhysicalNameArgNull() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, null, StreamConfig.SYSTEM, TEST_SYSTEM);
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    assertNull(spec.getPhysicalName());
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 10 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamEdge method testGenerateConfig.

@Test
public void testGenerateConfig() {
    // an example unbounded IO stream
    StreamSpec spec = new StreamSpec("stream-1", "physical-stream-1", "system-1", Collections.singletonMap("property1", "haha"));
    StreamEdge edge = new StreamEdge(spec, false, false, new MapConfig());
    Config config = edge.generateConfig();
    StreamConfig streamConfig = new StreamConfig(config);
    assertEquals(streamConfig.getSystem(spec.getId()), "system-1");
    assertEquals(streamConfig.getPhysicalName(spec.getId()), "physical-stream-1");
    assertEquals(streamConfig.getIsIntermediateStream(spec.getId()), false);
    assertEquals(streamConfig.getStreamProperties(spec.getId()).get("property1"), "haha");
    // bounded stream
    spec = new StreamSpec("stream-1", "physical-stream-1", "system-1", Collections.singletonMap("property1", "haha"));
    edge = new StreamEdge(spec, false, false, new MapConfig());
    config = edge.generateConfig();
    streamConfig = new StreamConfig(config);
    // intermediate stream
    edge = new StreamEdge(spec, true, false, new MapConfig());
    config = edge.generateConfig();
    streamConfig = new StreamConfig(config);
    assertEquals(streamConfig.getIsIntermediateStream(spec.getId()), true);
    assertEquals(streamConfig.getPriority(spec.toSystemStream()), Integer.MAX_VALUE);
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) ApplicationConfig(org.apache.samza.config.ApplicationConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) MapConfig(org.apache.samza.config.MapConfig) Test(org.junit.Test)

Aggregations

StreamConfig (org.apache.samza.config.StreamConfig)39 Config (org.apache.samza.config.Config)33 MapConfig (org.apache.samza.config.MapConfig)32 Test (org.junit.Test)31 JobConfig (org.apache.samza.config.JobConfig)23 StreamSpec (org.apache.samza.system.StreamSpec)17 Collection (java.util.Collection)11 StreamApplicationDescriptorImpl (org.apache.samza.application.descriptors.StreamApplicationDescriptorImpl)10 OperatorSpecGraph (org.apache.samza.operators.OperatorSpecGraph)10 SamzaSqlApplicationConfig (org.apache.samza.sql.runner.SamzaSqlApplicationConfig)9 SamzaSqlQueryParser (org.apache.samza.sql.util.SamzaSqlQueryParser)9 SamzaSqlTestConfig (org.apache.samza.sql.util.SamzaSqlTestConfig)9 SystemStream (org.apache.samza.system.SystemStream)6 HashMap (java.util.HashMap)3 ApplicationConfig (org.apache.samza.config.ApplicationConfig)3 HashMultimap (com.google.common.collect.HashMultimap)2 Multimap (com.google.common.collect.Multimap)2 ArrayList (java.util.ArrayList)2 Collections (java.util.Collections)2 HashSet (java.util.HashSet)2