Search in sources :

Example 1 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class DiagnosticsUtil method createDiagnosticsStream.

public static void createDiagnosticsStream(Config config) {
    if (!new JobConfig(config).getDiagnosticsEnabled()) {
        return;
    }
    // if diagnostics is enabled, create diagnostics stream if it doesnt exist
    String diagnosticsSystemStreamName = new MetricsConfig(config).getMetricsSnapshotReporterStream(MetricsConfig.METRICS_SNAPSHOT_REPORTER_NAME_FOR_DIAGNOSTICS).orElseThrow(() -> new ConfigException("Missing required config: " + String.format(MetricsConfig.METRICS_SNAPSHOT_REPORTER_STREAM, MetricsConfig.METRICS_SNAPSHOT_REPORTER_NAME_FOR_DIAGNOSTICS)));
    SystemStream diagnosticsSystemStream = StreamUtil.getSystemStreamFromNames(diagnosticsSystemStreamName);
    SystemConfig systemConfig = new SystemConfig(config);
    SystemAdmin diagnosticsSysAdmin = systemConfig.getSystemFactories().get(diagnosticsSystemStream.getSystem()).getAdmin(diagnosticsSystemStream.getSystem(), config, DiagnosticsUtil.class.getSimpleName());
    StreamSpec diagnosticsStreamSpec = new StreamSpec(DIAGNOSTICS_STREAM_ID, diagnosticsSystemStream.getStream(), diagnosticsSystemStream.getSystem(), new StreamConfig(config).getStreamProperties(DIAGNOSTICS_STREAM_ID));
    log.info("Creating diagnostics stream {}", diagnosticsSystemStream.getStream());
    diagnosticsSysAdmin.start();
    if (diagnosticsSysAdmin.createStream(diagnosticsStreamSpec)) {
        log.info("Created diagnostics stream {}", diagnosticsSystemStream.getStream());
    } else {
        log.info("Diagnostics stream {} already exists", diagnosticsSystemStream.getStream());
    }
    diagnosticsSysAdmin.stop();
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) SystemConfig(org.apache.samza.config.SystemConfig) SystemStream(org.apache.samza.system.SystemStream) StreamConfig(org.apache.samza.config.StreamConfig) ConfigException(org.apache.samza.config.ConfigException) SystemAdmin(org.apache.samza.system.SystemAdmin) JobConfig(org.apache.samza.config.JobConfig) MetricsConfig(org.apache.samza.config.MetricsConfig)

Example 2 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamStreamIdEmpty.

// Empty strings are NOT allowed for streamId, because it's used as an identifier in the config.
@Test(expected = IllegalArgumentException.class)
public void testGetStreamStreamIdEmpty() {
    Config config = buildStreamConfig("", StreamConfig.SYSTEM, TEST_SYSTEM);
    StreamUtil.getStreamSpec("", new StreamConfig(config));
}
Also used : JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 3 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testStreamConfigOverrides.

@Test
public void testStreamConfigOverrides() {
    final String sysStreamPrefix = String.format("systems.%s.streams.%s.", TEST_SYSTEM, TEST_PHYSICAL_NAME);
    Config config = addConfigs(buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME, StreamConfig.SYSTEM, TEST_SYSTEM, "systemProperty1", "systemValue1", "systemProperty2", "systemValue2", "systemProperty3", "systemValue3"), sysStreamPrefix + "systemProperty4", "systemValue4", sysStreamPrefix + "systemProperty2", "systemValue8");
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    Map<String, String> properties = spec.getConfig();
    assertEquals(4, properties.size());
    assertEquals("systemValue4", properties.get("systemProperty4"));
    assertEquals("systemValue2", properties.get("systemProperty2"));
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 4 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamPropertiesPassthrough.

// The properties in the config "streams.{streamId}.*" should be passed through to the spec.
@Test
public void testGetStreamPropertiesPassthrough() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME, StreamConfig.SYSTEM, TEST_SYSTEM, "systemProperty1", "systemValue1", "systemProperty2", "systemValue2", "systemProperty3", "systemValue3");
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    Map<String, String> properties = spec.getConfig();
    assertEquals(3, properties.size());
    assertEquals("systemValue1", properties.get("systemProperty1"));
    assertEquals("systemValue2", properties.get("systemProperty2"));
    assertEquals("systemValue3", properties.get("systemProperty3"));
    assertEquals("systemValue1", spec.get("systemProperty1"));
    assertEquals("systemValue2", spec.get("systemProperty2"));
    assertEquals("systemValue3", spec.get("systemProperty3"));
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Example 5 with StreamConfig

use of org.apache.samza.config.StreamConfig in project samza by apache.

the class TestStreamUtil method testGetStreamWithSystemAtDefaultScopeInConfig.

// If system isn't specified at stream scope, use the default system
@Test
public void testGetStreamWithSystemAtDefaultScopeInConfig() {
    Config config = addConfigs(buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME, TEST_PHYSICAL_NAME), JobConfig.JOB_DEFAULT_SYSTEM, TEST_DEFAULT_SYSTEM);
    StreamSpec spec = StreamUtil.getStreamSpec(STREAM_ID, new StreamConfig(config));
    assertEquals(TEST_DEFAULT_SYSTEM, spec.getSystemName());
}
Also used : StreamSpec(org.apache.samza.system.StreamSpec) JobConfig(org.apache.samza.config.JobConfig) Config(org.apache.samza.config.Config) MapConfig(org.apache.samza.config.MapConfig) StreamConfig(org.apache.samza.config.StreamConfig) StreamConfig(org.apache.samza.config.StreamConfig) Test(org.junit.Test)

Aggregations

StreamConfig (org.apache.samza.config.StreamConfig)39 Config (org.apache.samza.config.Config)33 MapConfig (org.apache.samza.config.MapConfig)32 Test (org.junit.Test)31 JobConfig (org.apache.samza.config.JobConfig)23 StreamSpec (org.apache.samza.system.StreamSpec)17 Collection (java.util.Collection)11 StreamApplicationDescriptorImpl (org.apache.samza.application.descriptors.StreamApplicationDescriptorImpl)10 OperatorSpecGraph (org.apache.samza.operators.OperatorSpecGraph)10 SamzaSqlApplicationConfig (org.apache.samza.sql.runner.SamzaSqlApplicationConfig)9 SamzaSqlQueryParser (org.apache.samza.sql.util.SamzaSqlQueryParser)9 SamzaSqlTestConfig (org.apache.samza.sql.util.SamzaSqlTestConfig)9 SystemStream (org.apache.samza.system.SystemStream)6 HashMap (java.util.HashMap)3 ApplicationConfig (org.apache.samza.config.ApplicationConfig)3 HashMultimap (com.google.common.collect.HashMultimap)2 Multimap (com.google.common.collect.Multimap)2 ArrayList (java.util.ArrayList)2 Collections (java.util.Collections)2 HashSet (java.util.HashSet)2