Examples with StreamSpec org.apache.samza.system.StreamSpec used on opensource projects

Example 31 with StreamSpec

use of org.apache.samza.system.StreamSpec in project samza by apache.

the class TestAbstractApplicationRunner method testGetStreamSystemNameArgValid.

// When the system name is provided explicitly, it should be used, regardless of whether it's also in the config
@Test
public void testGetStreamSystemNameArgValid() {
    Config config = buildStreamConfig(STREAM_ID, // This should be ignored because of the explicit arg
    StreamConfig.PHYSICAL_NAME(), // This should be ignored because of the explicit arg
    TEST_PHYSICAL_NAME2, StreamConfig.SYSTEM(), // This too
    TEST_SYSTEM2);
    AbstractApplicationRunner runner = new TestAbstractApplicationRunnerImpl(config);
    StreamSpec spec = runner.getStreamSpec(STREAM_ID, TEST_PHYSICAL_NAME, TEST_SYSTEM);
    assertEquals(STREAM_ID, spec.getId());
    assertEquals(TEST_PHYSICAL_NAME, spec.getPhysicalName());
    assertEquals(TEST_SYSTEM, spec.getSystemName());
}

Example 32 with StreamSpec

use of org.apache.samza.system.StreamSpec in project samza by apache.

the class TestAbstractApplicationRunner method testGetStreamPhysicalNameArgNull.

// Null is allowed for the physical name
@Test
public void testGetStreamPhysicalNameArgNull() {
    Config config = buildStreamConfig(STREAM_ID, StreamConfig.PHYSICAL_NAME(), TEST_PHYSICAL_NAME2, StreamConfig.SYSTEM(), TEST_SYSTEM);
    AbstractApplicationRunner runner = new TestAbstractApplicationRunnerImpl(config);
    StreamSpec spec = runner.getStreamSpec(STREAM_ID, null);
    assertNull(spec.getPhysicalName());
}

Example 33 with StreamSpec

use of org.apache.samza.system.StreamSpec in project samza by apache.

the class StreamGraphImpl method getIntermediateStream.

/**
   * Internal helper for {@link MessageStreamImpl} to add an intermediate {@link MessageStream} to the graph.
   * An intermediate {@link MessageStream} is both an output and an input stream.
   *
   * @param streamName the name of the stream to be created. Will be prefixed with job name and id to generate the
   *                   logical streamId.
   * @param keyExtractor the {@link Function} to extract the outgoing key from the intermediate message
   * @param msgExtractor the {@link Function} to extract the outgoing message from the intermediate message
   * @param msgBuilder the {@link BiFunction} to convert the incoming key and message to a message
   *                   in the intermediate {@link MessageStream}
   * @param <K> the type of key in the intermediate message
   * @param <V> the type of message in the intermediate message
   * @param <M> the type of messages in the intermediate {@link MessageStream}
   * @return  the intermediate {@link MessageStreamImpl}
   */
<K, V, M> MessageStreamImpl<M> getIntermediateStream(String streamName, Function<? super M, ? extends K> keyExtractor, Function<? super M, ? extends V> msgExtractor, BiFunction<? super K, ? super V, ? extends M> msgBuilder) {
    String streamId = String.format("%s-%s-%s", config.get(JobConfig.JOB_NAME()), config.get(JobConfig.JOB_ID(), "1"), streamName);
    if (msgBuilder == null) {
        throw new IllegalArgumentException("msgBuilder cannot be null for an intermediate stream");
    }
    if (keyExtractor == null) {
        throw new IllegalArgumentException("keyExtractor can't be null for an output stream.");
    }
    if (msgExtractor == null) {
        throw new IllegalArgumentException("msgExtractor can't be null for an output stream.");
    }
    StreamSpec streamSpec = runner.getStreamSpec(streamId);
    IntermediateStreamInternalImpl<K, V, M> intStream = (IntermediateStreamInternalImpl<K, V, M>) inStreams.computeIfAbsent(streamSpec, k -> new IntermediateStreamInternalImpl<>(this, streamSpec, (Function<M, K>) keyExtractor, (Function<M, V>) msgExtractor, (BiFunction<K, V, M>) msgBuilder));
    outStreams.putIfAbsent(streamSpec, intStream);
    return intStream;
}

Example 34 with StreamSpec

use of org.apache.samza.system.StreamSpec in project samza by apache.

the class TestKafkaStreamSpec method testUnsupportedConfigStrippedFromProperties.

@Test
public void testUnsupportedConfigStrippedFromProperties() {
    StreamSpec original = new StreamSpec("dummyId", "dummyPhysicalName", "dummySystemName", ImmutableMap.of("segment.bytes", "4", "replication.factor", "7"));
    // First verify the original
    assertEquals("7", original.get("replication.factor"));
    assertEquals("4", original.get("segment.bytes"));
    Map<String, String> config = original.getConfig();
    assertEquals("7", config.get("replication.factor"));
    assertEquals("4", config.get("segment.bytes"));
    // Now verify the Kafka spec
    KafkaStreamSpec spec = KafkaStreamSpec.fromSpec(original);
    assertNull(spec.get("replication.factor"));
    assertEquals("4", spec.get("segment.bytes"));
    Properties kafkaProperties = spec.getProperties();
    Map<String, String> kafkaConfig = spec.getConfig();
    assertNull(kafkaProperties.get("replication.factor"));
    assertEquals("4", kafkaProperties.get("segment.bytes"));
    assertNull(kafkaConfig.get("replication.factor"));
    assertEquals("4", kafkaConfig.get("segment.bytes"));
}

Example 35 with StreamSpec

use of org.apache.samza.system.StreamSpec in project samza by apache.

the class TestKafkaSystemAdminJava method testValidateStreamDoesNotExist.

@Test(expected = StreamValidationException.class)
public void testValidateStreamDoesNotExist() {
    SystemAdmin admin = this.basicSystemAdmin;
    StreamSpec spec = new StreamSpec("testId", "testStreamNameExist", "testSystem", 8);
    admin.validateStream(spec);
}