Search in sources :

Example 1 with SinglePartitionWithoutOffsetsSystemAdmin

use of org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin in project samza by apache.

the class TestCoordinatorStreamSystemConsumer method testOrderKeyRewrite.

/**
 * Verify that if a particular key-value is written, then another, then the original again,
 * that the original occurs last in the set.
 */
@Test
public void testOrderKeyRewrite() throws InterruptedException {
    final SystemStream systemStream = new SystemStream("system", "stream");
    final SystemStreamPartition ssp = new SystemStreamPartition(systemStream, new Partition(0));
    final SystemConsumer systemConsumer = mock(SystemConsumer.class);
    final List<IncomingMessageEnvelope> list = new ArrayList<>();
    SetConfig setConfig1 = new SetConfig("source", "key1", "value1");
    SetConfig setConfig2 = new SetConfig("source", "key1", "value2");
    SetConfig setConfig3 = new SetConfig("source", "key1", "value1");
    list.add(createIncomingMessageEnvelope(setConfig1, ssp));
    list.add(createIncomingMessageEnvelope(setConfig2, ssp));
    list.add(createIncomingMessageEnvelope(setConfig3, ssp));
    Map<SystemStreamPartition, List<IncomingMessageEnvelope>> messages = new HashMap<SystemStreamPartition, List<IncomingMessageEnvelope>>() {

        {
            put(ssp, list);
        }
    };
    when(systemConsumer.poll(anySet(), anyLong())).thenReturn(messages, Collections.<SystemStreamPartition, List<IncomingMessageEnvelope>>emptyMap());
    CoordinatorStreamSystemConsumer consumer = new CoordinatorStreamSystemConsumer(systemStream, systemConsumer, new SinglePartitionWithoutOffsetsSystemAdmin());
    consumer.bootstrap();
    Set<CoordinatorStreamMessage> bootstrappedMessages = consumer.getBootstrappedStream(SetConfig.TYPE);
    // First message should have been removed as a duplicate
    assertEquals(2, bootstrappedMessages.size());
    CoordinatorStreamMessage[] coordinatorStreamMessages = bootstrappedMessages.toArray(new CoordinatorStreamMessage[2]);
    assertEquals(setConfig2, coordinatorStreamMessages[0]);
    // Config 3 MUST be the last message, not config 2
    assertEquals(setConfig3, coordinatorStreamMessages[1]);
}
Also used : SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Partition(org.apache.samza.Partition) SystemConsumer(org.apache.samza.system.SystemConsumer) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) SystemStream(org.apache.samza.system.SystemStream) IncomingMessageEnvelope(org.apache.samza.system.IncomingMessageEnvelope) ArrayList(java.util.ArrayList) CoordinatorStreamMessage(org.apache.samza.coordinator.stream.messages.CoordinatorStreamMessage) SetConfig(org.apache.samza.coordinator.stream.messages.SetConfig) SinglePartitionWithoutOffsetsSystemAdmin(org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin) ArrayList(java.util.ArrayList) List(java.util.List) SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Test(org.junit.Test)

Example 2 with SinglePartitionWithoutOffsetsSystemAdmin

use of org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin in project samza by apache.

the class TestCoordinatorStreamSystemConsumer method testCoordinatorStreamSystemConsumer.

@Test
public void testCoordinatorStreamSystemConsumer() {
    Map<String, String> expectedConfig = new LinkedHashMap<String, String>();
    expectedConfig.put("job.id", "1234");
    SystemStream systemStream = new SystemStream("system", "stream");
    MockSystemConsumer systemConsumer = new MockSystemConsumer(new SystemStreamPartition(systemStream, new Partition(0)));
    CoordinatorStreamSystemConsumer consumer = new CoordinatorStreamSystemConsumer(systemStream, systemConsumer, new SinglePartitionWithoutOffsetsSystemAdmin());
    assertEquals(0, systemConsumer.getRegisterCount());
    consumer.register();
    assertEquals(1, systemConsumer.getRegisterCount());
    assertFalse(systemConsumer.isStarted());
    consumer.start();
    assertTrue(systemConsumer.isStarted());
    try {
        consumer.getConfig();
        fail("Should have failed when retrieving config before bootstrapping.");
    } catch (SamzaException e) {
    // Expected.
    }
    consumer.bootstrap();
    assertEquals(expectedConfig, consumer.getConfig());
    assertFalse(systemConsumer.isStopped());
    consumer.stop();
    assertTrue(systemConsumer.isStopped());
}
Also used : SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Partition(org.apache.samza.Partition) SystemStream(org.apache.samza.system.SystemStream) SinglePartitionWithoutOffsetsSystemAdmin(org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin) SamzaException(org.apache.samza.SamzaException) LinkedHashMap(java.util.LinkedHashMap) SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Test(org.junit.Test)

Example 3 with SinglePartitionWithoutOffsetsSystemAdmin

use of org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin in project samza by apache.

the class TestCoordinatorStreamSystemConsumer method testCoordinatorStreamSystemConsumerRegisterOnceOnly.

@Test
public void testCoordinatorStreamSystemConsumerRegisterOnceOnly() throws Exception {
    Map<String, String> expectedConfig = new LinkedHashMap<String, String>();
    expectedConfig.put("job.id", "1234");
    SystemStream systemStream = new SystemStream("system", "stream");
    MockSystemConsumer systemConsumer = new MockSystemConsumer(new SystemStreamPartition(systemStream, new Partition(0)));
    CoordinatorStreamSystemConsumer consumer = new CoordinatorStreamSystemConsumer(systemStream, systemConsumer, new SinglePartitionWithoutOffsetsSystemAdmin());
    assertEquals(0, systemConsumer.getRegisterCount());
    consumer.register();
    assertEquals(1, systemConsumer.getRegisterCount());
    assertFalse(systemConsumer.isStarted());
    consumer.start();
    assertTrue(systemConsumer.isStarted());
    consumer.register();
    assertEquals(1, systemConsumer.getRegisterCount());
}
Also used : SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Partition(org.apache.samza.Partition) SystemStream(org.apache.samza.system.SystemStream) SinglePartitionWithoutOffsetsSystemAdmin(org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin) LinkedHashMap(java.util.LinkedHashMap) SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Test(org.junit.Test)

Aggregations

LinkedHashMap (java.util.LinkedHashMap)3 Partition (org.apache.samza.Partition)3 SystemStream (org.apache.samza.system.SystemStream)3 SystemStreamPartition (org.apache.samza.system.SystemStreamPartition)3 SinglePartitionWithoutOffsetsSystemAdmin (org.apache.samza.util.SinglePartitionWithoutOffsetsSystemAdmin)3 Test (org.junit.Test)3 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 List (java.util.List)1 SamzaException (org.apache.samza.SamzaException)1 CoordinatorStreamMessage (org.apache.samza.coordinator.stream.messages.CoordinatorStreamMessage)1 SetConfig (org.apache.samza.coordinator.stream.messages.SetConfig)1 IncomingMessageEnvelope (org.apache.samza.system.IncomingMessageEnvelope)1 SystemConsumer (org.apache.samza.system.SystemConsumer)1