Examples with SystemStreamPartition org.apache.samza.system.SystemStreamPartition used on opensource projects

Example 16 with SystemStreamPartition

use of org.apache.samza.system.SystemStreamPartition in project samza by apache.

the class TestTaskCallbackManager method testUpdateCallbackInOrder.

@Test
public void testUpdateCallbackInOrder() {
    TaskName taskName = new TaskName("Partition 0");
    SystemStreamPartition ssp = new SystemStreamPartition("kafka", "topic", new Partition(0));
    ReadableCoordinator coordinator = new ReadableCoordinator(taskName);
    IncomingMessageEnvelope envelope0 = new IncomingMessageEnvelope(ssp, "0", null, null);
    TaskCallbackImpl callback0 = new TaskCallbackImpl(listener, taskName, envelope0, coordinator, 0, 0);
    List<TaskCallbackImpl> callbacksToUpdate = callbackManager.updateCallback(callback0);
    assertEquals(1, callbacksToUpdate.size());
    TaskCallbackImpl callback = callbacksToUpdate.get(0);
    assertTrue(callback.matchSeqNum(0));
    assertEquals(ssp, callback.envelope.getSystemStreamPartition());
    assertEquals("0", callback.envelope.getOffset());
    IncomingMessageEnvelope envelope1 = new IncomingMessageEnvelope(ssp, "1", null, null);
    TaskCallbackImpl callback1 = new TaskCallbackImpl(listener, taskName, envelope1, coordinator, 1, 0);
    callbacksToUpdate = callbackManager.updateCallback(callback1);
    assertEquals(1, callbacksToUpdate.size());
    callback = callbacksToUpdate.get(0);
    assertTrue(callback.matchSeqNum(1));
    assertEquals(ssp, callback.envelope.getSystemStreamPartition());
    assertEquals("1", callback.envelope.getOffset());
}

Example 17 with SystemStreamPartition

use of org.apache.samza.system.SystemStreamPartition in project samza by apache.

the class HdfsSystemConsumer method getPartitionDescriptor.

private List<String> getPartitionDescriptor(SystemStreamPartition systemStreamPartition) {
    String streamName = systemStreamPartition.getStream();
    Partition partition = systemStreamPartition.getPartition();
    try {
        return cachedPartitionDescriptorMap.get(streamName).get(partition);
    } catch (ExecutionException e) {
        throw new SamzaException("Failed to obtain descriptor for " + systemStreamPartition, e);
    }
}

Example 18 with SystemStreamPartition

use of org.apache.samza.system.SystemStreamPartition in project samza by apache.

the class HdfsSystemConsumer method doPoll.

private void doPoll(MultiFileHdfsReader reader) {
    SystemStreamPartition systemStreamPartition = reader.getSystemStreamPartition();
    while (reader.hasNext() && !isShutdown) {
        IncomingMessageEnvelope messageEnvelope = reader.readNext();
        offerMessage(systemStreamPartition, messageEnvelope);
        consumerMetrics.incNumEvents(systemStreamPartition);
        consumerMetrics.incTotalNumEvents();
    }
    offerMessage(systemStreamPartition, IncomingMessageEnvelope.buildEndOfStreamEnvelope(systemStreamPartition));
    reader.close();
}

Example 19 with SystemStreamPartition

use of org.apache.samza.system.SystemStreamPartition in project samza by apache.

the class TestMultiFileHdfsReader method testSequentialRead.

@Test
public void testSequentialRead() throws Exception {
    SystemStreamPartition ssp = new SystemStreamPartition("hdfs", "testStream", new Partition(0));
    MultiFileHdfsReader multiReader = new MultiFileHdfsReader(HdfsReaderFactory.ReaderType.AVRO, ssp, Arrays.asList(descriptors), "0:0");
    int index = 0;
    while (multiReader.hasNext()) {
        GenericRecord record = (GenericRecord) multiReader.readNext().getMessage();
        Assert.assertEquals(index % NUM_EVENTS, record.get(FIELD_1));
        Assert.assertEquals("string_" + (index % NUM_EVENTS), record.get(FIELD_2).toString());
        index++;
    }
    Assert.assertEquals(3 * NUM_EVENTS, index);
    multiReader.close();
}

Example 20 with SystemStreamPartition

use of org.apache.samza.system.SystemStreamPartition in project samza by apache.

the class TestMultiFileHdfsReader method testOutOfRangeFileIndex.

@Test(expected = SamzaException.class)
public void testOutOfRangeFileIndex() {
    SystemStreamPartition ssp = new SystemStreamPartition("hdfs", "testStream", new Partition(0));
    new MultiFileHdfsReader(HdfsReaderFactory.ReaderType.AVRO, ssp, Arrays.asList(descriptors), "3:0");
    Assert.fail();
}