Search in sources :

Example 11 with SubscriptionCursorWithoutToken

use of org.zalando.nakadi.view.SubscriptionCursorWithoutToken in project nakadi by zalando.

the class VersionZeroConverter method convertBatched.

public List<NakadiCursor> convertBatched(final List<SubscriptionCursorWithoutToken> cursors) throws InvalidCursorException, InternalNakadiException, NoSuchEventTypeException, ServiceUnavailableException {
    final NakadiCursor[] result = new NakadiCursor[cursors.size()];
    for (int idx = 0; idx < cursors.size(); ++idx) {
        final SubscriptionCursorWithoutToken cursor = cursors.get(idx);
        if (Cursor.BEFORE_OLDEST_OFFSET.equalsIgnoreCase(cursor.getOffset())) {
            // Preform begin checks afterwards to optimize calls
            continue;
        }
        if (!NUMBERS_ONLY_PATTERN.matcher(cursor.getOffset()).matches()) {
            throw new InvalidCursorException(CursorError.INVALID_OFFSET, cursor);
        }
    }
    // now it is time for massive convert.
    final LinkedHashMap<SubscriptionCursorWithoutToken, NakadiCursor> beginsToConvert = new LinkedHashMap<>();
    final Map<SubscriptionCursorWithoutToken, Timeline> cursorTimelines = new HashMap<>();
    final Map<TopicRepository, List<SubscriptionCursorWithoutToken>> repos = new HashMap<>();
    for (int i = 0; i < result.length; ++i) {
        if (null == result[i]) {
            // cursor requires database hit
            final SubscriptionCursorWithoutToken cursor = cursors.get(i);
            final Timeline timeline = timelineService.getActiveTimelinesOrdered(cursor.getEventType()).get(0);
            final TopicRepository topicRepo = timelineService.getTopicRepository(timeline);
            beginsToConvert.put(cursor, null);
            cursorTimelines.put(cursor, timeline);
            repos.computeIfAbsent(topicRepo, k -> new ArrayList<>()).add(cursor);
        }
    }
    for (final Map.Entry<TopicRepository, List<SubscriptionCursorWithoutToken>> entry : repos.entrySet()) {
        final List<Optional<PartitionStatistics>> stats = entry.getKey().loadPartitionStatistics(entry.getValue().stream().map(scwt -> new TopicRepository.TimelinePartition(cursorTimelines.get(scwt), scwt.getPartition())).collect(Collectors.toList()));
        for (int idx = 0; idx < entry.getValue().size(); ++idx) {
            // Reinsert doesn't change the order
            beginsToConvert.put(entry.getValue().get(idx), stats.get(idx).orElseThrow(() -> new InvalidCursorException(PARTITION_NOT_FOUND)).getBeforeFirst());
        }
    }
    final Iterator<NakadiCursor> missingBegins = beginsToConvert.values().iterator();
    return Stream.of(result).map(it -> null == it ? missingBegins.next() : it).collect(Collectors.toList());
}
Also used : NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) HashMap(java.util.HashMap) NUMBERS_ONLY_PATTERN(org.zalando.nakadi.util.CursorConversionUtils.NUMBERS_ONLY_PATTERN) Cursor(org.zalando.nakadi.view.Cursor) StringUtils(org.apache.commons.lang3.StringUtils) NoSuchEventTypeException(org.zalando.nakadi.exceptions.NoSuchEventTypeException) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) PARTITION_NOT_FOUND(org.zalando.nakadi.domain.CursorError.PARTITION_NOT_FOUND) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) Map(java.util.Map) TimelineService(org.zalando.nakadi.service.timeline.TimelineService) TopicRepository(org.zalando.nakadi.repository.TopicRepository) Iterator(java.util.Iterator) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) Collectors(java.util.stream.Collectors) List(java.util.List) Stream(java.util.stream.Stream) Timeline(org.zalando.nakadi.domain.Timeline) Optional(java.util.Optional) CursorError(org.zalando.nakadi.domain.CursorError) InternalNakadiException(org.zalando.nakadi.exceptions.InternalNakadiException) CursorConverter(org.zalando.nakadi.service.CursorConverter) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) Optional(java.util.Optional) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) Timeline(org.zalando.nakadi.domain.Timeline) ArrayList(java.util.ArrayList) List(java.util.List) TopicRepository(org.zalando.nakadi.repository.TopicRepository) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) Map(java.util.Map)

Example 12 with SubscriptionCursorWithoutToken

use of org.zalando.nakadi.view.SubscriptionCursorWithoutToken in project nakadi by zalando.

the class CursorsServiceAT method whenGetSubscriptionCursorsThenOk.

@Test
public void whenGetSubscriptionCursorsThenOk() throws Exception {
    setPartitions(new Partition[] { new Partition(etName, P1, "", "", Partition.State.UNASSIGNED), new Partition(etName, P2, "", "", Partition.State.UNASSIGNED) });
    final List<SubscriptionCursorWithoutToken> cursors = cursorsService.getSubscriptionCursors(sid);
    assertThat(ImmutableSet.copyOf(cursors), equalTo(ImmutableSet.of(new SubscriptionCursorWithoutToken(etName, P1, OLD_OFFSET), new SubscriptionCursorWithoutToken(etName, P2, OLD_OFFSET))));
}
Also used : Partition(org.zalando.nakadi.service.subscription.model.Partition) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) Test(org.junit.Test)

Example 13 with SubscriptionCursorWithoutToken

use of org.zalando.nakadi.view.SubscriptionCursorWithoutToken in project nakadi by zalando.

the class SubscriptionAT method testSubscriptionWithInitialCursors.

@Test
public void testSubscriptionWithInitialCursors() throws Exception {
    final EventType et1 = createBusinessEventTypeWithPartitions(2);
    final EventType et2 = createBusinessEventTypeWithPartitions(2);
    // write 10 events to each partition of two event-types
    publishBusinessEventWithUserDefinedPartition(et1.getName(), 10, i -> "dummy", i -> "0");
    publishBusinessEventWithUserDefinedPartition(et1.getName(), 10, i -> "dummy", i -> "1");
    publishBusinessEventWithUserDefinedPartition(et2.getName(), 10, i -> "dummy", i -> "0");
    publishBusinessEventWithUserDefinedPartition(et2.getName(), 10, i -> "dummy", i -> "1");
    // create subscription with initial cursors
    final SubscriptionBase subscriptionBase = RandomSubscriptionBuilder.builder().withEventTypes(ImmutableSet.of(et1.getName(), et2.getName())).withStartFrom(SubscriptionBase.InitialPosition.CURSORS).withInitialCursors(ImmutableList.of(new SubscriptionCursorWithoutToken(et1.getName(), "0", "000000000000000007"), new SubscriptionCursorWithoutToken(et1.getName(), "1", "000000000000000002"), new SubscriptionCursorWithoutToken(et2.getName(), "0", Cursor.BEFORE_OLDEST_OFFSET), new SubscriptionCursorWithoutToken(et2.getName(), "1", "000000000000000009"))).buildSubscriptionBase();
    final Subscription subscription = createSubscription(subscriptionBase);
    final TestStreamingClient client = TestStreamingClient.create(URL, subscription.getId(), "max_uncommitted_events=100").start();
    // we should read 19 events in total
    waitFor(() -> assertThat(client.getBatches(), hasSize(19)));
    final List<StreamBatch> batches = client.getBatches();
    // check that first events of each partition have correct offsets
    assertThat(getFirstBatchOffsetFor(batches, new EventTypePartition(et1.getName(), "0")), equalTo(Optional.of("001-0001-000000000000000008")));
    assertThat(getFirstBatchOffsetFor(batches, new EventTypePartition(et1.getName(), "1")), equalTo(Optional.of("001-0001-000000000000000003")));
    assertThat(getFirstBatchOffsetFor(batches, new EventTypePartition(et2.getName(), "0")), equalTo(Optional.of("001-0001-000000000000000000")));
    assertThat(getFirstBatchOffsetFor(batches, new EventTypePartition(et2.getName(), "1")), equalTo(Optional.empty()));
}
Also used : SubscriptionBase(org.zalando.nakadi.domain.SubscriptionBase) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) TestStreamingClient(org.zalando.nakadi.webservice.utils.TestStreamingClient) TestUtils.buildDefaultEventType(org.zalando.nakadi.utils.TestUtils.buildDefaultEventType) NakadiTestUtils.createSubscriptionForEventType(org.zalando.nakadi.webservice.utils.NakadiTestUtils.createSubscriptionForEventType) EventType(org.zalando.nakadi.domain.EventType) Subscription(org.zalando.nakadi.domain.Subscription) NakadiTestUtils.createSubscription(org.zalando.nakadi.webservice.utils.NakadiTestUtils.createSubscription) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) Test(org.junit.Test)

Example 14 with SubscriptionCursorWithoutToken

use of org.zalando.nakadi.view.SubscriptionCursorWithoutToken in project nakadi by zalando.

the class SubscriptionConsumptionTest method testAllEventsConsumed.

@Test
public void testAllEventsConsumed() throws IOException, InterruptedException {
    final String[] expected = new String[] { "001-0001-000000000000000000", "001-0001-000000000000000001", "001-0002-000000000000000000", "001-0002-000000000000000001", "001-0003-000000000000000000", "001-0003-000000000000000001", "001-0004-000000000000000000", "001-0004-000000000000000001" };
    // Do not test last case, because it makes no sense...
    for (int idx = -1; idx < expected.length - 1; ++idx) {
        final CountDownLatch finished = new CountDownLatch(1);
        final AtomicReference<String[]> receivedOffset = new AtomicReference<>();
        final Subscription subscription = createSubscription(RandomSubscriptionBuilder.builder().withEventType(eventType.getName()).withStartFrom(SubscriptionBase.InitialPosition.CURSORS).withInitialCursors(Collections.singletonList(new SubscriptionCursorWithoutToken(eventType.getName(), "0", idx == -1 ? "BEGIN" : expected[idx]))).build());
        createParallelConsumer(subscription, expected.length - 1 - idx, finished, receivedOffset::set);
        finished.await();
        final String[] testedOffsets = Arrays.copyOfRange(expected, idx + 1, expected.length);
        Assert.assertArrayEquals(testedOffsets, receivedOffset.get());
    }
}
Also used : SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) AtomicReference(java.util.concurrent.atomic.AtomicReference) CountDownLatch(java.util.concurrent.CountDownLatch) Subscription(org.zalando.nakadi.domain.Subscription) NakadiTestUtils.createSubscription(org.zalando.nakadi.webservice.utils.NakadiTestUtils.createSubscription) Test(org.junit.Test)

Example 15 with SubscriptionCursorWithoutToken

use of org.zalando.nakadi.view.SubscriptionCursorWithoutToken in project nakadi by zalando.

the class HilaAT method whenPatchThenCursorsAreInitializedAndPatched.

@Test(timeout = 15000)
public void whenPatchThenCursorsAreInitializedAndPatched() throws Exception {
    final EventType et = createEventType();
    publishEvents(et.getName(), 10, i -> "{\"foo\": \"bar\"}");
    final List<EventTypePartitionView> etStats = MAPPER.readValue(given().get("/event-types/{et}/partitions", et.getName()).getBody().asString(), new TypeReference<List<EventTypePartitionView>>() {
    });
    final EventTypePartitionView begin = etStats.get(0);
    final Subscription s = createSubscription(RandomSubscriptionBuilder.builder().withEventType(et.getName()).withStartFrom(END).buildSubscriptionBase());
    given().body(MAPPER.writeValueAsString(new ItemsWrapper<>(Collections.singletonList(new SubscriptionCursorWithoutToken(et.getName(), begin.getPartitionId(), begin.getOldestAvailableOffset()))))).contentType(JSON).patch("/subscriptions/{id}/cursors", s.getId()).then().statusCode(SC_NO_CONTENT);
    final ItemsWrapper<SubscriptionCursor> subscriptionCursors = MAPPER.readValue(given().get("/subscriptions/{id}/cursors", s.getId()).getBody().asString(), new TypeReference<ItemsWrapper<SubscriptionCursor>>() {
    });
    Assert.assertEquals(subscriptionCursors.getItems().size(), etStats.size());
    subscriptionCursors.getItems().forEach(item -> {
        if (item.getPartition().equals(begin.getPartitionId())) {
            Assert.assertEquals(begin.getOldestAvailableOffset(), item.getOffset());
        } else {
            Assert.assertEquals(begin.getNewestAvailableOffset(), item.getOffset());
        }
    });
}
Also used : SubscriptionCursor(org.zalando.nakadi.view.SubscriptionCursor) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) NakadiTestUtils.createEventType(org.zalando.nakadi.webservice.utils.NakadiTestUtils.createEventType) EventType(org.zalando.nakadi.domain.EventType) EventTypePartitionView(org.zalando.nakadi.view.EventTypePartitionView) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) Subscription(org.zalando.nakadi.domain.Subscription) NakadiTestUtils.createSubscription(org.zalando.nakadi.webservice.utils.NakadiTestUtils.createSubscription) ItemsWrapper(org.zalando.nakadi.domain.ItemsWrapper) Test(org.junit.Test)

Aggregations

SubscriptionCursorWithoutToken (org.zalando.nakadi.view.SubscriptionCursorWithoutToken)21 Test (org.junit.Test)12 NakadiCursor (org.zalando.nakadi.domain.NakadiCursor)9 EventTypePartition (org.zalando.nakadi.domain.EventTypePartition)8 Subscription (org.zalando.nakadi.domain.Subscription)5 Partition (org.zalando.nakadi.service.subscription.model.Partition)5 List (java.util.List)4 Map (java.util.Map)4 PartitionStatistics (org.zalando.nakadi.domain.PartitionStatistics)4 TopicRepository (org.zalando.nakadi.repository.TopicRepository)4 ImmutableList (com.google.common.collect.ImmutableList)3 HashMap (java.util.HashMap)3 Storage (org.zalando.nakadi.domain.Storage)3 InvalidCursorException (org.zalando.nakadi.exceptions.InvalidCursorException)3 ServiceUnavailableException (org.zalando.nakadi.exceptions.ServiceUnavailableException)3 NakadiTestUtils.createSubscription (org.zalando.nakadi.webservice.utils.NakadiTestUtils.createSubscription)3 UTF_8 (com.google.common.base.Charsets.UTF_8)2 IOException (java.io.IOException)2 ArrayList (java.util.ArrayList)2 Collection (java.util.Collection)2