Search in sources :

Example 1 with PartitionEndStatistics

use of org.zalando.nakadi.domain.PartitionEndStatistics in project nakadi by zalando.

the class CursorOperationsService method cursorsLag.

public List<NakadiCursorLag> cursorsLag(final String eventTypeName, final List<NakadiCursor> cursors) throws InvalidCursorOperation {
    try {
        final List<Timeline> timelines = timelineService.getActiveTimelinesOrdered(eventTypeName);
        // Next 2 calls could be optimized to 1 storage call, instead of possible 2 calls.
        // But it is simpler not to do anything, cause timelines are not switched every day and almost all the time
        // (except retention time after switch) there will be only 1 active timeline, and this option is covered.
        final List<PartitionStatistics> oldestStats = getStatsForTimeline(timelines.get(0));
        final List<PartitionStatistics> newestStats = timelines.size() == 1 ? oldestStats : getStatsForTimeline(timelines.get(timelines.size() - 1));
        return cursors.stream().map(c -> {
            final PartitionStatistics oldestStat = oldestStats.stream().filter(item -> item.getPartition().equalsIgnoreCase(c.getPartition())).findAny().orElseThrow(() -> new InvalidCursorOperation(PARTITION_NOT_FOUND));
            NakadiCursor newestPosition = newestStats.stream().filter(item -> item.getPartition().equalsIgnoreCase(c.getPartition())).map(PartitionEndStatistics::getLast).findAny().orElseThrow(() -> new InvalidCursorOperation(PARTITION_NOT_FOUND));
            // it
            while (numberOfEventsBeforeCursor(newestPosition) == -1) {
                final int prevOrder = newestPosition.getTimeline().getOrder() - 1;
                final Timeline prevTimeline = timelines.stream().filter(t -> t.getOrder() == prevOrder).findAny().orElse(null);
                if (null == prevTimeline) {
                    break;
                }
                // We moved back, so timeline definitely have latest position set
                newestPosition = prevTimeline.getLatestPosition().toNakadiCursor(prevTimeline, newestPosition.getPartition());
            }
            // calls (in case of kafka)
            return new NakadiCursorLag(oldestStat.getFirst(), newestPosition, calculateDistance(c, newestPosition));
        }).collect(Collectors.toList());
    } catch (final NakadiException e) {
        throw new MyNakadiRuntimeException1("error", e);
    }
}
Also used : Storage(org.zalando.nakadi.domain.Storage) Logger(org.slf4j.Logger) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) LoggerFactory(org.slf4j.LoggerFactory) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) Autowired(org.springframework.beans.factory.annotation.Autowired) NakadiException(org.zalando.nakadi.exceptions.NakadiException) MyNakadiRuntimeException1(org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1) InvalidCursorOperation(org.zalando.nakadi.exceptions.runtime.InvalidCursorOperation) Collectors(java.util.stream.Collectors) KafkaCursor(org.zalando.nakadi.repository.kafka.KafkaCursor) TIMELINE_NOT_FOUND(org.zalando.nakadi.exceptions.runtime.InvalidCursorOperation.Reason.TIMELINE_NOT_FOUND) List(java.util.List) UnknownStorageTypeException(org.zalando.nakadi.exceptions.runtime.UnknownStorageTypeException) Timeline(org.zalando.nakadi.domain.Timeline) PARTITION_NOT_FOUND(org.zalando.nakadi.exceptions.runtime.InvalidCursorOperation.Reason.PARTITION_NOT_FOUND) Service(org.springframework.stereotype.Service) NakadiCursorLag(org.zalando.nakadi.domain.NakadiCursorLag) ShiftedNakadiCursor(org.zalando.nakadi.domain.ShiftedNakadiCursor) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) TimelineService(org.zalando.nakadi.service.timeline.TimelineService) CURSORS_WITH_DIFFERENT_PARTITION(org.zalando.nakadi.exceptions.runtime.InvalidCursorOperation.Reason.CURSORS_WITH_DIFFERENT_PARTITION) Collections(java.util.Collections) Timeline(org.zalando.nakadi.domain.Timeline) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) ShiftedNakadiCursor(org.zalando.nakadi.domain.ShiftedNakadiCursor) MyNakadiRuntimeException1(org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) InvalidCursorOperation(org.zalando.nakadi.exceptions.runtime.InvalidCursorOperation) NakadiCursorLag(org.zalando.nakadi.domain.NakadiCursorLag) NakadiException(org.zalando.nakadi.exceptions.NakadiException)

Example 2 with PartitionEndStatistics

use of org.zalando.nakadi.domain.PartitionEndStatistics in project nakadi by zalando.

the class SubscriptionService method loadPartitionEndStatistics.

private List<PartitionEndStatistics> loadPartitionEndStatistics(final Collection<EventType> eventTypes) throws ServiceUnavailableException {
    final List<PartitionEndStatistics> topicPartitions = new ArrayList<>();
    final Map<TopicRepository, List<Timeline>> timelinesByRepo = eventTypes.stream().map(timelineService::getActiveTimeline).collect(Collectors.groupingBy(timelineService::getTopicRepository));
    for (final Map.Entry<TopicRepository, List<Timeline>> repoEntry : timelinesByRepo.entrySet()) {
        final TopicRepository topicRepository = repoEntry.getKey();
        final List<Timeline> timelinesForRepo = repoEntry.getValue();
        topicPartitions.addAll(topicRepository.loadTopicEndStatistics(timelinesForRepo));
    }
    return topicPartitions;
}
Also used : Timeline(org.zalando.nakadi.domain.Timeline) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) ArrayList(java.util.ArrayList) List(java.util.List) ArrayList(java.util.ArrayList) TopicRepository(org.zalando.nakadi.repository.TopicRepository) Map(java.util.Map)

Example 3 with PartitionEndStatistics

use of org.zalando.nakadi.domain.PartitionEndStatistics in project nakadi by zalando.

the class KafkaTopicRepository method loadTopicEndStatistics.

@Override
public List<PartitionEndStatistics> loadTopicEndStatistics(final Collection<Timeline> timelines) throws ServiceUnavailableException {
    try (Consumer<byte[], byte[]> consumer = kafkaFactory.getConsumer()) {
        final Map<TopicPartition, Timeline> backMap = new HashMap<>();
        for (final Timeline timeline : timelines) {
            consumer.partitionsFor(timeline.getTopic()).stream().map(p -> new TopicPartition(p.topic(), p.partition())).forEach(tp -> backMap.put(tp, timeline));
        }
        final List<TopicPartition> kafkaTPs = newArrayList(backMap.keySet());
        consumer.assign(kafkaTPs);
        consumer.seekToEnd(kafkaTPs);
        return backMap.entrySet().stream().map(e -> {
            final TopicPartition tp = e.getKey();
            final Timeline timeline = e.getValue();
            return new KafkaPartitionEndStatistics(timeline, tp.partition(), consumer.position(tp) - 1);
        }).collect(toList());
    } catch (final Exception e) {
        throw new ServiceUnavailableException("Error occurred when fetching partitions offsets", e);
    }
}
Also used : EventPublishingException(org.zalando.nakadi.exceptions.EventPublishingException) NotLeaderForPartitionException(org.apache.kafka.common.errors.NotLeaderForPartitionException) Collections.unmodifiableList(java.util.Collections.unmodifiableList) LoggerFactory(org.slf4j.LoggerFactory) TimeoutException(java.util.concurrent.TimeoutException) TopicRepositoryException(org.zalando.nakadi.exceptions.runtime.TopicRepositoryException) PARTITION_NOT_FOUND(org.zalando.nakadi.domain.CursorError.PARTITION_NOT_FOUND) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) Map(java.util.Map) RetryForSpecifiedTimeStrategy(org.echocat.jomon.runtime.concurrent.RetryForSpecifiedTimeStrategy) Consumer(org.apache.kafka.clients.consumer.Consumer) ZooKeeperHolder(org.zalando.nakadi.repository.zookeeper.ZooKeeperHolder) TopicPartition(org.apache.kafka.common.TopicPartition) TopicRepository(org.zalando.nakadi.repository.TopicRepository) Retryer(org.echocat.jomon.runtime.concurrent.Retryer) Collection(java.util.Collection) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) ConfigType(kafka.server.ConfigType) PartitionInfo(org.apache.kafka.common.PartitionInfo) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) Collectors(java.util.stream.Collectors) TopicDeletionException(org.zalando.nakadi.exceptions.TopicDeletionException) Objects(java.util.Objects) ZkUtils(kafka.utils.ZkUtils) TopicExistsException(org.apache.kafka.common.errors.TopicExistsException) List(java.util.List) Stream(java.util.stream.Stream) Lists.newArrayList(com.google.common.collect.Lists.newArrayList) Timeline(org.zalando.nakadi.domain.Timeline) ZookeeperSettings(org.zalando.nakadi.repository.zookeeper.ZookeeperSettings) NULL_OFFSET(org.zalando.nakadi.domain.CursorError.NULL_OFFSET) BatchItem(org.zalando.nakadi.domain.BatchItem) Optional(java.util.Optional) UnknownTopicOrPartitionException(org.apache.kafka.common.errors.UnknownTopicOrPartitionException) AdminUtils(kafka.admin.AdminUtils) IntStream(java.util.stream.IntStream) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) NetworkException(org.apache.kafka.common.errors.NetworkException) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) NakadiSettings(org.zalando.nakadi.config.NakadiSettings) TopicCreationException(org.zalando.nakadi.exceptions.TopicCreationException) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) TopicConfigException(org.zalando.nakadi.exceptions.runtime.TopicConfigException) UnknownServerException(org.apache.kafka.common.errors.UnknownServerException) ArrayList(java.util.ArrayList) ConcurrentMap(java.util.concurrent.ConcurrentMap) UUIDGenerator(org.zalando.nakadi.util.UUIDGenerator) InterruptException(org.apache.kafka.common.errors.InterruptException) EventPublishingStep(org.zalando.nakadi.domain.EventPublishingStep) Nullable(javax.annotation.Nullable) UNAVAILABLE(org.zalando.nakadi.domain.CursorError.UNAVAILABLE) NULL_PARTITION(org.zalando.nakadi.domain.CursorError.NULL_PARTITION) Logger(org.slf4j.Logger) Properties(java.util.Properties) Producer(org.apache.kafka.clients.producer.Producer) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) EventConsumer(org.zalando.nakadi.repository.EventConsumer) Collectors.toList(java.util.stream.Collectors.toList) EventPublishingStatus(org.zalando.nakadi.domain.EventPublishingStatus) Preconditions(com.google.common.base.Preconditions) Collections(java.util.Collections) RackAwareMode(kafka.admin.RackAwareMode) Timeline(org.zalando.nakadi.domain.Timeline) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) HashMap(java.util.HashMap) TopicPartition(org.apache.kafka.common.TopicPartition) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) EventPublishingException(org.zalando.nakadi.exceptions.EventPublishingException) NotLeaderForPartitionException(org.apache.kafka.common.errors.NotLeaderForPartitionException) TimeoutException(java.util.concurrent.TimeoutException) TopicRepositoryException(org.zalando.nakadi.exceptions.runtime.TopicRepositoryException) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) TopicDeletionException(org.zalando.nakadi.exceptions.TopicDeletionException) TopicExistsException(org.apache.kafka.common.errors.TopicExistsException) UnknownTopicOrPartitionException(org.apache.kafka.common.errors.UnknownTopicOrPartitionException) NetworkException(org.apache.kafka.common.errors.NetworkException) TopicCreationException(org.zalando.nakadi.exceptions.TopicCreationException) TopicConfigException(org.zalando.nakadi.exceptions.runtime.TopicConfigException) UnknownServerException(org.apache.kafka.common.errors.UnknownServerException) InterruptException(org.apache.kafka.common.errors.InterruptException) ExecutionException(java.util.concurrent.ExecutionException)

Example 4 with PartitionEndStatistics

use of org.zalando.nakadi.domain.PartitionEndStatistics in project nakadi by zalando.

the class KafkaTopicRepositoryTest method canLoadPartitionEndStatistics.

@Test
public void canLoadPartitionEndStatistics() throws Exception {
    final Timeline t1 = mock(Timeline.class);
    when(t1.getTopic()).thenReturn(MY_TOPIC);
    final Timeline t2 = mock(Timeline.class);
    when(t2.getTopic()).thenReturn(ANOTHER_TOPIC);
    final ImmutableList<Timeline> timelines = ImmutableList.of(t1, t2);
    final List<PartitionEndStatistics> stats = kafkaTopicRepository.loadTopicEndStatistics(timelines);
    final Set<PartitionEndStatistics> expected = PARTITIONS.stream().map(p -> {
        final Timeline timeline = p.topic.equals(MY_TOPIC) ? t1 : t2;
        return new KafkaPartitionEndStatistics(timeline, p.partition, p.latestOffset - 1);
    }).collect(Collectors.toSet());
    assertThat(newHashSet(stats), equalTo(expected));
}
Also used : EventPublishingException(org.zalando.nakadi.exceptions.EventPublishingException) GetChildrenBuilder(org.apache.curator.framework.api.GetChildrenBuilder) KafkaException(org.apache.kafka.common.KafkaException) NakadiException(org.zalando.nakadi.exceptions.NakadiException) Assert.assertThat(org.junit.Assert.assertThat) Future(java.util.concurrent.Future) Arrays.asList(java.util.Arrays.asList) Mockito.doAnswer(org.mockito.Mockito.doAnswer) Sets.newHashSet(com.google.common.collect.Sets.newHashSet) Assert.fail(org.junit.Assert.fail) Matchers.anyVararg(org.mockito.Matchers.anyVararg) Consumer(org.apache.kafka.clients.consumer.Consumer) ZooKeeperHolder(org.zalando.nakadi.repository.zookeeper.ZooKeeperHolder) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) Set(java.util.Set) PartitionInfo(org.apache.kafka.common.PartitionInfo) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) Collectors(java.util.stream.Collectors) Matchers.any(org.mockito.Matchers.any) List(java.util.List) CuratorFramework(org.apache.curator.framework.CuratorFramework) TestUtils.buildTimelineWithTopic(org.zalando.nakadi.utils.TestUtils.buildTimelineWithTopic) Timeline(org.zalando.nakadi.domain.Timeline) ZookeeperSettings(org.zalando.nakadi.repository.zookeeper.ZookeeperSettings) Matchers.containsInAnyOrder(org.hamcrest.Matchers.containsInAnyOrder) Matchers.equalTo(org.hamcrest.Matchers.equalTo) BatchItem(org.zalando.nakadi.domain.BatchItem) Node(org.apache.kafka.common.Node) Matchers.is(org.hamcrest.Matchers.is) Callback(org.apache.kafka.clients.producer.Callback) Mockito.mock(org.mockito.Mockito.mock) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) NakadiSettings(org.zalando.nakadi.config.NakadiSettings) Cursor(org.zalando.nakadi.view.Cursor) Matchers.anyString(org.mockito.Matchers.anyString) ArrayList(java.util.ArrayList) UUIDGenerator(org.zalando.nakadi.util.UUIDGenerator) HashSet(java.util.HashSet) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) ImmutableList(com.google.common.collect.ImmutableList) LinkedList(java.util.LinkedList) TimeoutException(org.apache.kafka.common.errors.TimeoutException) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) Test(org.junit.Test) BufferExhaustedException(org.apache.kafka.clients.producer.BufferExhaustedException) Mockito.when(org.mockito.Mockito.when) Mockito(org.mockito.Mockito) Collectors.toList(java.util.stream.Collectors.toList) EventPublishingStatus(org.zalando.nakadi.domain.EventPublishingStatus) CursorError(org.zalando.nakadi.domain.CursorError) Assert(org.junit.Assert) Collections(java.util.Collections) Timeline(org.zalando.nakadi.domain.Timeline) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) Test(org.junit.Test)

Example 5 with PartitionEndStatistics

use of org.zalando.nakadi.domain.PartitionEndStatistics in project nakadi by zalando.

the class SubscriptionControllerTest method whenGetSubscriptionStatThenOk.

@Test
public void whenGetSubscriptionStatThenOk() throws Exception {
    final Subscription subscription = builder().withEventType(TIMELINE.getEventType()).build();
    final Collection<Partition> partitions = Collections.singleton(new Partition(TIMELINE.getEventType(), "0", "xz", null, Partition.State.ASSIGNED));
    final ZkSubscriptionNode zkSubscriptionNode = new ZkSubscriptionNode(partitions, Arrays.asList(new Session("xz", 0)));
    when(subscriptionRepository.getSubscription(subscription.getId())).thenReturn(subscription);
    when(zkSubscriptionClient.getZkSubscriptionNodeLocked()).thenReturn(Optional.of(zkSubscriptionNode));
    final SubscriptionCursorWithoutToken currentOffset = new SubscriptionCursorWithoutToken(TIMELINE.getEventType(), "0", "3");
    final EventTypePartition etp = new EventTypePartition(TIMELINE.getEventType(), "0");
    final Map<EventTypePartition, SubscriptionCursorWithoutToken> offsets = new HashMap<>();
    offsets.put(etp, currentOffset);
    when(zkSubscriptionClient.getOffsets(Collections.singleton(etp))).thenReturn(offsets);
    when(eventTypeRepository.findByName(TIMELINE.getEventType())).thenReturn(EventTypeTestBuilder.builder().name(TIMELINE.getEventType()).build());
    final List<PartitionEndStatistics> statistics = Collections.singletonList(new KafkaPartitionEndStatistics(TIMELINE, 0, 13));
    when(topicRepository.loadTopicEndStatistics(eq(Collections.singletonList(TIMELINE)))).thenReturn(statistics);
    final NakadiCursor currentCursor = mock(NakadiCursor.class);
    when(currentCursor.getEventTypePartition()).thenReturn(new EventTypePartition(TIMELINE.getEventType(), "0"));
    when(cursorConverter.convert((List<SubscriptionCursorWithoutToken>) any())).thenReturn(Collections.singletonList(currentCursor));
    when(cursorOperationsService.calculateDistance(eq(currentCursor), eq(statistics.get(0).getLast()))).thenReturn(10L);
    final List<SubscriptionEventTypeStats> expectedStats = Collections.singletonList(new SubscriptionEventTypeStats(TIMELINE.getEventType(), Collections.singletonList(new SubscriptionEventTypeStats.Partition("0", "assigned", 10L, "xz", AUTO))));
    getSubscriptionStats(subscription.getId()).andExpect(status().isOk()).andExpect(content().string(TestUtils.JSON_TEST_HELPER.matchesObject(new ItemsWrapper<>(expectedStats))));
}
Also used : EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) Partition(org.zalando.nakadi.service.subscription.model.Partition) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) HashMap(java.util.HashMap) SubscriptionEventTypeStats(org.zalando.nakadi.domain.SubscriptionEventTypeStats) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) KafkaPartitionEndStatistics(org.zalando.nakadi.repository.kafka.KafkaPartitionEndStatistics) KafkaPartitionEndStatistics(org.zalando.nakadi.repository.kafka.KafkaPartitionEndStatistics) PartitionEndStatistics(org.zalando.nakadi.domain.PartitionEndStatistics) Subscription(org.zalando.nakadi.domain.Subscription) ZkSubscriptionNode(org.zalando.nakadi.service.subscription.zk.ZkSubscriptionNode) Session(org.zalando.nakadi.service.subscription.model.Session) Test(org.junit.Test)

Aggregations

PartitionEndStatistics (org.zalando.nakadi.domain.PartitionEndStatistics)7 List (java.util.List)5 NakadiCursor (org.zalando.nakadi.domain.NakadiCursor)5 Timeline (org.zalando.nakadi.domain.Timeline)5 ArrayList (java.util.ArrayList)4 Collections (java.util.Collections)4 Collectors (java.util.stream.Collectors)4 ServiceUnavailableException (org.zalando.nakadi.exceptions.ServiceUnavailableException)4 Map (java.util.Map)3 Set (java.util.Set)3 ZkSubscriptionNode (org.zalando.nakadi.service.subscription.zk.ZkSubscriptionNode)3 Collection (java.util.Collection)2 HashMap (java.util.HashMap)2 Optional (java.util.Optional)2 Collectors.toList (java.util.stream.Collectors.toList)2 Nullable (javax.annotation.Nullable)2 Test (org.junit.Test)2 Logger (org.slf4j.Logger)2 LoggerFactory (org.slf4j.LoggerFactory)2 EventType (org.zalando.nakadi.domain.EventType)2