Search in sources :

Example 1 with NakadiRuntimeException

use of org.zalando.nakadi.exceptions.NakadiRuntimeException in project nakadi by zalando.

the class StreamingState method reconfigureKafkaConsumer.

private void reconfigureKafkaConsumer(final boolean forceSeek) {
    if (eventConsumer == null) {
        throw new IllegalStateException("kafkaConsumer should not be null when calling reconfigureKafkaConsumer method");
    }
    final Set<EventTypePartition> newAssignment = offsets.keySet().stream().filter(o -> !this.releasingPartitions.containsKey(o)).collect(Collectors.toSet());
    if (forceSeek) {
        // removing all the current assignments for real consumer.
        try {
            eventConsumer.reassign(Collections.emptyList());
        } catch (final NakadiException | InvalidCursorException ex) {
            throw new NakadiRuntimeException(ex);
        }
    }
    final Set<EventTypePartition> currentAssignment = eventConsumer.getAssignment();
    getLog().info("Changing kafka assignment from {} to {}", Arrays.deepToString(currentAssignment.toArray()), Arrays.deepToString(newAssignment.toArray()));
    if (!currentAssignment.equals(newAssignment)) {
        try {
            final Map<EventTypePartition, NakadiCursor> beforeFirst = getBeforeFirstCursors(newAssignment);
            final List<NakadiCursor> cursors = newAssignment.stream().map(pk -> {
                final NakadiCursor beforeFirstAvailable = beforeFirst.get(pk);
                // Checks that current cursor is still available in storage
                offsets.get(pk).ensureDataAvailable(beforeFirstAvailable);
                return offsets.get(pk).getSentOffset();
            }).collect(Collectors.toList());
            eventConsumer.reassign(cursors);
        } catch (NakadiException | InvalidCursorException ex) {
            throw new NakadiRuntimeException(ex);
        }
    }
}
Also used : ConsumedEvent(org.zalando.nakadi.domain.ConsumedEvent) Arrays(java.util.Arrays) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) IdleStreamWatcher(org.zalando.nakadi.service.subscription.IdleStreamWatcher) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) LoggerFactory(org.slf4j.LoggerFactory) Collectors.groupingBy(java.util.stream.Collectors.groupingBy) HashMap(java.util.HashMap) NakadiException(org.zalando.nakadi.exceptions.NakadiException) ZkSubscriptionClient(org.zalando.nakadi.service.subscription.zk.ZkSubscriptionClient) Partition(org.zalando.nakadi.service.subscription.model.Partition) HashSet(java.util.HashSet) Meter(com.codahale.metrics.Meter) OptionalLong(java.util.OptionalLong) JSONObject(org.json.JSONObject) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) Map(java.util.Map) NakadiKpiPublisher(org.zalando.nakadi.service.NakadiKpiPublisher) ZkSubscription(org.zalando.nakadi.service.subscription.zk.ZkSubscription) PartitionStatistics(org.zalando.nakadi.domain.PartitionStatistics) StreamKpiData(org.zalando.nakadi.metrics.StreamKpiData) Set(java.util.Set) IOException(java.io.IOException) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) Collectors(java.util.stream.Collectors) TimeUnit(java.util.concurrent.TimeUnit) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) EventConsumer(org.zalando.nakadi.repository.EventConsumer) List(java.util.List) Stream(java.util.stream.Stream) Timeline(org.zalando.nakadi.domain.Timeline) Closeable(java.io.Closeable) Optional(java.util.Optional) Preconditions(com.google.common.base.Preconditions) SubscriptionCursor(org.zalando.nakadi.view.SubscriptionCursor) MetricUtils(org.zalando.nakadi.metrics.MetricUtils) Comparator(java.util.Comparator) Collections(java.util.Collections) Client(org.zalando.nakadi.security.Client) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) NakadiCursor(org.zalando.nakadi.domain.NakadiCursor) InvalidCursorException(org.zalando.nakadi.exceptions.InvalidCursorException) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) NakadiException(org.zalando.nakadi.exceptions.NakadiException)

Example 2 with NakadiRuntimeException

use of org.zalando.nakadi.exceptions.NakadiRuntimeException in project nakadi by zalando.

the class AbstractZkSubscriptionClient method commitOffsets.

@Override
public List<Boolean> commitOffsets(final List<SubscriptionCursorWithoutToken> cursors, final Comparator<SubscriptionCursorWithoutToken> comparator) {
    final Map<EventTypePartition, List<SubscriptionCursorWithoutToken>> grouped = cursors.stream().collect(Collectors.groupingBy(SubscriptionCursorWithoutToken::getEventTypePartition));
    try {
        final Map<EventTypePartition, Iterator<Boolean>> committedOverall = new HashMap<>();
        for (final Map.Entry<EventTypePartition, List<SubscriptionCursorWithoutToken>> entry : grouped.entrySet()) {
            final String offsetPath = getOffsetPath(entry.getKey());
            final List<Boolean> committed;
            committed = executeWithRetry(() -> {
                final Stat stat = new Stat();
                final byte[] currentOffsetData = getCurator().getData().storingStatIn(stat).forPath(offsetPath);
                final String currentMaxOffset = new String(currentOffsetData, UTF_8);
                SubscriptionCursorWithoutToken currentMaxCursor = new SubscriptionCursorWithoutToken(entry.getKey().getEventType(), entry.getKey().getPartition(), currentMaxOffset);
                final List<Boolean> commits = Lists.newArrayList();
                for (final SubscriptionCursorWithoutToken cursor : entry.getValue()) {
                    if (comparator.compare(cursor, currentMaxCursor) > 0) {
                        currentMaxCursor = cursor;
                        commits.add(true);
                    } else {
                        commits.add(false);
                    }
                }
                if (!currentMaxCursor.getOffset().equals(currentMaxOffset)) {
                    getLog().info("Committing {} to {}", currentMaxCursor.getOffset(), offsetPath);
                    getCurator().setData().withVersion(stat.getVersion()).forPath(offsetPath, currentMaxCursor.getOffset().getBytes(Charsets.UTF_8));
                }
                return commits;
            }, new RetryForSpecifiedCountStrategy<List<Boolean>>(COMMIT_CONFLICT_RETRY_TIMES).withExceptionsThatForceRetry(KeeperException.BadVersionException.class));
            committedOverall.put(entry.getKey(), Optional.ofNullable(committed).orElse(Collections.nCopies(entry.getValue().size(), false)).iterator());
        }
        return cursors.stream().map(cursor -> committedOverall.get(cursor.getEventTypePartition()).next()).collect(Collectors.toList());
    } catch (final Exception ex) {
        throw new NakadiRuntimeException(ex);
    }
}
Also used : CreateMode(org.apache.zookeeper.CreateMode) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) Arrays(java.util.Arrays) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) Session(org.zalando.nakadi.service.subscription.model.Session) BiFunction(java.util.function.BiFunction) LoggerFactory(org.slf4j.LoggerFactory) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) HashMap(java.util.HashMap) Callable(java.util.concurrent.Callable) NodeCache(org.apache.curator.framework.recipes.cache.NodeCache) InterProcessSemaphoreMutex(org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex) Stat(org.apache.zookeeper.data.Stat) MyNakadiRuntimeException1(org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1) UTF_8(com.google.common.base.Charsets.UTF_8) Function(java.util.function.Function) Retryer.executeWithRetry(org.echocat.jomon.runtime.concurrent.Retryer.executeWithRetry) RetryForSpecifiedCountStrategy(org.echocat.jomon.runtime.concurrent.RetryForSpecifiedCountStrategy) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) Lists(com.google.common.collect.Lists) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) Map(java.util.Map) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) Charsets(com.google.common.base.Charsets) Logger(org.slf4j.Logger) KeeperException(org.apache.zookeeper.KeeperException) Iterator(java.util.Iterator) Collection(java.util.Collection) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) Collectors(java.util.stream.Collectors) TimeUnit(java.util.concurrent.TimeUnit) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) CountDownLatch(java.util.concurrent.CountDownLatch) List(java.util.List) CuratorFramework(org.apache.curator.framework.CuratorFramework) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException) Closeable(java.io.Closeable) Optional(java.util.Optional) Comparator(java.util.Comparator) Collections(java.util.Collections) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) HashMap(java.util.HashMap) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) Stat(org.apache.zookeeper.data.Stat) Iterator(java.util.Iterator) List(java.util.List) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) HashMap(java.util.HashMap) Map(java.util.Map) KeeperException(org.apache.zookeeper.KeeperException)

Example 3 with NakadiRuntimeException

use of org.zalando.nakadi.exceptions.NakadiRuntimeException in project nakadi by zalando.

the class AbstractZkSubscriptionClient method runLocked.

@Override
public final <T> T runLocked(final Callable<T> function) {
    try {
        Exception releaseException = null;
        if (null == lock) {
            lock = new InterProcessSemaphoreMutex(curatorFramework, "/nakadi/locks/subscription_" + subscriptionId);
        }
        final boolean acquired = lock.acquire(SECONDS_TO_WAIT_FOR_LOCK, TimeUnit.SECONDS);
        if (!acquired) {
            throw new ServiceUnavailableException("Failed to acquire subscription lock within " + SECONDS_TO_WAIT_FOR_LOCK + " seconds");
        }
        final T result;
        try {
            result = function.call();
        } finally {
            try {
                lock.release();
            } catch (final Exception e) {
                log.error("Failed to release lock", e);
                releaseException = e;
            }
        }
        if (releaseException != null) {
            throw releaseException;
        }
        return result;
    } catch (final NakadiRuntimeException | MyNakadiRuntimeException1 e) {
        throw e;
    } catch (final Exception e) {
        throw new NakadiRuntimeException(e);
    }
}
Also used : NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) InterProcessSemaphoreMutex(org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex) MyNakadiRuntimeException1(org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException)

Example 4 with NakadiRuntimeException

use of org.zalando.nakadi.exceptions.NakadiRuntimeException in project nakadi by zalando.

the class AbstractZkSubscriptionClient method loadDataAsync.

protected <K, V> Map<K, V> loadDataAsync(final Collection<K> keys, final Function<K, String> keyConverter, final BiFunction<K, byte[], V> valueConverter) throws ServiceTemporarilyUnavailableException, NakadiRuntimeException {
    final Map<K, V> result = new HashMap<>();
    final CountDownLatch latch = new CountDownLatch(keys.size());
    try {
        for (final K key : keys) {
            final String zkKey = keyConverter.apply(key);
            getCurator().getData().inBackground((client, event) -> {
                try {
                    if (event.getResultCode() == KeeperException.Code.OK.intValue()) {
                        final V value = valueConverter.apply(key, event.getData());
                        synchronized (result) {
                            result.put(key, value);
                        }
                    } else {
                        getLog().error("Failed to get {} data from zk. status code: {}", zkKey, event.getResultCode());
                    }
                } catch (RuntimeException ex) {
                    getLog().error("Failed to memorize {} key value", key, ex);
                } finally {
                    latch.countDown();
                }
            }).forPath(zkKey);
        }
    } catch (Exception ex) {
        throw new NakadiRuntimeException(ex);
    }
    try {
        if (!latch.await(MAX_ZK_RESPONSE_SECONDS, TimeUnit.SECONDS)) {
            throw new ServiceTemporarilyUnavailableException("Failed to wait for zk response", null);
        }
    } catch (InterruptedException ex) {
        Thread.currentThread().interrupt();
        throw new ServiceTemporarilyUnavailableException("Failed to wait for zk response", ex);
    }
    if (result.size() != keys.size()) {
        throw new ServiceTemporarilyUnavailableException("Failed to wait for keys " + keys.stream().filter(v -> !result.containsKey(v)).map(String::valueOf).collect(Collectors.joining(", ")) + " to be in response", null);
    }
    return result;
}
Also used : CreateMode(org.apache.zookeeper.CreateMode) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) Arrays(java.util.Arrays) EventTypePartition(org.zalando.nakadi.domain.EventTypePartition) Session(org.zalando.nakadi.service.subscription.model.Session) BiFunction(java.util.function.BiFunction) LoggerFactory(org.slf4j.LoggerFactory) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) HashMap(java.util.HashMap) Callable(java.util.concurrent.Callable) NodeCache(org.apache.curator.framework.recipes.cache.NodeCache) InterProcessSemaphoreMutex(org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex) Stat(org.apache.zookeeper.data.Stat) MyNakadiRuntimeException1(org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1) UTF_8(com.google.common.base.Charsets.UTF_8) Function(java.util.function.Function) Retryer.executeWithRetry(org.echocat.jomon.runtime.concurrent.Retryer.executeWithRetry) RetryForSpecifiedCountStrategy(org.echocat.jomon.runtime.concurrent.RetryForSpecifiedCountStrategy) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) Lists(com.google.common.collect.Lists) SubscriptionCursorWithoutToken(org.zalando.nakadi.view.SubscriptionCursorWithoutToken) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) Map(java.util.Map) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) Charsets(com.google.common.base.Charsets) Logger(org.slf4j.Logger) KeeperException(org.apache.zookeeper.KeeperException) Iterator(java.util.Iterator) Collection(java.util.Collection) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) Collectors(java.util.stream.Collectors) TimeUnit(java.util.concurrent.TimeUnit) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) CountDownLatch(java.util.concurrent.CountDownLatch) List(java.util.List) CuratorFramework(org.apache.curator.framework.CuratorFramework) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException) Closeable(java.io.Closeable) Optional(java.util.Optional) Comparator(java.util.Comparator) Collections(java.util.Collections) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) HashMap(java.util.HashMap) CountDownLatch(java.util.concurrent.CountDownLatch) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException)

Example 5 with NakadiRuntimeException

use of org.zalando.nakadi.exceptions.NakadiRuntimeException in project nakadi by zalando.

the class AbstractZkSubscriptionClient method subscribeForCursorsReset.

@Override
public final Closeable subscribeForCursorsReset(final Runnable listener) throws NakadiRuntimeException, UnsupportedOperationException {
    final NodeCache cursorResetCache = new NodeCache(getCurator(), resetCursorPath);
    cursorResetCache.getListenable().addListener(listener::run);
    try {
        cursorResetCache.start();
    } catch (final Exception e) {
        throw new NakadiRuntimeException(e);
    }
    return () -> {
        try {
            cursorResetCache.getListenable().clear();
            cursorResetCache.close();
        } catch (final IOException e) {
            throw new NakadiRuntimeException(e);
        }
    };
}
Also used : NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) NodeCache(org.apache.curator.framework.recipes.cache.NodeCache) IOException(java.io.IOException) RequestInProgressException(org.zalando.nakadi.exceptions.runtime.RequestInProgressException) UnableProcessException(org.zalando.nakadi.exceptions.UnableProcessException) ServiceUnavailableException(org.zalando.nakadi.exceptions.ServiceUnavailableException) OperationInterruptedException(org.zalando.nakadi.exceptions.runtime.OperationInterruptedException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException) OperationTimeoutException(org.zalando.nakadi.exceptions.runtime.OperationTimeoutException) ZookeeperException(org.zalando.nakadi.exceptions.runtime.ZookeeperException) NakadiRuntimeException(org.zalando.nakadi.exceptions.NakadiRuntimeException) ServiceTemporarilyUnavailableException(org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException)

Aggregations

NakadiRuntimeException (org.zalando.nakadi.exceptions.NakadiRuntimeException)18 IOException (java.io.IOException)13 ServiceUnavailableException (org.zalando.nakadi.exceptions.ServiceUnavailableException)11 KeeperException (org.apache.zookeeper.KeeperException)10 ServiceTemporarilyUnavailableException (org.zalando.nakadi.exceptions.runtime.ServiceTemporarilyUnavailableException)10 UnableProcessException (org.zalando.nakadi.exceptions.UnableProcessException)8 OperationInterruptedException (org.zalando.nakadi.exceptions.runtime.OperationInterruptedException)8 OperationTimeoutException (org.zalando.nakadi.exceptions.runtime.OperationTimeoutException)8 RequestInProgressException (org.zalando.nakadi.exceptions.runtime.RequestInProgressException)8 ZookeeperException (org.zalando.nakadi.exceptions.runtime.ZookeeperException)8 EventTypePartition (org.zalando.nakadi.domain.EventTypePartition)6 Closeable (java.io.Closeable)5 Collections (java.util.Collections)5 HashMap (java.util.HashMap)5 List (java.util.List)5 Map (java.util.Map)5 TimeUnit (java.util.concurrent.TimeUnit)5 Collectors (java.util.stream.Collectors)5 MyNakadiRuntimeException1 (org.zalando.nakadi.exceptions.runtime.MyNakadiRuntimeException1)5 Arrays (java.util.Arrays)4