Search in sources :

Example 11 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class RecordCollectorTest method shouldThrowIfTopicIsUnknownOnSendWithPartitioner.

@Test
public void shouldThrowIfTopicIsUnknownOnSendWithPartitioner() {
    final RecordCollector collector = new RecordCollectorImpl(logContext, taskId, new StreamsProducer(config, processId + "-StreamThread-1", new MockClientSupplier() {

        @Override
        public Producer<byte[], byte[]> getProducer(final Map<String, Object> config) {
            return new MockProducer<byte[], byte[]>(cluster, true, new DefaultPartitioner(), byteArraySerializer, byteArraySerializer) {

                @Override
                public List<PartitionInfo> partitionsFor(final String topic) {
                    return Collections.emptyList();
                }
            };
        }
    }, null, null, logContext, Time.SYSTEM), productionExceptionHandler, streamsMetrics);
    collector.initialize();
    final StreamsException thrown = assertThrows(StreamsException.class, () -> collector.send(topic, "3", "0", null, null, stringSerializer, stringSerializer, streamPartitioner));
    assertThat(thrown.getMessage(), equalTo("Could not get partition information for topic topic for task 0_0." + " This can happen if the topic does not exist."));
}
Also used : MockProducer(org.apache.kafka.clients.producer.MockProducer) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) MockClientSupplier(org.apache.kafka.test.MockClientSupplier) StreamsException(org.apache.kafka.streams.errors.StreamsException) PartitionInfo(org.apache.kafka.common.PartitionInfo) Utils.mkMap(org.apache.kafka.common.utils.Utils.mkMap) Map(java.util.Map) Test(org.junit.Test)

Example 12 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class WindowedStreamPartitionerTest method testCopartitioning.

@Test
public void testCopartitioning() {
    final Random rand = new Random();
    final DefaultPartitioner defaultPartitioner = new DefaultPartitioner();
    final WindowedSerializer<Integer> timeWindowedSerializer = new TimeWindowedSerializer<>(intSerializer);
    final WindowedStreamPartitioner<Integer, String> streamPartitioner = new WindowedStreamPartitioner<>(timeWindowedSerializer);
    for (int k = 0; k < 10; k++) {
        final Integer key = rand.nextInt();
        final byte[] keyBytes = intSerializer.serialize(topicName, key);
        final String value = key.toString();
        final byte[] valueBytes = stringSerializer.serialize(topicName, value);
        final Integer expected = defaultPartitioner.partition("topic", key, keyBytes, value, valueBytes, cluster);
        for (int w = 1; w < 10; w++) {
            final TimeWindow window = new TimeWindow(10 * w, 20 * w);
            final Windowed<Integer> windowedKey = new Windowed<>(key, window);
            final Integer actual = streamPartitioner.partition(topicName, windowedKey, value, infos.size());
            assertEquals(expected, actual);
        }
    }
    defaultPartitioner.close();
}
Also used : TimeWindowedSerializer(org.apache.kafka.streams.kstream.TimeWindowedSerializer) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) Windowed(org.apache.kafka.streams.kstream.Windowed) Random(java.util.Random) Test(org.junit.Test)

Example 13 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class RecordCollectorTest method shouldRetryWhenTimeoutExceptionOccursOnSend.

@SuppressWarnings("unchecked")
@Test
public void shouldRetryWhenTimeoutExceptionOccursOnSend() throws Exception {
    final AtomicInteger attempt = new AtomicInteger(0);
    RecordCollectorImpl collector = new RecordCollectorImpl(new MockProducer(cluster, true, new DefaultPartitioner(), byteArraySerializer, byteArraySerializer) {

        @Override
        public synchronized Future<RecordMetadata> send(final ProducerRecord record, final Callback callback) {
            if (attempt.getAndIncrement() == 0) {
                throw new TimeoutException();
            }
            return super.send(record, callback);
        }
    }, "test");
    collector.send("topic1", "3", "0", null, null, stringSerializer, stringSerializer, streamPartitioner);
    final Long offset = collector.offsets().get(new TopicPartition("topic1", 0));
    assertEquals(Long.valueOf(0L), offset);
}
Also used : MockProducer(org.apache.kafka.clients.producer.MockProducer) Callback(org.apache.kafka.clients.producer.Callback) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) TopicPartition(org.apache.kafka.common.TopicPartition) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) Future(java.util.concurrent.Future) TimeoutException(org.apache.kafka.common.errors.TimeoutException) Test(org.junit.Test)

Example 14 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class RecordCollectorTest method shouldThrowStreamsExceptionAfterMaxAttempts.

@SuppressWarnings("unchecked")
@Test(expected = StreamsException.class)
public void shouldThrowStreamsExceptionAfterMaxAttempts() throws Exception {
    RecordCollector collector = new RecordCollectorImpl(new MockProducer(cluster, true, new DefaultPartitioner(), byteArraySerializer, byteArraySerializer) {

        @Override
        public synchronized Future<RecordMetadata> send(final ProducerRecord record, final Callback callback) {
            throw new TimeoutException();
        }
    }, "test");
    collector.send("topic1", "3", "0", null, null, stringSerializer, stringSerializer, streamPartitioner);
}
Also used : MockProducer(org.apache.kafka.clients.producer.MockProducer) Callback(org.apache.kafka.clients.producer.Callback) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) Future(java.util.concurrent.Future) TimeoutException(org.apache.kafka.common.errors.TimeoutException) Test(org.junit.Test)

Example 15 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class RecordCollectorTest method shouldThrowStreamsExceptionOnFlushIfASendFailed.

@SuppressWarnings("unchecked")
@Test(expected = StreamsException.class)
public void shouldThrowStreamsExceptionOnFlushIfASendFailed() throws Exception {
    final RecordCollector collector = new RecordCollectorImpl(new MockProducer(cluster, true, new DefaultPartitioner(), byteArraySerializer, byteArraySerializer) {

        @Override
        public synchronized Future<RecordMetadata> send(final ProducerRecord record, final Callback callback) {
            callback.onCompletion(null, new Exception());
            return null;
        }
    }, "test");
    collector.send("topic1", "3", "0", null, null, stringSerializer, stringSerializer, streamPartitioner);
    collector.flush();
}
Also used : MockProducer(org.apache.kafka.clients.producer.MockProducer) Callback(org.apache.kafka.clients.producer.Callback) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) Future(java.util.concurrent.Future) TimeoutException(org.apache.kafka.common.errors.TimeoutException) StreamsException(org.apache.kafka.streams.errors.StreamsException) Test(org.junit.Test)

Aggregations

DefaultPartitioner (org.apache.kafka.clients.producer.internals.DefaultPartitioner)23 Test (org.junit.Test)21 MockProducer (org.apache.kafka.clients.producer.MockProducer)17 Future (java.util.concurrent.Future)12 Callback (org.apache.kafka.clients.producer.Callback)12 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)12 StreamsException (org.apache.kafka.streams.errors.StreamsException)10 KafkaException (org.apache.kafka.common.KafkaException)7 DefaultProductionExceptionHandler (org.apache.kafka.streams.errors.DefaultProductionExceptionHandler)7 TimeoutException (org.apache.kafka.common.errors.TimeoutException)5 AlwaysContinueProductionExceptionHandler (org.apache.kafka.streams.errors.AlwaysContinueProductionExceptionHandler)4 PartitionInfo (org.apache.kafka.common.PartitionInfo)3 TopicPartition (org.apache.kafka.common.TopicPartition)3 List (java.util.List)2 Map (java.util.Map)2 Random (java.util.Random)2 Cluster (org.apache.kafka.common.Cluster)2 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)2 LogContext (org.apache.kafka.common.utils.LogContext)2 Utils.mkMap (org.apache.kafka.common.utils.Utils.mkMap)2