Search in sources :

Example 21 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project apache-kafka-on-k8s by banzaicloud.

the class WindowedStreamPartitionerTest method testCopartitioning.

@Test
public void testCopartitioning() {
    final Random rand = new Random();
    final DefaultPartitioner defaultPartitioner = new DefaultPartitioner();
    final WindowedSerializer<Integer> timeWindowedSerializer = new TimeWindowedSerializer<>(intSerializer);
    final WindowedStreamPartitioner<Integer, String> streamPartitioner = new WindowedStreamPartitioner<>(topicName, timeWindowedSerializer);
    for (int k = 0; k < 10; k++) {
        Integer key = rand.nextInt();
        byte[] keyBytes = intSerializer.serialize(topicName, key);
        String value = key.toString();
        byte[] valueBytes = stringSerializer.serialize(topicName, value);
        Integer expected = defaultPartitioner.partition("topic", key, keyBytes, value, valueBytes, cluster);
        for (int w = 1; w < 10; w++) {
            TimeWindow window = new TimeWindow(10 * w, 20 * w);
            Windowed<Integer> windowedKey = new Windowed<>(key, window);
            Integer actual = streamPartitioner.partition(windowedKey, value, infos.size());
            assertEquals(expected, actual);
        }
    }
    defaultPartitioner.close();
}
Also used : TimeWindowedSerializer(org.apache.kafka.streams.kstream.TimeWindowedSerializer) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) Windowed(org.apache.kafka.streams.kstream.Windowed) Random(java.util.Random) Test(org.junit.Test)

Example 22 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project apache-kafka-on-k8s by banzaicloud.

the class MockProducerTest method testPartitioner.

@Test
public void testPartitioner() throws Exception {
    PartitionInfo partitionInfo0 = new PartitionInfo(topic, 0, null, null, null);
    PartitionInfo partitionInfo1 = new PartitionInfo(topic, 1, null, null, null);
    Cluster cluster = new Cluster(null, new ArrayList<Node>(0), asList(partitionInfo0, partitionInfo1), Collections.<String>emptySet(), Collections.<String>emptySet());
    MockProducer<String, String> producer = new MockProducer<>(cluster, true, new DefaultPartitioner(), new StringSerializer(), new StringSerializer());
    ProducerRecord<String, String> record = new ProducerRecord<>(topic, "key", "value");
    Future<RecordMetadata> metadata = producer.send(record);
    assertEquals("Partition should be correct", 1, metadata.get().partition());
    producer.clear();
    assertEquals("Clear should erase our history", 0, producer.history().size());
    producer.close();
}
Also used : DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner) Node(org.apache.kafka.common.Node) Cluster(org.apache.kafka.common.Cluster) PartitionInfo(org.apache.kafka.common.PartitionInfo) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) Test(org.junit.Test)

Example 23 with DefaultPartitioner

use of org.apache.kafka.clients.producer.internals.DefaultPartitioner in project kafka by apache.

the class MockClientSupplier method getProducer.

@Override
public Producer<byte[], byte[]> getProducer(final Map<String, Object> config) {
    if (applicationId != null) {
        assertThat((String) config.get(ProducerConfig.TRANSACTIONAL_ID_CONFIG), startsWith(applicationId + "-"));
    } else {
        assertFalse(config.containsKey(ProducerConfig.TRANSACTIONAL_ID_CONFIG));
    }
    final MockProducer<byte[], byte[]> producer = new MockProducer<>(cluster, true, new DefaultPartitioner(), BYTE_ARRAY_SERIALIZER, BYTE_ARRAY_SERIALIZER);
    producers.add(producer);
    return producer;
}
Also used : MockProducer(org.apache.kafka.clients.producer.MockProducer) DefaultPartitioner(org.apache.kafka.clients.producer.internals.DefaultPartitioner)

Aggregations

DefaultPartitioner (org.apache.kafka.clients.producer.internals.DefaultPartitioner)23 Test (org.junit.Test)21 MockProducer (org.apache.kafka.clients.producer.MockProducer)17 Future (java.util.concurrent.Future)12 Callback (org.apache.kafka.clients.producer.Callback)12 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)12 StreamsException (org.apache.kafka.streams.errors.StreamsException)10 KafkaException (org.apache.kafka.common.KafkaException)7 DefaultProductionExceptionHandler (org.apache.kafka.streams.errors.DefaultProductionExceptionHandler)7 TimeoutException (org.apache.kafka.common.errors.TimeoutException)5 AlwaysContinueProductionExceptionHandler (org.apache.kafka.streams.errors.AlwaysContinueProductionExceptionHandler)4 PartitionInfo (org.apache.kafka.common.PartitionInfo)3 TopicPartition (org.apache.kafka.common.TopicPartition)3 List (java.util.List)2 Map (java.util.Map)2 Random (java.util.Random)2 Cluster (org.apache.kafka.common.Cluster)2 StringSerializer (org.apache.kafka.common.serialization.StringSerializer)2 LogContext (org.apache.kafka.common.utils.LogContext)2 Utils.mkMap (org.apache.kafka.common.utils.Utils.mkMap)2