Search in sources :

Example 46 with RecordMetadata

use of org.apache.kafka.clients.producer.RecordMetadata in project testcases by coheigea.

the class KafkaAuthorizerSASLGSSTest method testUnauthorizedWrite.

// "client" is not authorized to write to "dev"
@org.junit.Test
public void testUnauthorizedWrite() throws Exception {
    // Create the Producer
    Properties producerProps = new Properties();
    producerProps.put("bootstrap.servers", "localhost:" + port);
    producerProps.put("acks", "all");
    producerProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    producerProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
    producerProps.put("sasl.mechanism", "GSSAPI");
    producerProps.put("sasl.kerberos.service.name", "kafka");
    producerProps.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "JKS");
    producerProps.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, this.getClass().getResource("/servicestore.jks").getPath());
    producerProps.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "sspass");
    producerProps.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "skpass");
    producerProps.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, this.getClass().getResource("/truststore.jks").getPath());
    producerProps.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "security");
    final Producer<String, String> producer = new KafkaProducer<>(producerProps);
    // Send a message
    try {
        Future<RecordMetadata> record = producer.send(new ProducerRecord<String, String>("dev", "somekey", "somevalue"));
        producer.flush();
        record.get();
        Assert.fail("Authorization failure expected");
    } catch (Exception ex) {
        Assert.assertTrue(ex.getMessage().contains("Not authorized to access topics"));
    }
    producer.close();
}
Also used : KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) Properties(java.util.Properties)

Example 47 with RecordMetadata

use of org.apache.kafka.clients.producer.RecordMetadata in project testcases by coheigea.

the class KafkaSentryAuthorizerTest method testUnauthorizedWrite.

// The "public" group can't write to "test" or "dev"
@org.junit.Test
public void testUnauthorizedWrite() throws Exception {
    // Create the Producer
    Properties producerProps = new Properties();
    producerProps.put("bootstrap.servers", "localhost:" + port);
    producerProps.put("acks", "all");
    producerProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    producerProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
    producerProps.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "JKS");
    producerProps.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, this.getClass().getResource("/clientstore.jks").getPath());
    producerProps.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "cspass");
    producerProps.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "ckpass");
    producerProps.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, this.getClass().getResource("/truststore.jks").getPath());
    producerProps.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "security");
    final Producer<String, String> producer = new KafkaProducer<>(producerProps);
    // Send a message
    try {
        Future<RecordMetadata> record = producer.send(new ProducerRecord<String, String>("test", "somekey2", "somevalue2"));
        producer.flush();
        record.get();
        Assert.fail("Authorization failure expected");
    } catch (Exception ex) {
        Assert.assertTrue(ex.getMessage().contains("Not authorized to access topics"));
    }
    producer.close();
}
Also used : KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) Properties(java.util.Properties)

Example 48 with RecordMetadata

use of org.apache.kafka.clients.producer.RecordMetadata in project drill by apache.

the class KafkaMessageGenerator method populateJsonMsgWithTimestamps.

public void populateJsonMsgWithTimestamps(String topic, int numMsg) throws ExecutionException, InterruptedException {
    try (KafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties)) {
        int halfCount = numMsg / 2;
        for (PartitionInfo tpInfo : producer.partitionsFor(topic)) {
            for (int i = 1; i <= numMsg; ++i) {
                JsonObject object = new JsonObject();
                object.addProperty("stringKey", UUID.randomUUID().toString());
                object.addProperty("intKey", numMsg - i);
                object.addProperty("boolKey", i % 2 == 0);
                long timestamp = i < halfCount ? (halfCount - i) : i;
                ProducerRecord<String, String> message = new ProducerRecord<>(tpInfo.topic(), tpInfo.partition(), timestamp, "key" + i, object.toString());
                logger.info("Publishing message : {}", message);
                Future<RecordMetadata> future = producer.send(message);
                logger.info("Committed offset of the message : {}", future.get().offset());
            }
        }
    }
}
Also used : KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) JsonObject(com.google.gson.JsonObject) PartitionInfo(org.apache.kafka.common.PartitionInfo)

Example 49 with RecordMetadata

use of org.apache.kafka.clients.producer.RecordMetadata in project flink by apache.

the class FlinkKafkaProducer method open.

// ----------------------------------- Utilities --------------------------
/**
 * Initializes the connection to Kafka.
 */
@Override
public void open(Configuration configuration) throws Exception {
    if (logFailuresOnly) {
        callback = new Callback() {

            @Override
            public void onCompletion(RecordMetadata metadata, Exception e) {
                if (e != null) {
                    LOG.error("Error while sending record to Kafka: " + e.getMessage(), e);
                }
                acknowledgeMessage();
            }
        };
    } else {
        callback = new Callback() {

            @Override
            public void onCompletion(RecordMetadata metadata, Exception exception) {
                if (exception != null && asyncException == null) {
                    asyncException = exception;
                }
                acknowledgeMessage();
            }
        };
    }
    RuntimeContext ctx = getRuntimeContext();
    if (flinkKafkaPartitioner != null) {
        flinkKafkaPartitioner.open(ctx.getIndexOfThisSubtask(), ctx.getNumberOfParallelSubtasks());
    }
    if (kafkaSchema instanceof KafkaContextAware) {
        KafkaContextAware<IN> contextAwareSchema = (KafkaContextAware<IN>) kafkaSchema;
        contextAwareSchema.setParallelInstanceId(ctx.getIndexOfThisSubtask());
        contextAwareSchema.setNumParallelInstances(ctx.getNumberOfParallelSubtasks());
    }
    if (kafkaSchema != null) {
        kafkaSchema.open(RuntimeContextInitializationContextAdapters.serializationAdapter(getRuntimeContext(), metricGroup -> metricGroup.addGroup("user")));
    }
    super.open(configuration);
}
Also used : RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) TwoPhaseCommitSinkFunction(org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction) LoggerFactory(org.slf4j.LoggerFactory) ExceptionUtils(org.apache.flink.util.ExceptionUtils) FunctionSnapshotContext(org.apache.flink.runtime.state.FunctionSnapshotContext) StringUtils(org.apache.commons.lang3.StringUtils) NetUtils(org.apache.flink.util.NetUtils) Lists(org.apache.flink.shaded.guava30.com.google.common.collect.Lists) ListState(org.apache.flink.api.common.state.ListState) TypeSerializerSnapshot(org.apache.flink.api.common.typeutils.TypeSerializerSnapshot) Duration(java.time.Duration) Map(java.util.Map) Metric(org.apache.kafka.common.Metric) MetricName(org.apache.kafka.common.MetricName) ListStateDescriptor(org.apache.flink.api.common.state.ListStateDescriptor) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) FlinkFixedPartitioner(org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner) TransactionalIdsGenerator(org.apache.flink.streaming.connectors.kafka.internals.TransactionalIdsGenerator) FunctionInitializationContext(org.apache.flink.runtime.state.FunctionInitializationContext) Collection(java.util.Collection) Set(java.util.Set) PartitionInfo(org.apache.kafka.common.PartitionInfo) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) Preconditions(org.apache.flink.util.Preconditions) KeyedSerializationSchema(org.apache.flink.streaming.util.serialization.KeyedSerializationSchema) KafkaMetricMutableWrapper(org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper) MetricGroup(org.apache.flink.metrics.MetricGroup) List(java.util.List) InvalidTxnStateException(org.apache.kafka.common.errors.InvalidTxnStateException) ProducerFencedException(org.apache.kafka.common.errors.ProducerFencedException) SimpleTypeSerializerSnapshot(org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Optional(java.util.Optional) Callback(org.apache.kafka.clients.producer.Callback) Time(org.apache.flink.api.common.time.Time) SerializationSchema(org.apache.flink.api.common.serialization.SerializationSchema) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) TemporaryClassLoaderContext(org.apache.flink.util.TemporaryClassLoaderContext) PublicEvolving(org.apache.flink.annotation.PublicEvolving) HashMap(java.util.HashMap) RuntimeContextInitializationContextAdapters(org.apache.flink.api.common.serialization.RuntimeContextInitializationContextAdapters) DataOutputView(org.apache.flink.core.memory.DataOutputView) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) KafkaSerializationSchemaWrapper(org.apache.flink.streaming.connectors.kafka.internals.KafkaSerializationSchemaWrapper) ByteArraySerializer(org.apache.kafka.common.serialization.ByteArraySerializer) DataInputView(org.apache.flink.core.memory.DataInputView) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) Nullable(javax.annotation.Nullable) Preconditions.checkState(org.apache.flink.util.Preconditions.checkState) Logger(org.slf4j.Logger) Properties(java.util.Properties) Producer(org.apache.kafka.clients.producer.Producer) BlockingDeque(java.util.concurrent.BlockingDeque) Configuration(org.apache.flink.configuration.Configuration) FlinkKafkaInternalProducer(org.apache.flink.streaming.connectors.kafka.internals.FlinkKafkaInternalProducer) IOException(java.io.IOException) FlinkKafkaPartitioner(org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner) VisibleForTesting(org.apache.flink.annotation.VisibleForTesting) AtomicLong(java.util.concurrent.atomic.AtomicLong) LinkedBlockingDeque(java.util.concurrent.LinkedBlockingDeque) Internal(org.apache.flink.annotation.Internal) TypeSerializerSingleton(org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton) ClosureCleaner(org.apache.flink.api.java.ClosureCleaner) Comparator(java.util.Comparator) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) Collections(java.util.Collections) Callback(org.apache.kafka.clients.producer.Callback) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) InvalidTxnStateException(org.apache.kafka.common.errors.InvalidTxnStateException) ProducerFencedException(org.apache.kafka.common.errors.ProducerFencedException) IOException(java.io.IOException)

Example 50 with RecordMetadata

use of org.apache.kafka.clients.producer.RecordMetadata in project flink by apache.

the class FlinkKafkaProducerBase method open.

// ----------------------------------- Utilities --------------------------
/**
 * Initializes the connection to Kafka.
 */
@Override
public void open(Configuration configuration) throws Exception {
    if (schema instanceof KeyedSerializationSchemaWrapper) {
        ((KeyedSerializationSchemaWrapper<IN>) schema).getSerializationSchema().open(RuntimeContextInitializationContextAdapters.serializationAdapter(getRuntimeContext(), metricGroup -> metricGroup.addGroup("user")));
    }
    producer = getKafkaProducer(this.producerConfig);
    RuntimeContext ctx = getRuntimeContext();
    if (null != flinkKafkaPartitioner) {
        flinkKafkaPartitioner.open(ctx.getIndexOfThisSubtask(), ctx.getNumberOfParallelSubtasks());
    }
    LOG.info("Starting FlinkKafkaProducer ({}/{}) to produce into default topic {}", ctx.getIndexOfThisSubtask() + 1, ctx.getNumberOfParallelSubtasks(), defaultTopicId);
    // register Kafka metrics to Flink accumulators
    if (!Boolean.parseBoolean(producerConfig.getProperty(KEY_DISABLE_METRICS, "false"))) {
        Map<MetricName, ? extends Metric> metrics = this.producer.metrics();
        if (metrics == null) {
            // MapR's Kafka implementation returns null here.
            LOG.info("Producer implementation does not support metrics");
        } else {
            final MetricGroup kafkaMetricGroup = getRuntimeContext().getMetricGroup().addGroup("KafkaProducer");
            for (Map.Entry<MetricName, ? extends Metric> metric : metrics.entrySet()) {
                kafkaMetricGroup.gauge(metric.getKey().name(), new KafkaMetricWrapper(metric.getValue()));
            }
        }
    }
    if (flushOnCheckpoint && !((StreamingRuntimeContext) this.getRuntimeContext()).isCheckpointingEnabled()) {
        LOG.warn("Flushing on checkpoint is enabled, but checkpointing is not enabled. Disabling flushing.");
        flushOnCheckpoint = false;
    }
    if (logFailuresOnly) {
        callback = new Callback() {

            @Override
            public void onCompletion(RecordMetadata metadata, Exception e) {
                if (e != null) {
                    LOG.error("Error while sending record to Kafka: " + e.getMessage(), e);
                }
                acknowledgeMessage();
            }
        };
    } else {
        callback = new Callback() {

            @Override
            public void onCompletion(RecordMetadata metadata, Exception exception) {
                if (exception != null && asyncException == null) {
                    asyncException = exception;
                }
                acknowledgeMessage();
            }
        };
    }
}
Also used : RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KeyedSerializationSchemaWrapper(org.apache.flink.streaming.connectors.kafka.internals.KeyedSerializationSchemaWrapper) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) RuntimeContextInitializationContextAdapters(org.apache.flink.api.common.serialization.RuntimeContextInitializationContextAdapters) FunctionSnapshotContext(org.apache.flink.runtime.state.FunctionSnapshotContext) NetUtils(org.apache.flink.util.NetUtils) ArrayList(java.util.ArrayList) KafkaMetricWrapper(org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricWrapper) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) ByteArraySerializer(org.apache.kafka.common.serialization.ByteArraySerializer) Map(java.util.Map) Objects.requireNonNull(java.util.Objects.requireNonNull) Metric(org.apache.kafka.common.Metric) MetricName(org.apache.kafka.common.MetricName) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) Logger(org.slf4j.Logger) Properties(java.util.Properties) CheckpointedFunction(org.apache.flink.streaming.api.checkpoint.CheckpointedFunction) FunctionInitializationContext(org.apache.flink.runtime.state.FunctionInitializationContext) Configuration(org.apache.flink.configuration.Configuration) PartitionInfo(org.apache.kafka.common.PartitionInfo) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) FlinkKafkaPartitioner(org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner) KeyedSerializationSchema(org.apache.flink.streaming.util.serialization.KeyedSerializationSchema) RichSinkFunction(org.apache.flink.streaming.api.functions.sink.RichSinkFunction) VisibleForTesting(org.apache.flink.annotation.VisibleForTesting) SerializableObject(org.apache.flink.util.SerializableObject) MetricGroup(org.apache.flink.metrics.MetricGroup) List(java.util.List) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) Internal(org.apache.flink.annotation.Internal) ClosureCleaner(org.apache.flink.api.java.ClosureCleaner) Comparator(java.util.Comparator) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) Callback(org.apache.kafka.clients.producer.Callback) Collections(java.util.Collections) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) KeyedSerializationSchemaWrapper(org.apache.flink.streaming.connectors.kafka.internals.KeyedSerializationSchemaWrapper) MetricGroup(org.apache.flink.metrics.MetricGroup) RecordMetadata(org.apache.kafka.clients.producer.RecordMetadata) MetricName(org.apache.kafka.common.MetricName) Callback(org.apache.kafka.clients.producer.Callback) KafkaMetricWrapper(org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricWrapper) RuntimeContext(org.apache.flink.api.common.functions.RuntimeContext) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) HashMap(java.util.HashMap) Map(java.util.Map)

Aggregations

RecordMetadata (org.apache.kafka.clients.producer.RecordMetadata)191 Test (org.junit.Test)64 Node (org.apache.kafka.common.Node)50 Test (org.junit.jupiter.api.Test)50 TopicPartition (org.apache.kafka.common.TopicPartition)48 ProducerRecord (org.apache.kafka.clients.producer.ProducerRecord)46 ExecutionException (java.util.concurrent.ExecutionException)35 Callback (org.apache.kafka.clients.producer.Callback)33 KafkaProducer (org.apache.kafka.clients.producer.KafkaProducer)31 Properties (java.util.Properties)30 HashMap (java.util.HashMap)24 TimeoutException (org.apache.kafka.common.errors.TimeoutException)23 ArrayList (java.util.ArrayList)21 KafkaException (org.apache.kafka.common.KafkaException)19 List (java.util.List)15 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)15 Metrics (org.apache.kafka.common.metrics.Metrics)15 LinkedHashMap (java.util.LinkedHashMap)13 Future (java.util.concurrent.Future)13 Map (java.util.Map)12