Examples with JmxPrometheusExporterMetrics io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics used on opensource projects

Search in sources :

Example 6 with JmxPrometheusExporterMetrics

use of io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics in project strimzi-kafka-operator by strimzi.

the class KafkaAssemblyOperatorTest method testUpdateZkClusterMetricsConfig.

@ParameterizedTest
@MethodSource("data")
public void testUpdateZkClusterMetricsConfig(Params params, VertxTestContext context) {
    setFields(params);
    Kafka kafkaAssembly = getKafkaAssembly("bar");
    JmxPrometheusExporterMetrics jmxMetricsConfig = io.strimzi.operator.cluster.TestUtils.getJmxPrometheusExporterMetrics("metrics-config.yml", differentMetricsCMName);
    kafkaAssembly.getSpec().getKafka().setMetricsConfig(jmxMetricsConfig);
    updateCluster(context, getKafkaAssembly("bar"), kafkaAssembly);
}
Also used : JmxPrometheusExporterMetrics(io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics) Kafka(io.strimzi.api.kafka.model.Kafka) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 7 with JmxPrometheusExporterMetrics

use of io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics in project strimzi-kafka-operator by strimzi.

the class MetricsIsolatedST method testKafkaMetricsSettings.

/**
 * 1. Update metrics form whatever it is to @metricsConfigYaml in spec.kafka.metricsConfig
 * 2. Check, whether the metrics ConfigMap is changed
 * 3. Updates ConfigMap linked as metrics on
 * 4. Check, whether the metrics ConfigMap is changed
 */
@ParallelTest
void testKafkaMetricsSettings() {
    String metricsConfigJson = "{\"lowercaseOutputName\":true}";
    String metricsConfigYaml = "lowercaseOutputName: true";
    ObjectMapper mapper = new ObjectMapper();
    mapper.configure(JsonReadFeature.ALLOW_UNESCAPED_CONTROL_CHARS.mappedFeature(), true);
    ConfigMap externalMetricsCm = new ConfigMapBuilder().withData(Collections.singletonMap(Constants.METRICS_CONFIG_YAML_NAME, metricsConfigYaml)).withNewMetadata().withName("external-metrics-cm").withNamespace(SECOND_NAMESPACE).endMetadata().build();
    kubeClient().getClient().configMaps().inNamespace(SECOND_NAMESPACE).createOrReplace(externalMetricsCm);
    // spec.kafka.metrics -> spec.kafka.jmxExporterMetrics
    ConfigMapKeySelector cmks = new ConfigMapKeySelectorBuilder().withName("external-metrics-cm").withKey(Constants.METRICS_CONFIG_YAML_NAME).build();
    JmxPrometheusExporterMetrics jmxPrometheusExporterMetrics = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(cmks).endValueFrom().build();
    KafkaResource.replaceKafkaResourceInSpecificNamespace(SECOND_CLUSTER, k -> {
        k.getSpec().getKafka().setMetricsConfig(jmxPrometheusExporterMetrics);
    }, SECOND_NAMESPACE);
    PodUtils.verifyThatRunningPodsAreStable(SECOND_NAMESPACE, SECOND_CLUSTER);
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(SECOND_CLUSTER, 1)) {
        ConfigMap actualCm = kubeClient(SECOND_NAMESPACE).getConfigMap(cmName);
        assertThat(actualCm.getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(metricsConfigJson));
    }
    // update metrics
    ConfigMap externalMetricsUpdatedCm = new ConfigMapBuilder().withData(Collections.singletonMap(Constants.METRICS_CONFIG_YAML_NAME, metricsConfigYaml.replace("true", "false"))).withNewMetadata().withName("external-metrics-cm").withNamespace(SECOND_NAMESPACE).endMetadata().build();
    kubeClient().getClient().configMaps().inNamespace(SECOND_NAMESPACE).createOrReplace(externalMetricsUpdatedCm);
    PodUtils.verifyThatRunningPodsAreStable(SECOND_NAMESPACE, SECOND_CLUSTER);
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(SECOND_CLUSTER, 1)) {
        ConfigMap actualCm = kubeClient(SECOND_NAMESPACE).getConfigMap(cmName);
        assertThat(actualCm.getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(metricsConfigJson.replace("true", "false")));
    }
}
Also used : JmxPrometheusExporterMetrics(io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics) ConfigMap(io.fabric8.kubernetes.api.model.ConfigMap) ConfigMapKeySelectorBuilder(io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder) JmxPrometheusExporterMetricsBuilder(io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsBuilder) ConfigMapBuilder(io.fabric8.kubernetes.api.model.ConfigMapBuilder) Matchers.emptyString(org.hamcrest.Matchers.emptyString) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ConfigMapKeySelector(io.fabric8.kubernetes.api.model.ConfigMapKeySelector) ParallelTest(io.strimzi.systemtest.annotations.ParallelTest)

Example 8 with JmxPrometheusExporterMetrics

use of io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics in project strimzi-kafka-operator by strimzi.

the class RollingUpdateST method testMetricsChange.

@IsolatedTest
@Tag(ROLLING_UPDATE)
@KRaftNotSupported("Zookeeper is not supported by KRaft mode and is used in this test class")
@SuppressWarnings("checkstyle:MethodLength")
void testMetricsChange(ExtensionContext extensionContext) throws JsonProcessingException {
    final TestStorage testStorage = new TestStorage(extensionContext, namespace);
    // Kafka
    Map<String, Object> kafkaRule = new HashMap<>();
    kafkaRule.put("pattern", "kafka.(\\w+)<type=(.+), name=(.+)><>Count");
    kafkaRule.put("name", "kafka_$1_$2_$3_count");
    kafkaRule.put("type", "COUNTER");
    Map<String, Object> kafkaMetrics = new HashMap<>();
    kafkaMetrics.put("lowercaseOutputName", true);
    kafkaMetrics.put("rules", Collections.singletonList(kafkaRule));
    String metricsCMNameK = "k-metrics-cm";
    ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
    String yaml = mapper.writeValueAsString(kafkaMetrics);
    ConfigMap metricsCMK = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameK).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", yaml)).build();
    JmxPrometheusExporterMetrics kafkaMetricsConfig = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(new ConfigMapKeySelectorBuilder().withName(metricsCMNameK).withKey("metrics-config.yml").withOptional(true).build()).endValueFrom().build();
    // Zookeeper
    Map<String, Object> zookeeperLabels = new HashMap<>();
    zookeeperLabels.put("replicaId", "$2");
    Map<String, Object> zookeeperRule = new HashMap<>();
    zookeeperRule.put("labels", zookeeperLabels);
    zookeeperRule.put("name", "zookeeper_$3");
    zookeeperRule.put("pattern", "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)");
    Map<String, Object> zookeeperMetrics = new HashMap<>();
    zookeeperMetrics.put("lowercaseOutputName", true);
    zookeeperMetrics.put("rules", Collections.singletonList(zookeeperRule));
    String metricsCMNameZk = "zk-metrics-cm";
    ConfigMap metricsCMZk = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameZk).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(zookeeperMetrics))).build();
    JmxPrometheusExporterMetrics zkMetricsConfig = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(new ConfigMapKeySelectorBuilder().withName(metricsCMNameZk).withKey("metrics-config.yml").withOptional(true).build()).endValueFrom().build();
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMK);
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMZk);
    resourceManager.createResource(extensionContext, KafkaTemplates.kafkaEphemeral(testStorage.getClusterName(), 3, 3).editMetadata().withNamespace(testStorage.getNamespaceName()).endMetadata().editSpec().editKafka().withMetricsConfig(kafkaMetricsConfig).endKafka().editOrNewZookeeper().withMetricsConfig(zkMetricsConfig).endZookeeper().withNewKafkaExporter().endKafkaExporter().endSpec().build());
    Map<String, String> kafkaPods = PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getKafkaSelector());
    Map<String, String> zkPods = PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getZookeeperSelector());
    resourceManager.createResource(extensionContext, ScraperTemplates.scraperPod(testStorage.getNamespaceName(), testStorage.getScraperName()).build());
    String metricsScraperPodName = PodUtils.getPodsByPrefixInNameWithDynamicWait(testStorage.getNamespaceName(), testStorage.getScraperName()).get(0).getMetadata().getName();
    MetricsCollector metricsCollector = new MetricsCollector.Builder().withNamespaceName(testStorage.getNamespaceName()).withScraperPodName(metricsScraperPodName).withComponentName(testStorage.getClusterName()).withComponentType(ComponentType.Kafka).build();
    LOGGER.info("Check if metrics are present in pod of Kafka and Zookeeper");
    Map<String, String> kafkaMetricsOutput = metricsCollector.collectMetricsFromPods();
    Map<String, String> zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPods();
    assertThat(kafkaMetricsOutput.values().toString().contains("kafka_"), is(true));
    assertThat(zkMetricsOutput.values().toString().contains("replicaId"), is(true));
    LOGGER.info("Changing metrics to something else");
    kafkaRule.replace("pattern", "kafka.(\\w+)<type=(.+), name=(.+)><>Count", "kafka.(\\w+)<type=(.+), name=(.+)Percent\\w*><>MeanRate");
    kafkaRule.replace("name", "kafka_$1_$2_$3_count", "kafka_$1_$2_$3_percent");
    kafkaRule.replace("type", "COUNTER", "GAUGE");
    zookeeperRule.replace("pattern", "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)", "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)");
    zookeeperRule.replace("name", "zookeeper_$3", "zookeeper_$2");
    zookeeperRule.replace("labels", zookeeperLabels, null);
    metricsCMZk = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameZk).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(zookeeperMetrics))).build();
    metricsCMK = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameK).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(kafkaMetrics))).build();
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMK);
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMZk);
    PodUtils.verifyThatRunningPodsAreStable(testStorage.getNamespaceName(), KafkaResources.zookeeperStatefulSetName(testStorage.getClusterName()));
    PodUtils.verifyThatRunningPodsAreStable(testStorage.getNamespaceName(), KafkaResources.kafkaStatefulSetName(testStorage.getClusterName()));
    LOGGER.info("Check if Kafka and Zookeeper pods didn't roll");
    assertThat(PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getZookeeperSelector()), is(zkPods));
    assertThat(PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getKafkaSelector()), is(kafkaPods));
    LOGGER.info("Check if Kafka and Zookeeper metrics are changed");
    ObjectMapper yamlReader = new ObjectMapper(new YAMLFactory());
    String kafkaMetricsConf = kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(metricsCMNameK).get().getData().get("metrics-config.yml");
    String zkMetricsConf = kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(metricsCMNameZk).get().getData().get("metrics-config.yml");
    Object kafkaMetricsJsonToYaml = yamlReader.readValue(kafkaMetricsConf, Object.class);
    Object zkMetricsJsonToYaml = yamlReader.readValue(zkMetricsConf, Object.class);
    ObjectMapper jsonWriter = new ObjectMapper();
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(testStorage.getClusterName(), 3)) {
        assertThat(kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(cmName).get().getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(jsonWriter.writeValueAsString(kafkaMetricsJsonToYaml)));
    }
    assertThat(kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(KafkaResources.zookeeperMetricsAndLogConfigMapName(testStorage.getClusterName())).get().getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(jsonWriter.writeValueAsString(zkMetricsJsonToYaml)));
    LOGGER.info("Check if metrics are present in pod of Kafka and Zookeeper");
    kafkaMetricsOutput = metricsCollector.collectMetricsFromPods();
    zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPods();
    assertThat(kafkaMetricsOutput.values().toString().contains("kafka_"), is(true));
    assertThat(zkMetricsOutput.values().toString().contains("replicaId"), is(true));
    LOGGER.info("Removing metrics from Kafka and Zookeeper and setting them to null");
    KafkaResource.replaceKafkaResourceInSpecificNamespace(testStorage.getClusterName(), kafka -> {
        kafka.getSpec().getKafka().setMetricsConfig(null);
        kafka.getSpec().getZookeeper().setMetricsConfig(null);
    }, testStorage.getNamespaceName());
    LOGGER.info("Wait if Kafka and Zookeeper pods will roll");
    RollingUpdateUtils.waitTillComponentHasRolledAndPodsReady(testStorage.getNamespaceName(), testStorage.getZookeeperSelector(), 3, zkPods);
    RollingUpdateUtils.waitTillComponentHasRolled(testStorage.getNamespaceName(), testStorage.getKafkaSelector(), 3, kafkaPods);
    LOGGER.info("Check if metrics are not existing in pods");
    kafkaMetricsOutput = metricsCollector.collectMetricsFromPodsWithoutWait();
    zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPodsWithoutWait();
    kafkaMetricsOutput.values().forEach(value -> assertThat(value, is("")));
    zkMetricsOutput.values().forEach(value -> assertThat(value, is("")));
}
Also used : MetricsCollector(io.strimzi.systemtest.metrics.MetricsCollector) ConfigMap(io.fabric8.kubernetes.api.model.ConfigMap) HashMap(java.util.HashMap) ConfigMapKeySelectorBuilder(io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder) JmxPrometheusExporterMetrics(io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics) JmxPrometheusExporterMetricsBuilder(io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsBuilder) ConfigMapBuilder(io.fabric8.kubernetes.api.model.ConfigMapBuilder) YAMLFactory(com.fasterxml.jackson.dataformat.yaml.YAMLFactory) TestStorage(io.strimzi.systemtest.storage.TestStorage) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) KRaftNotSupported(io.strimzi.systemtest.annotations.KRaftNotSupported) IsolatedTest(io.strimzi.systemtest.annotations.IsolatedTest) Tag(org.junit.jupiter.api.Tag)

Example 9 with JmxPrometheusExporterMetrics

use of io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics in project strimzi by strimzi.

the class RollingUpdateST method testMetricsChange.

@IsolatedTest
@Tag(ROLLING_UPDATE)
@KRaftNotSupported("Zookeeper is not supported by KRaft mode and is used in this test class")
@SuppressWarnings("checkstyle:MethodLength")
void testMetricsChange(ExtensionContext extensionContext) throws JsonProcessingException {
    final TestStorage testStorage = new TestStorage(extensionContext, namespace);
    // Kafka
    Map<String, Object> kafkaRule = new HashMap<>();
    kafkaRule.put("pattern", "kafka.(\\w+)<type=(.+), name=(.+)><>Count");
    kafkaRule.put("name", "kafka_$1_$2_$3_count");
    kafkaRule.put("type", "COUNTER");
    Map<String, Object> kafkaMetrics = new HashMap<>();
    kafkaMetrics.put("lowercaseOutputName", true);
    kafkaMetrics.put("rules", Collections.singletonList(kafkaRule));
    String metricsCMNameK = "k-metrics-cm";
    ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
    String yaml = mapper.writeValueAsString(kafkaMetrics);
    ConfigMap metricsCMK = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameK).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", yaml)).build();
    JmxPrometheusExporterMetrics kafkaMetricsConfig = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(new ConfigMapKeySelectorBuilder().withName(metricsCMNameK).withKey("metrics-config.yml").withOptional(true).build()).endValueFrom().build();
    // Zookeeper
    Map<String, Object> zookeeperLabels = new HashMap<>();
    zookeeperLabels.put("replicaId", "$2");
    Map<String, Object> zookeeperRule = new HashMap<>();
    zookeeperRule.put("labels", zookeeperLabels);
    zookeeperRule.put("name", "zookeeper_$3");
    zookeeperRule.put("pattern", "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)");
    Map<String, Object> zookeeperMetrics = new HashMap<>();
    zookeeperMetrics.put("lowercaseOutputName", true);
    zookeeperMetrics.put("rules", Collections.singletonList(zookeeperRule));
    String metricsCMNameZk = "zk-metrics-cm";
    ConfigMap metricsCMZk = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameZk).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(zookeeperMetrics))).build();
    JmxPrometheusExporterMetrics zkMetricsConfig = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(new ConfigMapKeySelectorBuilder().withName(metricsCMNameZk).withKey("metrics-config.yml").withOptional(true).build()).endValueFrom().build();
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMK);
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMZk);
    resourceManager.createResource(extensionContext, KafkaTemplates.kafkaEphemeral(testStorage.getClusterName(), 3, 3).editMetadata().withNamespace(testStorage.getNamespaceName()).endMetadata().editSpec().editKafka().withMetricsConfig(kafkaMetricsConfig).endKafka().editOrNewZookeeper().withMetricsConfig(zkMetricsConfig).endZookeeper().withNewKafkaExporter().endKafkaExporter().endSpec().build());
    Map<String, String> kafkaPods = PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getKafkaSelector());
    Map<String, String> zkPods = PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getZookeeperSelector());
    resourceManager.createResource(extensionContext, ScraperTemplates.scraperPod(testStorage.getNamespaceName(), testStorage.getScraperName()).build());
    String metricsScraperPodName = PodUtils.getPodsByPrefixInNameWithDynamicWait(testStorage.getNamespaceName(), testStorage.getScraperName()).get(0).getMetadata().getName();
    MetricsCollector metricsCollector = new MetricsCollector.Builder().withNamespaceName(testStorage.getNamespaceName()).withScraperPodName(metricsScraperPodName).withComponentName(testStorage.getClusterName()).withComponentType(ComponentType.Kafka).build();
    LOGGER.info("Check if metrics are present in pod of Kafka and Zookeeper");
    Map<String, String> kafkaMetricsOutput = metricsCollector.collectMetricsFromPods();
    Map<String, String> zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPods();
    assertThat(kafkaMetricsOutput.values().toString().contains("kafka_"), is(true));
    assertThat(zkMetricsOutput.values().toString().contains("replicaId"), is(true));
    LOGGER.info("Changing metrics to something else");
    kafkaRule.replace("pattern", "kafka.(\\w+)<type=(.+), name=(.+)><>Count", "kafka.(\\w+)<type=(.+), name=(.+)Percent\\w*><>MeanRate");
    kafkaRule.replace("name", "kafka_$1_$2_$3_count", "kafka_$1_$2_$3_percent");
    kafkaRule.replace("type", "COUNTER", "GAUGE");
    zookeeperRule.replace("pattern", "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)", "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)");
    zookeeperRule.replace("name", "zookeeper_$3", "zookeeper_$2");
    zookeeperRule.replace("labels", zookeeperLabels, null);
    metricsCMZk = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameZk).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(zookeeperMetrics))).build();
    metricsCMK = new ConfigMapBuilder().withNewMetadata().withName(metricsCMNameK).withNamespace(testStorage.getNamespaceName()).endMetadata().withData(singletonMap("metrics-config.yml", mapper.writeValueAsString(kafkaMetrics))).build();
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMK);
    kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).createOrReplace(metricsCMZk);
    PodUtils.verifyThatRunningPodsAreStable(testStorage.getNamespaceName(), KafkaResources.zookeeperStatefulSetName(testStorage.getClusterName()));
    PodUtils.verifyThatRunningPodsAreStable(testStorage.getNamespaceName(), KafkaResources.kafkaStatefulSetName(testStorage.getClusterName()));
    LOGGER.info("Check if Kafka and Zookeeper pods didn't roll");
    assertThat(PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getZookeeperSelector()), is(zkPods));
    assertThat(PodUtils.podSnapshot(testStorage.getNamespaceName(), testStorage.getKafkaSelector()), is(kafkaPods));
    LOGGER.info("Check if Kafka and Zookeeper metrics are changed");
    ObjectMapper yamlReader = new ObjectMapper(new YAMLFactory());
    String kafkaMetricsConf = kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(metricsCMNameK).get().getData().get("metrics-config.yml");
    String zkMetricsConf = kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(metricsCMNameZk).get().getData().get("metrics-config.yml");
    Object kafkaMetricsJsonToYaml = yamlReader.readValue(kafkaMetricsConf, Object.class);
    Object zkMetricsJsonToYaml = yamlReader.readValue(zkMetricsConf, Object.class);
    ObjectMapper jsonWriter = new ObjectMapper();
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(testStorage.getClusterName(), 3)) {
        assertThat(kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(cmName).get().getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(jsonWriter.writeValueAsString(kafkaMetricsJsonToYaml)));
    }
    assertThat(kubeClient().getClient().configMaps().inNamespace(testStorage.getNamespaceName()).withName(KafkaResources.zookeeperMetricsAndLogConfigMapName(testStorage.getClusterName())).get().getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(jsonWriter.writeValueAsString(zkMetricsJsonToYaml)));
    LOGGER.info("Check if metrics are present in pod of Kafka and Zookeeper");
    kafkaMetricsOutput = metricsCollector.collectMetricsFromPods();
    zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPods();
    assertThat(kafkaMetricsOutput.values().toString().contains("kafka_"), is(true));
    assertThat(zkMetricsOutput.values().toString().contains("replicaId"), is(true));
    LOGGER.info("Removing metrics from Kafka and Zookeeper and setting them to null");
    KafkaResource.replaceKafkaResourceInSpecificNamespace(testStorage.getClusterName(), kafka -> {
        kafka.getSpec().getKafka().setMetricsConfig(null);
        kafka.getSpec().getZookeeper().setMetricsConfig(null);
    }, testStorage.getNamespaceName());
    LOGGER.info("Wait if Kafka and Zookeeper pods will roll");
    RollingUpdateUtils.waitTillComponentHasRolledAndPodsReady(testStorage.getNamespaceName(), testStorage.getZookeeperSelector(), 3, zkPods);
    RollingUpdateUtils.waitTillComponentHasRolled(testStorage.getNamespaceName(), testStorage.getKafkaSelector(), 3, kafkaPods);
    LOGGER.info("Check if metrics are not existing in pods");
    kafkaMetricsOutput = metricsCollector.collectMetricsFromPodsWithoutWait();
    zkMetricsOutput = metricsCollector.toBuilder().withComponentType(ComponentType.Zookeeper).build().collectMetricsFromPodsWithoutWait();
    kafkaMetricsOutput.values().forEach(value -> assertThat(value, is("")));
    zkMetricsOutput.values().forEach(value -> assertThat(value, is("")));
}
Also used : MetricsCollector(io.strimzi.systemtest.metrics.MetricsCollector) ConfigMap(io.fabric8.kubernetes.api.model.ConfigMap) HashMap(java.util.HashMap) ConfigMapKeySelectorBuilder(io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder) JmxPrometheusExporterMetrics(io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics) JmxPrometheusExporterMetricsBuilder(io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsBuilder) ConfigMapBuilder(io.fabric8.kubernetes.api.model.ConfigMapBuilder) YAMLFactory(com.fasterxml.jackson.dataformat.yaml.YAMLFactory) TestStorage(io.strimzi.systemtest.storage.TestStorage) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) KRaftNotSupported(io.strimzi.systemtest.annotations.KRaftNotSupported) IsolatedTest(io.strimzi.systemtest.annotations.IsolatedTest) Tag(org.junit.jupiter.api.Tag)

Example 10 with JmxPrometheusExporterMetrics

use of io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics in project strimzi by strimzi.

the class MetricsIsolatedST method testKafkaMetricsSettings.

/**
 * 1. Update metrics form whatever it is to @metricsConfigYaml in spec.kafka.metricsConfig
 * 2. Check, whether the metrics ConfigMap is changed
 * 3. Updates ConfigMap linked as metrics on
 * 4. Check, whether the metrics ConfigMap is changed
 */
@ParallelTest
void testKafkaMetricsSettings() {
    String metricsConfigJson = "{\"lowercaseOutputName\":true}";
    String metricsConfigYaml = "lowercaseOutputName: true";
    ObjectMapper mapper = new ObjectMapper();
    mapper.configure(JsonReadFeature.ALLOW_UNESCAPED_CONTROL_CHARS.mappedFeature(), true);
    ConfigMap externalMetricsCm = new ConfigMapBuilder().withData(Collections.singletonMap(Constants.METRICS_CONFIG_YAML_NAME, metricsConfigYaml)).withNewMetadata().withName("external-metrics-cm").withNamespace(SECOND_NAMESPACE).endMetadata().build();
    kubeClient().getClient().configMaps().inNamespace(SECOND_NAMESPACE).createOrReplace(externalMetricsCm);
    // spec.kafka.metrics -> spec.kafka.jmxExporterMetrics
    ConfigMapKeySelector cmks = new ConfigMapKeySelectorBuilder().withName("external-metrics-cm").withKey(Constants.METRICS_CONFIG_YAML_NAME).build();
    JmxPrometheusExporterMetrics jmxPrometheusExporterMetrics = new JmxPrometheusExporterMetricsBuilder().withNewValueFrom().withConfigMapKeyRef(cmks).endValueFrom().build();
    KafkaResource.replaceKafkaResourceInSpecificNamespace(SECOND_CLUSTER, k -> {
        k.getSpec().getKafka().setMetricsConfig(jmxPrometheusExporterMetrics);
    }, SECOND_NAMESPACE);
    PodUtils.verifyThatRunningPodsAreStable(SECOND_NAMESPACE, SECOND_CLUSTER);
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(SECOND_CLUSTER, 1)) {
        ConfigMap actualCm = kubeClient(SECOND_NAMESPACE).getConfigMap(cmName);
        assertThat(actualCm.getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(metricsConfigJson));
    }
    // update metrics
    ConfigMap externalMetricsUpdatedCm = new ConfigMapBuilder().withData(Collections.singletonMap(Constants.METRICS_CONFIG_YAML_NAME, metricsConfigYaml.replace("true", "false"))).withNewMetadata().withName("external-metrics-cm").withNamespace(SECOND_NAMESPACE).endMetadata().build();
    kubeClient().getClient().configMaps().inNamespace(SECOND_NAMESPACE).createOrReplace(externalMetricsUpdatedCm);
    PodUtils.verifyThatRunningPodsAreStable(SECOND_NAMESPACE, SECOND_CLUSTER);
    for (String cmName : StUtils.getKafkaConfigurationConfigMaps(SECOND_CLUSTER, 1)) {
        ConfigMap actualCm = kubeClient(SECOND_NAMESPACE).getConfigMap(cmName);
        assertThat(actualCm.getData().get(Constants.METRICS_CONFIG_JSON_NAME), is(metricsConfigJson.replace("true", "false")));
    }
}
Also used : JmxPrometheusExporterMetrics(io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics) ConfigMap(io.fabric8.kubernetes.api.model.ConfigMap) ConfigMapKeySelectorBuilder(io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder) JmxPrometheusExporterMetricsBuilder(io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsBuilder) ConfigMapBuilder(io.fabric8.kubernetes.api.model.ConfigMapBuilder) Matchers.emptyString(org.hamcrest.Matchers.emptyString) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ConfigMapKeySelector(io.fabric8.kubernetes.api.model.ConfigMapKeySelector) ParallelTest(io.strimzi.systemtest.annotations.ParallelTest)

Aggregations

JmxPrometheusExporterMetrics (io.strimzi.api.kafka.model.JmxPrometheusExporterMetrics)14 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)8 YAMLFactory (com.fasterxml.jackson.dataformat.yaml.YAMLFactory)6 ConfigMap (io.fabric8.kubernetes.api.model.ConfigMap)6 ConfigMapBuilder (io.fabric8.kubernetes.api.model.ConfigMapBuilder)6 ConfigMapKeySelectorBuilder (io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder)6 JmxPrometheusExporterMetricsBuilder (io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsBuilder)6 Kafka (io.strimzi.api.kafka.model.Kafka)6 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)4 ConfigMapKeySelector (io.fabric8.kubernetes.api.model.ConfigMapKeySelector)4 IntOrString (io.fabric8.kubernetes.api.model.IntOrString)4 KafkaBuilder (io.strimzi.api.kafka.model.KafkaBuilder)2 KafkaExporterSpec (io.strimzi.api.kafka.model.KafkaExporterSpec)2 IsolatedTest (io.strimzi.systemtest.annotations.IsolatedTest)2 KRaftNotSupported (io.strimzi.systemtest.annotations.KRaftNotSupported)2 ParallelTest (io.strimzi.systemtest.annotations.ParallelTest)2 MetricsCollector (io.strimzi.systemtest.metrics.MetricsCollector)2 TestStorage (io.strimzi.systemtest.storage.TestStorage)2 Checkpoint (io.vertx.junit5.Checkpoint)2 HashMap (java.util.HashMap)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial