Search in sources :

Example 1 with MetricReporterFactory

use of org.apache.flink.metrics.reporter.MetricReporterFactory in project flink by apache.

the class ReporterSetup method setupReporters.

private static List<ReporterSetup> setupReporters(Map<String, MetricReporterFactory> reporterFactories, List<Tuple2<String, Configuration>> reporterConfigurations) {
    List<ReporterSetup> reporterSetups = new ArrayList<>(reporterConfigurations.size());
    for (Tuple2<String, Configuration> reporterConfiguration : reporterConfigurations) {
        String reporterName = reporterConfiguration.f0;
        Configuration reporterConfig = reporterConfiguration.f1;
        try {
            Optional<MetricReporter> metricReporterOptional = loadReporter(reporterName, reporterConfig, reporterFactories);
            // massage user variables keys into scope format for parity to variable exclusion
            Map<String, String> additionalVariables = reporterConfig.get(ADDITIONAL_VARIABLES).entrySet().stream().collect(Collectors.toMap(e -> ScopeFormat.asVariable(e.getKey()), Entry::getValue));
            metricReporterOptional.ifPresent(reporter -> {
                MetricConfig metricConfig = new MetricConfig();
                reporterConfig.addAllToProperties(metricConfig);
                reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter, additionalVariables));
            });
        } catch (Throwable t) {
            LOG.error("Could not instantiate metrics reporter {}. Metrics might not be exposed/reported.", reporterName, t);
        }
    }
    return reporterSetups;
}
Also used : ConfigOptions(org.apache.flink.configuration.ConfigOptions) MetricConfig(org.apache.flink.metrics.MetricConfig) Tuple2(org.apache.flink.api.java.tuple.Tuple2) DelegatingConfiguration(org.apache.flink.configuration.DelegatingConfiguration) LoggerFactory(org.slf4j.LoggerFactory) InterceptInstantiationViaReflection(org.apache.flink.metrics.reporter.InterceptInstantiationViaReflection) HashMap(java.util.HashMap) MetricReporter(org.apache.flink.metrics.reporter.MetricReporter) TreeSet(java.util.TreeSet) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) Matcher(java.util.regex.Matcher) InstantiateViaFactory(org.apache.flink.metrics.reporter.InstantiateViaFactory) ScopeFormat(org.apache.flink.runtime.metrics.scope.ScopeFormat) Map(java.util.Map) ConfigConstants(org.apache.flink.configuration.ConfigConstants) ServiceConfigurationError(java.util.ServiceConfigurationError) ConfigOption(org.apache.flink.configuration.ConfigOption) Nullable(javax.annotation.Nullable) MetricReporterFactory(org.apache.flink.metrics.reporter.MetricReporterFactory) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) Configuration(org.apache.flink.configuration.Configuration) Set(java.util.Set) ServiceLoader(java.util.ServiceLoader) MetricOptions(org.apache.flink.configuration.MetricOptions) Collectors(java.util.stream.Collectors) File(java.io.File) Iterators(org.apache.flink.shaded.guava30.com.google.common.collect.Iterators) VisibleForTesting(org.apache.flink.annotation.VisibleForTesting) List(java.util.List) PluginManager(org.apache.flink.core.plugin.PluginManager) Entry(java.util.Map.Entry) Optional(java.util.Optional) Pattern(java.util.regex.Pattern) Collections(java.util.Collections) MetricConfig(org.apache.flink.metrics.MetricConfig) DelegatingConfiguration(org.apache.flink.configuration.DelegatingConfiguration) Configuration(org.apache.flink.configuration.Configuration) ArrayList(java.util.ArrayList) MetricReporter(org.apache.flink.metrics.reporter.MetricReporter)

Example 2 with MetricReporterFactory

use of org.apache.flink.metrics.reporter.MetricReporterFactory in project flink by apache.

the class ReporterSetup method loadAvailableReporterFactories.

private static Map<String, MetricReporterFactory> loadAvailableReporterFactories(@Nullable PluginManager pluginManager) {
    final Map<String, MetricReporterFactory> reporterFactories = new HashMap<>(2);
    final Iterator<MetricReporterFactory> factoryIterator = getAllReporterFactories(pluginManager);
    // non-existing factory class
    while (factoryIterator.hasNext()) {
        try {
            MetricReporterFactory factory = factoryIterator.next();
            String factoryClassName = factory.getClass().getName();
            MetricReporterFactory existingFactory = reporterFactories.get(factoryClassName);
            if (existingFactory == null) {
                reporterFactories.put(factoryClassName, factory);
                LOG.debug("Found reporter factory {} at {} ", factoryClassName, new File(factory.getClass().getProtectionDomain().getCodeSource().getLocation().toURI()).getCanonicalPath());
            } else {
                LOG.warn("Multiple implementations of the same reporter were found in 'lib' and/or 'plugins' directories for {}. It is recommended to remove redundant reporter JARs to resolve used versions' ambiguity.", factoryClassName);
            }
        } catch (Exception | ServiceConfigurationError e) {
            LOG.warn("Error while loading reporter factory.", e);
        }
    }
    return Collections.unmodifiableMap(reporterFactories);
}
Also used : HashMap(java.util.HashMap) ServiceConfigurationError(java.util.ServiceConfigurationError) MetricReporterFactory(org.apache.flink.metrics.reporter.MetricReporterFactory) File(java.io.File)

Aggregations

File (java.io.File)2 HashMap (java.util.HashMap)2 ServiceConfigurationError (java.util.ServiceConfigurationError)2 MetricReporterFactory (org.apache.flink.metrics.reporter.MetricReporterFactory)2 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 HashSet (java.util.HashSet)1 Iterator (java.util.Iterator)1 List (java.util.List)1 Map (java.util.Map)1 Entry (java.util.Map.Entry)1 Optional (java.util.Optional)1 ServiceLoader (java.util.ServiceLoader)1 Set (java.util.Set)1 TreeSet (java.util.TreeSet)1 Matcher (java.util.regex.Matcher)1 Pattern (java.util.regex.Pattern)1 Collectors (java.util.stream.Collectors)1 Nullable (javax.annotation.Nullable)1 VisibleForTesting (org.apache.flink.annotation.VisibleForTesting)1