prometheus/prometheus
kubernetes_sd_config (alertmanagers): reloading config doesn't work
Closed
#11,013 opened on Jul 13, 2022
component/service discoverycomponent/service discovery/kuberneteshelp wantedkind/bugpriority/P2
Repository metrics
- Stars
- (64,042 stars)
- PR merge metrics
- (Avg merge 11d 5h) (118 merged PRs in 30d)
Description
What did you do?
It is possible to automatically discovery alertmanagers if they are hosted in Kubernetes, something like this:
alerting:
alertmanagers:
- kubernetes_sd_configs:
- role: pod
namespaces:
names:
- monitoring
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
regex: alertmanager
action: keep
What did you expect to see?
Updating the configuration when adding new alertmanagers.
What did you see instead? Under which circumstances?
Prometheus only discovery alertmanagers at startup. If the pods with the alertmanagers were recreated (and their ip changed), then Prometheus will not know about them.
System information
No response
Prometheus version
Version 2.36.2
Revision d7e7b8e04b5ecdc1dd153534ba376a622b72741b
Branch HEAD
BuildUser root@f051ce0d6050
BuildDate 20220620-13:21:35
GoVersion go1.18.3
Prometheus configuration file
...
alerting:
alertmanagers:
- kubernetes_sd_configs:
- role: pod
namespaces:
names:
- monitoring
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
regex: alertmanager
action: keep
...
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response