普罗米修斯针对未发现目标的警报规则

Question

我在使用配置监控模块时收到警报prometheus/kube-prometheus-stack 25.1.0。

\n

警报

\n

[FIRING:1] KubeProxyDown - critical\nAlert: Target disappeared from Prometheus target discovery. - critical\n Description: KubeProxy has disappeared from Prometheus target discovery.\n Details:\n  \xe2\x80\xa2 alertname: KubeProxyDown\n  \xe2\x80\xa2 prometheus: monitoring/prometheus-kube-prometheus-prometheus\n  \xe2\x80\xa2 severity: critical\n

\n

我认为这是一个新的默认规则kube-prometheus-stack 25.x.x。它不存在于prometheus/kube-prometheus-stack 21.x.x.

\n

EKS 和 minikube 中也发生了同样的问题。

\n

KubeProxyDown规则

\n

alert: KubeProxyDown\nexpr: absent(up{job="kube-proxy"}\n  == 1)\nfor: 15m\nlabels:\n  severity: critical\nannotations:\n  description: KubeProxy has disappeared from Prometheus target discovery.\n  runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeproxydown\n  summary: Target disappeared from Prometheus target discovery.\n

\n

我该如何解决这个问题？

\n

如果有人能帮助我，我将不胜感激

\n

Answer 1

metrics-bind-address发生了变化kube-proxy。以下是此处、此处和此处发布的问题。我可以提出以下建议。将 ConfigMap更改kube-proxy为不同的值：

$ kubectl edit cm/kube-proxy -n kube-system
## Change from
    metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
## Change to
    metricsBindAddress: 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

Answer 2

这是在 AWS EKS 集群 v1.21 中对我有用的方法：

$ kubectl edit cm/kube-proxy-config -n kube-system
---
metricsBindAddress: 127.0.0.1:10249 ### <--- change to 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

注意，配置映射的名称是kube-proxy-config，而不是kube-proxy