使用prometheus和alertmanager在松弛状态下不显示警报消息

Question

我试图通过使用alertmanager获得Prometheus发现的警报以获得松弛通知.

这是alert.rules文件,工作正常

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

它成功地显示了一个实例.

但是我的alertmanager.yml中的问题是它没有向slack发送通知.我也成功设置了松弛的webhook,甚至测试了钩子是否正常工作,同时创建了一个钩子与松弛提供的服务

alertmanager.yml

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'
[tgurung@ip131 prometheus_graphana_myversion]$ cat alertmanager/alertmanager.yml 


route: 
  receiver: 'slack-notifications'
  #group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: https://hooks.slack.com/services/T52GRFN3F/B93KTCUHH/JC
    channel: #general
    send_resolved: true

    # Alertmanager templates apply here.
    text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

当运行docker-compose时,我得到了关注

prometheus_1     | level=error ts=2018-02-06T09:36:35.580565429Z caller=notifier.go:454 component=notifier alertmanager=http://x.x.x.x:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://x.X.x.x:9093/api/v1/alerts: dial tcp x.x.x.x:9093: getsockopt: no route to host"

解决上述错误: 为了解决上述路由问题,我在全新的实例中运行了alertmanager,并且克服了该错误

转到错误消息中的API链接我可以看到这一点

{"status":"success","data":[]}

这是一个alert_manager,看起来很棒.

alertmanager_1   | level=info ts=2018-02-06T09:36:37.66654544Z caller=main.go:141 msg="Starting Alertmanager" version="(version=0.13.0, branch=HEAD, revision=fb713f6d8239b57c646cae30f78e8b4b8861a1aa)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.66661402Z caller=main.go:142 build_context="(go=go1.9.2, user=root@d83981af1d3d, date=20180112-10:32:46)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.668103448Z caller=main.go:279 msg="Loading configuration file" file=/alertmanager/alertmanager.yml
alertmanager_1   | level=info ts=2018-02-06T09:36:37.673288146Z caller=main.go:354 msg=Listening address=:9093

这是prometheus.yml配置文件

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

#alerting rules file
rule_files:
  - '/alertmanager/alert.rules'


scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
       - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']


alerting:
  alertmanagers:
    - static_configs:
      - targets: ["54.36.X.X:9093"]  #this is the alertmanager service url 

这是我的docker-compose.yml

version: '2'
volumes:
    grafana_data: {}

services:
    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanager/alert.rules:/alertmanager/alert.rules
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'
        links:
            - "alertmanager"


    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/alertmanager/alertmanager.yml'
        ports:
            - '9093:9093'

alertmanager状态链接不显示在docker-composer中从卷传递的配置.它显示默认配置

Answer 1

正如有些人已经指出的那样，alertmanager 配置仅包括如何发送警报，而不包括创建警报。这就是普罗米修斯的工作。看看这个repo，它非常简单地设置了prometheus和alertmanager

https://github.com/stefanprodan/dockprom