使用 prometheus 抓取 kubernetes 容器的 HTTP 和 HTTPS 端口指标

Question

我们希望 Prometheus 安装能够抓取 Pod 中两个容器的指标。一个容器通过 HTTPS 在端口 443 公开指标，而另一个容器通过 HTTP 在端口 8080 公开指标。两个容器在同一路径提供指标，即/metrics.

如果我们将prometheus.io/scheme声明为 http 或 https，则只会抓取一个容器。对于另一个我们总是收到：如果我们根本server returned HTTP status 400 Bad Request 不定义prometheus.io/scheme，也会发生同样的情况。然后，Prometheus 将为两个端口使用 http，并且对于在端口 443 公开指标的容器会失败，因为它只期望 HTTPS 请求。

有没有办法告诉普罗米修斯如何准确地抓取我们部署中的各个容器？获取两个容器的指标有哪些可行的解决方法？

版本

库伯内特斯：1.10.2

普罗米修斯：2.2.1

部署摘录

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxx
  namespace: xxx
spec:
  selector:
    matchLabels:
      app: xxx
  template:
    metadata:
      labels:
        app: xxx
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: container-1
        image: xxx
        ports:
        - containerPort: 443
      - name: container-2
        image: xxx
        ports:
        - containerPort: 8080

普罗米修斯配置：

- job_name: kubernetes-pods
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names: []
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace

Answer 1

我发现了一个 GIST 片段，如果它被命名为“metrics”，它会直接从容器中获取端口，而不是依赖于每个 pod 注释。它还包含一条注释，使其成为任何以“metrics”开头的端口的正则表达式。

也许您可以扩展它以从端口名称中提取架构，例如“metrics-http”和“metrics-https”。

https://gist.github.com/bakins/5bf7d4e719f36c1c555d81134d8887eb

# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. This
#    will be the same for every container in the pod that is scraped.
# * this will scrape every container in a pod with `prometheus.io/scrape` set to true and the
    port is name `metrics` in the container
# * note `prometheus.io/port` is no longer honored. You must name the port(s) to scrape `metrics`
#   Also, in some of the issues I read, there was mention of a container role, but I couldn't get 
#   that to work - or find any more info on it.
- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: metrics
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name