Getting error "Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused"

Question

I'm trying to configure Prometheus and Grafana with my Hyperledger fabric v1.4 network to analyze the peer and chaincode mertics. I've mapped peer container's port 9443 to my host machine's port 9443 after following this documentation. I've also changed the provider entry to prometheus under metrics section in core.yml of peer. I've configured prometheus and grafana in docker-compose.yml in the following way.

  prometheus:
    image: prom/prometheus:v2.6.1
    container_name: prometheus
    volumes:
    - ./prometheus/:/etc/prometheus/
    - prometheus_data:/prometheus
    command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/etc/prometheus/console_libraries'
    - '--web.console.templates=/etc/prometheus/consoles'
    - '--storage.tsdb.retention=200h'
    - '--web.enable-lifecycle'
    restart: unless-stopped
    ports:
    - 9090:9090
    networks:
    - basic
    labels:
    org.label-schema.group: "monitoring"

  grafana:
    image: grafana/grafana:5.4.3
    container_name: grafana
    volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana/datasources:/etc/grafana/datasources
    - ./grafana/dashboards:/etc/grafana/dashboards
    - ./grafana/setup.sh:/setup.sh
    entrypoint: /setup.sh
    environment:
    - GF_SECURITY_ADMIN_USER={ADMIN_USER}
    - GF_SECURITY_ADMIN_PASSWORD={ADMIN_PASS}
    - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped
    ports:
    - 3000:3000
    networks:
    - basic
    labels:
    org.label-schema.group: "monitoring"

When I curl 0.0.0.0:9443/metrics on my remote centos machine, I get all the list of metrics. However, when I run Prometheus with the above configuration, it throws the error Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused. This is what my prometheus.yml looks like.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'peer_metrics'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9443']

Even, when I go to endpoint http://localhost:9443/metrics in my browser, I get all the metrics. What am I doing wrong here. How come Prometheus metrics are being shown on its interface and not peer's?

Answer 1

由于目标不在 prometheus 容器内运行，因此无法通过 localhost 访问它们。您需要通过主机私有 IP 或替换localhost为docker.for.mac.localhost或来访问它们host.docker.internal。

Answer 2

您的 prometheus 容器未在主机网络上运行。它在自己的桥（由 docker-compose 创建的桥）上运行。因此，peer 的抓取配置应该指向 peer 容器的 IP。

推荐的解决方法：

• 在与结构网络相同的网络中运行 prometheus 和 grafana。在您 docker-compose for prometheus stack 中，您可以像这样引用它：

networks:
  default:
    external:
      name: <your-hyperledger-network>

（用于docker network ls查找网络名称）

然后你可以http://<peer_container_name>:9443在你的刮配置中使用

Answer 3

问题：在 Prometheus 上，您添加了用于抓取的服务，但在http://localhost:9090/targets端点状态为Down 并出现错误：

获取 http://localhost:9091/metrics: dial tcp 127.0.0.1:9091: connect: connection denied

解决方案：在prometheus.yml您需要验证

1. 抓取指向正确端点的详细信息。
2. yml 缩进是正确的。
3. usingcurl -v http://<serviceip>:<port>/metrics应该在终端中以纯文本形式提示指标。

注意：如果您指向另一个 docker 容器中的某个服务，则您的 localhost 可能不会表示为 localhost，而是表示为servicename（显示在 中的服务名称docker ps）或docker.host.internal（运行 docker 容器的内部 ip）。

对于此示例：我将使用 2 个 dockers 容器 prometheus 和“myService”。

sudo docker ps

CONTAINER ID        IMAGE                     CREATED                        PORTS                    NAMES
abc123        prom/prometheus:latest        2 hours ago               0.0.0.0:9090->9090/tcp         prometheus
def456        myService/myService:latest         2 hours ago               0.0.0.0:9091->9091/tcp         myService

然后编辑文件prometheus.yml（并重新运行 prometheus）

- job_name: myService
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  static_configs:
    - targets: // Presenting you 3 options
      - localhost:9091 // simple localhost 
      - docker.host.internal:9091 // the localhost of agent that runs the docker container
      - myService:9091 // docker container name (worked in my case)