0/1 个节点可用：启动 promethus 导出程序时，1 个节点没有用于请求的 pod 端口的可用端口

Question

在 kubernetes 集群中使用 helm 安装 promethus 后，pod 显示如下错误：

\n

0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.\n

\n

这是部署 yaml：

\n

apiVersion: v1\nkind: Pod\nmetadata:\n  name: kube-prometheus-1660560589-node-exporter-n7rzg\n  generateName: kube-prometheus-1660560589-node-exporter-\n  namespace: reddwarf-monitor\n  uid: 73986565-ccd8-421c-bcbb-33879437c4f3\n  resourceVersion: '71494023'\n  creationTimestamp: '2022-08-15T10:51:07Z'\n  labels:\n    app.kubernetes.io/instance: kube-prometheus-1660560589\n    app.kubernetes.io/managed-by: Helm\n    app.kubernetes.io/name: node-exporter\n    controller-revision-hash: 65c69f9b58\n    helm.sh/chart: node-exporter-3.0.8\n    pod-template-generation: '1'\n  ownerReferences:\n    - apiVersion: apps/v1\n      kind: DaemonSet\n      name: kube-prometheus-1660560589-node-exporter\n      uid: 921f98b9-ccc9-4e84-b092-585865bca024\n      controller: true\n      blockOwnerDeletion: true\nstatus:\n  phase: Pending\n  conditions:\n    - type: PodScheduled\n      status: 'False'\n      lastProbeTime: null\n      lastTransitionTime: '2022-08-15T10:51:07Z'\n      reason: Unschedulable\n      message: >-\n        0/1 nodes are available: 1 node(s) didn't have free ports for the\n        requested pod ports.\n  qosClass: BestEffort\nspec:\n  volumes:\n    - name: proc\n      hostPath:\n        path: /proc\n        type: ''\n    - name: sys\n      hostPath:\n        path: /sys\n        type: ''\n    - name: kube-api-access-9fj8v\n      projected:\n        sources:\n          - serviceAccountToken:\n              expirationSeconds: 3607\n              path: token\n          - configMap:\n              name: kube-root-ca.crt\n              items:\n                - key: ca.crt\n                  path: ca.crt\n          - downwardAPI:\n              items:\n                - path: namespace\n                  fieldRef:\n                    apiVersion: v1\n                    fieldPath: metadata.namespace\n        defaultMode: 420\n  containers:\n    - name: node-exporter\n      image: docker.io/bitnami/node-exporter:1.3.1-debian-11-r23\n      args:\n        - '--path.procfs=/host/proc'\n        - '--path.sysfs=/host/sys'\n        - '--web.listen-address=0.0.0.0:9100'\n        - >-\n          --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\n        - >-\n          --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\n      ports:\n        - name: metrics\n          hostPort: 9100\n          containerPort: 9100\n          protocol: TCP\n      resources: {}\n      volumeMounts:\n        - name: proc\n          readOnly: true\n          mountPath: /host/proc\n        - name: sys\n          readOnly: true\n          mountPath: /host/sys\n        - name: kube-api-access-9fj8v\n          readOnly: true\n          mountPath: /var/run/secrets/kubernetes.io/serviceaccount\n      livenessProbe:\n        httpGet:\n          path: /\n          port: metrics\n          scheme: HTTP\n        initialDelaySeconds: 120\n        timeoutSeconds: 5\n        periodSeconds: 10\n        successThreshold: 1\n        failureThreshold: 6\n      readinessProbe:\n        httpGet:\n          path: /\n          port: metrics\n          scheme: HTTP\n        initialDelaySeconds: 30\n        timeoutSeconds: 5\n        periodSeconds: 10\n        successThreshold: 1\n        failureThreshold: 6\n      terminationMessagePath: /dev/termination-log\n      terminationMessagePolicy: File\n      imagePullPolicy: IfNotPresent\n      securityContext:\n        runAsUser: 1001\n        runAsNonRoot: true\n  restartPolicy: Always\n  terminationGracePeriodSeconds: 30\n  dnsPolicy: ClusterFirst\n  serviceAccountName: kube-prometheus-1660560589-node-exporter\n  serviceAccount: kube-prometheus-1660560589-node-exporter\n  hostNetwork: true\n  hostPID: true\n  securityContext:\n    fsGroup: 1001\n  affinity:\n    nodeAffinity:\n      requiredDuringSchedulingIgnoredDuringExecution:\n        nodeSelectorTerms:\n          - matchFields:\n              - key: metadata.name\n                operator: In\n                values:\n                  - k8smasterone\n    podAntiAffinity:\n      preferredDuringSchedulingIgnoredDuringExecution:\n        - weight: 1\n          podAffinityTerm:\n            labelSelector:\n              matchLabels:\n                app.kubernetes.io/instance: kube-prometheus-1660560589\n                app.kubernetes.io/name: node-exporter\n            namespaces:\n              - reddwarf-monitor\n            topologyKey: kubernetes.io/hostname\n  schedulerName: default-scheduler\n  tolerations:\n    - key: node.kubernetes.io/not-ready\n      operator: Exists\n      effect: NoExecute\n    - key: node.kubernetes.io/unreachable\n      operator: Exists\n      effect: NoExecute\n    - key: node.kubernetes.io/disk-pressure\n      operator: Exists\n      effect: NoSchedule\n    - key: node.kubernetes.io/memory-pressure\n      operator: Exists\n      effect: NoSchedule\n    - key: node.kubernetes.io/pid-pressure\n      operator: Exists\n      effect: NoSchedule\n    - key: node.kubernetes.io/unschedulable\n      operator: Exists\n      effect: NoSchedule\n    - key: node.kubernetes.io/network-unavailable\n      operator: Exists\n      effect: NoSchedule\n  priority: 0\n  enableServiceLinks: true\n  preemptionPolicy: PreemptLowerPriority\n

\n

我检查了主机，发现9100端口是空闲的，为什么还是提示这个pod没有端口？我应该怎么做才能避免这个问题？这是主机端口 9100 检查命令：

\n

[root@k8smasterone grafana]# lsof -i:9100\n[root@k8smasterone grafana]#\n

\n

这是 pod 描述信息：

\n

\xe2\x9e\x9c  ~ kubectl describe pod kube-prometheus-1660560589-node-exporter-n7rzg -n reddwarf-monitor\nName:           kube-prometheus-1660560589-node-exporter-n7rzg\nNamespace:      reddwarf-monitor\nPriority:       0\nNode:           <none>\nLabels:         app.kubernetes.io/instance=kube-prometheus-1660560589\n                app.kubernetes.io/managed-by=Helm\n                app.kubernetes.io/name=node-exporter\n                controller-revision-hash=65c69f9b58\n                helm.sh/chart=node-exporter-3.0.8\n                pod-template-generation=1\nAnnotations:    <none>\nStatus:         Pending\nIP:\nIPs:            <none>\nControlled By:  DaemonSet/kube-prometheus-1660560589-node-exporter\nContainers:\n  node-exporter:\n    Image:      docker.io/bitnami/node-exporter:1.3.1-debian-11-r23\n    Port:       9100/TCP\n    Host Port:  9100/TCP\n    Args:\n      --path.procfs=/host/proc\n      --path.sysfs=/host/sys\n      --web.listen-address=0.0.0.0:9100\n      --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\n      --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\n    Liveness:     http-get http://:metrics/ delay=120s timeout=5s period=10s #success=1 #failure=6\n    Readiness:    http-get http://:metrics/ delay=30s timeout=5s period=10s #success=1 #failure=6\n    Environment:  <none>\n    Mounts:\n      /host/proc from proc (ro)\n      /host/sys from sys (ro)\n      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9fj8v (ro)\nConditions:\n  Type           Status\n  PodScheduled   False\nVolumes:\n  proc:\n    Type:          HostPath (bare host directory volume)\n    Path:          /proc\n    HostPathType:\n  sys:\n    Type:          HostPath (bare host directory volume)\n    Path:          /sys\n    HostPathType:\n  kube-api-access-9fj8v:\n    Type:                    Projected (a volume that contains injected data from multiple sources)\n    TokenExpirationSeconds:  3607\n    ConfigMapName:           kube-root-ca.crt\n    ConfigMapOptional:       <nil>\n    DownwardAPI:             true\nQoS Class:                   BestEffort\nNode-Selectors:              <none>\nTolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists\n                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists\n                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists\n                             node.kubernetes.io/not-ready:NoExecute op=Exists\n                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists\n                             node.kubernetes.io/unreachable:NoExecute op=Exists\n                             node.kubernetes.io/unschedulable:NoSchedule op=Exists\nEvents:\n  Type     Reason            Age                      From               Message\n  ----     ------            ----                     ----               -------\n  Warning  FailedScheduling  2m54s (x233 over 3h53m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.\n

\n

这是网络统计：

\n

[root@k8smasterone ~]# netstat -plant |grep 9100\n[root@k8smasterone ~]#\n

\n

我还尝试通过添加以下配置来允许 Pod 在主节点中运行：

\n

tolerations:\n  - effect: NoSchedule\n    key: node-role.kubernetes.io/master\n

\n

还是没有解决这个问题。

\n

Answer 1

当您使用 来配置 pod 时hostNetwork: true，该 pod 中运行的容器可以直接看到启动该 pod 的主机的网络接口。

容器端口将暴露在外网：，hostPort是用户在配置中请求的端口hostPort。

要绕过您的问题，您有两种选择：

• 环境hostNetwork: false
• 选择不同的hostPort（最好在 49152 到 65535 范围内）