kubeadm 连接失败，http://localhost:10248/healthz 连接被拒绝

Question

我正在尝试在三个虚拟机上设置 kubernetes（来自 centos7 的教程），\n不幸的是工作线程的加入失败。我希望有人已经遇到过这个问题（在网上发现了两次但没有答案），或者可能猜出出了什么问题。

\n\n

这是我通过 kubeadm join 得到的结果：

\n\n

[preflight] running pre-flight checks\n        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]\nyou can solve this problem with following methods:\n 1. Run \'modprobe -- \' to load missing kernel modules;\n2. Provide the missing builtin kernel ipvs support\n\nI0902 20:31:15.401693    2032 kernel_validator.go:81] Validating kernel version\nI0902 20:31:15.401768    2032 kernel_validator.go:96] Validating kernel config\n        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.1-ce. Max validated version: 17.03\n[discovery] Trying to connect to API Server "192.168.1.30:6443"\n[discovery] Created cluster-info discovery client, requesting info from "https://192.168.1.30:6443"\n[discovery] Requesting info from "https://192.168.1.30:6443" again to validate TLS against the pinned public key\n[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.1.30:6443"\n[discovery] Successfully established connection with API Server "192.168.1.30:6443"\n[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace\n[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"\n[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"\n[preflight] Activating the kubelet service\n[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...\n[kubelet-check] It seems like the kubelet isn\'t running or healthy.\n[kubelet-check] The HTTP call equal to \'curl -sSL http://localhost:10248/healthz\' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.\n[kubelet-check] It seems like the kubelet isn\'t running or healthy.\n

\n\n

虽然 kublet 正在运行：

\n\n

[root@k8s-worker1 nodesetup]# systemctl status kubelet -l\n\xe2\x97\x8f kubelet.service - kubelet: The Kubernetes Node Agent\n   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)\n  Drop-In: /etc/systemd/system/kubelet.service.d\n           \xe2\x94\x94\xe2\x94\x8010-kubeadm.conf\n   Active: active (running) since So 2018-09-02 20:31:15 CEST; 19min ago\n     Docs: https://kubernetes.io/docs/\n Main PID: 2093 (kubelet)\n    Tasks: 7\n   Memory: 12.1M\n   CGroup: /system.slice/kubelet.service\n           \xe2\x94\x94\xe2\x94\x802093 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\n\nSep 02 20:31:15 k8s-worker1 systemd[1]: Started kubelet: The Kubernetes Node Agent.\nSep 02 20:31:15 k8s-worker1 systemd[1]: Starting kubelet: The Kubernetes Node Agent...\nSep 02 20:31:15 k8s-worker1 kubelet[2093]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet\'s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\nSep 02 20:31:15 k8s-worker1 kubelet[2093]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet\'s --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\nSep 02 20:31:16 k8s-worker1 kubelet[2093]: I0902 20:31:16.440010    2093 server.go:408] Version: v1.11.2\nSep 02 20:31:16 k8s-worker1 kubelet[2093]: I0902 20:31:16.440314    2093 plugins.go:97] No cloud provider specified.\n[root@k8s-worker1 nodesetup]# \n

\n\n

据我所知，工作程序可以连接到主程序，但它尝试对某些尚未启动的本地 servlet 运行运行状况检查。有任何想法吗？

\n\n

这是我配置我的工作人员所做的事情：

\n\n

exec bash\nsetenforce 0\nsed -i --follow-symlinks \'s/SELINUX=enforcing/SELINUX=disabled/g\' /etc/sysconfig/selinux\n\n\necho "Setting Firewallrules"\nfirewall-cmd --permanent --add-port=10250/tcp\nfirewall-cmd --permanent --add-port=10255/tcp\nfirewall-cmd --permanent --add-port=30000-32767/tcp\nfirewall-cmd --permanent --add-port=6783/tcp\nfirewall-cmd --reload\n\n\necho "And enable br filtering"\nmodprobe br_netfilter\necho \'1\' > /proc/sys/net/bridge/bridge-nf-call-iptables\n\n\necho "disable swap"\nswapoff -a\necho "### You need to edit /etc/fstab and comment the swapline!! ###"\n\n\necho "Adding kubernetes repo for download"\ncat <<EOF > /etc/yum.repos.d/kubernetes.repo\n[kubernetes]\nname=Kubernetes\nbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\nenabled=1\ngpgcheck=1\nrepo_gpgcheck=1\ngpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\n        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\nEOF\n\n\necho "install the Docker-ce dependencies"\nyum install -y yum-utils device-mapper-persistent-data lvm2\n\necho "add docker-ce repository"\nyum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo\n\necho "install docker ce"\nyum install -y docker-ce\n\necho "Install kubeadm kubelet kubectl"\nyum install kubelet kubeadm kubectl -y\n\necho "start and enable kubectl"\nsystemctl restart docker && systemctl enable docker\nsystemctl restart kubelet && systemctl enable kubelet\n\necho "Now we need to ensure that both Docker-ce and Kubernetes belong to the same control group (cgroup)"\n\necho "We assume that docker is using cgroupfs ... assuming kubelet does so too"\ndocker info | grep -i cgroup\ngrep -i cgroup /var/lib/kubelet/kubeadm-flags.env\n#  old style\n# sed -i \'s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g\' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n\nsystemctl daemon-reload\nsystemctl restart kubelet\n\n# There has been an issue reported that traffic in iptable is been routed incorrectly.\n# Below settings will make sure IPTable is configured correctly.\n#\nsudo bash -c \'cat <<EOF >  /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nEOF\'\n\n# Make changes effective\nsudo sysctl --system\n

\n\n

感谢您提前提供的任何帮助。

\n\n

更新一

\n\n

来自工作人员的 Journalctl 输出：

\n\n

[root@k8s-worker1 ~]# journalctl -xeu kubelet\nSep 02 21:19:56 k8s-worker1 systemd[1]: Started kubelet: The Kubernetes Node Agent.\n-- Subject: Unit kubelet.service has finished start-up\n-- Defined-By: systemd\n-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\n-- \n-- Unit kubelet.service has finished starting up.\n-- \n-- The start-up result is done.\nSep 02 21:19:56 k8s-worker1 systemd[1]: Starting kubelet: The Kubernetes Node Agent...\n-- Subject: Unit kubelet.service has begun start-up\n-- Defined-By: systemd\n-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\n-- \n-- Unit kubelet.service has begun starting up.\nSep 02 21:19:56 k8s-worker1 kubelet[3082]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet\'s --confi\nSep 02 21:19:56 k8s-worker1 kubelet[3082]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet\'s --confi\nSep 02 21:19:56 k8s-worker1 kubelet[3082]: I0902 21:19:56.788059    3082 server.go:408] Version: v1.11.2\nSep 02 21:19:56 k8s-worker1 kubelet[3082]: I0902 21:19:56.788214    3082 plugins.go:97] No cloud provider specified.\nSep 02 21:19:56 k8s-worker1 kubelet[3082]: F0902 21:19:56.814469    3082 server.go:262] failed to run Kubelet: cannot create certificate signing request: Unauthorized\nSep 02 21:19:56 k8s-worker1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a\nSep 02 21:19:56 k8s-worker1 systemd[1]: Unit kubelet.service entered failed state.\nSep 02 21:19:56 k8s-worker1 systemd[1]: kubelet.service failed.\n

\n\n

主站上的 get pod 会导致：

\n\n

[root@k8s-master ~]# kubectl get pods --all-namespaces=true\nNAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE\nkube-system   coredns-78fcdf6894-79n2m             0/1       Pending   0          1d\nkube-system   coredns-78fcdf6894-tlngr             0/1       Pending   0          1d\nkube-system   etcd-k8s-master                      1/1       Running   3          1d\nkube-system   kube-apiserver-k8s-master            1/1       Running   0          1d\nkube-system   kube-controller-manager-k8s-master   0/1       Evicted   0          1d\nkube-system   kube-proxy-2x8cx                     1/1       Running   3          1d\nkube-system   kube-scheduler-k8s-master            1/1       Running   0          1d\n[root@k8s-master ~]# \n

\n\n

更新二 \n作为下一步，我在主端生成了一个新令牌，并在加入命令中使用了这个令牌。尽管主令牌列表显示该令牌是有效的，但工作节点坚持认为主节点不知道该令牌或者它已过期......停止！是时候从主设置开始一切重新开始了。

\n\n

这就是我所做的：

\n\n

1) 重置主虚拟机，这意味着在 virtualbox 上安装全新的 centos7 (CentOS-7-x86_64-Minimal-1804.iso)。配置 virtualbox 的网络：adapter1 作为主机系统的 NAT（以便能够安装这些内容），adapter2 作为内部网络（与 kubernetes 网络的主节点和工作节点同名）。

\n\n

2) 安装新映像后，基础接口 enp0s3 未配置为在引导时运行（因此 ifup enp03s，并在 /etc/sysconfig/network-script 中重新配置为在引导时运行）。

\n\n

3）配置内部kubernetes网络的第二个接口：

\n\n

/etc/主机：

\n\n

#!/bin/sh\necho \'192.168.1.30 k8s-master\' >> /etc/hosts\necho \'192.168.1.40 k8s-worker1\' >> /etc/hosts\necho \'192.168.1.50 k8s-worker2\' >> /etc/hosts\n

\n\n

通过“ip -color - human addr”识别了我的第二个接口，它向我展示了我的案例中的 enp0S8，所以：

\n\n

#!/bin/sh\necho "Setting up internal Interface"\ncat <<EOF > /etc/sysconfig/network-scripts/ifcfg-enp0s8\nDEVICE=enp0s8\nIPADDR=192.168.1.30\nNETMASK=255.255.255.0\nNETWORK=192.168.1.0\nBROADCAST=192.168.1.255\nONBOOT=yes\nNAME=enp0s8\nEOF\n\necho "Activate interface"\nifup enp0s8\n

\n\n

4) 主机名、交换、禁用 SELinux

\n\n

#!/bin/sh\necho "Setting hostname und deactivate SELinux"\nhostnamectl set-hostname \'k8s-master\'\nexec bash\nsetenforce 0\nsed -i --follow-symlinks \'s/SELINUX=enforcing/SELINUX=disabled/g\' /etc/sysconfig/selinux\n\necho "disable swap"\nswapoff -a\n\necho "### You need to edit /etc/fstab and comment the swapline!! ###"\n

\n\n

这里有一些注释：我重新启动，因为我看到后来的预检检查似乎解析 /etc/fstab 以查看交换不存在。另外，centos 似乎重新激活了 SElinux（我需要稍后检查这一点）作为解决方法，我在每次重新启动后再次禁用它。

\n\n

5) 建立所需的防火墙设置

\n\n

#!/bin/sh\necho "Setting Firewallrules"\nfirewall-cmd --permanent --add-port=6443/tcp\nfirewall-cmd --permanent --add-port=2379-2380/tcp\nfirewall-cmd --permanent --add-port=10250/tcp\nfirewall-cmd --permanent --add-port=10251/tcp\nfirewall-cmd --permanent --add-port=10252/tcp\nfirewall-cmd --permanent --add-port=10255/tcp\nfirewall-cmd --reload\n\necho "And enable br filtering"\nmodprobe br_netfilter\necho \'1\' > /proc/sys/net/bridge/bridge-nf-call-iptables\n

\n\n

6）添加kubernetes存储库

\n\n

#!/bin/sh\necho "Adding kubernetes repo for download"\ncat <<EOF > /etc/yum.repos.d/kubernetes.repo\n[kubernetes]\nname=Kubernetes\nbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\nenabled=1\ngpgcheck=1\nrepo_gpgcheck=1\ngpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\n        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\nEOF\n

\n\n

7) 安装所需的包并配置服务

\n\n

#!/bin/sh\n\necho "install the Docker-ce dependencies"\nyum install -y yum-utils device-mapper-persistent-data lvm2\n\necho "add docker-ce repository"\nyum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo\n\necho "install docker ce"\nyum install -y docker-ce\n\necho "Install kubeadm kubelet kubectl"\nyum install kubelet kubeadm kubectl -y\n\necho "start and enable kubectl"\nsystemctl restart docker && systemctl enable docker\nsystemctl restart kubelet && systemctl enable kubelet\n\necho "Now we need to ensure that both Docker-ce and Kubernetes belong to the same control group (cgroup)"\necho "We assume that docker is using cgroupfs ... assuming kubelet does so too"\ndocker info | grep -i cgroup\ngrep -i cgroup /var/lib/kubelet/kubeadm-flags.env\n#  old style\n# sed -i \'s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g\' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n\nsystemctl daemon-reload\nsystemctl restart kubelet\n\n# There has been an issue reported that traffic in iptable is been routed incorrectly.\n# Below settings will make sure IPTable is configured correctly.\n#\nsudo bash -c \'cat <<EOF >  /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nEOF\'\n\n# Make changes effective\nsudo sysctl --system\n

\n\n

8）初始化集群

\n\n

#!/bin/sh\necho "Init kubernetes. Check join cmd in initProtocol.txt"\nkubeadm init --apiserver-advertise-address=192.168.1.30 --pod-network-cidr=192.168.1.0/16 | tee initProtocol.txt\n

\n\n

这里要验证的是这个命令的结果：

\n\n

Init kubernetes. Check join cmd in initProtocol.txt\n[init] using Kubernetes version: v1.11.2\n[preflight] running pre-flight checks\n        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly\nI0904 21:53:15.271999    1526 kernel_validator.go:81] Validating kernel version\nI0904 21:53:15.272165    1526 kernel_validator.go:96] Validating kernel config\n        [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.1-ce. Max validated version: 17.03\n[preflight/images] Pulling images required for setting up a Kubernetes cluster\n[preflight/images] This might take a minute or two, depending on the speed of your internet connection\n[preflight/images] You can also perform this action in beforehand using \'kubeadm config images pull\'\n[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"\n[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"\n[preflight] Activating the kubelet service\n[certificates] Generated ca certificate and key.\n[certificates] Generated apiserver certificate and key.\n[certificates] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.30]\n[certificates] Generated apiserver-kubelet-client certificate and key.\n[certificates] Generated sa key and public key.\n[certificates] Generated front-proxy-ca certificate and key.\n[certificates] Generated front-proxy-client certificate and key.\n[certificates] Generated etcd/ca certificate and key.\n[certificates] Generated etcd/server certificate and key.\n[certificates] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [127.0.0.1 ::1]\n[certificates] Generated etcd/peer certificate and key.\n[certificates] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.1.30 127.0.0.1 ::1]\n[certificates] Generated etcd/healthcheck-client certificate and key.\n[certificates] Generated apiserver-etcd-client certificate and key.\n[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"\n[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"\n[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"\n[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"\n[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"\n[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"\n[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"\n[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"\n[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"\n[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" \n[init] this might take a minute or longer if the control plane images have to be pulled\n[apiclient] All control plane components are healthy after 43.504792 seconds\n[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace\n[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster\n[markmaster] Marking the node k8s-master as master by adding the label "node-role.kubernetes.io/master=\'\'"\n[markmaster] Marking the node k8s-master as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]\n[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master" as an annotation\n[bootstraptoken] using token: n4yt3r.3c8tuj11nwszts2d\n[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials\n[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token\n[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster\n[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace\n[addons] Applied essential addon: CoreDNS\n[addons] Applied essential addon: kube-proxy\n\nYour Kubernetes master has initialized successfully!\n\nTo start using your cluster, you need to run the following as a regular user:\n\n  mkdir -p $HOME/.kube\n  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n  sudo chown $(id -u):$(id -g) $HOME/.kube/config\n\nYou should now deploy a pod network to the cluster.\nRun "kubectl apply -f [podnetwork].yaml" with one of the options listed at:\n  https://kubernetes.io/docs/concepts/cluster-administration/addons/\n\nYou can now join any number of machines by running the following on each node\nas root:\n\n  kubeadm join 192.168.1.30:6443 --token n4yt3r.3c8tuj11nwszts2d --discovery-token-ca-cert-hash sha256:466e7972a4b6997651ac1197fdde68d325a7bc41f2fccc2b1efc17515af61172\n

\n\n

备注：到目前为止对我来说看起来不错，尽管我有点担心最新的 docker-ce 版本可能会在这里带来麻烦......

\n\n

9）部署Pod网络

\n\n

#!/bin/bash\n\necho "Configure demo cluster usage as root"\nmkdir -p $HOME/.kube\ncp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nchown $(id -u):$(id -g) $HOME/.kube/config\n\n# Deploy-Network using flanel\n# Taken from first matching two tutorials on the web\n# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml\n# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml\n\n# taken from https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network\nkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml\nkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml\n\necho "Try to run kubectl get pods --all-namespaces"\necho "After joining nodes: try to run kubectl get nodes to verify the status"\n

\n\n

这是该命令的输出：

\n\n

Configure demo cluster usage as root\nclusterrole.rbac.authorization.k8s.io/flannel created\nclusterrolebinding.

    

Answer 1

根据所附日志，您的似乎kubeadm token已过期。kubelet

\n\n

\n

Sep 02 21:19:56 k8s-worker1 kubelet[3082]: F0902 21:19:56.814469
\n 3082 server.go:262] 无法运行 Kubelet：无法创建证书\n 签名请求：未经授权

\n

\n\n

该令牌的 TTL 在命令发布后保留 24 小时kubeadm init，请查看此链接以获取更多信息。

\n\n

主节点\xe2\x80\x99s系统运行时组件看起来不健康，不确定集群是否可以正常运行。尽管CoreDNS服务处于挂起状态，但请查看kubeadm故障排除文档以检查集群上是否安装了任何Pod 网络提供程序。

\n\n

我建议重建集群，以便kubeadm token从头开始刷新和引导集群系统模块。

\n