Kubernetes：证书过期

Question

我们的Kubernetes 1.6集群具有在2017年4月13日构建集群时生成的证书。

2017年12月13日，我们的集群升级到了版本1.8，并生成了新证书[显然，证书集不完整]。

从2018年4月13日起，我们开始在api服务器的Kubernetes仪表板中看到以下消息：

[authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]

试图指向客户端证书和客户端密钥内/etc/kubernetes/kubelet.conf，在上12月13日[生成的证书apiserver-kubelet-client.crt和apiserver-kubelet-client.crt]，但继续看到上述错误。

试图指向客户端证书和客户端密钥内/etc/kubernetes/kubelet.conf，在不同的12月13日[生成的证书apiserver.crt和apiserver.crt（I真的不理解这些2台证书/密钥之间的差），但是继续看到上述错误。

试图指向客户端证书和客户端密钥中/etc/kubernetes/kubelet.conf的不存在的文件，并没有KUBE的*服务将启动，有/var/log/syslog抱怨这一点：

Apr 17 17:50:08 kuber01 kubelet[2422]: W0417 17:50:08.181326 2422 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /tmp/this/cert/does/not/exist.crt for system:node:node01 due to open /tmp/this/cert/does/not/exist.crt: no such file or directory, unable to read client-key /tmp/this/key/does/not/exist.key for system:node:node01 due to open /tmp/this/key/does/not/exist.key: no such file or directory]

关于如何克服此错误，甚至更详细地解决问题的任何建议？kubeadm alpha phase certs apiserver根据https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs中的说明，正在考虑为api-server（）重新生成证书...但不确定是否我会造成更大的伤害。

对Kubernetes来说这是一个相对较新的东西，设置此设置的绅士不可以咨询...非常感谢任何帮助。谢谢。

Answer 1

对于将来偶然发现此问题的任何人（正在运行较新版本的 kubernetes >1.17），这可能是更新证书的最简单方法。

以下内容更新所有证书，重新启动 kubelet，备份旧的管理配置并应用新的管理配置：

kubeadm certs renew all
systemctl restart kubelet
cp /root/.kube/config /root/.kube/.old-$(date --iso)-config
cp /etc/kubernetes/admin.conf /root/.kube/config

Answer 2

该主题也在以下内容中讨论：

• https://github.com/kubernetes/kubeadm/issues/581
  • 1.15 kubeadm upgrade 之后会自动为你更新证书！
  • 1.15 还添加了一个命令来检查 kubeadm 中的证书过期时间
• 过期后更新kubernetes pki

Kubernetes v1.15提供了“使用 kubeadm 进行证书管理”的文档：

• https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
• 检查证书有效期：

kubeadm alpha certs check-expiration

• 自动证书更新：
  • kubeadm 在控制平面升级期间更新所有证书。
• 手动证书更新：
  • 您可以随时使用该kubeadm alpha certs renew命令手动更新证书。
  • 此命令使用存储在 /etc/kubernetes/pki 中的 CA（或前端代理 CA）证书和密钥执行更新。

对于Kubernetes v1.14，我发现这个过程最有帮助：

• /sf/answers/3943431271/
• 备份并重新生成所有证书：

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>

• 备份并重新生成所有 kubeconfig 文件：

$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot

• 复制新的 admin.conf：

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

Answer 3

Kubernetes集群中的每个节点都包含一个用于运行kubelet ... 的配置文件/etc/kubernetes/kubelet.conf，该文件由kubeadm自动生成。在此自动生成过程中，kubeadm用于/etc/kubernetes/ca.key创建特定/etc/kubernetes/kubelet.conf于节点的文件，该文件中有两个非常重要的部分：client-certificate-data和client-key-data。我最初的思考过程使我相信，我需要找到相应的证书文件和密钥文件，续订这些文件，将它们都转换为base64，并在kubelet.conf整个群集的文件中使用这些值……这是不正确的。

相反，修复是利用kubeadm再生kubectl.conf所有节点上，以及admin.conf，controller-manager.conf以及scheduler.conf集群的主节点上。您将需要/etc/kubernetes/pki/ca.key在每个节点上，以便配置文件包括client-certificate-data和client-key-data的有效数据。

专家提示：使用该--apiserver-advertise-address参数可确保新的配置文件包含托管kube-apiserver服务的节点的正确IP地址。

Answer 4

对于版本 1.21.5，这是我的解决方案：

步骤1：

ssh 到主节点，然后在步骤 2 中检查证书。

第2步：

运行这个命令：kubeadm certs check-expiration

root@kube-master-1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                         EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                          Oct 21, 2022 16:05 UTC   <invalid>                               no      
apiserver                           Oct 21, 2022 16:05 UTC   <invalid>       ca                      no      
!MISSING! apiserver-etcd-client                                                                      
apiserver-kubelet-client            Oct 21, 2022 16:05 UTC   <invalid>       ca                      no      
controller-manager.conf             Oct 21, 2022 16:05 UTC   <invalid>                               no      
!MISSING! etcd-healthcheck-client                                                                    
!MISSING! etcd-peer                                                                                  
!MISSING! etcd-server                                                                                
front-proxy-client                  Oct 21, 2022 16:05 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf                      Oct 21, 2022 16:05 UTC   <invalid>                               no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 19, 2031 16:05 UTC   8y              no      
!MISSING! etcd-ca                                                
front-proxy-ca          Oct 19, 2031 16:05 UTC   8y              no      

并看到昨天所有的都过期了。

步骤3：

所有现有证书的备份：

root@kube-master-1:~# cp -R /etc/kubernetes/ssl /etc/kubernetes/ssl.backup
root@kube-master-1:~# cp /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.backup
root@kube-master-1:~# cp /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.backup
root@kube-master-1:~# cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.backup
root@kube-master-1:~# cp /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.backup

步骤4：

要全部更新，请运行以下命令： kubeadm certs renew all

root@kube-master-1:~# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1023 15:15:16.234334 2175921 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

步骤 5：步骤 4 的最后一行告诉我们重要的注意事项：

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates

完成本次运行：

kubectl -n kube-system delete pod -l 'component=kube-apiserver'
kubectl -n kube-system delete pod -l 'component=kube-controller-manager'
kubectl -n kube-system delete pod -l 'component=kube-scheduler'
kubectl -n kube-system delete pod -l 'component=etcd'

步骤6：然后重新启动主节点。

第7步：查看结果：

root@kube-master-1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1023 15:15:23.141925 2177263 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 23, 2023 07:15 UTC   364d                                    no      
apiserver                  Oct 23, 2023 07:15 UTC   364d            ca                      no      
apiserver-kubelet-client   Oct 23, 2023 07:15 UTC   364d            ca                      no      
controller-manager.conf    Oct 23, 2023 07:15 UTC   364d                                    no      
front-proxy-client         Oct 23, 2023 07:15 UTC   364d            front-proxy-ca          no      
scheduler.conf             Oct 23, 2023 07:15 UTC   364d                                    no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 19, 2031 16:05 UTC   8y              no      
front-proxy-ca          Oct 19, 2031 16:05 UTC   8y              no     

全部续订至 2023 年

Answer 5

在 k8s 1.7 上，我遇到了类似的问题（/var/log/kube-apiserver.log 中包含 x509 过期错误）并且找不到任何过期的证书。我们决定只重启主节点上的 apiserver docker。它解决了问题。

$ sudo docker ps -a | grep apiserver
af99f816c7ec        gcr.io/google_containers/kube-apiserver@sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85               "/bin/sh -c '/usr/loc"   15 months ago       Up 19 hours                                     k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_0
40f3a18050c3        gcr.io/google_containers/pause-amd64:3.0                                                                                      "/pause"                 15 months ago       Up 15 months                                    k8s_POD_kube-apiserver-ip-xxxc_0
$ sudo docker restart af99f816c7ec
af99f816c7ec
$ 

Answer 6

我认为您需要重新生成apiserver证书，/etc/kubernetes/pki/apiserver.crt您可以像这样查看当前的到期日期。

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Dec 20 14:32:00 2017 GMT
            Not After : Dec 20 14:32:00 2018 GMT

这是我用来在v1.11.5群集上重新生成证书的步骤。从这里编译的步骤https://github.com/kubernetes/kubeadm/issues/581

检查所有证书的过期日期：

find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'

在Maser节点上续订证书。

*）续证

mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old
mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old
mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old


kubeadm alpha phase certs apiserver  --config /root/kubeadm-kubetest.yaml
kubeadm alpha phase certs apiserver-kubelet-client
kubeadm alpha phase certs front-proxy-client

mv /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt.old
mv /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key.old
kubeadm alpha phase certs  apiserver-etcd-client


mv /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.crt.old
mv /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.key.old
kubeadm alpha phase certs  etcd-server --config /root/kubeadm-kubetest.yaml

mv /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt.old
mv /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.key.old
kubeadm alpha phase certs  etcd-healthcheck-client --config /root/kubeadm-kubetest.yaml


mv /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/peert.crt.old
mv /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.key.old
kubeadm alpha phase certs  etcd-peer --config /root/kubeadm-kubetest.yaml

*)  Backup old configuration files
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old

kubeadm alpha phase kubeconfig all  --config /root/kubeadm-kubetest.yaml

mv $HOME/.kube/config .$HOMEkube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 777 $HOME/.kube/config
export KUBECONFIG=.kube/config

重新启动节点，并检查etcd，kubeapi和kubelet的日志。

注意： 请记住要更新CI / CD作业kubeconfig文件。如果您使用的是helm命令，请进行测试。