无法创建 pod 沙箱：rpc 错误：代码 = 未知描述 = 无法设置沙箱容器

Question

无法创建 pod 沙箱：rpc 错误：代码 = 未知描述 = 无法设置沙箱容器

我们正在尝试创建 POD，但 Pod 的状态在 ContainerCreating 上很长时间受到打击。

这是我们运行命令后得到的输出：kubectl describe pod

Name:           demo-6c59fb8f77-9x6sr
Namespace:      default
Priority:       0
Node:           k8-slave2/10.0.0.5
Start Time:     Wed, 23 Dec 2020 10:16:23 +0000
Labels:         app=demo
                pod-template-hash=6c59fb8f77
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/demo-6c59fb8f77
Containers:
  private-docker-registry:
    Container ID:
    Image:          private-docker-registry:5000/mahin/mof-docker-demo:v1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p94zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p94zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p94zw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  

  Events:
      Type     Reason                  Age                  From               Message
      ----     ------                  ----                 ----               -------
      Normal   Scheduled               10m                  default-scheduler  Successfully assigned default/demo-6c59fb8f77-9x6sr to k8-slave2
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "95e72bfc6f6c13de7f5c96eb76b012c2e6639ca03f4c2f270b23ed1a09b90413" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "566370012e4a1d32af2ef9035ff64d743cd81f36f25d2724e7b033e393b8247e" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d499e40f572cfc29ecfb44f8376493df56a44213b1c1e9333b65499a0c288cd" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "53241e64de1e4470712b4061e2c82f44916d654bc532f8f1d12e5d5d4e136914" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd168faab4546f988dc38fc56df2f71cf80c922e86d3f869be15a43f08328f99" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e578afe329abb0cba64802dfa480e00f2bbbb8c80be537791c24a31c853eb62f" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a3cb32dba55907ca907fc4f38f7ca05ef6db10a6af2dd1fa3c4db166e4ab9ffe" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e4368ba8ec460b3c94de24ab0a04b6c799eb28df885cbbacfc3bb3ffa8c1e67" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m (x4 over 10m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c4aaa8f8cd2dc1eff788baf04774c4ecc845568d00ed1b386df311ec224eb6f3" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Normal   SandboxChanged          56s (x551 over 10m)  kubelet            Pod sandbox changed, it will be killed and re-created.



azureuser@k8-master:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                demo-6c59fb8f77-2jq6k                        0/1     ContainerCreating   0          5m23s
kube-system            coredns-f9fd979d6-q8s9b                      1/1     Running             2          27h
kube-system            coredns-f9fd979d6-qnm4j                      1/1     Running             2          27h
kube-system            etcd-k8-master                               1/1     Running             2          27h
kube-system            kube-apiserver-k8-master                     1/1     Running             3          27h
kube-system            kube-controller-manager-k8-master            1/1     Running             3          27h
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
kube-system            kube-proxy-4mb47                             1/1     Running             2          27h
kube-system            kube-proxy-54m9b                             1/1     Running             2          27h
kube-system            kube-proxy-wdxfz                             1/1     Running             1          27h
kube-system            kube-scheduler-k8-master                     1/1     Running             3          27h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-zmlvs   0/1     ContainerCreating   0          27h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-cnsvn         0/1     ContainerCreating   0          6h3m

Run Code Online (Sandbox Code Playgroud)

为了修复 flannel crashloopbackoff，我们执行了 Kubeadm reset，一段时间后这个问题再次出现。

目前我们正在使用一个主节点和两个工作节点。

我的集群详细信息如下：azureuser@k8-master:~$ kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+ OMITTED server: https://52.150.11.168:6443 name: kubernetes contexts: - 上下文：集群：kubernetes 用户：kubernetes-admin 名称：kubernetes-admin@kubernetes 当前上下文：kubernetes-admin@kubernetes kind：配置首选项：{} 用户：-名称：kubernetes-admin 用户：client-certificate-data：已编辑的客户端密钥数据：已编辑

码头工人版本：

azureuser@k8-master:~$ sudo docker version
[sudo] password for azureuser: 
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Wed Oct 14 19:00:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Oct 14 16:52:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

Run Code Online (Sandbox Code Playgroud)

kubeadm 版本：

azureuser@k8-master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:15:05Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Run Code Online (Sandbox Code Playgroud)

每当我尝试安排 pod 创建时，法兰绒就会崩溃。

Answer 1

Pjo*_*erS 6

背景

我认为您的问题是由您的2 Flannel CNIPodCrashLoopBackOff状态引起的。

你的错误

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory

Run Code Online (Sandbox Code Playgroud)

指出由于缺少/run/flannel/subnet.env文件而无法创建 pod。在Flannel Github文档中你可以找到：

Flannel 在每台主机上运行一个名为 flanneld 的小型单一二进制代理，并负责从更大的预配置地址空间中为每台主机分配子网租约。

这意味着，为了正常工作，Flannelpod 应该在每个节点上运行，因为它包含子网信息。从您的输出中我可以看到3 个 pod 中只有 1 个正常工作。 Flannel

NAMESPACE NAME READY STATUS RESTARTS AGE ... kube-system kube-flannel-ds-kqz4t 0/1 CrashLoopBackOff 92 27h kube-system kube-flannel-ds-szqzn 1/1 Running 3 27h kube-system kube-flannel-ds-v9q47 0/1 CrashLoopBackOff 142 27h
Run Code Online (Sandbox Code Playgroud)
如果提到的 pod 被安排在 flannel pod 不工作的节点上，则由于CNI network issues. 除了您的demopod 之外，kubernetes-dashboardpod 也存在同样的状态问题ContainerCreating。

结论

由于 Kubernetes 遇到一些与 flannel 配置文件 ( ) 相关的网络问题，您的demopod 无法调度...network: open /run/flannel/subnet.env: no such file or directory。

您的法兰绒吊舱重新启动次数非常高27 hours。您必须确定原因并解决它。这可能是由于缺乏资源、基础设施的网络问题或许多其他原因。一旦所有flannelPod 都能正常工作，您就不会遇到此错误。

解决方案

您必须使 flannel pod 在每个节点上正常工作。

其他故障排除详细信息

如需详细调查，请提供

$ kubectl describe kube-flannel-ds-kqz4t -n kube-system $ kubectl describe kube-flannel-ds-v9q47 -n kube-system
Run Code Online (Sandbox Code Playgroud)
日志详细信息也会有帮助

$ kubectl logs kube-flannel-ds-kqz4t -n kube-system $ kubectl logs kube-flannel-ds-v9q47 -n kube-system
Run Code Online (Sandbox Code Playgroud)
请替换kubectl get pods --all-namespaces为kubectl get pods -o wide -A和的输出kubectl get nodes -o wide。

如果您提供这些信息，应该可以确定flannelPod 问题的根本原因，我将使用确切的解决方案编辑此答案。

归档时间：	5 年，6 月前
查看次数：	11756 次
最近记录：	5 年，6 月前