创建 pod 沙箱失败：rpc 错误：代码 = 未知描述 = NetworkPlugin cni 无法设置 pod 网络

Question

在 k8s(v1.10) 集群上创建 Redis POD，并在“ContainerCreating”中创建 POD

Type     Reason                  Age                   From                Message
  ----     ------                  ----                  ----                -------
  Normal   Scheduled               30m                   default-scheduler   Successfully assigned redis to k8snode02
  Normal   SuccessfulMountVolume   30m                   kubelet, k8snode02  MountVolume.SetUp succeeded for volume "default-token-f8tcg"
  Warning  FailedCreatePodSandBox  5m (x1202 over 30m)   kubelet, k8snode02  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "redis_default" network: failed to find plugin "loopback" in path [/opt/loopback/bin /opt/cni/bin]
  Normal   SandboxChanged          47s (x1459 over 30m)  kubelet, k8snode02  Pod sandbox changed, it will be killed and re-created.

Answer 1

当我使用 calico 作为 CNI 时，我遇到了类似的问题。

\n

容器仍处于创建状态，我检查了主节点上的 /etc/cni/net.d 和 /opt/cni/bin 都存在，但不确定工作节点上是否也需要这样做。

\n

root@KubernetesMaster:/opt/cni/bin# kubectl get pods\nNAME                   READY   STATUS              RESTARTS   AGE\nnginx-5c7588df-5zds6   0/1     ContainerCreating   0          21m\nroot@KubernetesMaster:/opt/cni/bin# kubectl get nodes\nNAME               STATUS   ROLES    AGE   VERSION\nkubernetesmaster   Ready    master   26m   v1.13.4\nkubernetesslave1   Ready    <none>   22m   v1.13.4\nroot@KubernetesMaster:/opt/cni/bin#\n\nkubectl describe pods\nName:               nginx-5c7588df-5zds6\nNamespace:          default\nPriority:           0\nPriorityClassName:  <none>\nNode:               kubernetesslave1/10.0.3.80\nStart Time:         Sun, 17 Mar 2019 05:13:30 +0000\nLabels:             app=nginx\n                    pod-template-hash=5c7588df\nAnnotations:        <none>\nStatus:             Pending\nIP:\nControlled By:      ReplicaSet/nginx-5c7588df\nContainers:\n  nginx:\n    Container ID:\n    Image:          nginx\n    Image ID:\n    Port:           <none>\n    Host Port:      <none>\n    State:          Waiting\n      Reason:       ContainerCreating\n    Ready:          False\n    Restart Count:  0\n    Environment:    <none>\n    Mounts:\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qtfbs (ro)\nConditions:\n  Type              Status\n  Initialized       True\n  Ready             False\n  ContainersReady   False\n  PodScheduled      True\nVolumes:\n  default-token-qtfbs:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  default-token-qtfbs\n    Optional:    false\nQoS Class:       BestEffort\nNode-Selectors:  <none>\nTolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n                 node.kubernetes.io/unreachable:NoExecute for 300s\nEvents:\n  Type     Reason                  Age                    From                       Message\n  ----     ------                  ----                   ----                       -------\n  Normal   Scheduled               18m                    default-scheduler          Successfully assigned default/nginx-5c7588df-5zds6 to kubernetesslave1\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "123d527490944d80f44b1976b82dbae5dc56934aabf59cf89f151736d7ea8adc" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8cc5e62ebaab7075782c2248e00d795191c45906cc9579464a00c09a2bc88b71" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "30ffdeace558b0935d1ed3c2e59480e2dd98e983b747dacae707d1baa222353f" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "630e85451b6ce2452839c4cfd1ecb9acce4120515702edf29421c123cf231213" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "820b919b7edcfc3081711bb78b79d33e5be3f7dafcbad29fe46b6d7aa22227aa" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abbfb5d2756f12802072039dec20ba52f546ae755aaa642a9a75c86577be589f" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dfeb46ffda4d0f8a434f3f3af04328fcc4b6c7cafaa62626e41b705b06d98cc4" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9ae3f47bb0282a56e607779d3267127ee8b0ae1d7f416f5a184682119203b1c8" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Warning  FailedCreatePodSandBox  18m                    kubelet, kubernetesslave1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "546d07f1864728b2e2675c066775f94d658e221ada5fb4ed6bf6689ec7b8de23" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\n  Normal   SandboxChanged          18m (x12 over 18m)     kubelet, kubernetesslave1  Pod sandbox changed, it will be killed and re-created.\n  Warning  FailedCreatePodSandBox  3m39s (x829 over 18m)  kubelet, kubernetesslave1  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f586be437843537a3082f37ad139c88d0eacfbe99ddf00621efd4dc049a268cc" network for pod "nginx-5c7588df-5zds6": NetworkPlugin cni failed to set up pod "nginx-5c7588df-5zds6_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\nroot@KubernetesMaster:/etc/cni/net.d#\n

\n

在工作节点 NGINX 试图启动但退出，我不确定这里发生了什么 - 我是 kubernetes 的新手并且无法解决这个问题 -

\n

root@kubernetesslave1:/home/ubuntu# docker ps\nCONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES\n5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   3 minutes ago       Up 3 minutes                            k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\nb1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\nceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\nroot@kubernetesslave1:/home/ubuntu# docker ps\nCONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES\n5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   3 minutes ago       Up 3 minutes                            k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\nb1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\nceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\nroot@kubernetesslave1:/home/ubuntu# docker ps\nCONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES\n5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   3 minutes ago       Up 3 minutes                            k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\nb1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\nceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n\n    root@kubernetesslave1:/home/ubuntu# docker ps\n    CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                  PORTS               NAMES\n    94b2994401d0        k8s.gcr.io/pause:3.1   "/pause"                 1 second ago        Up Less than a second                       k8s_POD_nginx-5c7588df-5zds6_default_677a722b-4873-11e9-a33a-06516e7d78c4_534\n    5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   4 minutes ago       Up 4 minutes                                k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    b1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                                k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\n    ceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                                k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    root@kubernetesslave1:/home/ubuntu# docker ps\n    CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES\n    5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   4 minutes ago       Up 4 minutes                            k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    b1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\n    ceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    root@kubernetesslave1:/home/ubuntu# docker ps\n    CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                  PORTS               NAMES\n    f72500cae2b7        k8s.gcr.io/pause:3.1   "/pause"                 1 second ago        Up Less than a second                       k8s_POD_nginx-5c7588df-5zds6_default_677a722b-4873-11e9-a33a-06516e7d78c4_585\n    5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   4 minutes ago       Up 4 minutes                                k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    b1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                                k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\n    ceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 4 minutes ago       Up 4 minutes                                k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    root@kubernetesslave1:/home/ubuntu# docker ps\n    CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES\n    5ad5500e8270        fadcc5d2b066           "/usr/local/bin/kube\xe2\x80\xa6"   5 minutes ago       Up 5 minutes                            k8s_kube-proxy_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n    b1c9929ebe9e        k8s.gcr.io/pause:3.1   "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_calico-node-749qx_kube-system_4e2d8c9c-4873-11e9-a33a-06516e7d78c4_1\n    ceb78340b563        k8s.gcr.io/pause:3.1   "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_kube-proxy-f24gd_kube-system_4e2d313a-4873-11e9-a33a-06516e7d78c4_1\n

\n

我还检查了工作节点上的 /etc/cni/net.d 和 /opt/cni/bin ，它就在那里 -

\n

root@kubernetesslave1:/home/ubuntu# cd /etc/cni\nroot@kubernetesslave1:/etc/cni# ls -ltr\ntotal 4\ndrwxr-xr-x 2 root root 4096 Mar 17 05:19 net.d\nroot@kubernetesslave1:/etc/cni# cd /opt/cni\nroot@kubernetesslave1:/opt/cni# ls -ltr\ntotal 4\ndrwxr-xr-x 2 root root 4096 Mar 17 05:19 bin\nroot@kubernetesslave1:/opt/cni# cd bin\nroot@kubernetesslave1:/opt/cni/bin# ls -ltr\ntotal 107440\n-rwxr-xr-x 1 root root  3890407 Aug 17  2017 bridge\n-rwxr-xr-x 1 root root  3475802 Aug 17  2017 ipvlan\n-rwxr-xr-x 1 root root  3520724 Aug 17  2017 macvlan\n-rwxr-xr-x 1 root root  3877986 Aug 17  2017 ptp\n-rwxr-xr-x 1 root root  3475750 Aug 17  2017 vlan\n-rwxr-xr-x 1 root root  9921982 Aug 17  2017 dhcp\n-rwxr-xr-x 1 root root  2605279 Aug 17  2017 sample\n-rwxr-xr-x 1 root root 32351072 Mar 17 05:19 calico\n-rwxr-xr-x 1 root root 31490656 Mar 17 05:19 calico-ipam\n-rwxr-xr-x 1 root root  2856252 Mar 17 05:19 flannel\n-rwxr-xr-x 1 root root  3084347 Mar 17 05:19 loopback\n-rwxr-xr-x 1 root root  3036768 Mar 17 05:19 host-local\n-rwxr-xr-x 1 root root  3550877 Mar 17 05:19 portmap\n-rwxr-xr-x 1 root root  2850029 Mar 17 05:19 tuning\nroot@kubernetesslave1:/opt/cni/bin#\n

\n

Answer 2

确保它/etc/cni/net.d和它的/opt/cni/bin朋友都存在，并且在所有节点上正确填充了CNI配置文件和二进制文件。对于法兰绒，可以使用法兰绒 cni repo