kubectl port-forward: "pod does not exist" at the first time running?

Question

Follow this guide, I'm trying to start minikube and forward port at the boot time.

My script:

#!/bin/bash

set -eux

export PATH=/usr/local/bin:$PATH

minikube status || minikube start
minikube ssh 'grep docker.for.mac.localhost /etc/hosts || echo -e "127.0.0.1\tdocker.for.mac.localhost" | sudo tee -a /etc/hosts'
minikube ssh 'test -f wait-for-it.sh || curl -O https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh'
minikube ssh 'chmod +x wait-for-it.sh && ./wait-for-it.sh 127.0.1.1:10250'
POD=$(kubectl get po --namespace kube-system | awk '/kube-registry-v0/ { print $1 }')
kubectl port-forward --namespace kube-system $POD 5000:5000

Everything works fine except that kubectl port-forward said that pod does not exist at the first time running:

++ kubectl get po --namespace kube-system
++ awk '/kube-registry-v0/ { print $1 }'
+ POD=kube-registry-v0-qr2ml
+ kubectl port-forward --namespace kube-system kube-registry-v0-qr2ml 5000:5000
error: error upgrading connection: unable to upgrade connection: pod does not exist

If I re-run:

+ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
+ minikube ssh 'grep docker.for.mac.localhost /etc/hosts || echo -e "127.0.0.1\tdocker.for.mac.localhost" | sudo tee -a /etc/hosts'
127.0.0.1   docker.for.mac.localhost
+ minikube ssh 'test -f wait-for-it.sh || curl -O https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh'
+ minikube ssh 'chmod +x wait-for-it.sh && ./wait-for-it.sh 127.0.1.1:10250'
wait-for-it.sh: waiting 15 seconds for 127.0.1.1:10250
wait-for-it.sh: 127.0.1.1:10250 is available after 0 seconds
++ kubectl get po --namespace kube-system
++ awk '/kube-registry-v0/ { print $1 }'
+ POD=kube-registry-v0-qr2ml
+ kubectl port-forward --namespace kube-system kube-registry-v0-qr2ml 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

I added a debug line before forwarding:

kubectl describe pod --namespace kube-system $POD

and saw this:

+ POD=kube-registry-v0-qr2ml
+ kubectl describe pod --namespace kube-system kube-registry-v0-qr2ml
Name:       kube-registry-v0-qr2ml
Namespace:  kube-system
Node:       minikube/192.168.99.100
Start Time: Thu, 28 Dec 2017 10:00:00 +0700
Labels:     k8s-app=kube-registry
        version=v0
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"kube-system","name":"kube-registry-v0","uid":"317ecc42-eb7b-11e7-a8ce-...
Status:     Running
IP:     172.17.0.6
Controllers:    ReplicationController/kube-registry-v0
Containers:
  registry:
    Container ID:   docker://6e8f3f33399605758354f3f546996067d834459781235d51eef3ffa9c6589947
    Image:      registry:2.5.1
    Image ID:       docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
    Port:       5000/TCP
    State:      Running
      Started:      Thu, 28 Dec 2017 13:22:44 +0700

Why kubectl said that it does not exist?

---

Fri Dec 29 04:58:06 +07 2017

Looking carefully at the events, I found something:

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath                   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------                   --------        ------                  -------
  20m           20m             1       kubelet, minikube                                       Normal          SuccessfulMountVolume   MountVolume.SetUp succ
eeded for volume "image-store"
  20m           20m             1       kubelet, minikube                                       Normal          SuccessfulMountVolume   MountVolume.SetUp succ
eeded for volume "default-token-fs7kr"
  20m           20m             1       kubelet, minikube                                       Normal          SandboxChanged          Pod sandbox changed, it will be killed and re-created.
  20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Pulled                  Container image "registry:2.5.1" already present on machine
  20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Created                 Created container
  20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Started                 Started container

Pod sandbox changed, it will be killed and re-created.

Before:

Containers:
  registry:
    Container ID:       docker://47c510dce00c6c2c29c9fe69665e1241c457d0666174a7723062c534e7229c58
    Image:              registry:2.5.1
    Image ID:           docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
    Port:               5000/TCP
    State:              Running
      Started:          Thu, 28 Dec 2017 13:47:02 +0700
    Last State:         Terminated
      Reason:           Error
      Exit Code:        2
      Started:          Thu, 28 Dec 2017 13:22:44 +0700
      Finished:         Thu, 28 Dec 2017 13:45:18 +0700
    Ready:              True
    Restart Count:      14

After:

Containers:
  registry:
    Container ID:       docker://3a7da784d3d596796111348757725f5af22b47c5edd0fc29a4ffbb84f3f08956
    Image:              registry:2.5.1
    Image ID:           docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
    Port:               5000/TCP
    State:              Running
      Started:          Thu, 28 Dec 2017 19:03:04 +0700
    Last State:         Terminated
      Reason:           Error
      Exit Code:        2
      Started:          Thu, 28 Dec 2017 13:47:02 +0700
      Finished:         Thu, 28 Dec 2017 19:00:48 +0700
    Ready:              True
    Restart Count:      15

minikube logs:

Dec 28 22:15:41 minikube localkube[3250]: W1228 22:15:41.102038
3250 docker_sandbox.go:343] 无法从插件/docker 读取 pod IP：找不到 kube-system/kube-registry 的网络状态-v0-qr2ml 通过插件：无效的网络状态

Answer 1

POD=$(kubectl get po --namespace kube-system | awk '/kube-registry-v0/ { print $1 }')

请注意，使用选择器几乎肯定比使用文本实用程序更好，尤其是从kubectl. 我不知道他们对默认输出的格式做出任何承诺，这就是为什么--output=json和朋友存在的原因。但是，在您只想要名称的情况下，有一个特殊的功能--output=name可以按照它所说的做，但需要注意的是 Resource 前缀将位于名称前面（pods/kube-registry-v0-qr2ml在您的情况下）

另外，我看到您有“等待”，但仅仅因为端口正在接受连接并不意味着 Pod 已就绪。您实际上会想要使用--output=json（或更多awk脚本，我猜）来确保 Pod 处于运行状态和就绪状态，当 kubernetes 和 Pod 同意一切都很酷时，就会达到后一种状态。

我怀疑，但必须通过实验才能确定，错误消息只是具有误导性；并不是 kubernetes 对你的 Pod 一无所知，只是它无法在它所处的状态下将其转发到它。

您还可以通过创建一个Serviceoftype: NodePort然后在分配的端口上与节点的 IP 通信来获得更好的成功；该侧步骤全部由这种kubectl壳一塌糊涂，但确实没有侧步就绪部分-只豆荚就绪状态将接收来自服务业务

作为一个次要的，迂腐的注释，--namespace是一个参数kubectl，而不是port-forward，所以最正确的调用是kubectl --namespace=kube-system port-forward kube-registry-v0-qr2ml 5000:5000确保参数不会被错误解析