bay*_*man 4 kubernetes kubernetes-helm nginx-ingress
我使用 Kubespray 部署了一个裸机集群，并启用了 kubernetes 1.22.2、MetalLB 和 ingress-nginx。404 Not found当设置时尝试访问通过 helm 部署的任何服务时，我得到了ingressClassName: nginx。但是，如果我不使用ingressClassName: nginx而是kubernetes.io/ingress.class: nginx在 helm 图表中使用，一切都会正常工作。yaml。我怎样才能让它工作ingressClassName？
这些是我的 kubespray 设置inventory/mycluster/group_vars/k8s_cluster/addons.yml
# Nginx ingress controller deployment
ingress_nginx_enabled: true
ingress_nginx_host_network: false
ingress_publish_status_address: ""
ingress_nginx_nodeselector:
kubernetes.io/os: "linux"
ingress_nginx_tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Equal"
value: ""
effect: "NoSchedule"
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_configmap:
map-hash-bucket-size: "128"
ssl-protocols: "TLSv1.2 TLSv1.3"
ingress_nginx_configmap_tcp_services:
9000: "default/example-go:8080"
ingress_nginx_configmap_udp_services:
53: "kube-system/coredns:53"
ingress_nginx_extra_args:
- --default-ssl-certificate=default/mywildcard-tls
ingress_nginx_class: "nginx"
grafana helm value.yaml
ingress:
enabled: true
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
ingressClassName: nginx
# Values can be templated
annotations:
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
labels: {}
path: /
# pathType is only for k8s >= 1.1=
pathType: Prefix
hosts:
- grafana.mycluster.org
tls:
- secretName: mywildcard-tls
hosts:
- grafana.mycluster.org
kubectl describe pod grafana-679bbfd94-p2dd7
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25m default-scheduler Successfully assigned default/grafana-679bbfd94-p2dd7 to node1
Normal Pulled 25m kubelet Container image "grafana/grafana:8.2.2" already present on machine
Normal Created 25m kubelet Created container grafana
Normal Started 25m kubelet Started container grafana
Warning Unhealthy 24m (x3 over 25m) kubelet Readiness probe failed: Get "http://10.233.90.33:3000/api/health": dial tcp 10.233.90.33:3000: connect: connection refused
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 10.233.14.90 10.10.30.52 80:30285/TCP 55m
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 9d
kubectl get ing（未分配节点地址）
NAME CLASS HOSTS ADDRESS PORTS AGE
grafana nginx grafana.mycluster.org 80, 443 25m
kubectl describe ing grafana（未分配节点地址）
Name: grafana
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
mywildcard-tls terminates grafana.mycluster.org
Rules:
Host Path Backends
---- ---- --------
grafana.mycluster.org
/ grafana:80 (10.233.90.33:3000)
Annotations: meta.helm.sh/release-name: grafana
meta.helm.sh/release-namespace: default
Events: <none>
kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/grafana-b988b9b6-pxccw 1/1 Running 0 2m53s
default pod/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4-wjlpv 1/1 Running 0 17h
ingress-nginx pod/ingress-nginx-controller-6m2vt 1/1 Running 0 17h
ingress-nginx pod/ingress-nginx-controller-xkgxl 1/1 Running 0 17h
kube-system pod/calico-kube-controllers-684bcfdc59-kmsst 1/1 Running 0 17h
kube-system pod/calico-node-dhlnt 1/1 Running 0 17h
kube-system pod/calico-node-r8ktz 1/1 Running 0 17h
kube-system pod/coredns-8474476ff8-9sbwh 1/1 Running 0 17h
kube-system pod/coredns-8474476ff8-fdgcb 1/1 Running 0 17h
kube-system pod/dns-autoscaler-5ffdc7f89d-vskvq 1/1 Running 0 17h
kube-system pod/kube-apiserver-node1 1/1 Running 0 17h
kube-system pod/kube-controller-manager-node1 1/1 Running 1 17h
kube-system pod/kube-proxy-hbjz6 1/1 Running 0 16h
kube-system pod/kube-proxy-lfqzt 1/1 Running 0 16h
kube-system pod/kube-scheduler-node1 1/1 Running 1 17h
kube-system pod/kubernetes-dashboard-548847967d-qqngw 1/1 Running 0 17h
kube-system pod/kubernetes-metrics-scraper-6d49f96c97-2h7hc 1/1 Running 0 17h
kube-system pod/nginx-proxy-node2 1/1 Running 0 17h
kube-system pod/nodelocaldns-64cqs 1/1 Running 0 17h
kube-system pod/nodelocaldns-t5vv6 1/1 Running 0 17h
kube-system pod/registry-proxy-kljvw 1/1 Running 0 17h
kube-system pod/registry-proxy-nz4qk 1/1 Running 0 17h
kube-system pod/registry-xzh9d 1/1 Running 0 17h
metallb-system pod/controller-77c44876d-c92lb 1/1 Running 0 17h
metallb-system pod/speaker-fkjqp 1/1 Running 0 17h
metallb-system pod/speaker-pqjgt 1/1 Running 0 17h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/grafana LoadBalancer 10.233.1.104 10.10.30.52 80:31116/TCP 2m53s
default service/kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 17h
kube-system service/coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 17h
kube-system service/dashboard-metrics-scraper ClusterIP 10.233.35.124 <none> 8000/TCP 17h
kube-system service/kubernetes-dashboard ClusterIP 10.233.32.133 <none> 443/TCP 17h
kube-system service/registry ClusterIP 10.233.30.221 <none> 5000/TCP 17h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
ingress-nginx daemonset.apps/ingress-nginx-controller 2 2 2 2 2 kubernetes.io/os=linux 17h
kube-system daemonset.apps/calico-node 2 2 2 2 2 kubernetes.io/os=linux 17h
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 kubernetes.io/os=linux 17h
kube-system daemonset.apps/nodelocaldns 2 2 2 2 2 kubernetes.io/os=linux 17h
kube-system daemonset.apps/registry-proxy 2 2 2 2 2 <none> 17h
metallb-system daemonset.apps/speaker 2 2 2 2 2 kubernetes.io/os=linux 17h
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/grafana 1/1 1 1 2m53s
default deployment.apps/nfs-client-nfs-subdir-external-provisioner 1/1 1 1 17h
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 17h
kube-system deployment.apps/coredns 2/2 2 2 17h
kube-system deployment.apps/dns-autoscaler 1/1 1 1 17h
kube-system deployment.apps/kubernetes-dashboard 1/1 1 1 17h
kube-system deployment.apps/kubernetes-metrics-scraper 1/1 1 1 17h
metallb-system deployment.apps/controller 1/1 1 1 17h
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/grafana-b988b9b6 1 1 1 2m53s
default replicaset.apps/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4 1 1 1 17h
kube-system replicaset.apps/calico-kube-controllers-684bcfdc59 1 1 1 17h
kube-system replicaset.apps/coredns-8474476ff8 2 2 2 17h
kube-system replicaset.apps/dns-autoscaler-5ffdc7f89d 1 1 1 17h
kube-system replicaset.apps/kubernetes-dashboard-548847967d 1 1 1 17h
kube-system replicaset.apps/kubernetes-metrics-scraper-6d49f96c97 1 1 1 17h
kube-system replicaset.apps/registry 1 1 1 17h
metallb-system replicaset.apps/controller-77c44876d 1 1 1 17h
kubectl get ing grafana -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/tls-acme: "true"
meta.helm.sh/release-name: grafana
meta.helm.sh/release-namespace: default
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
creationTimestamp: "2021-11-11T07:16:12Z"
generation: 1
labels:
app.kubernetes.io/instance: grafana
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 8.2.2
helm.sh/chart: grafana-6.17.5
name: grafana
namespace: default
resourceVersion: "3137"
uid: 6c34d3bd-9ab6-42fe-ac1b-7620a9566f62
spec:
ingressClassName: nginx
rules:
- host: grafana.mycluster.org
http:
paths:
- backend:
service:
name: ssl-redirect
port:
name: use-annotation
path: /*
pathType: Prefix
- backend:
service:
name: grafana
port:
number: 80
path: /
pathType: Prefix
status:
loadBalancer: {}
运行
kubectl get ingressclass返回“未找到资源”。
这就是你的问题的主要原因。
为什么？
ingressClassName: nginx当您在 Grafana文件中指定时，values.yaml您将Ingress 资源设置为使用 nginx不存在的 Ingress 类。
我使用通过修改后的deploy.yaml文件安装的minikube、MetalLB和NGINX Ingress复制了您的问题，并添加了注释资源 + 将 NGINX Ingress 控制器名称设置为您的示例中的内容。结果完全相同 -不起作用（没有地址），但注释起作用了。IngressClassnginxingressClassName: nginxkubernetes.io/ingress.class: nginx
（对于下面的解决方案，我使用的是控制器 Pod name ingress-nginx-controller-86c865f5c4-qwl2b，但在你的情况下，它会有所不同 - 使用kubectl get pods -n ingress-nginx命令检查它。另请记住，这是一种解决方法 - 通常ingressClass资源应该在 NGINX Ingress 的整个安装中自动安装.我提出这个解决方案是为了了解为什么它以前不适合你，以及为什么它可以与使用 helm 安装的 NGINX Ingress 一起使用）
在 Ingress NGINX 控制器的日志中我发现 ( kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx):
"Ignoring ingress because of error while validating ingress class" ingress="default/minimal-ingress" error="no object matching key \"nginx\" in local store"
因此，它清楚地表明控制器类没有匹配的键nginx- 因为不存在ingressClass作为 NGINX Ingress 控制器和正在运行的 Ingress 资源之间的“链接”的资源。
您可以通过运行以下命令来验证向控制器投标的控制器类名称kubectl get pod ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx -o yaml：
"Ignoring ingress because of error while validating ingress class" ingress="default/minimal-ingress" error="no object matching key \"nginx\" in local store"
现在我将创建并应用以下 Ingress 类资源：
...
spec:
containers:
- args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/nginx
...
现在在日志中我可以看到它已正确配置：
I1115 12:13:42.410384 7 main.go:101] "successfully validated configuration, accepting" ingress="minimal-ingress/default"
I1115 12:13:42.420408 7 store.go:371] "Found valid IngressClass" ingress="default/minimal-ingress" ingressclass="nginx"
I1115 12:13:42.421487 7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"minimal-ingress", UID:"c708a672-a8dd-45d3-a2ec-f2e2881623ea", APIVersion:"networking.k8s.io/v1", ResourceVersion:"454362", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
我重新应用了入口资源定义，获得了入口资源的 IP 地址。
正如我之前所说，我建议不要使用此解决方法，而是使用自动安装的解决方案来安装 NGINX Ingress 资源IngressClass。由于您选择了 Helm Chart，它具有Ingress Class资源，因此问题就消失了。其他可能的安装方法请参见此处。
| 归档时间： |
|
| 查看次数： |
5806 次 |
| 最近记录： |