操作系统信息
例如:虚拟机,Oracle8,4C/8G

Kubernetes版本信息
v1.22.2

KubeSphere版本信息
3.3.2

问题是什么

我这边是有kubesphere的低版本升级到新的kubespherev3.2.0版本,使用 kubesphere的项目网关,使用的 NodePort 模式,不启用链路追踪。

默认创建的项目网关使用的image版本是: kubesphere/nginx-ingress-controller:v0.48.1@sha256:e9fb216ace49dfa4a5983b183067e97496e7a8b307d2093f4278cd550c303899,导致kubesphere-router-xxxxxxxx容器无法启动,日志报错:

 -------------------------------------------------------------------------------

 NGINX Ingress controller

   Release:       v0.48.1

   Build:         30809c066cd027079cbb32dccc8a101d6fbffdcb

   Repository:    https://github.com/kubernetes/ingress-nginx

   nginx version: nginx/1.20.1

 

 -------------------------------------------------------------------------------

 

 I0404 10:24:02.977959       7 flags.go:211] "Watching for Ingress" class="nginx"

 W0404 10:24:02.978014       7 flags.go:216] Ingresses with an empty class will also be processed by this Ingress controller

 W0404 10:24:02.978330       7 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

 I0404 10:24:02.978589       7 main.go:241] "Creating API client" host="https://172.23.0.1:443"

 I0404 10:24:02.986453       7 main.go:285] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.2" state="clean" commit="8b5a19147530eaac9476b0ab82980b4088bbc1b2" platform="linux/amd64"

 I0404 10:24:03.227803       7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"

 I0404 10:24:03.229505       7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"

 W0404 10:24:03.232104       7 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.

 I0404 10:24:03.265319       7 nginx.go:254] "Starting NGINX Ingress controller"

 E0404 10:24:04.366977       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 E0404 10:24:05.892260       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 E0404 10:24:09.066939       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 E0404 10:24:13.411711       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 2023/04/04 10:24:23 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 W0404 10:24:23.093611       7 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 E0404 10:24:26.038268       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 2023/04/04 10:24:27 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 W0404 10:24:27.226895       7 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 E0404 10:24:39.168476       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

 2023/04/04 10:24:53 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 W0404 10:24:53.093053       7 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 2023/04/04 10:24:57 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 W0404 10:24:57.226374       7 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

 I0404 10:25:01.783134       7 main.go:187] "Received SIGTERM, shutting down"

 I0404 10:25:01.783163       7 nginx.go:372] "Shutting down controller queues"

 E0404 10:25:01.783625       7 store.go:178] timed out waiting for caches to sync

 I0404 10:25:01.783678       7 nginx.go:296] "Starting NGINX process"

 I0404 10:25:01.784876       7 leaderelection.go:243] attempting to acquire leader lease kubesphere-controls-system/ingress-controller-leader-kubesphere-router-gitinfra-nginx...

 I0404 10:25:01.785190       7 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"

 I0404 10:25:01.796300       7 status.go:204] "POD is not ready" pod="kubesphere-controls-system/kubesphere-router-gitinfra-6898577c6d-6jfkt" node="10.26.9.44"

 I0404 10:25:01.797118       7 leaderelection.go:253] successfully acquired lease kubesphere-controls-system/ingress-controller-leader-kubesphere-router-gitinfra-nginx

 I0404 10:25:01.797159       7 status.go:84] "New leader elected" identity="kubesphere-router-gitinfra-6898577c6d-6jfkt"

 I0404 10:25:01.797336       7 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"

 I0404 10:25:01.800152       7 status.go:132] "removing value from ingress status" address=[]

 I0404 10:25:01.800188       7 nginx.go:388] "Stopping NGINX process"

 2023/04/04 10:25:01 [notice] 38#38: signal process started

 I0404 10:25:02.804683       7 nginx.go:401] "NGINX process has stopped"

 I0404 10:25:02.804718       7 main.go:195] "Handled quit, awaiting Pod deletion"

 I0404 10:25:12.804806       7 main.go:198] "Exiting" code=0

经过调查得知:

bitnami/charts#7264
经过调查发现可能是 kubesphere/nginx-ingress-controller:v0.48.1@sha256:e9fb216ace49dfa4a5983b183067e97496e7a8b307d2093f4278cd550c303899 镜像的版本和k8s的版本造成的,所以我想问下怎么升级下在 kubesphere的创建路由默认使用的 nginx-ingress版本?

我使用了


发现没有用。新建的路由还是使用的v0.48.1的版本

    xuanyuanaosheng
    镜像仓库和tag可以在 kubesphere-config 中的 gateway 下配置指定,但这样指定只适合版本跨度不大的情况下不会有问题。如果版本跨度太大最好是替换掉 ks-controller-manger 内置所使用的 nginx ingress helm 文件的方式处理。
    3.2 的网关用的 0.48.1 的版本不完全兼容 K8s 1.22,建议升级到 ks 3.3 ,ks 3.3 中用的 1.1.0 版本可以兼容 K8s 1.22