kubesphere从3.0升级到3.1的时候报错

wiliiwin

操作系统信息

 ubuntu 18.04 tls

Kubernetes版本信息

   1.17.3      3个master     多个node节点

KubeSphere版本信息

3.0

问题是什么

 我现在从3.0版本升级到3.1版本的过程中出现了一下的错误，报错请看截图

这个执行失败次数过后出现了

fatal: [localhost]: FAILED! => {“attempts”: 90, “changed”: true, “cmd”: “/usr/local/bin/kubectl get pod -n kubesphere-system -l job-name=ks-upgrade –no-headers | awk ‘{print $3}’\n”, “delta”: “0:00:00.097521”, “end”: “2021-05-10 11:22:44.004625”, “rc”: 0, “start”: “2021-05-10 11:22:43.907104”, “stderr”: “No resources found in kubesphere-system namespace.”, “stderr_lines”: [“No resources found in kubesphere-system namespace.”], “stdout”: "", “stdout_lines”: []}

Cauchy

wiliiwin

kubectl get all -n kubesphere-system

wiliiwin

Cauchy
NAME READY STATUS RESTARTS AGE
pod/etcd-f988bdb6f-sn6cc 1/1 Running 2 163d
pod/ks-account-c449648c5-kz64l 2/2 Running 13 170d
pod/ks-alerting-migration-jq99h 0/1 Completed 0 5h14m
pod/ks-apigateway-846f6bff97-xm2tr 1/1 Running 0 224d
pod/ks-apiserver-6847666944-7s499 1/1 Running 0 3h55m
pod/ks-console-9bc9c5df8-dhfbx 1/1 Running 9 223d
pod/ks-controller-manager-854dcddb5c-stl42 1/1 Running 0 3h55m
pod/ks-installer-576475b6cc-nr67n 1/1 Running 0 5h19m
pod/minio-845b7bd867-9ld5z 1/1 Running 18 163d
pod/mysql-67cd66d5d-lqbdb 1/1 Running 1 163d
pod/openldap-0 1/1 Running 0 402d
pod/openpitrix-import-job-4xbzh 0/1 Completed 6 4h
pod/redis-6fd6c6d6f9-bh8gk 1/1 Running 0 402d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/etcd ClusterIP 10.98.160.179 <none> 2379/TCP 401d
service/ks-account ClusterIP 10.104.120.114 <none> 80/TCP 402d
service/ks-apigateway ClusterIP 10.104.232.116 <none> 80/TCP 402d
service/ks-apiserver ClusterIP 10.102.226.6 <none> 80/TCP 402d
service/ks-console NodePort 10.100.8.114 <none> 80:30880/TCP 402d
service/ks-controller-manager ClusterIP 10.103.63.218 <none> 443/TCP 223d
service/minio ClusterIP 10.105.207.39 <none> 9000/TCP 401d
service/mysql ClusterIP 10.98.137.225 <none> 3306/TCP 401d
service/openldap ClusterIP None <none> 389/TCP 402d
service/redis ClusterIP 10.105.37.115 <none> 6379/TCP 402d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/etcd 1/1 1 1 401d
deployment.apps/ks-account 1/1 1 1 402d
deployment.apps/ks-apigateway 1/1 1 1 402d
deployment.apps/ks-apiserver 1/1 1 1 223d
deployment.apps/ks-console 1/1 1 1 223d
deployment.apps/ks-controller-manager 1/1 1 1 223d
deployment.apps/ks-installer 1/1 1 1 402d
deployment.apps/minio 1/1 1 1 401d
deployment.apps/mysql 1/1 1 1 401d
deployment.apps/redis 1/1 1 1 402d

NAME replicaset.apps/etcd-5769d4997f replicaset.apps/etcd-f988bdb6f replicaset.apps/ks-account-56698f8f6f replicaset.apps/ks-account-567d88b5b replicaset.apps/ks-account-5c7c88959 replicaset.apps/ks-account-5cf586f4b9 replicaset.apps/ks-account-678d8c6965 replicaset.apps/ks-account-67988cb7cd replicaset.apps/ks-account-79cd8fc4ff replicaset.apps/ks-account-7d676c8c replicaset.apps/ks-account-7dbdf54799 replicaset.apps/ks-account-84d464456 replicaset.apps/ks-account-c449648c5 replicaset.apps/ks-apigateway-5664c4b76f replicaset.apps/ks-apigateway-6b47787b9c replicaset.apps/ks-apigateway-6b9d88bd6d replicaset.apps/ks-apigateway-6ddd598bd4 replicaset.apps/ks-apigateway-77cd4c4c5 replicaset.apps/ks-apigateway-846f6bff97 replicaset.apps/ks-apigateway-854c885b44 replicaset.apps/ks-apigateway-85d67f6488 replicaset.apps/ks-apigateway-8d66ddf58 replicaset.apps/ks-apiserver-545f9cf7c9 replicaset.apps/ks-apiserver-5685c5b479 replicaset.apps/ks-apiserver-596678b86d replicaset.apps/ks-apiserver-645687cb48 replicaset.apps/ks-apiserver-6847666944 replicaset.apps/ks-apiserver-765c47fbf4 replicaset.apps/ks-apiserver-77557d8dbd replicaset.apps/ks-apiserver-7c6bd7cd7b replicaset.apps/ks-apiserver-85b75597f replicaset.apps/ks-apiserver-8bb465988 replicaset.apps/ks-apiserver-9c5849b56 replicaset.apps/ks-console-9bc9c5df8 replicaset.apps/ks-controller-manager-5b7947f4b7 replicaset.apps/ks-controller-manager-5f6dfffc76 replicaset.apps/ks-controller-manager-685c474f48 replicaset.apps/ks-controller-manager-6f4b94cb6f replicaset.apps/ks-controller-manager-6f79dc8c7f replicaset.apps/ks-controller-manager-77f887c46b replicaset.apps/ks-controller-manager-78f7cfc867 replicaset.apps/ks-controller-manager-7b8ff48774 replicaset.apps/ks-controller-manager-7d4dbf8fdc replicaset.apps/ks-controller-manager-854dcddb5c replicaset.apps/ks-controller-manager-dc57bb86 replicaset.apps/ks-installer-576475b6cc replicaset.apps/ks-installer-64f4dff579 replicaset.apps/ks-installer-6c6f485448 replicaset.apps/ks-installer-6c84548bb8 replicaset.apps/ks-installer-75b8d89dff replicaset.apps/ks-installer-7b8bfd5fbc replicaset.apps/ks-installer-85854b8c8 replicaset.apps/ks-installer-8696cb86bb replicaset.apps/ks-installer-d66bcbf87 replicaset.apps/minio-845b7bd867 replicaset.apps/mysql-66df969d replicaset.apps/mysql-67cd66d5d replicaset.apps/redis-6fd6c6d6f9 DESIRED CURRENT READY AGE
0 0 0 401d
1 1 1 401d
0 0 0 401d
0 0 0 401d
0 0 0 170d
0 0 0 223d
0 0 0 224d
0 0 0 170d
0 0 0 224d
0 0 0 170d
0 0 0 170d
0 0 0 170d
1 1 1 170d
0 0 0 402d
0 0 0 401d
0 0 0 402d
0 0 0 401d
0 0 0 401d
1 1 1 224d
0 0 0 401d
0 0 0 401d
0 0 0 224d
0 0 0 170d
0 0 0 3h56m
0 0 0 5h14m
0 0 0 170d
1 1 1 3h55m
0 0 0 170d
0 0 0 170d
0 0 0 170d
0 0 0 170d
0 0 0 4h34m
0 0 0 170d
1 1 1 223d
0 0 0 170d
0 0 0 4h34m
0 0 0 4h33m
0 0 0 4h35m
0 0 0 4h40m
0 0 0 4h18m
0 0 0 170d
0 0 0 4h32m
0 0 0 3h56m
1 1 1 3h55m
0 0 0 5h14m
1 1 1 5h19m
0 0 0 170d
0 0 0 194d
0 0 0 194d
0 0 0 402d
0 0 0 170d
0 0 0 223d
0 0 0 170d
0 0 0 194d
1 1 1 401d
0 0 0 401d
1 1 1 224d
1 1 1 402d

NAME READY AGE
statefulset.apps/openldap 1/1 402d

NAME COMPLETIONS DURATION AGE
job.batch/ks-alerting-migration 1/1 21s 5h14m
job.batch/ks-devops-migration 0/1 223d 223d
job.batch/ks-upgrade 1/1 77s 223d
job.batch/openpitrix-import-job 1/1 6m12s 4h

Cauchy

wiliiwin

还有ks-account和ks-apigateway，这个环境是2.1的吧，不是3.0的

wiliiwin

Cauchy
当时升级到3.0的时候没有看着2个还是2.1的这个在3.0版本已经没有用了应该不影响到这个问题吧

Cauchy

正常情况升级到3.0之后这两个组件应该会自动关掉呀~

可以把这两个副本数调整到 0，然后重启下ks-installer试试

wiliiwin

Cauchy
我现在把2.1这2个组件删除了，目前卡在了这个地方

Cauchy

wiliiwin

嗯稍等会看起来是monitoring还没执行完

wiliiwin

Cauchy
已经卡了很久了这个monitor本身就很慢吗？

Cauchy

wiliiwin
进到ks-installer里边

kubectl exec -it -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') /bin/bash

看下 results/monitoring 里的 stdout

wiliiwin

Cauchy

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Downloading items] ********************************************
skipping: [localhost]

TASK [download : Synchronizing container] **************************************
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [ks-monitor : Monitoring | Checking old installation files] ***************
ok: [localhost]

TASK [ks-monitor : Monitoring | Deleting old prometheus-operator files] ********
skipping: [localhost]

TASK [ks-monitor : Monitoring | Deleting old prometheus-operator] **************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Deleting old prometheus-operator files] ********
skipping: [localhost] => (item=prometheus) 

TASK [ks-monitor : Monitoring | Getting ks-monitoring installation files] ******
changed: [localhost] => (item=prometheus)
changed: [localhost] => (item=monitoring-dashboard)

TASK [ks-monitor : Monitoring | Checking Prometheus PersistentVolumeClaim] *****
changed: [localhost]

TASK [ks-monitor : Monitoring | Setting Prometheus data pv size] ***************
ok: [localhost]

TASK [ks-monitor : Monitoring | Checking Prometheus retention days] ************
changed: [localhost]

TASK [ks-monitor : Monitoring | Setting Prometheus retention days] *************
ok: [localhost]

TASK [ks-monitor : Monitoring | Checking Prometheus node selector] *************
changed: [localhost]

TASK [ks-monitor : Monitoring | Setting Prometheus node selector] **************
ok: [localhost]

TASK [ks-monitor : Monitoring | Creating manifests] ****************************
changed: [localhost] => (item={'path': 'prometheus/init', 'file': 'prometheus-operator-deployment.yaml'})
changed: [localhost] => (item={'path': 'prometheus/prometheus', 'file': 'prometheus-prometheus.yaml'})
changed: [localhost] => (item={'path': 'prometheus/kube-state-metrics', 'file': 'kube-state-metrics-deployment.yaml'})
changed: [localhost] => (item={'path': 'prometheus/node-exporter', 'file': 'node-exporter-daemonset.yaml'})
changed: [localhost] => (item={'path': 'prometheus/alertmanager', 'file': 'alertmanager-alertmanager.yaml'})
changed: [localhost] => (item={'path': 'prometheus/prometheus-adapter', 'file': 'prometheus-adapter-deployment.yaml'})
changed: [localhost] => (item={'path': 'prometheus/grafana', 'file': 'grafana-deployment.yaml'})
changed: [localhost] => (item={'path': 'prometheus/etcd', 'file': 'prometheus-serviceMonitorEtcd.yaml'})
changed: [localhost] => (item={'path': 'prometheus/etcd', 'file': 'prometheus-endpointsEtcd.yaml'})
changed: [localhost] => (item={'path': 'prometheus/thanosruler', 'file': 'thanosruler.yaml'})

TASK [ks-monitor : Monitoring | Initing prometheus-operator] *******************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing node-exporter] **********************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing kube-state-metrics] *****************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing grafana] ****************************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing Prometheus] *************************
skipping: [localhost] => (item=prometheus) 
skipping: [localhost] => (item=prometheus) 

TASK [ks-monitor : Monitoring | Installing devops monitoring] ******************
changed: [localhost] => (item=devops)

TASK [ks-monitor : Monitoring | Installing etcd monitoring] ********************
skipping: [localhost] => (item=etcd) 

TASK [ks-monitor : Monitoring | Checking Kubernetes version] *******************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Creating Prometheus rules] *********************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Creating Prometheus rules (v1.16.0+)] **********
skipping: [localhost]
                            
TASK [ks-monitor : Monitoring | Installing alertmanager] ***********************
skipping: [localhost]

TASK [ks-monitor : notification-manager | Getting notification-manager installation files] ***
changed: [localhost]

TASK [ks-monitor : notification-manager | Getting federated files] *************
changed: [localhost]

TASK [ks-monitor : notification-manager | Getting migrator files] **************
changed: [localhost]

TASK [ks-monitor : notification-manager | Getting notification-manager installation files] ***
changed: [localhost] => (item={'path': 'notification-manager', 'file': 'custom-values-notification.yaml'})

TASK [ks-monitor : notification-manager | Checking notification-manager] *******
changed: [localhost]

TASK [ks-monitor : KubeSphere | Backuping old notification-manager crd] ********
skipping: [localhost]

TASK [ks-monitor : notification-manager | Uninstalling old notification-manager] ***
skipping: [localhost]

TASK [ks-monitor : notification-manager | Deploying notification-manager] ******
changed: [localhost]

TASK [ks-monitor : notification-manager | Deploying notification-manager crds] ***
skipping: [localhost]

TASK [ks-monitor : notification-manager | Checking federatednamespace crd installed] ***
changed: [localhost]
                            
TASK [ks-monitor : notification-manager | Deploying federatednamespace crd] ****
skipping: [localhost]

TASK [ks-monitor : notification-manager | Checking federated namespace deployed] ***
changed: [localhost]

TASK [ks-monitor : notification-manager | Deploying federated namespace] *******
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing monitoring-dashboard] ***************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Importing ks-monitoring status] ****************
skipping: [localhost]

TASK [ks-monitor : Monitoring | Installing thanosruler] ************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=20   changed=15   unreachable=0    failed=0    skipped=24   rescued=0    ignored=0

Cauchy

wiliiwin

这个看起来已经跑完了，再看下日志，如果还卡着的话重启下ks-installer再观察下日志。

wiliiwin

Cauchy

这个升级完之后，看版本怎么还是3.0.0呢？

Cauchy

wiliiwin

咋升级的呢？直接apply了那两个yaml么？

wiliiwin

Cauchy
是的
按照这个地址https://github.com/kubesphere/ks-installer/blob/master/README_zh.md 来进行升级的

wiliiwin

Cauchy

ks-install是3.1.0的

我发现ks-controller-manager和ks-apiserver还是3.0.0

Cauchy

wiliiwin
多apply了一个yaml哦，文档里有写：
https://github.com/kubesphere/ks-installer#upgrade

现在的话可以通过下边方法重新触发下升级

执行 kubectl edit cc -n kubesphere-system ks-installer，删掉里边status中的内容。
重启ks-installer

wiliiwin

Cauchy
现在升级到了3.1版本了但是我接入到openldap，登录的时候出现了
users.iam.kubesphere.io is forbidden: User “system:pre-registration” cannot create resource “users” in API group “iam.kubesphere.io” at the cluster scope

wiliiwin

hongming
Cauchy

非常感谢2位的帮助，我的问题已经解决了

roger-perper

Cauchy 请教下，我是kubekey安装的，我想kubesphere和k8s一起升级，这个错误可能是啥原因

hongming

wiliiwin 看看这个issue kubesphere/kubesphere#3850

wiliiwin

hongming

原来外接openldap直接就可以登录了，升级到3.1版本之后，还会跳出来这么一个输入框，然后还无法登录，这个能取消吗？

hongming

wiliiwin

for user in `kubectl get users -l \!iam.kubesphere.io/origin-uid,iam.kubesphere.io/identify-provider -o jsonpath="{.items[*].metadata.name}"`; do kubectl label user $user iam.kubesphere.io/origin-uid=$user; done