kubesphere访问不了，集群正常

jwangkun

不小心删除了kubesphere的mysql，导致kubesphere访问不了，如何恢复kubesphere，或者kubesphere如何重装，不影响k8s

Forest-L

jwangkun 重新跑下ks-installer pod

jwangkun

Forest-L 不好意思，没太明白你说的，我没有ks-installer这个pod，我是用kubekey安装的，您能说详细点么

hongming

kubectl -n kubesphere-system rollout restart deploy ks-installer

jwangkun

hongming 你好，执行成功了，但是还是一样，页面提示Internal Server Error

jwangkun

hongming [root@node1 hngc]# kubectl logs -f ks-apiserver-85445c576d-vcpq7
Error from server (NotFound): pods “ks-apiserver-85445c576d-vcpq7” not found
[root@node1 hngc]# kubectl logs -f ks-apiserver-85445c576d-vcpq7 -n kubesphere-system
W1210 15:21:02.794412 1 client_config.go:543] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.
Error: failed to connect to redis service, please check redis status, error: dial tcp 10.233.62.215:6379: connect: connection refused
2020/12/10 15:21:02 failed to connect to redis service, please check redis status, error: dial tcp 10.233.62.215:6379: connect: connection refused

Forest-L

jwangkun
1、kubectl edit cc -n kubesphere-system ks-installer 把mysql状态信息都删除，如mysql字样等下面三行内容删除。

status:
  alerting:
    enabledTime: 2020-12-09T17:53:10CST
    status: enabled
  mysql:
    enabledTime: 2020-11-30T16:36:10CST
    status: enabled

2、再执行如下语句，kubectl -n kubesphere-system rollout restart deploy ks-installer

jwangkun

[root@node1 hngc]# kubectl get deployment -n kubesphere-system NAME READY UP-TO-DATE AVAILABLE AGE etcd 0/1 1 0 86d ks-apiserver 1/3 1 1 86d ks-console 3/3 3 3 86d ks-controller-manager 3/3 3 3 86d ks-installer 1/1 1 1 86d minio 1/1 1 1 86d

jwangkun

Forest-L 已经没有mysql 了，如果重装这个呢

jwangkun

Forest-L 执行完了，mysql 和redis都起来了，但是还是访问不了

jwangkun

Forest-L mysql-7f64d9f584-v5nnb 0/1 Pending 0 18m
这个好像装不上

jwangkun

kubectl delete deployment mysql -n kubesphere-system
kubectl delete deployment redis-ha-haproxy -n kubesphere-system
kubectl delete minio redis-ha-haproxy -n kubesphere-system
执行过这个三个

Jeff

jwangkun 你删除这些干啥呢，redis mysql minio 都是集群依赖的数据库啊，这些删除了当然启动不了了

jwangkun

Jeff 这些启动报错，没想到是他系统依赖的，现在如何恢复呢

jwangkun

[root@node1 hngc]# kubectl get pods -n kubesphere-system NAME READY STATUS RESTARTS AGE etcd-65796969c7-65s2c 0/1 Pending 0 125m ks-apiserver-68676cc974-b76np 0/1 CrashLoopBackOff 4 2m29s ks-apiserver-6dd46fc878-2kxgg 0/1 CrashLoopBackOff 4 2m21s ks-apiserver-c6d5554cb-9gmx5 1/1 Running 0 97m ks-console-786b9846d4-b9pkk 1/1 Running 4 49d ks-console-786b9846d4-d62f9 1/1 Running 2 49d ks-console-786b9846d4-kjslt 1/1 Running 1 7d18h ks-controller-manager-8549c9699-dfxf2 1/1 Running 0 2m17s ks-controller-manager-8549c9699-knhkv 1/1 Running 0 2m19s ks-controller-manager-8549c9699-sq4jg 1/1 Running 0 2m21s ks-installer-7f59c47bbf-jgs7n 1/1 Running 0 5m18s minio-7bfdb5968b-kdp5p 1/1 Running 5 86d mysql-7f64d9f584-v5nnb 0/1 Pending 0 4m4s openldap-0 0/1 CrashLoopBackOff 2198 7d19h openldap-1 1/1 Running 5 86d redis-ha-server-0 2/2 Running 10 86d redis-ha-server-1 2/2 Running 2 7d19h redis-ha-server-2 2/2 Running 6 49d [root@node1 hngc]# kubectl logs mysql-7f64d9f584-v5nnb -n kubesphere-system [root@node1 hngc]# kubectl logs mysql-7f64d9f584-v5nnb -f -n kubesphere-system [root@node1 hngc]#

jwangkun

Warning FailedScheduling 80s (x2 over 80s) default-scheduler 0/6 nodes are available: 6 node(s) had volume node affinity conflict.

Forest-L

jwangkun 参考这个文章排查呗， https://github.com/kubernetes/kubernetes/issues/81020

rayzhou2017

6 node(s) had volume node affinity conflict 看看这个，错误提示已经很清楚了