kubesphere3.0登录报错：Internal Server Error #kubesphere-system空间下redis卸载与安装

kopnono

kubesphere3.0运行了一段时间了，今天登录出现了这个错误

查看kubesphere的pod发现好多没有运行：

[root@master ~]# kubectl get pod -n kubesphere-system
NAME                                    READY   STATUS             RESTARTS   AGE
etcd-85c98fb695-ddqkm                   1/1     Running            0          22d
ks-apiserver-7995f455bb-2kmx8           1/1     Terminating        0          22d
ks-apiserver-7995f455bb-wfzmz           0/1     CrashLoopBackOff   48         3h45m
ks-console-fb4c655cf-r9xmr              1/1     Running            0          31h
ks-console-fb4c655cf-z8xgb              1/1     Terminating        0          33d
ks-controller-manager-885bb48b4-g9m9v   1/1     Terminating        10         22d
ks-controller-manager-885bb48b4-wbbn7   0/1     CrashLoopBackOff   61         5h2m
ks-installer-85854b8c8-xx6pd            1/1     Running            3          22d
minio-764b67f6fb-fjwpl                  1/1     Running            0          30d
mysql-67cd66d5d-dg6gt                   1/1     Running            0          22d
openldap-0                              1/1     Terminating        0          33d
redis-6fd6c6d6f9-h4klr                  1/1     Terminating        0          33d
redis-6fd6c6d6f9-ws5bh                  0/1     Pending            0          3h26m

各pod的log如下：

ks-apiserver的log：

[root@master ~]# kubectl logs ks-apiserver-7995f455bb-2kmx8 -n kubesphere-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

[root@master ~]# kubectl logs ks-apiserver-7995f455bb-wfzmz -n kubesphere-system
W0207 15:59:09.342848       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
Error: failed to connect to redis service, please check redis status, error: dial tcp 10.233.28.61:6379: connect: connection refused
2021/02/07 15:59:09 failed to connect to redis service, please check redis status, error: dial tcp 10.233.28.61:6379: connect: connection refused

ks-console 部分log:

[root@master ~]# kubectl logs ks-console-fb4c655cf-z8xgb -n kubesphere-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)
[root@master ~]# kubectl logs ks-console-fb4c655cf-r9xmr -n kubesphere-system
···
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.233.28.61:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
{ redis.set: Executed timeout 5000 ms
    at Timeout._onTimeout (/opt/kubesphere/console/server/server.js:64530:19)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
  name: 'redis.set',
  args: 
   [ 'sid-FBjWrmrCnZXk1rtD3AsPRpMNHJkPxWCZ',
     '{"salt":"fDzzMkcpg-GtBgJY"}',
     'EX',
     7200000 ] }
  --> GET /login 200 6,017ms 14.82kb 2021/02/07T16:15:20.107
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.233.28.61:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.233.28.61:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.233.28.61:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.233.28.61:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
  <-- GET / 2021/02/07T16:15:24.086
{ UnauthorizedError: Not Login
    at Object.throw (/opt/kubesphere/console/server/server.js:31701:11)
    at getCurrentUser (/opt/kubesphere/console/server/server.js:9037:14)
    at renderView (/opt/kubesphere/console/server/server.js:23231:46)
    at dispatch (/opt/kubesphere/console/server/server.js:6870:32)
    at next (/opt/kubesphere/console/server/server.js:6871:18)
    at /opt/kubesphere/console/server/server.js:70183:16
    at dispatch (/opt/kubesphere/console/server/server.js:6870:32)
    at next (/opt/kubesphere/console/server/server.js:6871:18)
    at /opt/kubesphere/console/server/server.js:77986:37
    at dispatch (/opt/kubesphere/console/server/server.js:6870:32)
    at next (/opt/kubesphere/console/server/server.js:6871:18)
    at /opt/kubesphere/console/server/server.js:70183:16
    at dispatch (/opt/kubesphere/console/server/server.js:6870:32)
    at next (/opt/kubesphere/console/server/server.js:6871:18)
    at /opt/kubesphere/console/server/server.js:77986:37
    at dispatch (/opt/kubesphere/console/server/server.js:6870:32) message: 'Not Login' }
  --> GET / 302 2ms 43b 2021/02/07T16:15:24.088
  <-- GET /login 2021/02/07T16:15:24.091
{ FetchError: request to http://ks-apiserver.kubesphere-system.svc/kapis/config.kubesphere.io/v1alpha2/configs/oauth failed, reason: connect ECONNREFUSED 10.233.37.111:80
    at ClientRequest.<anonymous> (/opt/kubesphere/console/server/server.js:80604:11)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:73:8)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)
  message: 'request to http://ks-apiserver.kubesphere-system.svc/kapis/config.kubesphere.io/v1alpha2/configs/oauth failed, reason: connect ECONNREFUSED 10.233.37.111:80',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }

ks-controller-manager的log:

[root@master ~]# kubectl logs ks-controller-manager-885bb48b4-wbbn7 -n kubesphere-system
W0207 16:17:43.116464       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0207 16:17:55.138753       1 server.go:81] failed to connect to ldap service, please check ldap status, error: factory is not able to fill the pool: LDAP Result Code 200 "Network Error": dial tcp: lookup openldap.kubesphere-system.svc on 169.254.25.10:53: server misbehaving
[root@master ~]# kubectl logs ks-controller-manager-885bb48b4-g9m9v -n kubesphere-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

redis的log:

[root@master ~]# kubectl logs redis-6fd6c6d6f9-h4klr -n kubesphere-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)
[root@master ~]# kubectl logs redis-6fd6c6d6f9-ws5bh -n kubesphere-system
[root@master ~]#

我在docker中重启了这些容器，但是没用，这种问题怎么解决？

RolandMa1986

kopnono 盲猜是由于前一个redis的pvc 没有卸载成功，导致另外一个redis无法挂载。你可以describe 一下这两个redis pod。

kopnono

RolandMa1986
确实，两个redis用的一个数据卷：

[root@master ~]# kubectl describe pod redis-6fd6c6d6f9-zx2gn -n kubesphere-system
Name:           redis-6fd6c6d6f9-zx2gn
Namespace:      kubesphere-system
Priority:       0
Node:           <none>
Labels:         app=redis
                pod-template-hash=6fd6c6d6f9
                tier=database
                version=redis-4.0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/redis-6fd6c6d6f9
Containers:
  redis:
    Image:      redis:5.0.5-alpine
    Port:       6379/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        20m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /data from redis-pvc (rw,path="redis-data")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mk7sx (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  redis-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-pvc
    ReadOnly:   false
  default-token-mk7sx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mk7sx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/5 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 4 node(s) had volume node affinity conflict.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/5 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 4 node(s) had volume node affinity conflict.

[root@master ~]# kubectl describe pod redis-6fd6c6d6f9-h4klr -n kubesphere-system
Name:                      redis-6fd6c6d6f9-h4klr
Namespace:                 kubesphere-system
Priority:                  0
Node:                      master/10.0.3.238
Start Time:                Mon, 04 Jan 2021 17:56:18 +0800
Labels:                    app=redis
                           pod-template-hash=6fd6c6d6f9
                           tier=database
                           version=redis-4.0
Annotations:               cni.projectcalico.org/podIP: 10.233.70.29/32
                           cni.projectcalico.org/podIPs: 10.233.70.29/32
Status:                    Terminating (lasts 32h)
Termination Grace Period:  30s
IP:                        10.233.70.3
IPs:
  IP:           10.233.70.3
Controlled By:  ReplicaSet/redis-6fd6c6d6f9
Containers:
  redis:
    Container ID:   docker://d015b3ed5e126c8da875e73b3909c21cfa1b7f20a237a2f3cd656b3c243fa56f
    Image:          redis:5.0.5-alpine
    Image ID:       docker-pullable://redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 04 Jan 2021 17:56:22 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        20m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /data from redis-pvc (rw,path="redis-data")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mk7sx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  redis-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-pvc
    ReadOnly:   false
  default-token-mk7sx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mk7sx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

接下来，怎么做？

RolandMa1986

kopnono 可以检查一下为什么redis-6fd6c6d6f9-h4klr 一直是Terminating无法删除。或者试一下强制删除。这个也可能跟你用的csi有关。

kopnono

RolandMa1986
RolandMa1986
我强制删除了redis-6fd6c6d6f9-h4klr，但是另一个redis的pod还是起不来，新建的redis的pod还是pending状态。我查到了redis-pvs的数据卷是local类型，使用kubectl删除了该卷。现在有点乱了，由于redis-6fd6c6d6f9-h4klr的容器一直再自动的重建，退出，删除，再重建，不断循环。我把redis的deloyment也删了，但redis-6fd6c6d6f9-h4klr的容器依旧重复的创建删除。

[root@master ~]# docker ps -a|grep redis
e207b7d86b42        kubesphere/pause:3.1      "/pause"                 2 seconds ago        Created                                              k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7356
83096429a229        kubesphere/pause:3.1      "/pause"                 10 seconds ago       Exited (0) 5 seconds ago                             k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7355
[root@master ~]# docker ps -a|grep redis
a7a291c49fbe        kubesphere/pause:3.1      "/pause"                 14 seconds ago      Exited (0) 2 seconds ago                             k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7380
5bb50a4db81f        kubesphere/pause:3.1      "/pause"                 23 seconds ago      Exited (0) 15 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7379
9e789809f1e4        kubesphere/pause:3.1      "/pause"                 53 seconds ago      Exited (0) 43 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7378
[root@master ~]# docker ps -a|grep redis
4884e5e16522        kubesphere/pause:3.1      "/pause"                 5 seconds ago       Created                                              k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7381
a7a291c49fbe        kubesphere/pause:3.1      "/pause"                 19 seconds ago      Exited (0) 7 seconds ago                             k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7380
5bb50a4db81f        kubesphere/pause:3.1      "/pause"                 28 seconds ago      Exited (0) 20 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7379
9e789809f1e4        kubesphere/pause:3.1      "/pause"                 58 seconds ago      Exited (0) 48 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7378
[root@master ~]# docker ps -a|grep redis
7aa6fd8d7e94        kubesphere/pause:3.1      "/pause"                 Less than a second ago   Created                                              k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7382
4884e5e16522        kubesphere/pause:3.1      "/pause"                 10 seconds ago           Exited (0) 3 seconds ago                             k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7381
a7a291c49fbe        kubesphere/pause:3.1      "/pause"                 24 seconds ago           Exited (0) 13 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7380
5bb50a4db81f        kubesphere/pause:3.1      "/pause"                 33 seconds ago           Exited (0) 26 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7379
9e789809f1e4        kubesphere/pause:3.1      "/pause"                 About a minute ago       Exited (0) 54 seconds ago                            k8s_POD_redis-6fd6c6d6f9-h4klr_kubesphere-system_c967f966-f3e5-4f6c-810a-efbae2a79d78_7378

现在问题有点多了，redis的deployment没有备份，我试着在主节点使用kk从新创建一遍集群，但并没有恢复redis的deployment。

kopnono

重启了ks-installer,redis还是没有装上，所有的和redis相关的job都被忽略。

kopnono

进去ks-installer的pod

# kubectl exec -it ks-installer-85854b8c8-gsr8d -n kubesphere-system sh

在/kubesphere/kubesphere/common目录下找到了redis.yml，直接在容器里发布了redis

$ kubectl apply -f redis.yaml -n kubesphere-system

退出容器，查看redis资源

[root@master ~]# kubectl get pod -n kubesphere-system
NAME                                     READY   STATUS              RESTARTS   AGE
etcd-85c98fb695-ddqkm                    1/1     Running             0          24d
ks-apiserver-9cc65745-mq4tx              0/1     CrashLoopBackOff    22         91m
ks-console-fb4c655cf-r9xmr               1/1     Running             0          2d9h
ks-controller-manager-5f6c9b4d78-7rknn   1/1     Running             0          91m
ks-installer-85854b8c8-gsr8d             1/1     Running             0          94m
minio-764b67f6fb-fjwpl                   1/1     Running             0          31d
mysql-67cd66d5d-dg6gt                    1/1     Running             0          24d
openldap-0                               1/1     Running             0          4h31m
redis-6fd6c6d6f9-4knck                   0/1     ContainerCreating   0          7m50s

[root@master ~]# kubectl get pvc -n kubesphere-system
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
etcd-pvc                  Bound    pvc-65d8852b-319e-40f8-b4e1-a48f4c5dc4b6   20Gi       RWO            local          34d
minio                     Bound    pvc-d42a2f81-388b-45d1-a09d-d8fbdbb1c322   20Gi       RWO            local          34d
mysql-pvc                 Bound    pvc-5760a349-db4a-4c81-9d6d-5ef91d879695   20Gi       RWO            local          34d
openldap-pvc-openldap-0   Bound    pvc-585dc650-7220-422d-9638-5c3863d63bf4   2Gi        RWO            local          34d
redis-pvc                 Bound    pvc-ba3d57ec-3fc0-4394-8608-168081c544e0   2Gi        RWO            csi-rbd-sc     9m36s

[root@master ~]# kubectl get svc -n kubesphere-system
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
etcd                    ClusterIP   10.233.40.159   <none>        2379/TCP       34d
ks-apiserver            ClusterIP   10.233.37.111   <none>        80/TCP         34d
ks-console              NodePort    10.233.4.252    <none>        80:30880/TCP   34d
ks-controller-manager   ClusterIP   10.233.1.10     <none>        443/TCP        34d
minio                   ClusterIP   10.233.39.59    <none>        9000/TCP       34d
mysql                   ClusterIP   10.233.20.133   <none>        3306/TCP       34d
openldap                ClusterIP   None            <none>        389/TCP        34d
redis                   ClusterIP   10.233.38.229   <none>        6379/TCP       10m

redis的pod一直在创建中，openldap的clusterIP为什么是none?

kopnono

[root@master ~]# kubectl describe pod redis-6fd6c6d6f9-4knck -n kubesphere-system
Name:           redis-6fd6c6d6f9-4knck
Namespace:      kubesphere-system
Priority:       0
Node:           master/10.0.3.238
Start Time:     Mon, 08 Feb 2021 17:30:31 +0800
Labels:         app=redis
                pod-template-hash=6fd6c6d6f9
                tier=database
                version=redis-4.0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/redis-6fd6c6d6f9
Containers:
  redis:
    Container ID:   
    Image:          redis:5.0.5-alpine
    Image ID:       
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        20m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /data from redis-pvc (rw,path="redis-data")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mk7sx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  redis-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-pvc
    ReadOnly:   false
  default-token-mk7sx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mk7sx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Warning  FailedScheduling        <unknown>            default-scheduler        error while running "VolumeBinding" filter plugin for pod "redis-6fd6c6d6f9-4knck": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling        <unknown>            default-scheduler        error while running "VolumeBinding" filter plugin for pod "redis-6fd6c6d6f9-4knck": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               <unknown>            default-scheduler        Successfully assigned kubesphere-system/redis-6fd6c6d6f9-4knck to master
  Normal   SuccessfulAttachVolume  21m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ba3d57ec-3fc0-4394-8608-168081c544e0"
  Warning  FailedMount             5m15s (x4 over 18m)  kubelet, master          Unable to attach or mount volumes: unmounted volumes=[redis-pvc], unattached volumes=[default-token-mk7sx redis-pvc]: timed out waiting for the condition
  Warning  FailedMount             40s (x5 over 16m)    kubelet, master          Unable to attach or mount volumes: unmounted volumes=[redis-pvc], unattached volumes=[redis-pvc default-token-mk7sx]: timed out waiting for the condition
  Warning  FailedMount             33s (x18 over 21m)   kubelet, master          MountVolume.MountDevice failed for volume "pvc-ba3d57ec-3fc0-4394-8608-168081c544e0" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rbd.csi.ceph.com not found in the list of registered CSI drivers

看来和ceph-csi有关，继续排查

kopnono

因为ceph-rbd的数据卷挂载超时，所以先把默认存储类改成了local。

[root@master ~]# kubectl get sc -n kube-system
NAME                   PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc (default)   rbd.csi.ceph.com                                  Delete          Immediate              true                   20d
local                  openebs.io/local                                  Delete          WaitForFirstConsumer   false                  35d
nfs-client             cluster.local/nfs-client-nfs-client-provisioner   Delete          Immediate              true                   35d
[root@master ~]# kubectl edit sc csi-rbd-sc  -n kube-system
storageclass.storage.k8s.io/csi-rbd-sc edited
[root@master ~]# kubectl edit sc local  -n kube-system
storageclass.storage.k8s.io/local edited
[root@master ~]# kubectl get sc -n kube-system
NAME              PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc        rbd.csi.ceph.com                                  Delete          Immediate              true                   20d
local (default)   openebs.io/local                                  Delete          WaitForFirstConsumer   false                  35d
nfs-client        cluster.local/nfs-client-nfs-client-provisioner   Delete          Immediate              true                   35d

进入ks-installer删除redis,重新部署。

[root@master ~]# kubectl exec -it ks-installer-85854b8c8-gsr8d -n kubesphere-system sh
/kubesphere $ cd kubesphere/common
/kubesphere/kubesphere/common $ ls
etcd.yaml     mysql.yaml    password.txt  redis.yaml
/kubesphere/kubesphere/common $ kubectl delete -f redis.yaml 
persistentvolumeclaim "redis-pvc" deleted
deployment.apps "redis" deleted
service "redis" deleted
/kubesphere/kubesphere/common $ kubectl apply -f redis.yaml -n kubesphere-system
persistentvolumeclaim/redis-pvc created
deployment.apps/redis created
service/redis created
/kubesphere/kubesphere/common $ exit
[root@master ~]# kubectl get pod -n kubesphere-system
NAME                                     READY   STATUS    RESTARTS   AGE
etcd-85c98fb695-ddqkm                    1/1     Running   0          24d
ks-apiserver-9cc65745-mq4tx              1/1     Running   78         6h16m
ks-console-fb4c655cf-r9xmr               1/1     Running   0          2d14h
ks-controller-manager-5f6c9b4d78-7rknn   1/1     Running   0          6h16m
ks-installer-85854b8c8-gsr8d             1/1     Running   0          6h19m
minio-764b67f6fb-fjwpl                   1/1     Running   0          32d
mysql-67cd66d5d-dg6gt                    1/1     Running   0          24d
openldap-0                               1/1     Running   0          9h
redis-6fd6c6d6f9-5t4dj                   1/1     Running   0          45s

现在可以登录了，剩下就是解决cephfs存储巨量数据引起的一系列影响ceph性能的问题。