• 安装部署
  • kubesphere安装报错redis-ha-server-0启动失败

Ulysses-Z

可以先排查下redis和openldap的pvc挂盘是否正常




longhorn的pod启动都没问题,pvc没挂载上应该是

1 年 后

我遇到了相同的问题!我执行 kubectl -n kubesphere-system logs redis-ha-server-0 config-init 命令后输出如下:

    qumy

    kubectl describe 一下这个pod瞅瞅,是不是也是存储不正常 挂不上盘呀?

    • qumy 回复了此帖

      Cauchy

      kubectl get po -A

      kubectl describe po -n kubesphere-system redis-ha-server-0

      kubectl -n kubesphere-system logs redis-ha-server-0 config-init

      造成这个问题的原因:

      1、领导告知 k8s 上的系统不可用;

      2、检查发现 1个master 节点磁盘超 85%且存在2000多个Evicted状态的pod

      3、通过脚本删除Evicted状态的pod;

      4、发现redis-ha-server中第0个存在为题,第1、2个正常;

      5、通过ks面板调整redis-ha-server数量为0,ks面板被强制踢出,且无法再次登录;

      6、通过kubectl edit 调整 redis-ha-server有状态副本集数量;

      7、redis-ha-server-0还是有问题,且ks无法登陆;

      Cauchy

      相关版本:

      ks 版本:3.1.1

      安装方式:k8s 之上安装ks

      我也遇到这个问题,k8s v1.21.11 多master集群上安装kubesphere v3.2.1,ks-apiserver和redis-ha-server-0起不来

      这是在测试环境部署的,准备引入生产环境。

      帮忙看下,感谢!

      [root@ks-master3 heketi]# kubectl get pods -n kubesphere-system 
      NAME                                     READY   STATUS             RESTARTS   AGE
      ks-apiserver-5578d6cf6f-nvgck            0/1     CrashLoopBackOff   10         33m
      ks-apiserver-5578d6cf6f-qlwzp            0/1     CrashLoopBackOff   10         33m
      ks-apiserver-794b78d766-mxqxr            0/1     CrashLoopBackOff   10         33m
      ks-console-65f4d44d88-86lmg              1/1     Running            0          37m
      ks-console-65f4d44d88-8ffkr              1/1     Running            0          37m
      ks-console-65f4d44d88-t8pr8              1/1     Running            0          37m
      ks-controller-manager-7598dd7db7-2dwd4   1/1     Running            0          33m
      ks-controller-manager-7598dd7db7-fgdfx   1/1     Running            0          33m
      ks-controller-manager-7598dd7db7-jpwhk   1/1     Running            0          32m
      ks-installer-85dcfff87d-t6vsc            1/1     Running            0          48m
      redis-ha-haproxy-868fdbddd4-g6zj6        1/1     Running            0          40m
      redis-ha-haproxy-868fdbddd4-rbqv6        1/1     Running            0          40m
      redis-ha-haproxy-868fdbddd4-rtgms        1/1     Running            0          40m
      redis-ha-server-0                        0/2     Init:0/1           0          40m

      kubectl -n kubesphere-system describe pod redis-ha-server-0

      ...
      status 32
      Mounting command: systemd-run
      Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f4dc9c60-8742-45f5-9541-0f41c235d4d7/volumes/kubernetes.io~glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=192.168.142.138:192.168.142.139:192.168.142.96,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b/redis-ha-server-0-glusterfs.log,log-level=ERROR 192.168.142.96:vol_5dba7beb565c1bf41dc19563531d9b97 /var/lib/kubelet/pods/f4dc9c60-8742-45f5-9541-0f41c235d4d7/volumes/kubernetes.io~glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b
      Output: Running scope as unit run-31764.scope.
      mount: unknown filesystem type 'glusterfs'
      , the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod redis-ha-server-0
        Warning  FailedMount  10m (x18 over 38m)  kubelet  (combined from similar events): MountVolume.SetUp failed for volume "pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b" : mount failed: mount failed: exit status 32
      Mounting command: systemd-run
      Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f4dc9c60-8742-45f5-9541-0f41c235d4d7/volumes/kubernetes.io~glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=192.168.142.138:192.168.142.139:192.168.142.96,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b/redis-ha-server-0-glusterfs.log,log-level=ERROR 192.168.142.138:vol_5dba7beb565c1bf41dc19563531d9b97 /var/lib/kubelet/pods/f4dc9c60-8742-45f5-9541-0f41c235d4d7/volumes/kubernetes.io~glusterfs/pvc-e9c083df-bb41-4580-b89d-ec8edfdcd20b
      Output: Running scope as unit run-25511.scope.
      mount: unknown filesystem type 'glusterfs'
      , the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod redis-ha-server-0
        Warning  FailedMount  4m40s (x9 over 31m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[config data kube-api-access-w85dk]: timed out waiting for the condition
        Warning  FailedMount  7s (x5 over 38m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-w85dk config data]: timed out waiting for the condition

      kubectl -n kubesphere-system logs -f ks-apiserver-5578d6cf6f-nvgck

      [root@ks-master1 kubesphere]#  kubectl -n kubesphere-system  logs -f  ks-apiserver-5578d6cf6f-nvgck 
      W0325 10:19:02.245683       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
      W0325 10:19:02.253978       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
      W0325 10:19:02.280574       1 metricsserver.go:238] Metrics API not available.
      Error: failed to connect to redis service, please check redis status, error: EOF
      2022/03/25 10:19:02 failed to connect to redis service, please check redis status, error: EOF

        peteradmin

        ks-apiserver 起不来是因为redis没起来,redis没起来是因为集群里存储不正常,看起来这个集群里的存储是glusterfs,建议先把存储调好,这些服务应该就可以正常运行了。

          peteradmin 你应该检查一下你的存储系统:unknown filesystem type ‘glusterfs’

          5 个月 后

          Cauchy

          [upl-image-preview url=

          大佬可以帮我看下么,我是用kk离线装的k8s和kubesphere,k8s没有问题,但是kubesphere的redis一直起不来,不知道怎么回事]

            6 天 后