• 安装部署
  • v4.1.2 离线部署高可用时 ks-apiserver 部署失败

创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
发帖前请点击 发表主题 右边的 预览(👀) 按钮,确保帖子格式正确。

操作系统信息
虚拟机,Centos7.9,4C/16G

Kubernetes版本信息
v1.28.15 3节点 高可用安装

容器运行时
containerd 1.7.13

KubeSphere版本信息
v4.1.2 离线安装 高可用安装

kubekey 版本

v3.1.7

问题:离线部署好k8s集群,后在高可用部署kubesphere 4.1.2的时候ks-apiserver时报错

config-sample.yaml配置文件 :internalLoadbalancer: haproxy 开启

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: 192-168-11-84, address: 192.168.11.84, internalAddress: 192.168.11.84, user: root, password: "Testing@123"}
  - {name: 192-168-11-85, address: 192.168.11.85, internalAddress: 192.168.11.85, user: root, password: "Testing@123"}
  - {name: 192-168-11-86, address: 192.168.11.86, internalAddress: 192.168.11.86, user: root, password: "Testing@123"}
  - {name: 192-168-11-88, address: 192.168.11.88, internalAddress: 192.168.11.88, user: root, password: "Testing@123"}
  roleGroups:
    etcd:
    - 192-168-11-84
    - 192-168-11-85
    - 192-168-11-86
    control-plane: 
    - 192-168-11-84
    - 192-168-11-85
    - 192-168-11-86
    worker:
    - 192-168-11-84
    - 192-168-11-85
    - 192-168-11-86
    registry:
    - 192-168-11-88
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.28.15
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    type: harbor
    auths:
      "dockerhub.kubekey.local":
        username: admin
        password: Harbor12345
        skipTLSVerify: true
    privateRegistry: "dockerhub.kubekey.local"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
  addons: []

离线部署k8s代码:./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-local-storage

离线部署k8s状态: 成功

离线部署kubesphere代码:

如需高可用部署 KubeSphere,可在命令中添加 --set ha.enabled=true,redisHA.enabled=true

只要添加了 --set ha.enabled=true,redisHA.enabled=true 就会报错,去掉的话部署成功

helm upgrade --install -n kubesphere-system --create-namespace ks-core ks-core-1.1.3.tgz \
     --set global.imageRegistry=dockerhub.kubekey.local/ks \
     --set extension.imageRegistry=dockerhub.kubekey.local/ks \
     --set ksExtensionRepository.image.tag=v1.1.2 \
     --debug \
     --wait \
     --set ha.enabled=true,redisHA.enabled=true

离线部署kubesphere状态:报错,

错误截图如下:

检查下存储吧,看看有没有 storageclass,有的话 看看 kube-system 里边 openebs-localpv-provisioner 的状态

    Cauchy

    pod状态:

    kube-system名称空间下的openebs-localpv-provisioner运行为running

    [root@192-168-11-85 ~]# kubectl get pods -A
    NAMESPACE           NAME                                            READY   STATUS             RESTARTS      AGE
    kube-system         calico-kube-controllers-69449756dd-qvt4r        1/1     Running            1 (21h ago)   46h
    kube-system         calico-node-2mdg7                               1/1     Running            0             46h
    kube-system         calico-node-6bzq5                               1/1     Running            0             46h
    kube-system         calico-node-httch                               1/1     Running            0             46h
    kube-system         coredns-8477f66f9d-h44wk                        1/1     Running            0             46h
    kube-system         coredns-8477f66f9d-t284l                        1/1     Running            0             46h
    kube-system         init-pvc-959ea330-6369-48b0-b814-ebea19ae3bbc   0/1     Completed          0             46h
    kube-system         kube-apiserver-192-168-11-84                    1/1     Running            1 (21h ago)   46h
    kube-system         kube-apiserver-192-168-11-85                    1/1     Running            1 (21h ago)   46h
    kube-system         kube-apiserver-192-168-11-86                    1/1     Running            1 (21h ago)   46h
    kube-system         kube-controller-manager-192-168-11-84           1/1     Running            2 (21h ago)   46h
    kube-system         kube-controller-manager-192-168-11-85           1/1     Running            1 (19h ago)   46h
    kube-system         kube-controller-manager-192-168-11-86           1/1     Running            2 (21h ago)   46h
    kube-system         kube-proxy-7c56q                                1/1     Running            0             46h
    kube-system         kube-proxy-brtdd                                1/1     Running            0             46h
    kube-system         kube-proxy-vn75c                                1/1     Running            0             46h
    kube-system         kube-scheduler-192-168-11-84                    1/1     Running            1 (46h ago)   46h
    kube-system         kube-scheduler-192-168-11-85                    1/1     Running            2 (19h ago)   46h
    kube-system         kube-scheduler-192-168-11-86                    1/1     Running            3 (19h ago)   46h
    kube-system         nodelocaldns-hpm7r                              1/1     Running            0             46h
    kube-system         nodelocaldns-jdx97                              1/1     Running            1             46h
    kube-system         nodelocaldns-qfgr6                              1/1     Running            0             46h
    kube-system         openebs-localpv-provisioner-68bd976677-tk4hs    1/1     Running            4 (19h ago)   46h
    kubesphere-system   extensions-museum-757866db87-tp9xk              1/1     Running            0             3m8s
    kubesphere-system   ks-apiserver-5cddf58865-4dcfx                   0/1     Error              5 (89s ago)   3m8s
    kubesphere-system   ks-apiserver-5cddf58865-7tl49                   0/1     CrashLoopBackOff   4 (84s ago)   3m8s
    kubesphere-system   ks-apiserver-5cddf58865-c7cq4                   0/1     CrashLoopBackOff   5 (11s ago)   3m8s
    kubesphere-system   ks-console-69647dc987-2tfjp                     1/1     Running            0             3m8s
    kubesphere-system   ks-console-69647dc987-2xhvm                     1/1     Running            0             3m8s
    kubesphere-system   ks-console-69647dc987-nc2st                     1/1     Running            0             3m8s
    kubesphere-system   ks-controller-manager-dffb6c49f-cwqw4           1/1     Running            0             3m8s
    kubesphere-system   ks-controller-manager-dffb6c49f-j9q8n           1/1     Running            0             3m8s
    kubesphere-system   ks-controller-manager-dffb6c49f-nsst7           1/1     Running            0             3m8s
    kubesphere-system   ks-core-redisha-haproxy-8486ff5fcf-n9wr8        1/1     Running            0             3m8s
    kubesphere-system   ks-core-redisha-haproxy-8486ff5fcf-s8fcs        1/1     Running            0             3m8s
    kubesphere-system   ks-core-redisha-haproxy-8486ff5fcf-thxxw        1/1     Running            0             3m8s
    kubesphere-system   ks-core-redisha-server-0                        0/3     Pending            0             3m8s

    sc状态:

    [root@192-168-11-85 ~]# kubectl get sc -A
    NAME              PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    local (default)   openebs.io/local   Delete          WaitForFirstConsumer   false                  46h

    报错信息:

    [root@192-168-11-85 ~]# kubectl describe -n kubesphere-system pods ks-apiserver-5cddf58865-c7cq4
    Name:             ks-apiserver-5cddf58865-c7cq4
    Namespace:        kubesphere-system
    Priority:         0
    Service Account:  kubesphere
    Node:             192-168-11-84/192.168.11.84
    Start Time:       Wed, 27 Nov 2024 14:29:58 +0800
    Labels:           app=ks-apiserver
                      pod-template-hash=5cddf58865
                      tier=backend
    Annotations:      checksum/config: 38889a4f3131a4f427343227c459c8e8a28793538fd2c805ff71cafbe5f7eb93
                      cni.projectcalico.org/containerID: a6f508b6f612138fa57ae4d317d96f81dda678e0dc5a6f768247cf8d7f692729
                      cni.projectcalico.org/podIP: 10.233.104.17/32
                      cni.projectcalico.org/podIPs: 10.233.104.17/32
    Status:           Running
    IP:               10.233.104.17
    IPs:
      IP:           10.233.104.17
    Controlled By:  ReplicaSet/ks-apiserver-5cddf58865
    Containers:
      ks-apiserver:
        Container ID:  containerd://cdee2e0422c4af31038ec51f69c48dea0c4fd6f7e6ccdef6fa47a6814a7a46a5
        Image:         dockerhub.kubekey.local/ks/kubesphere/ks-apiserver:v4.1.2
        Image ID:      dockerhub.kubekey.local/ks/kubesphere/ks-apiserver@sha256:2294402448f1e5561fbe1c0862ad2a8eaab5860e5f1284df7876ed1b6e1fc3d7
        Port:          9090/TCP
        Host Port:     0/TCP
        Command:
          ks-apiserver
          --logtostderr=true
        State:          Waiting
          Reason:       CrashLoopBackOff
        Last State:     Terminated
          Reason:       Error
          Exit Code:    1
          Started:      Wed, 27 Nov 2024 14:35:47 +0800
          Finished:     Wed, 27 Nov 2024 14:35:48 +0800
        Ready:          False
        Restart Count:  6
        Limits:
          cpu:     1
          memory:  1Gi
        Requests:
          cpu:     20m
          memory:  100Mi
        Liveness:  http-get http://:9090/version delay=15s timeout=15s period=10s #success=1 #failure=8
        Environment:
          KUBESPHERE_CACHE_OPTIONS_PASSWORD:  <set to the key 'auth' in secret 'redis-secret'>  Optional: false
        Mounts:
          /etc/kubesphere/ from kubesphere-config (rw)
          /etc/localtime from host-time (ro)
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j56px (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             False 
      ContainersReady   False 
      PodScheduled      True 
    Volumes:
      kubesphere-config:
        Type:      ConfigMap (a volume populated by a ConfigMap)
        Name:      kubesphere-config
        Optional:  false
      host-time:
        Type:          HostPath (bare host directory volume)
        Path:          /etc/localtime
        HostPathType:  
      kube-api-access-j56px:
        Type:                    Projected (a volume that contains injected data from multiple sources)
        TokenExpirationSeconds:  3607
        ConfigMapName:           kube-root-ca.crt
        ConfigMapOptional:       <nil>
        DownwardAPI:             true
    QoS Class:                   Burstable
    Node-Selectors:              <none>
    Tolerations:                 CriticalAddonsOnly op=Exists
                                 node-role.kubernetes.io/control-plane:NoSchedule
                                 node-role.kubernetes.io/master:NoSchedule
                                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
      Type     Reason     Age                    From               Message
      ----     ------     ----                   ----               -------
      Normal   Scheduled  6m42s                  default-scheduler  Successfully assigned kubesphere-system/ks-apiserver-5cddf58865-c7cq4 to 192-168-11-84
      Normal   Pulled     5m12s (x5 over 6m41s)  kubelet            Container image "dockerhub.kubekey.local/ks/kubesphere/ks-apiserver:v4.1.2" already present on machine
      Normal   Created    5m12s (x5 over 6m41s)  kubelet            Created container ks-apiserver
      Normal   Started    5m12s (x5 over 6m40s)  kubelet            Started container ks-apiserver
      Warning  BackOff    98s (x27 over 6m38s)   kubelet            Back-off restarting failed container ks-apiserver in pod ks-apiserver-5cddf58865-c7cq4_kubesphere-system(199c164a-b913-4ecd-9ad5-c4df85ad0d50)

    ks-apiserver pod日志信息:

    [root@192-168-11-85 ~]# kubectl logs -n kubesphere-system ks-apiserver-5cddf58865-c7cq4
    W1127 14:41:00.080967       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
    E1127 14:41:00.125376       1 cache.go:53] failed to create cache, error: EOF
    E1127 14:41:00.125668       1 run.go:74] "command failed" err="failed to create cache, error: EOF"

    ks-core-redisha-server报错

    [root@192-168-11-84 412]# kubectl describe -n kubesphere-system pods ks-core-redisha-server-0  
    Name:             ks-core-redisha-server-0
    Namespace:        kubesphere-system
    Priority:         0
    Service Account:  ks-core-redisha
    Node:             <none>
    Labels:           app=redisHA
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=ks-core-redisha-server-9d84bf6df
                      ks-core-redisha=replica
                      release=ks-core
                      statefulset.kubernetes.io/pod-name=ks-core-redisha-server-0
    Annotations:      checksum/init-config: b575128b781c88419615117734b100b0e22315ae9c22364b766842d127472c0a
    Status:           Pending
    IP:               
    IPs:              <none>
    Controlled By:    StatefulSet/ks-core-redisha-server
    Init Containers:
      config-init:
        Image:           dockerhub.kubekey.local/ks/kubesphere/redis:7.2.4-alpine
        Port:            <none>
        Host Port:       <none>
        SeccompProfile:  RuntimeDefault
        Command:
          sh
        Args:
          /readonly-config/init.sh
        Environment:
          SENTINEL_ID_0:  86d4ca8ed5d49dc46f3d4eed6209b41a17e7be00
          SENTINEL_ID_1:  345bd3dd364ff9c38af3bffc5a6bcf2f2b366bf6
          SENTINEL_ID_2:  39e7ff69e1b2542e3b9b122c4ce1275d40d3937c
          AUTH:           <set to the key 'auth' in secret 'redis-secret'>  Optional: false
        Mounts:
          /data from data (rw)
          /readonly-config from config (ro)
    Containers:
      redis:
        Image:           dockerhub.kubekey.local/ks/kubesphere/redis:7.2.4-alpine
        Port:            6379/TCP
        Host Port:       0/TCP
        SeccompProfile:  RuntimeDefault
        Command:
          redis-server
        Args:
          /data/conf/redis.conf
        Liveness:   exec [sh -c /health/redis_liveness.sh] delay=30s timeout=15s period=15s #success=1 #failure=5
        Readiness:  exec [sh -c /health/redis_readiness.sh] delay=30s timeout=15s period=15s #success=1 #failure=5
        Environment:
          AUTH:  <set to the key 'auth' in secret 'redis-secret'>  Optional: false
        Mounts:
          /data from data (rw)
          /health from health (rw)
          /readonly-config from config (ro)
      sentinel:
        Image:           dockerhub.kubekey.local/ks/kubesphere/redis:7.2.4-alpine
        Port:            26379/TCP
        Host Port:       0/TCP
        SeccompProfile:  RuntimeDefault
        Command:
          redis-sentinel
        Args:
          /data/conf/sentinel.conf
        Liveness:   exec [sh -c /health/sentinel_liveness.sh] delay=30s timeout=15s period=15s #success=1 #failure=5
        Readiness:  exec [sh -c /health/sentinel_liveness.sh] delay=30s timeout=15s period=15s #success=3 #failure=5
        Environment:
          AUTH:  <set to the key 'auth' in secret 'redis-secret'>  Optional: false
        Mounts:
          /data from data (rw)
          /health from health (rw)
      split-brain-fix:
        Image:           dockerhub.kubekey.local/ks/kubesphere/redis:7.2.4-alpine
        Port:            <none>
        Host Port:       <none>
        SeccompProfile:  RuntimeDefault
        Command:
          sh
        Args:
          /readonly-config/fix-split-brain.sh
        Environment:
          SENTINEL_ID_0:  86d4ca8ed5d49dc46f3d4eed6209b41a17e7be00
          SENTINEL_ID_1:  345bd3dd364ff9c38af3bffc5a6bcf2f2b366bf6
          SENTINEL_ID_2:  39e7ff69e1b2542e3b9b122c4ce1275d40d3937c
          AUTH:           <set to the key 'auth' in secret 'redis-secret'>  Optional: false
        Mounts:
          /data from data (rw)
          /readonly-config from config (ro)
    Conditions:
      Type           Status
      PodScheduled   False 
    Volumes:
      data:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  data-ks-core-redisha-server-0
        ReadOnly:   false
      config:
        Type:      ConfigMap (a volume populated by a ConfigMap)
        Name:      ks-core-redisha-configmap
        Optional:  false
      health:
        Type:        ConfigMap (a volume populated by a ConfigMap)
        Name:        ks-core-redisha-health-configmap
        Optional:    false
    QoS Class:       BestEffort
    Node-Selectors:  <none>
    Tolerations:     CriticalAddonsOnly op=Exists
                     node-role.kubernetes.io/control-plane:NoSchedule
                     node-role.kubernetes.io/master:NoSchedule
                     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                     node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
      Type     Reason            Age    From               Message
      ----     ------            ----   ----               -------
      Warning  FailedScheduling  15m    default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
      Warning  FailedScheduling  5m38s  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

    已经解决了,是pvc创建的错误,将 init-pvc 删除后,重新执行 helm安装 kubesphere即可