KubeSphere 3.1 HA 部署之后无法完成HA测试

klaylin

部署

使用KubeKey部署3 masters + 2 workers的HA集群部署
以下是部分配置

kubernetes:
    version: v1.20.4
    imageRepo: kubesphere
    clusterName: cluster.local
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18

部署完成之后可以正常使用
##测试
shutdown VIP 所在的master节点，能够正常使用kubectl命令查看创建资源，可以登录到KubeSphere管理页面，但是无法查看具体资源
kubectl查看资源

   root@km99:~# kubectl get node
NAME   STATUS     ROLES                  AGE   VERSION
k8s1   NotReady   control-plane,master   11d   v1.20.4
k8s2   Ready      control-plane,master   11d   v1.20.4
k8s3   Ready      control-plane,master   11d   v1.20.4
k8s4   Ready      worker                 11d   v1.20.4
k8s5   Ready      worker                 11d   v1.20.4
root@km99:~# kubectl get all -A
NAMESPACE                      NAME                                                   READY   STATUS             RESTARTS   AGE
default                        pod/minio-ha-5d854c85f4-g9782                          1/1     Running            0          2d19h
default                        pod/ng2-5c4c569779-42px4                               1/1     Running            0          11d
default                        pod/ng2-5c4c569779-gxtcc                               1/1     Running            0          11d
kube-system                    pod/coredns-65944cbcb8-8d5zz                           1/1     Terminating        0          2d18h
kube-system                    pod/coredns-65944cbcb8-fnrwm                           1/1     Running            0          2d18h
kube-system                    pod/coredns-65944cbcb8-rjjs8                           0/1     Running            0          2d17h
kube-system                    pod/kube-apiserver-k8s1                                1/1     Running            2          11d
kube-system                    pod/kube-apiserver-k8s2                                1/1     Running            4          11d
kube-system                    pod/kube-apiserver-k8s3                                1/1     Running            1          11d
kube-system                    pod/kube-controller-manager-k8s1                       1/1     Running            3          11d
kube-system                    pod/kube-controller-manager-k8s2                       1/1     Running            6          11d
kube-system                    pod/kube-controller-manager-k8s3                       1/1     Running            3          11d
kube-system                    pod/kube-flannel-ds-82rg9                              1/1     Running            10         11d
kube-system                    pod/kube-flannel-ds-b8n5r                              1/1     Running            3          11d
kube-system                    pod/kube-flannel-ds-k96q9                              1/1     Running            1          11d
kube-system                    pod/kube-flannel-ds-pz7xc                              1/1     Running            1          11d
kube-system                    pod/kube-flannel-ds-qq6d4                              1/1     Running            2          11d
kube-system                    pod/kube-proxy-9mkq2                                   1/1     Running            4          11d
kube-system                    pod/kube-proxy-d69t9                                   1/1     Running            1          11d
kube-system                    pod/kube-proxy-dt8tq                                   1/1     Running            2          11d
kube-system                    pod/kube-proxy-j22wl                                   1/1     Running            1          11d
kube-system                    pod/kube-proxy-t9c89                                   1/1     Running            3          11d
kube-system                    pod/kube-scheduler-k8s1                                1/1     Running            3          11d
kube-system                    pod/kube-scheduler-k8s2                                1/1     Running            5          11d
kube-system                    pod/kube-scheduler-k8s3                                1/1     Running            4          11d
kube-system                    pod/linstor-csi-controller-0                           5/5     Running            0          2d19h
kube-system                    pod/linstor-csi-node-tdr7d                             2/2     Running            0          4d19h
kube-system                    pod/linstor-csi-node-twwm9                             2/2     Running            4          4d19h
kube-system                    pod/nodelocaldns-fsxh8                                 1/1     Running            1          11d
kube-system                    pod/nodelocaldns-gvngp                                 1/1     Running            1          11d
kube-system                    pod/nodelocaldns-mfx4s                                 1/1     Running            4          11d
kube-system                    pod/nodelocaldns-n7s2m                                 1/1     Running            3          11d
kube-system                    pod/nodelocaldns-nhkxt                                 1/1     Running            2          11d
kube-system                    pod/openebs-localpv-provisioner-7cfc686bc5-lr6jb       0/1     CrashLoopBackOff   763        11d
kube-system                    pod/snapshot-controller-0                              1/1     Running            0          11d
kubesphere-controls-system     pod/default-http-backend-76d9fb4bb7-zrcpm              1/1     Running            0          11d
kubesphere-controls-system     pod/kubectl-admin-7b69cb97d5-ksdgh                     1/1     Running            0          11d
kubesphere-monitoring-system   pod/alertmanager-main-0                                2/2     Running            0          2d19h
kubesphere-monitoring-system   pod/alertmanager-main-1                                2/2     Running            0          11d
kubesphere-monitoring-system   pod/alertmanager-main-2                                2/2     Running            0          2d19h
kubesphere-monitoring-system   pod/kube-state-metrics-687c7c4d86-8xtwl                3/3     Running            0          11d
kubesphere-monitoring-system   pod/node-exporter-6jhgd                                2/2     Running            0          11d
kubesphere-monitoring-system   pod/node-exporter-7cdz2                                2/2     Running            2          11d
kubesphere-monitoring-system   pod/node-exporter-88wmg                                2/2     Running            6          11d
kubesphere-monitoring-system   pod/node-exporter-bltwd                                2/2     Running            4          11d
kubesphere-monitoring-system   pod/node-exporter-kpbqq                                2/2     Running            0          11d
kubesphere-monitoring-system   pod/notification-manager-deployment-7bd887ffb4-cglzx   1/1     Running            0          11d
kubesphere-monitoring-system   pod/notification-manager-deployment-7bd887ffb4-v88ql   1/1     Running            0          2d20h
kubesphere-monitoring-system   pod/notification-manager-operator-78595d8666-7rhz4     1/2     CrashLoopBackOff   767        11d
kubesphere-monitoring-system   pod/prometheus-k8s-0                                   3/3     Running            1          11d
kubesphere-monitoring-system   pod/prometheus-k8s-1                                   3/3     Running            1          2d19h
kubesphere-monitoring-system   pod/prometheus-operator-d7fdfccbf-qltt8                2/2     Running            0          2d20h
kubesphere-system              pod/ks-apiserver-5b797944-2976m                        1/1     Running            0          4d18h
kubesphere-system              pod/ks-apiserver-5b797944-nk69v                        0/1     CrashLoopBackOff   766        2d17h
kubesphere-system              pod/ks-apiserver-5b797944-rqhgh                        1/1     Terminating        6          2d18h
kubesphere-system              pod/ks-apiserver-5b797944-vnsjh                        1/1     Running            0          4d18h
kubesphere-system              pod/ks-console-7877478685-46s74                        1/1     Running            0          2d17h
kubesphere-system              pod/ks-console-7877478685-4bhm9                        1/1     Running            0          2d19h
kubesphere-system              pod/ks-console-7877478685-qd4bh                        1/1     Running            0          11d
kubesphere-system              pod/ks-console-7877478685-x2bs7                        1/1     Terminating        0          2d18h
kubesphere-system              pod/ks-controller-manager-858c95f978-br8t5             1/1     Terminating        0          2d18h
kubesphere-system              pod/ks-controller-manager-858c95f978-c4x5f             1/1     Running            13         4d18h
kubesphere-system              pod/ks-controller-manager-858c95f978-hv2xx             0/1     CrashLoopBackOff   786        2d19h
kubesphere-system              pod/ks-controller-manager-858c95f978-pbclm             0/1     CrashLoopBackOff   774        2d17h
kubesphere-system              pod/ks-installer-5d65c99d54-9dhr2                      1/1     Running            0          11d
kubesphere-system              pod/redis-ha-haproxy-68ff446d79-g2h9c                  1/1     Running            0          2d19h
kubesphere-system              pod/redis-ha-haproxy-68ff446d79-j8cmn                  1/1     Terminating        0          2d18h
kubesphere-system              pod/redis-ha-haproxy-68ff446d79-qj2gc                  1/1     Running            0          11d
kubesphere-system              pod/redis-ha-haproxy-68ff446d79-zms9b                  1/1     Running            0          2d17h
kubesphere-system              pod/redis-ha-server-0                                  2/2     Terminating        0          2d18h
kubesphere-system              pod/redis-ha-server-1                                  2/2     Running            0          11d
kubesphere-system              pod/redis-ha-server-2                                  2/2     Running            0          11d

NAMESPACE                      NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
default                        service/kubernetes                                ClusterIP   10.233.0.1      <none>        443/TCP                        11d
default                        service/minio-ha                                  NodePort    10.233.6.226    <none>        9000:30539/TCP                 4d16h
default                        service/ng2                                       NodePort    10.233.11.34    <none>        80:32060/TCP                   11d
kube-system                    service/coredns                                   ClusterIP   10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP         11d
kube-system                    service/kube-controller-manager-svc               ClusterIP   None            <none>        10257/TCP                      11d
kube-system                    service/kube-scheduler-svc                        ClusterIP   None            <none>        10259/TCP                      11d
kube-system                    service/kubelet                                   ClusterIP   None            <none>        10250/TCP,10255/TCP,4194/TCP   11d
kubesphere-controls-system     service/default-http-backend                      ClusterIP   10.233.8.179    <none>        80/TCP                         11d
kubesphere-monitoring-system   service/alertmanager-main                         ClusterIP   10.233.3.185    <none>        9093/TCP                       11d
kubesphere-monitoring-system   service/alertmanager-operated                     ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP     11d
kubesphere-monitoring-system   service/kube-state-metrics                        ClusterIP   None            <none>        8443/TCP,9443/TCP              11d
kubesphere-monitoring-system   service/node-exporter                             ClusterIP   None            <none>        9100/TCP                       11d
kubesphere-monitoring-system   service/notification-manager-controller-metrics   ClusterIP   10.233.7.212    <none>        8443/TCP                       11d
kubesphere-monitoring-system   service/notification-manager-svc                  ClusterIP   10.233.56.33    <none>        19093/TCP                      11d
kubesphere-monitoring-system   service/prometheus-k8s                            ClusterIP   10.233.54.100   <none>        9090/TCP                       11d
kubesphere-monitoring-system   service/prometheus-operated                       ClusterIP   None            <none>        9090/TCP                       11d
kubesphere-monitoring-system   service/prometheus-operator                       ClusterIP   None            <none>        8443/TCP                       11d
kubesphere-system              service/ks-apiserver                              ClusterIP   10.233.50.243   <none>        80/TCP                         11d
kubesphere-system              service/ks-console                                NodePort    10.233.51.171   <none>        80:30880/TCP                   11d
kubesphere-system              service/ks-controller-manager                     ClusterIP   10.233.62.183   <none>        443/TCP                        11d
kubesphere-system              service/redis                                     ClusterIP   10.233.23.145   <none>        6379/TCP                       11d
kubesphere-system              service/redis-ha                                  ClusterIP   None            <none>        6379/TCP,26379/TCP             11d
kubesphere-system              service/redis-ha-announce-0                       ClusterIP   10.233.44.240   <none>        6379/TCP,26379/TCP             11d
kubesphere-system              service/redis-ha-announce-1                       ClusterIP   10.233.22.58    <none>        6379/TCP,26379/TCP             11d
kubesphere-system              service/redis-ha-announce-2                       ClusterIP   10.233.34.142   <none>        6379/TCP,26379/TCP             11d

NAMESPACE                      NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system                    daemonset.apps/kube-flannel-ds    5         5         4       5            4           <none>                   11d
kube-system                    daemonset.apps/kube-proxy         5         5         4       5            4           kubernetes.io/os=linux   11d
kube-system                    daemonset.apps/linstor-csi-node   2         2         2       2            2           <none>                   4d19h
kube-system                    daemonset.apps/nodelocaldns       5         5         4       5            4           <none>                   11d
kubesphere-monitoring-system   daemonset.apps/node-exporter      5         5         4       5            4           kubernetes.io/os=linux   11d

NAMESPACE                      NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
default                        deployment.apps/minio-ha                          1/1     1            1           2d19h
default                        deployment.apps/ng2                               2/2     2            2           11d
kube-system                    deployment.apps/coredns                           1/2     2            1           11d
kube-system                    deployment.apps/openebs-localpv-provisioner       0/1     1            0           11d
kubesphere-controls-system     deployment.apps/default-http-backend              1/1     1            1           11d
kubesphere-controls-system     deployment.apps/kubectl-admin                     1/1     1            1           11d
kubesphere-monitoring-system   deployment.apps/kube-state-metrics                1/1     1            1           11d
kubesphere-monitoring-system   deployment.apps/notification-manager-deployment   2/2     2            2           11d
kubesphere-monitoring-system   deployment.apps/notification-manager-operator     0/1     1            0           11d
kubesphere-monitoring-system   deployment.apps/prometheus-operator               1/1     1            1           11d
kubesphere-system              deployment.apps/ks-apiserver                      2/3     3            2           11d
kubesphere-system              deployment.apps/ks-console                        3/3     3            3           11d
kubesphere-system              deployment.apps/ks-controller-manager             1/3     3            1           11d
kubesphere-system              deployment.apps/ks-installer                      1/1     1            1           11d
kubesphere-system              deployment.apps/redis-ha-haproxy                  3/3     3            3           11d

NAMESPACE                      NAME                                                         DESIRED   CURRENT   READY   AGE
default                        replicaset.apps/minio-ha-5d854c85f4                          1         1         1       2d19h
default                        replicaset.apps/ng2-5c4c569779                               2         2         2       11d
kube-system                    replicaset.apps/coredns-65944cbcb8                           2         2         1       11d
kube-system                    replicaset.apps/openebs-localpv-provisioner-7cfc686bc5       1         1         0       11d
kubesphere-controls-system     replicaset.apps/default-http-backend-76d9fb4bb7              1         1         1       11d
kubesphere-controls-system     replicaset.apps/kubectl-admin-7b69cb97d5                     1         1         1       11d
kubesphere-monitoring-system   replicaset.apps/kube-state-metrics-687c7c4d86                1         1         1       11d
kubesphere-monitoring-system   replicaset.apps/notification-manager-deployment-7bd887ffb4   2         2         2       11d
kubesphere-monitoring-system   replicaset.apps/notification-manager-operator-78595d8666     1         1         0       11d
kubesphere-monitoring-system   replicaset.apps/prometheus-operator-d7fdfccbf                1         1         1       11d
kubesphere-system              replicaset.apps/ks-apiserver-5b797944                        3         3         2       4d18h
kubesphere-system              replicaset.apps/ks-apiserver-5f87f4dc7c                      0         0         0       11d
kubesphere-system              replicaset.apps/ks-apiserver-746bf8d8c8                      0         0         0       11d
kubesphere-system              replicaset.apps/ks-apiserver-74d7ccbccd                      0         0         0       11d
kubesphere-system              replicaset.apps/ks-apiserver-78f7c44667                      0         0         0       4d18h
kubesphere-system              replicaset.apps/ks-console-7877478685                        3         3         3       11d
kubesphere-system              replicaset.apps/ks-controller-manager-667bc699b5             0         0         0       11d
kubesphere-system              replicaset.apps/ks-controller-manager-6b549d7bc5             0         0         0       4d18h
kubesphere-system              replicaset.apps/ks-controller-manager-74fb49f447             0         0         0       11d
kubesphere-system              replicaset.apps/ks-controller-manager-7dff69f4c7             0         0         0       11d
kubesphere-system              replicaset.apps/ks-controller-manager-858c95f978             3         3         1       4d18h
kubesphere-system              replicaset.apps/ks-installer-5d65c99d54                      1         1         1       11d
kubesphere-system              replicaset.apps/redis-ha-haproxy-68ff446d79                  3         3         3       11d

NAMESPACE                      NAME                                      READY   AGE
kube-system                    statefulset.apps/linstor-csi-controller   1/1     4d19h
kube-system                    statefulset.apps/snapshot-controller      1/1     11d
kubesphere-monitoring-system   statefulset.apps/alertmanager-main        3/3     11d
kubesphere-monitoring-system   statefulset.apps/prometheus-k8s           2/2     11d
kubesphere-system              statefulset.apps/redis-ha-server          2/3     11d

进入KubeSphere界面后查看资源

ruiyaoOps

高可用是需要自己设置的，不是三个节点就行，文档见https://kubesphere.com.cn/docs/installing-on-linux/high-availability-configurations/ha-configuration/

klaylin

ruiyaoOps 已经用Keepalived和haproxy做负载均衡器了，所以shutdown vip
当前所在节点可以保证kubectl能继续使用，就是KubeSphere有点问题

ruiyaoOps

那你看看 shutdown vip 当前所在节点后，各节点访问10.233.0.1:443 端口是不是通的，这个报错是集群内部的kube-apiserver的svc访问不到，看节点上有没有开防火墙、安全组啥的，一般都是防火墙的锅

klaylin

ruiyaoOps 防火墙都是一开始就关闭的。。

ruiyaoOps

你看下你的 default nm 有没有一个名为 kubernetes 的 svc, 这个 svc ip 端口是不是 10.233.0.1:443，然后试试是不是每个节点都能访问到这个ip 端口

klaylin

ruiyaoOps 只要shutdown 当前 VIP所在节点就访问不了10.233.0.1:443

klaylin

ruiyaoOps 查看svc发现这个endpoint一直是shutdown的那个节点的IP

root@km99:~# kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Families:       <none>
IP:                10.233.0.1
IPs:               10.233.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.203.1.82:6443
Session Affinity:  None
Events:            <none>

能不能修改这个endpoint变成VIP加端口

klaylin

应该是shutdown节点之后，节点上的api-server Pod还是处于running的状态，所以svc
不会调endpoint到其他master节点，不知道为什么节点shutdown，上面的pod
还是running的

klaylin

还是这个svc的endpoint只有一个master的

kevendeng

创建KS集群时，是否在config.yaml里将.spec.controlPlaneEndpoint.address设为Keepalived的VIP？

如果正确设置了这个值，kubernetes service 的 endpoint应该就是VIP，shutdown某个master不会导致服务不可达。

至于Node已NotReady，但Pod未被驱逐，是由于K8s控制面的Pod都设置了通用的toleration，可以通过以下命令看到：

kubectl get pod -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].spec.tolerations}'

klaylin

kevendeng 谢谢回复，.spec.controlPlaneEndpoint.address 设定的一直是 VIP 的值，关于Node已NotReady，但Pod未被驱逐，通过命令查看到是这样
kubectl get pod -n kube-system -l component=kube-apiserver -o jsonpath=‘{.items[0].spec.tolerations}’

[{“effect”:“NoExecute”,“operator”:“Exists”}]

24sama

kk用的是什么版本呢？

./kk version

现在可以尝试编辑一下 kubeadm-config的configmap:

kubectl edit cm -n kube-system kubeadm-config

如果文件中有下面这行，就删了：

apiServer:
  extraArgs:
    advertise-address: {{ 猜测是master1的地址}} // 删除该行

并且在每台master节点上，修改/etc/kubernetes/kube-apiserver.yaml文件：

metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.1:6443 // 改成vip地址
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.1 // 改成当vip地址

klaylin

24sama 谢谢回复，太久没登陆了
之前通过修改每个master节点上的 /etc/kubernetes/manifests/kube-apiserver.yaml 里的advertise-address 为本地节点的地址，解决了
后续可以按照你这边的方法检查一下看看