1、部署方式是3master多node
2、故障原因是网络波动导致虚拟ip从master1漂移到master2. 目前已经恢复，虚拟ip已经回到master1
3、从ks的console看有众多ks组件不可用，点击重新部署一直卡在init

…

node状态：

NAME               STATUS   ROLES    AGE     VERSION
k8s-master-01   Ready    master   5d17h   v1.15.5
k8s-master-02   Ready    master   5d17h   v1.15.5
k8s-master-03   Ready    master   5d17h   v1.15.5
k8s-node-01     Ready    worker   5d17h   v1.15.5
k8s-node-02     Ready    worker   5d17h   v1.15.5
...

ETC状态：

192.168.23.46:2379, ee7487c8b74342d5, 3.2.18, 19 MB, true, 5, 1716575
192.168.23.39:2379, a00baa75c98a110d, 3.2.18, 19 MB, false, 5, 1716575
192.168.23.41:2379, c4d06d628786cbfc, 3.2.18, 19 MB, false, 5, 1716576

全部POD状态：

NAMESPACE                      NAME                                                              READY   STATUS                  RESTARTS   AGE
istio-system                   istio-citadel-59c5db9b77-2rl77                                    1/1     Running                 0          5d17h
istio-system                   istio-galley-f469f7d4f-d2qw5                                      1/1     Running                 0          5d17h
istio-system                   istio-ingressgateway-6c89d587bf-s5w8v                             1/1     Running                 0          5d17h
istio-system                   istio-init-crd-10-1.3.3-8hkd5                                     0/1     Completed               0          5d17h
istio-system                   istio-init-crd-11-1.3.3-zzwld                                     0/1     Completed               0          5d17h
istio-system                   istio-init-crd-12-1.3.3-gfmtr                                     0/1     Completed               0          5d17h
istio-system                   istio-pilot-5bb797c4bb-jzjgb                                      2/2     Running                 0          5d17h
istio-system                   istio-policy-86b448c565-ck9fl                                     2/2     Running                 0          5d17h
istio-system                   istio-sidecar-injector-d98dbbb8b-lkhfx                            1/1     Running                 0          5d17h
istio-system                   istio-telemetry-676c67ff64-z5jkm                                  2/2     Running                 0          5d17h
istio-system                   jaeger-collector-8698b58b55-pqcnh                                 1/1     Running                 0          5d17h
istio-system                   jaeger-operator-66ff476758-rnpdr                                  1/1     Running                 0          5d17h
istio-system                   jaeger-query-79958fb5cb-xbwvf                                     2/2     Running                 0          17m
kube-system                    coredns-647d4f64c4-2lxw7                                          1/1     Running                 0          5d17h
kube-system                    coredns-647d4f64c4-6k7nv                                          1/1     Running                 0          5d17h
kube-system                    coredns-647d4f64c4-sj9vc                                          1/1     Running                 0          5d17h
kube-system                    dns-autoscaler-6d695d6998-ffgwt                                   1/1     Running                 0          5d17h
kube-system                    kube-apiserver-smyk8s-master-01                                   1/1     Running                 0          5d17h
kube-system                    kube-apiserver-smyk8s-master-02                                   1/1     Running                 0          5d17h
kube-system                    kube-apiserver-smyk8s-master-03                                   1/1     Running                 0          5d17h
kube-system                    kube-controller-manager-smyk8s-master-01                          1/1     Running                 0          5d17h
kube-system                    kube-controller-manager-smyk8s-master-02                          1/1     Running                 0          5d17h
kube-system                    kube-controller-manager-smyk8s-master-03                          1/1     Running                 0          5d17h
kube-system                    kube-flannel-bhzsv                                                2/2     Running                 0          5d17h
kube-system                    kube-flannel-c9dcw                                                2/2     Running                 0          5d17h
kube-system                    kube-flannel-gbxkz                                                2/2     Running                 0          5d17h
kube-system                    kube-flannel-slpnw                                                2/2     Running                 0          5d17h
kube-system                    kube-flannel-v286l                                                2/2     Running                 0          5d17h
kube-system                    kube-proxy-24hr5                                                  1/1     Running                 0          5d17h
kube-system                    kube-proxy-7gdtp                                                  1/1     Running                 0          5d17h
kube-system                    kube-proxy-92r52                                                  1/1     Running                 0          5d17h
kube-system                    kube-proxy-ds5mm                                                  1/1     Running                 0          5d17h
kube-system                    kube-proxy-vkh9l                                                  1/1     Running                 0          5d17h
kube-system                    kube-scheduler-smyk8s-master-01                                   1/1     Running                 0          5d17h
kube-system                    kube-scheduler-smyk8s-master-02                                   1/1     Running                 0          5d17h
kube-system                    kube-scheduler-smyk8s-master-03                                   1/1     Running                 0          5d17h
kube-system                    metrics-server-749c9798c6-wskhw                                   1/1     Running                 0          5d17h
kube-system                    nfs-client-nfs-client-provisioner-648bdc7747-h6rw8                1/1     Running                 0          5d17h
kube-system                    nodelocaldns-47hjf                                                1/1     Running                 0          5d17h
kube-system                    nodelocaldns-4cshk                                                1/1     Running                 0          5d17h
kube-system                    nodelocaldns-gzc5r                                                1/1     Running                 0          5d17h
kube-system                    nodelocaldns-kjdbt                                                1/1     Running                 0          5d17h
kube-system                    nodelocaldns-zfwrn                                                1/1     Running                 0          5d17h
kube-system                    tiller-deploy-55fc49c595-wk96m                                    1/1     Running                 0          5d17h
kubesphere-alerting-system     alerting-client-57767c4b4f-zljnt                                  1/1     Running                 0          5d17h
kubesphere-alerting-system     alerting-db-ctrl-job-c85lj                                        0/1     Completed               0          5d17h
kubesphere-alerting-system     alerting-db-init-job-gzglc                                        0/1     Completed               0          5d17h
kubesphere-alerting-system     alerting-executor-5786d84d45-wjh89                                2/2     Running                 0          5d17h
kubesphere-alerting-system     alerting-manager-66c874ddb4-8zltx                                 1/1     Running                 0          5d17h
kubesphere-alerting-system     alerting-watcher-85df56b849-7g989                                 1/1     Running                 0          5d17h
kubesphere-alerting-system     notification-db-ctrl-job-66wm2                                    0/1     Completed               2          5d17h
kubesphere-alerting-system     notification-db-init-job-7qt66                                    0/1     Completed               0          5d17h
kubesphere-alerting-system     notification-deployment-66b98fc47-wzfm5                           1/1     Running                 2          5d17h
kubesphere-controls-system     default-http-backend-6555ff6898-w5q8t                             1/1     Running                 0          5d17h
kubesphere-controls-system     kubectl-admin-74fdfc47c7-pj9rd                                    1/1     Running                 0          5d17h
kubesphere-devops-system       ks-devops-db-ctrl-job-ll2dq                                       0/1     Completed               0          5d17h
kubesphere-devops-system       ks-devops-db-init-job-jn6zb                                       0/1     Completed               0          5d17h
kubesphere-devops-system       ks-jenkins-7c7878c77d-hp64k                                       0/1     Init:0/1                4          28m
kubesphere-devops-system       ks-sonarqube-postgresql-5c4db956d7-b8z95                          1/1     Running                 0          5d17h
kubesphere-devops-system       ks-sonarqube-sonarqube-588f4bbd45-7s2z7                           1/1     Running                 1          5d17h
kubesphere-devops-system       ks-sonarqube-sonarqube-588f4bbd45-sdrc2                           0/1     Init:CrashLoopBackOff   7          15m
kubesphere-devops-system       s2ioperator-0                                                     1/1     Running                 1          5d17h
kubesphere-devops-system       uc-jenkins-update-center-759b5b5885-rcflx                         1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-curator-elasticsearch-curator-157723v5q6v   0/1     Completed               0          2d5h
kubesphere-logging-system      elasticsearch-logging-curator-elasticsearch-curator-157732btzl9   0/1     Completed               0          29h
kubesphere-logging-system      elasticsearch-logging-curator-elasticsearch-curator-157740rvpvb   0/1     Completed               0          5h53m
kubesphere-logging-system      elasticsearch-logging-data-0                                      1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-data-1                                      1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-discovery-0                                 1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-discovery-1                                 1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-discovery-2                                 1/1     Running                 0          5d17h
kubesphere-logging-system      elasticsearch-logging-kibana-75c755b6c-nw9dn                      1/1     Running                 0          5d17h
kubesphere-logging-system      fluent-bit-5sxzk                                                  2/2     Running                 0          5d17h
kubesphere-logging-system      fluent-bit-899hr                                                  2/2     Running                 0          5d17h
kubesphere-logging-system      fluent-bit-brpxc                                                  2/2     Running                 0          5d17h
kubesphere-logging-system      fluent-bit-qkvns                                                  2/2     Running                 0          5d17h
kubesphere-logging-system      fluent-bit-sql7z                                                  2/2     Running                 0          5d17h
kubesphere-logging-system      logging-fluentbit-operator-6798c5bf8-7qzvf                        1/1     Running                 0          5d17h
kubesphere-logging-system      logsidecar-injector-67bbcdd8c6-kzsh7                              1/1     Running                 0          5d17h
kubesphere-logging-system      logsidecar-injector-67bbcdd8c6-nb4wx                              1/1     Running                 0          5d17h
kubesphere-monitoring-system   kube-state-metrics-6f47d67cd-r5hfl                                4/4     Running                 0          5d17h
kubesphere-monitoring-system   node-exporter-7vzz9                                               2/2     Running                 0          5d17h
kubesphere-monitoring-system   node-exporter-bpk9n                                               2/2     Running                 0          5d17h
kubesphere-monitoring-system   node-exporter-cfhc5                                               2/2     Running                 0          5d17h
kubesphere-monitoring-system   node-exporter-kvfhk                                               2/2     Running                 0          5d17h
kubesphere-monitoring-system   node-exporter-r9k9n                                               2/2     Running                 0          5d17h
kubesphere-monitoring-system   prometheus-k8s-0                                                  3/3     Running                 1          5d17h
kubesphere-monitoring-system   prometheus-k8s-1                                                  3/3     Running                 1          5d17h
kubesphere-monitoring-system   prometheus-k8s-system-0                                           3/3     Running                 1          5d17h
kubesphere-monitoring-system   prometheus-k8s-system-1                                           3/3     Running                 1          5d17h
kubesphere-monitoring-system   prometheus-operator-685bc484cb-ffg8j                              1/1     Running                 0          5d17h
kubesphere-system              etcd-555778878f-n9s94                                             1/1     Running                 0          5d17h
kubesphere-system              ks-account-75fdc96f65-4rxr4                                       1/1     Running                 0          5d17h
kubesphere-system              ks-account-75fdc96f65-lwhjj                                       1/1     Running                 0          5d17h
kubesphere-system              ks-account-75fdc96f65-mtf2f                                       1/1     Running                 0          5d17h
kubesphere-system              ks-apigateway-65dd54f989-494m7                                    1/1     Running                 0          5d17h
kubesphere-system              ks-apigateway-65dd54f989-njpmj                                    1/1     Running                 0          5d17h
kubesphere-system              ks-apigateway-65dd54f989-zjzk7                                    1/1     Running                 0          5d17h
kubesphere-system              ks-apiserver-5d98f5d7cb-kmfxr                                     1/1     Running                 0          5d17h
kubesphere-system              ks-apiserver-5d98f5d7cb-vvfz8                                     1/1     Running                 0          5d17h
kubesphere-system              ks-apiserver-5d98f5d7cb-wc6mp                                     1/1     Running                 0          5d17h
kubesphere-system              ks-console-6f7f75bb48-5j6b2                                       1/1     Running                 0          5d17h
kubesphere-system              ks-console-6f7f75bb48-5qbmv                                       1/1     Running                 0          5d17h
kubesphere-system              ks-console-6f7f75bb48-hgzf4                                       1/1     Running                 0          5d17h
kubesphere-system              ks-controller-manager-6dd9b76d75-2s5lz                            1/1     Running                 0          5d17h
kubesphere-system              ks-controller-manager-6dd9b76d75-m6b96                            1/1     Running                 0          5d17h
kubesphere-system              ks-controller-manager-6dd9b76d75-p9zq8                            1/1     Running                 0          5d17h
kubesphere-system              ks-installer-7987c659d6-5fkf9                                     1/1     Running                 0          5d17h
kubesphere-system              minio-8cd46c8d9-z6lxd                                             1/1     Running                 0          5d17h
kubesphere-system              mysql-b5597d996-mzsrx                                             1/1     Running                 0          5d17h
kubesphere-system              openldap-0                                                        1/1     Running                 0          5d17h
kubesphere-system              openldap-1                                                        1/1     Running                 0          5d17h
kubesphere-system              redis-ha-haproxy-75776f44c4-9q8jg                                 1/1     Running                 0          5d17h
kubesphere-system              redis-ha-haproxy-75776f44c4-dtsqf                                 1/1     Running                 0          5d17h
kubesphere-system              redis-ha-haproxy-75776f44c4-wjv6n                                 1/1     Running                 1          5d17h
kubesphere-system              redis-ha-server-0                                                 2/2     Running                 0          5d17h
kubesphere-system              redis-ha-server-1                                                 2/2     Running                 0          5d17h
kubesphere-system              redis-ha-server-2                                                 2/2     Running                 0          5d17h
openpitrix-system              hyperpitrix-release-app-job-jjrjd                                 0/1     Completed               0          5d17h
openpitrix-system              openpitrix-app-db-ctrl-job-mb6rh                                  0/1     Completed               2          5d17h
openpitrix-system              openpitrix-attachment-db-ctrl-job-c8rrh                           0/1     Completed               2          5d17h
openpitrix-system              openpitrix-cluster-db-ctrl-job-s8qlv                              0/1     Completed               1          5d17h
openpitrix-system              openpitrix-db-init-job-bzsdm                                      0/1     Completed               0          5d17h
openpitrix-system              openpitrix-hyperpitrix-deployment-dddd548bd-92sbh                 1/1     Running                 0          5d17h
openpitrix-system              openpitrix-isv-db-ctrl-job-g88xm                                  0/1     Completed               1          5d17h
openpitrix-system              openpitrix-job-db-ctrl-job-cpkwx                                  0/1     Completed               1          5d17h
openpitrix-system              openpitrix-repo-db-ctrl-job-hr6hw                                 0/1     Completed               1          5d17h
openpitrix-system              openpitrix-rp-kubernetes-deployment-5cdf9c56c8-mzxm8              1/1     Running                 0          5d17h
openpitrix-system              openpitrix-runtime-db-ctrl-job-t8zxs                              0/1     Completed               1          5d17h
openpitrix-system              openpitrix-task-db-ctrl-job-vbfv5                                 0/1     Completed               0          5d17h