master节点故障重启后，ks平台出现无法登录

rysinal

环境：ks2.1.1
部署方式：3master 多worker
现象：今天由于其中一台master故障导致虚拟机重启，重启后，ks平台登录返回Internal Server Error

#查询异常pod，目前看起来只有ks-apigateway无法启动，如下：
root@smyk8s-master-01 admin]# kubectl get po -n kubesphere-system
NAME                                   READY   STATUS             RESTARTS   AGE
etcd-f988bdb6f-sg695                   1/1     Running            1          134d
ks-account-7478f89875-8wpnd            1/1     Running            0          135m
ks-account-7478f89875-9l2nd            1/1     Running            0          135m
ks-account-7478f89875-wcflg            1/1     Running            0          134m
ks-apigateway-5664c4b76f-4rqd2         1/1     Running            0          134d
ks-apigateway-5664c4b76f-lxkkc         1/1     Running            0          134d
ks-apigateway-77fbd6ff4d-66wtb         0/1     CrashLoopBackOff   7          11m
ks-apigateway-77fbd6ff4d-wm5nc         0/1     CrashLoopBackOff   10         26m
ks-apiserver-659746cf7d-8x4gr          1/1     Running            0          15d
ks-apiserver-659746cf7d-c9bcm          1/1     Running            0          178m
ks-apiserver-659746cf7d-fwnfp          1/1     Running            0          15d
ks-console-bf7975d87-5lpx5             1/1     Running            0          134m
ks-console-bf7975d87-8zs77             1/1     Running            0          135m
ks-console-bf7975d87-ps7tl             1/1     Running            0          135m
ks-controller-manager-d4788677-8drsm   1/1     Running            0          134d
ks-controller-manager-d4788677-9s7dt   1/1     Running            0          178m
ks-controller-manager-d4788677-nhgxt   1/1     Running            0          134d
ks-installer-669b4ff46-rj2g2           1/1     Running            0          134d
minio-8cd46c8d9-7rfx7                  1/1     Running            0          168m
mysql-b5597d996-ndmbl                  1/1     Running            1          134d
openldap-0                             1/1     Running            0          166m
openldap-1                             1/1     Running            3          262d
redis-ha-haproxy-f5747f989-gns98       1/1     Running            0          148m
redis-ha-haproxy-f5747f989-t2d6s       1/1     Running            0          147m
redis-ha-haproxy-f5747f989-xs6qj       1/1     Running            0          148m
redis-ha-server-0                      2/2     Running            0          140m
redis-ha-server-1                      2/2     Running            0          140m
redis-ha-server-2                      2/2     Running            0          140m

#进一步查看gateway日志，初步看是redis无法连接，但是redis的pod运行中，尝试重启redis deploy后仍无法连接。
[root@smyk8s-master-01 admin]# kubectl -n kubesphere-system  logs ks-apigateway-77fbd6ff4d-66wtb
[DEV NOTICE] Registered directive 'authenticate' before 'jwt'
[DEV NOTICE] Registered directive 'authentication' before 'jwt'
[DEV NOTICE] Registered directive 'swagger' before 'jwt'
Activating privacy features... done.
2020/09/09 07:05:22 [INFO][cache:0xc0000c76d0] Started certificate maintenance routine
E0909 07:05:22.770468       1 redis.go:51] unable to reach redis hostEOF
2020/09/09 07:05:22 EOF

#进一步查看redis日志报错（哨兵选举失败？）：
#kubectl -n kubesphere-system  logs redis-ha-server-0 redis
1:S 09 Sep 2020 07:09:18.517 * Master replied to PING, replication can continue...
1:S 09 Sep 2020 07:09:18.518 * Trying a partial resynchronization (request 2fcd515ac3f837eaffa39a657bca481bb454a5ca:4576914547).
1:S 09 Sep 2020 07:09:18.519 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 09 Sep 2020 07:09:19.519 * Connecting to MASTER 10.223.43.48:6379
1:S 09 Sep 2020 07:09:19.519 * MASTER <-> REPLICA sync started
1:S 09 Sep 2020 07:09:19.520 * Non blocking connect for SYNC fired the event.
1:S 09 Sep 2020 07:09:19.521 * Master replied to PING, replication can continue...
1:S 09 Sep 2020 07:09:19.522 * Trying a partial resynchronization (request 2fcd515ac3f837eaffa39a657bca481bb454a5ca:4576914547).
1:S 09 Sep 2020 07:09:19.522 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 09 Sep 2020 07:09:20.521 * Connecting to MASTER 10.223.43.48:6379
1:S 09 Sep 2020 07:09:20.521 * MASTER <-> REPLICA sync started
1:S 09 Sep 2020 07:09:20.522 * Non blocking connect for SYNC fired the event.
1:S 09 Sep 2020 07:09:20.523 * Master replied to PING, replication can continue...
1:S 09 Sep 2020 07:09:20.523 * Trying a partial resynchronization (request 2fcd515ac3f837eaffa39a657bca481bb454a5ca:4576914547).
1:S 09 Sep 2020 07:09:20.524 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 09 Sep 2020 07:09:21.528 * Connecting to MASTER 10.223.43.48:6379
1:S 09 Sep 2020 07:09:21.528 * MASTER <-> REPLICA sync started
1:S 09 Sep 2020 07:09:21.529 * Non blocking connect for SYNC fired the event.
1:S 09 Sep 2020 07:09:21.529 * Master replied to PING, replication can continue...
1:S 09 Sep 2020 07:09:21.531 * Trying a partial resynchronization (request 2fcd515ac3f837eaffa39a657bca481bb454a5ca:4576914547).
1:S 09 Sep 2020 07:09:21.531 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master

#kubectl -n kubesphere-system  logs -f redis-ha-haproxy-5d9d9c9d55-x5r8z
[NOTICE] 252/094142 (1) : New worker #1 (6) forked
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R0 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R1 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R2 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 252/094143 (6) : backend 'check_if_redis_is_master_0' has no server available!
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_1/R0 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.31.54')", check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_1/R1 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.31.54')", check duration: 1000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_1/R2 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.31.54')", check duration: 1000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 252/094143 (6) : backend 'check_if_redis_is_master_1' has no server available!
[WARNING] 252/094143 (6) : Server bk_redis_master/R0 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string 'role:master')", check duration: 1001ms. 2 active and 0 backup servers left.0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094144 (6) : Server bk_redis_master/R1 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string 'role:master')", check duration: 1000ms. 1 active and 0 backup servers left.0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094144 (6) : Server bk_redis_master/R2 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string 'role:master')", check duration: 1001ms. 0 active and 0 backup servers left.0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 252/094144 (6) : backend 'bk_redis_master' has no server available!

请教下大佬们redis这个如何解决呢

hongming

rysinal 端口都通不通呢，看看重启的节点上的kube-proxy有没有异常

kubectl -n kubesphere-system  exec -it redis-ha-server-0 redis-cli info replication
kubectl -n kubesphere-system  exec -it redis-ha-server-0 -- sh -c 'for i in `seq 0 2`; do nc -vz redis-ha-server-$i.redis-ha.kubesphere-system.svc 6379; done'

rysinal

hongming
当前机器的proxy状态是Running

[root@smyk8s-master-01 admin]# kubectl -n kube-system  get po |grep proxy
kube-proxy-24zv7                                     1/1     Running   4          134d
kube-proxy-7nngk                                     1/1     Running   3          134d
kube-proxy-87xsj                                     1/1     Running   1          135d
kube-proxy-fdggk                                     1/1     Running   2          134d
kube-proxy-k4484                                     1/1     Running   1          135d
kube-proxy-m5pw9                                     1/1     Running   3          135d
kube-proxy-njh6q                                     1/1     Running   2          135d
kube-proxy-p2vb4                                     1/1     Running   1          135d

#proxy日志

[root@smyk8s-master-01 admin]# kubectl -n kube-system  logs -f kube-proxy-p2vb4
I0909 04:09:00.416394       1 node.go:135] Successfully retrieved node IP: 192.168.23.46
I0909 04:09:00.416501       1 server_others.go:177] Using ipvs Proxier.
I0909 04:09:00.417688       1 server.go:529] Version: v1.16.7
I0909 04:09:00.418407       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
I0909 04:09:00.418462       1 conntrack.go:52] Setting nf_conntrack_max to 524288
I0909 04:09:00.475071       1 conntrack.go:83] Setting conntrack hashsize to 131072
I0909 04:09:00.491387       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0909 04:09:00.491483       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0909 04:09:00.494338       1 config.go:131] Starting endpoints config controller
I0909 04:09:00.495678       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0909 04:09:00.495402       1 config.go:313] Starting service config controller
I0909 04:09:00.495722       1 shared_informer.go:197] Waiting for caches to sync for service config
I0909 04:09:00.595957       1 shared_informer.go:204] Caches are synced for service config
I0909 04:09:00.598530       1 shared_informer.go:204] Caches are synced for endpoints config
I0909 04:16:00.420664       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.65.41:6379
I0909 04:16:00.420861       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.66.26:6379
I0909 04:39:00.429406       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.64.65:6379
I0909 04:48:00.434272       1 graceful_termination.go:93] lw: remote out of the list: 10.223.31.54:26379/TCP/10.223.66.31:26379
I0909 04:48:00.434516       1 graceful_termination.go:93] lw: remote out of the list: 10.223.2.232:6379/TCP/10.223.65.38:6379
I0909 04:48:00.434628       1 graceful_termination.go:93] lw: remote out of the list: 10.223.2.232:26379/TCP/10.223.65.38:26379
I0909 04:48:00.434753       1 graceful_termination.go:93] lw: remote out of the list: 10.223.43.48:6379/TCP/10.223.64.66:6379
I0909 09:41:00.535059       1 graceful_termination.go:93] lw: remote out of the list: 10.223.43.48:6379/TCP/10.223.64.68:6379
I0909 09:41:00.535267       1 graceful_termination.go:93] lw: remote out of the list: 10.223.31.54:26379/TCP/10.223.66.213:26379
I0909 09:42:00.536191       1 graceful_termination.go:93] lw: remote out of the list: 10.223.2.232:6379/TCP/10.223.65.67:6379
I0909 09:42:00.536780       1 graceful_termination.go:93] lw: remote out of the list: 10.223.2.232:26379/TCP/10.223.65.67:26379
I0909 09:43:00.537454       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.65.66:6379
I0909 09:44:00.538617       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.66.212:6379
I0909 09:44:00.538843       1 graceful_termination.go:93] lw: remote out of the list: 10.223.26.61:6379/TCP/10.223.67.2:6379

[root@smyk8s-master-01 admin]# kubectl -n kubesphere-system  exec -it redis-ha-server-0 redis-cli info replication
Defaulting container name to redis.
Use 'kubectl describe pod/redis-ha-server-0 -n kubesphere-system' to see all of the containers in this pod.
# Replication
role:slave
master_host:10.223.43.48
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:4576914546
master_link_down_since_seconds:1599648575
slave_priority:100
slave_read_only:1
connected_slaves:0
min_slaves_good_slaves:0
master_replid:2fcd515ac3f837eaffa39a657bca481bb454a5ca
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:4576914546
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0


[root@smyk8s-master-01 admin]# kubectl -n kubesphere-system  exec -it redis-ha-server-0 -- sh -c 'for i in `seq 0 2`; do nc -vz redis-ha-server-$i.redis-ha.kubesphere-system.svc 6379; done'
Defaulting container name to redis.
Use 'kubectl describe pod/redis-ha-server-0 -n kubesphere-system' to see all of the containers in this pod.
redis-ha-server-0.redis-ha.kubesphere-system.svc (10.223.65.68:6379) open
redis-ha-server-1.redis-ha.kubesphere-system.svc (10.223.66.214:6379) open
redis-ha-server-2.redis-ha.kubesphere-system.svc (10.223.64.73:6379) open

rysinal

hongming 可以抽空帮忙看下吗，谢谢

hongming

rysinal 可以把远程的方式发到kubesphere@yunify.com

rysinal

感谢hongming 大佬的指导，原因是REDIS完全挂了起不来，正常挂一个是可以恢复的
简单记录下版主操作给有需要的人参考一下：

kubectl -n kubesphere-system scale sts redis-ha-server --replicas=0
kubectl -n kubesphere-system scale deploy redis-ha-haproxy --replicas=0


kubectl -n kubesphere-system scale sts redis-ha-server --replicas=3
kubectl -n kubesphere-system scale deploy redis-ha-haproxy --replicas=3

#集群状态
kubectl -n kubesphere-system exec -it redis-ha-server-1 redis-cli info replication

#redis正常后
kubectl -n kubesphere-system scale deploy ks-apigateway --replicas=0
kubectl -n kubesphere-system scale deploy ks-apigateway --replicas=1

kubectl -n kubesphere-system rollout restart deploy ks-account

kubectl -n kubesphere-system scale deploy ks-apigateway --replicas=3

guangxilaobiao

我碰上过一次是因为有个机器的防火墙忘了关

zhangpeiyao

你好我想问下为什么有的用scale sts有的用scale deploy有什么说法和区别嘛

Cauchy

zhangpeiyao
建议查阅kubernetes文档：
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

zhangpeiyao

Cauchy 好的谢谢回复

rysinal

@hongming 大佬，redis集群报这个错怎么解决呢，按照之前的方法不好使了

1:S 02 Feb 2021 10:47:09.118 * Connecting to MASTER 10.223.2.232:6379
1:S 02 Feb 2021 10:47:09.119 * MASTER <-> REPLICA sync started
1:S 02 Feb 2021 10:48:10.318 # Timeout connecting to the MASTER...
1:S 02 Feb 2021 10:48:10.318 * Connecting to MASTER 10.223.2.232:6379
1:S 02 Feb 2021 10:48:10.318 * MASTER <-> REPLICA sync started
1:S 02 Feb 2021 10:48:13.222 # Error condition on socket for SYNC: Host is unreachable
1:S 02 Feb 2021 10:48:13.329 * Connecting to MASTER 10.223.2.232:6379
1:S 02 Feb 2021 10:48:13.329 * MASTER <-> REPLICA sync started
1:S 02 Feb 2021 10:48:19.234 # Error condition on socket for SYNC: Host is unreachable

另外这个10.223.2.232看着好像是写死的：

slave-announce-ip 10.223.2.232
slave-announce-port 6379

hongming

rysinal 正常重启一个节点的话，不应该出现故障，是重启了几台master吗？先等节点就绪，再重新scale 一下redis集群（如果有异常的话 endpoint 摘除会比较慢）。redis里没有太多值得持久化的数据（记录的是前端session和签发的token），可以重建一下 redis 用单副本（维护方便），保证 redis.kubesphere-system.svc:6379 可以正常访问到redis

rysinal

hongming 是的，重启了两台master，现在看着是所有节点都是 Ready 状态的

hongming

rysinal 那就等这两个节点完全就绪， kube-proxy 必须得正常，再scale 一下

sunlijun

hongming 大佬可以看看我主页的问题么我也不知道怎么搞了了啊