3master 多主安装 redis报错。请问该如何解决

curetxy

[root@huashan235 tools]# kubectl get pod –all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-578894d4cd-w7bgt 1/1 Running 0 7m48s
kube-system calico-node-6rs27 1/1 Running 0 7m39s
kube-system calico-node-cm95t 1/1 Running 0 7m39s
kube-system calico-node-cqqbj 1/1 Running 0 7m36s
kube-system calico-node-nvkx7 1/1 Running 0 7m39s
kube-system calico-node-pq2hr 1/1 Running 0 7m34s
kube-system calico-node-z5vk4 1/1 Running 0 7m48s
kube-system coredns-6b55b6764d-h82hh 1/1 Running 0 8m3s
kube-system coredns-6b55b6764d-xrkgr 1/1 Running 0 8m3s
kube-system kube-apiserver-master1 1/1 Running 0 8m12s
kube-system kube-apiserver-master2 1/1 Running 0 7m34s
kube-system kube-apiserver-master3 1/1 Running 0 7m31s
kube-system kube-controller-manager-master1 1/1 Running 0 8m12s
kube-system kube-controller-manager-master2 1/1 Running 0 7m34s
kube-system kube-controller-manager-master3 1/1 Running 0 7m32s
kube-system kube-proxy-67dl6 1/1 Running 0 7m39s
kube-system kube-proxy-8m4bg 1/1 Running 0 7m39s
kube-system kube-proxy-gskdm 1/1 Running 0 7m34s
kube-system kube-proxy-gzwt4 1/1 Running 0 7m39s
kube-system kube-proxy-hnczv 1/1 Running 0 8m3s
kube-system kube-proxy-mlhkq 1/1 Running 0 7m36s
kube-system kube-scheduler-master1 1/1 Running 0 8m12s
kube-system kube-scheduler-master2 1/1 Running 0 7m34s
kube-system kube-scheduler-master3 1/1 Running 0 7m31s
kube-system metrics-server-f47758d8f-8z96z 1/1 Running 0 5m
kube-system nodelocaldns-f9jz6 1/1 Running 0 7m36s
kube-system nodelocaldns-j44w2 1/1 Running 0 7m39s
kube-system nodelocaldns-j5rtc 1/1 Running 0 7m34s
kube-system nodelocaldns-lchlv 1/1 Running 0 7m39s
kube-system nodelocaldns-mwg27 1/1 Running 0 8m3s
kube-system nodelocaldns-rxnpz 1/1 Running 0 7m39s
kube-system openebs-localpv-provisioner-84446d4bd7-hbflf 1/1 Running 0 7m31s
kube-system openebs-ndm-h4vkw 1/1 Running 0 7m19s
kube-system openebs-ndm-nmbp7 1/1 Running 0 7m19s
kube-system openebs-ndm-operator-6456dc9db-wpxts 1/1 Running 1 7m31s
kube-system openebs-ndm-rkz72 1/1 Running 0 7m18s
kube-system snapshot-controller-0 1/1 Running 0 2m45s
kubesphere-controls-system default-http-backend-857d7b6856-x4sfx 1/1 Running 0 64s
kubesphere-system ks-apiserver-675bc84757-dnbck 0/1 ContainerCreating 0 33s
kubesphere-system ks-apiserver-675bc84757-np448 0/1 ContainerCreating 0 33s
kubesphere-system ks-apiserver-675bc84757-nvglk 0/1 ContainerCreating 0 33s
kubesphere-system ks-console-786b9846d4-dz6dv 1/1 Running 0 28s
kubesphere-system ks-console-786b9846d4-r5vnr 1/1 Running 0 28s
kubesphere-system ks-console-786b9846d4-xnt6n 1/1 Running 0 28s
kubesphere-system ks-controller-manager-69586754d4-q4hj9 0/1 ContainerCreating 0 32s
kubesphere-system ks-controller-manager-69586754d4-wdnc2 0/1 ContainerCreating 0 32s
kubesphere-system ks-controller-manager-69586754d4-wfqgf 0/1 ContainerCreating 0 32s
kubesphere-system ks-installer-7cb866bd-fd4c5 1/1 Running 0 7m27s
kubesphere-system openldap-0 0/1 Running 0 101s
kubesphere-system redis-ha-haproxy-5c6559d588-cfbwk 1/1 Running 0 118s
kubesphere-system redis-ha-haproxy-5c6559d588-mhbw5 0/1 Init:0/1 0 118s
kubesphere-system redis-ha-haproxy-5c6559d588-qlwtw 0/1 Init:0/1 0 118s
kubesphere-system redis-ha-server-0 0/2 Init:Error 2 118s

[root@huashan235 tools]# kubectl logs ks-apiserver-5d5dcc7d78-hr47f -n kubesphere-system
W0316 20:58:50.619974 1 client_config.go:543] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.
Error: failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout
2021/03/16 20:58:55 failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout

[root@huashan235 tools]#
[root@huashan235 tools]# kubectl logs redis-ha-haproxy-5c6559d588-cfbwk -n kubesphere-system
[NOTICE] 074/125355 (1) : New worker #1 (7) forked
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_0/R0 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_0/R1 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_0/R2 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 074/125356 (7) : backend ‘check_if_redis_is_master_0’ has no server available!
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_1/R0 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_1/R1 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_1/R2 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 074/125356 (7) : backend ‘check_if_redis_is_master_1’ has no server available!
[WARNING] 074/125356 (7) : Server check_if_redis_is_master_2/R0 is DOWN, reason: Layer4 timeout, info: “ at step 1 of tcp-check (connect)”, check duration: 1001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125357 (7) : Server check_if_redis_is_master_2/R1 is DOWN, reason: Layer4 connection problem, info: “Connection refused at step 1 of tcp-check (connect)”, check duration: 1001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125357 (7) : Server check_if_redis_is_master_2/R2 is DOWN, reason: Layer4 timeout, info: “ at step 1 of tcp-check (connect)”, check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 074/125357 (7) : backend ‘check_if_redis_is_master_2’ has no server available!
[WARNING] 074/125357 (7) : Server bk_redis_master/R0 is DOWN, reason: Layer4 timeout, info: “ at step 1 of tcp-check (connect)”, check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125357 (7) : Server bk_redis_master/R1 is DOWN, reason: Layer4 timeout, info: “ at step 1 of tcp-check (connect)”, check duration: 1001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 074/125357 (7) : Server bk_redis_master/R2 is DOWN, reason: Layer4 timeout, info: “ at step 1 of tcp-check (connect)”, check duration: 1000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 074/125357 (7) : backend ‘bk_redis_master’ has no server available!
[root@huashan235 tools]#
[root@huashan235 tools]#
[root@huashan235 tools]# kubectl logs redis-ha-server-0 -n kubesphere-system
error: a container name must be specified for pod redis-ha-server-0, choose one of: [redis sentinel] or one of the init containers: [config-init]

zhu733756

curetxy redis属于存储组件，可能跟网络和存储有关

curetxy

zhu733756 对我发现DNS 配置一个本地的DNS 就无法成功，设置成公网公网的DNS 就oK

zheng1

您好，如果在使用 KubeSphere 的过程中碰到了问题，希望快速得到技术专家的协助和支持，欢迎访问 kubesphere.cloud 平台，我们提供及时专业的在线工单支持，通过工单可以与 KubeSphere 资深工程师在线直接沟通，帮助您解决涉及 Kubernetes 以及运行在 KubeSphere 之上的应用服务相关的运维、管理等问题，为您的业务保驾护航~

curetxy

zheng1 谢谢

curetxy

找到问题了。我centos7中配置的DNS 是192.168.5.x 内网DNS 修改为公共DNS 重新安装就OK了
但是新问题又来了。。如果我再换回内网DNS重启就会报

[root@huashan235 tools]# kubectl get pod –all-namespaces
Unable to connect to the server: dial tcp: lookup lb.kubesphere.local on 202.103.24.68:53: no such host
[root@huashan235 tools]#
[root@huashan235 tools]#
[root@huashan235 tools]#
[root@huashan235 tools]# journalctl -f -u kubelet.service
– Logs begin at Thu 2021-03-18 08:54:50 CST. –
Mar 18 09:27:28 huashan235.bw580.com kubelet[2144]: E0318 09:27:28.996547 2144 kubelet.go:2268] node “master1” not found
Mar 18 09:27:29 huashan235.bw580.com kubelet[2144]: E0318 09:27:29.012266 2144 webhook.go:111] Failed to make webhook authenticator request: Post https://lb.kubesphere.local:6443/apis/authentication.k8s.io/v1/tokenreviews: dial tcp: lookup lb.kubesphere.local: no such host
Mar 18 09:27:29 huashan235.bw580.com kubelet[2144]: E0318 09:27:29.012378 2144 server.go:253] Unable to authenticate the request due to an error: Post https://lb.kubesphere.local:6443/apis/authentication.k8s.io/v1/tokenreviews: dial tcp: lookup lb.kubesphere.local: no such host
Mar 18 09:27:29 huashan235.bw580.com kubelet[2144]: E0318 09:27:29.096907 2144 kubelet.go:2268] node “master1” not found

RolandMa1986

curetxy 如果你是3master，就需要做master的ha，“lb.kubesphere.local” 地址可以通过修改host 实现或者修改为你ha的IP地址、DNS。根据你的环境可以选择使用 haproxy或者其他负载均衡器。
可以参考：https://kubesphere.com.cn/forum/d/1566-kubernetes-keepalived-haproxy

curetxy

确实是DNS的问题。第一次碰见，在这里把问题出现和解决过程略微写一下吧、

1、内网搭建bind并配置内网DNS，比如
[root@master1 ~]# cat /etc/resolv.conf

nameserver 192.168.5.192
nameserver 114.114.114.114

安装到最后会报 redis-ha-haproxy 和 redis-ha-server 起不来，解决比较简单，重新配置好可访问公网DNS，重新安装就好了。

2、不甘心还想配置一下内网DNS ，重新切换DNS 后重启发现问题依旧，解决办法修改DNS

sed -i “s/^DNS1=./DNS1=192.168.5.192/” /etc/sysconfig/network-scripts/ifcfg-eth0
sed -i “s/^DNS2=./DNS2=202.103.24.68/” /etc/sysconfig/network-scripts/ifcfg-eth0

sed -i “s/^DNS1=./DNS1=192.168.5.192/” /etc/sysconfig/network-scripts/ifcfg-eth1
sed -i “s/^DNS2=./DNS2=202.103.24.68/” /etc/sysconfig/network-scripts/ifcfg-eth1

reboot

启动以后会报redis 错误。如果起来了可能会密码错误。。

重新 scale redis

kubectl -n kubesphere-system scale sts redis-ha-server –replicas=0

kubectl -n kubesphere-system scale deploy redis-ha-haproxy –replicas=0

kubectl -n kubesphere-system scale sts redis-ha-server –replicas=3

kubectl -n kubesphere-system scale deploy redis-ha-haproxy –replicas=3

#集群状态
kubectl -n kubesphere-system exec -it redis-ha-server-0 redis-cli info replication

#问题解决

#如果还不行那么需要修改 DNS

执行 kubectl get pods –namespace=kube-system -l k8s-app=kube-dns

执行 kubectl get svc -n kube-system coredns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 269d

修改每个节点的 /var/lib/kubelet/config.yaml 里的clusterDNS 为coredns的cluster-ip

sed -i “s/^- 169.254.25.10/- 10.233.0.3/” /var/lib/kubelet/config.yaml

重启所有节点 kubelet

systemctl restart kubelet

kubectl get pod –all-namespaces

问题解决