kubekey部署KubeSphere3.0，minio部署失败

ethan_x

检查coredns发现，coredns ping本地的dns超时

[root@crm-server ~]# kubectl logs --namespace=kube-system coredns-6d444f4677-sfrfh 
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 152539192193979321.8927508496407462216. HINFO: read udp 10.233.71.3:34216->10.21.9.241:53: i/o timeout
[root@crm-server ~]# 
[root@crm-server ~]# 
[root@crm-server ~]# kubectl logs --namespace=kube-system coredns-6d444f4677-rnzxc 
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 509654373290296709.787190718157698380. HINFO: read udp 10.233.71.2:41470->10.21.9.241:53: i/o timeout

检查nodeloacaldns，连接coredns svc ip被拒：

[root@crm-server ~]# kubectl logs --namespace=kube-system nodelocaldns-kcwgr
2020/10/13 06:43:20 [INFO] Using Corefile /etc/coredns/Corefile
2020/10/13 06:43:20 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2020/10/13 06:43:20 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter INPUT [-p tcp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter INPUT [-p udp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added interface - nodelocaldns
.:53 on 169.254.25.10
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[ERROR] plugin/errors: 2 4616685356003502933.393542240170391009.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 3954823728226110956.2439121849018967759.cluster.local. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 7817138522425820044.931794010996272835.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 3954823728226110956.2439121849018967759.cluster.local. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 4616685356003502933.393542240170391009.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 7817138522425820044.931794010996272835.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused

主机上的防火墙都已经关闭，selinux也置为disabled。

yuswift

10.21.9.241是你自己的dns server吗如果没用可以临时禁用掉然后重启coredns pod
另外你集群是怎么安装的方便的话可以用kubekey重装试试

ethan_x

yuswift 就算禁掉dns server，重启coredns后，nodelocaldns与coredns间仍然不通

[root@crm-server ~]# kubectl logs --namespace=kube-system nodelocaldns-jkd9l -f
2020/10/13 07:48:12 [INFO] Using Corefile /etc/coredns/Corefile
2020/10/13 07:48:12 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2020/10/13 07:48:12 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw PREROUTING [-p tcp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw PREROUTING [-p udp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter INPUT [-p tcp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter INPUT [-p udp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p udp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -d 169.254.25.10 --dport 8080 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -s 169.254.25.10 --sport 8080 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added interface - nodelocaldns
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
.:53 on 169.254.25.10
[ERROR] Recovered from panic in server: "dns://169.254.25.10:53"
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[ERROR] plugin/errors: 2 7843590585922501453.7235693862240383215.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 7843590585922501453.7235693862240383215.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. A: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. A: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout

ethan_x

10.21.9.241是我们自己的dns server。集群全部使用kk命令部署的（k8s+KubeSphere）

yuswift

ethan_x
你去coredns 所在的节点用 telenet试试感觉是网络问题

ethan_x

集群节点都检查过了，与dns server连接都是通的：

[root@perf-db-0001 ~]# telnet 10.21.9.241 53
Trying 10.21.9.241...
Connected to 10.21.9.241.
Escape character is '^]'.

^]
telnet> quit
Connection closed.

[root@crm-server ~]# telnet 10.21.9.241 53
Trying 10.21.9.241...
Connected to 10.21.9.241.
Escape character is '^]'.

^]
telnet> quit
Connection closed.

yuswift

你去nodelocaldns所在节点手动连接一下cordons试试呢

ethan_x

在节点上直接ping codedns ip没问题，但Telnet就没反应了：

[root@crm-server ~]# kubectl get svc --all-namespaces 
NAMESPACE           NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default             kubernetes       ClusterIP   10.233.0.1      <none>        443/TCP                  118m
kube-system         coredns          ClusterIP   10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP   117m
kube-system         metrics-server   ClusterIP   10.233.0.239    <none>        443/TCP                  114m
kubesphere-system   minio            ClusterIP   10.233.34.202   <none>        9000/TCP                 111m
kubesphere-system   openldap         ClusterIP   None            <none>        389/TCP                  111m
kubesphere-system   redis            ClusterIP   10.233.51.70    <none>        6379/TCP                 112m

[root@crm-server ~]# ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3) 56(84) bytes of data.
64 bytes from 10.233.0.3: icmp_seq=1 ttl=64 time=0.075 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from 10.233.0.3: icmp_seq=3 ttl=64 time=0.046 ms
^C
--- 10.233.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.046/0.060/0.075/0.014 ms
[root@crm-server ~]# 
[root@crm-server ~]# telnet 10.233.0.3 53
Trying 10.233.0.3...
^C

[root@perf-db-0001 ~]# ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3) 56(84) bytes of data.
64 bytes from 10.233.0.3: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=64 time=0.054 ms
^C
--- 10.233.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.054/0.060/0.066/0.006 ms
[root@perf-db-0001 ~]# 
[root@perf-db-0001 ~]# 
[root@perf-db-0001 ~]# telnet 10.233.0.3 53
Trying 10.233.0.3...
^C

yuswift

那还是网络问题啊相当于localdns连不上coredns 这个得自行排查一下

xxm13504577723

是网络问题，建议你查看一下网络组件是否正常运行，ping ip肯定是没问题的，ping 一下svc name试一试，检查一下各个节点的网卡是否选择正确，我遇到过由于某个节点的网卡名字不同导致的解析失败问题。

pyfs

这种问题有没有解决思路；

看了几个帖子，都是烂尾的问题；

yuswift

pyfs 把你遇到的问题详细描述清楚

pyfs

yuswift

跟楼主一样， minio 安装报错；看日志提示说：

[root@crm-server storage]# kubectl logs --namespace=kubesphere-system minio-make-bucket-job-nkstp 
Connecting to Minio server: http://minio:9000
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-fs5ok5gjghb1/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 1"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-gtt6ktoe4jsz/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 2"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-6yibhu1w9vjs/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 3"

Jeff

pyfs https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ 你跟着k8s官方文档检查一遍，90%的网络问题看过这个文档都能解决

pyfs

Jeff

收到，先排查排查；多谢

lhq8620

解决了吗，方便提供一下解决方案吗？

pyfs

lhq8620

一直没解决，后来释放掉所有资源，重新申请一套虚拟机就可以了；

感觉是网络的问题；

Getup233

我是随便搭的一个，也碰到这个，然后就跟着提示信息

先给minio的搭建任务给删了，后面其他的都正常安装好了，然后就是kubesphere-system里面少了一个minio

目前我就用kubesphere搭了一个mysql，是可以正常使用的