kubekey部署KubeSphere3.0，minio部署失败

ethan_x

kubekey部署KubeSphere3.0，minio部署失败：

查看minio-make日志：

[root@crm-server storage]# kubectl logs --namespace=kubesphere-system minio-make-bucket-job-nkstp 
Connecting to Minio server: http://minio:9000
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-fs5ok5gjghb1/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 1"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-gtt6ktoe4jsz/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 2"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-6yibhu1w9vjs/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 3"

本地的helm中minio应用部署状态为failed：

[root@crm-server storage]# helm list -n kubesphere-system
NAME       	NAMESPACE        	REVISION	UPDATED                                	STATUS  	CHART            	APP VERSION                 
ks-minio   	kubesphere-system	1       	2020-10-13 11:04:43.883682184 +0800 CST	failed  	minio-2.5.16     	RELEASE.2019-08-07T01-59-21Z
ks-openldap	kubesphere-system	1       	2020-10-13 10:28:10.679122552 +0800 CST	deployed	openldap-ha-0.1.0	1.0

NFS作为存储，redis和openldap的存储没问题。

[root@crm-server storage]# kubectl get pods --namespace=kubesphere-system 
NAME                            READY   STATUS    RESTARTS   AGE
ks-installer-5d4fd57c59-hspt8   1/1     Running   0          34m
minio-55fc45b5ff-86ktq          1/1     Running   0          32m
minio-make-bucket-job-nkstp     1/1     Running   1          31m
openldap-0                      1/1     Running   0          68m
redis-69ccf5cbc8-z585j          1/1     Running   0          69m

[root@crm-server storage]# ls -l
total 0
drwxrwxrwx 3 root root 32 Oct 13 10:28 archived-kubesphere-system-minio-pvc-cab94a4f-bfa2-4773-a2cd-905b74c01f9f
drwxrwxrwx 3 root root 32 Oct 13 11:04 kubesphere-system-minio-pvc-1c5c5190-5d5f-40b4-bc6d-b370b94360ce
drwxrwxrwx 4 root root 54 Oct 13 10:28 kubesphere-system-openldap-pvc-openldap-0-pvc-cce51014-56d5-4d70-ba87-202c66b9feec
drwxrwxrwx 3 root root 32 Oct 13 10:28 kubesphere-system-redis-pvc-pvc-b0db013c-1cc7-43cf-909a-65b0ee823fc1

yuswift

看报错是网络连接超时先排查一下网络问题吧

ethan_x

检查发现有个coredns与本地dns连接超时，但我从主机上ping本地的dns是正常的，请问这个还有什么配置吗？

[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# kubectl logs --namespace=kube-system coredns-6d444f4677-6q785
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# 
[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# kubectl logs --namespace=kube-system coredns-6d444f4677-lm6ch 
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:33497->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:53555->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:55348->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:42338->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:57723->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:47521->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:34598->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:51137->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:52009->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:53003->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:46453->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:56227->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:59011->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:45478->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:48338->10.21.9.241:53: i/o timeout
[ERROR] plugin/errors: 2 . NS: read udp 10.233.82.3:47915->10.21.9.241:53: i/o timeout

[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# ping 10.21.9.241
PING 10.21.9.241 (10.21.9.241) 56(84) bytes of data.
64 bytes from 10.21.9.241: icmp_seq=1 ttl=125 time=0.678 ms
64 bytes from 10.21.9.241: icmp_seq=2 ttl=125 time=0.879 ms
64 bytes from 10.21.9.241: icmp_seq=3 ttl=125 time=1.22 ms
^C
--- 10.21.9.241 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.678/0.927/1.226/0.229 ms
[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# 
[root@crm-server kubesphere-all-v3.0.0-offline-linux-amd64]# telnet 10.21.9.241 53
Trying 10.21.9.241...
Connected to 10.21.9.241.
Escape character is '^]'.

ethan_x

检查coredns发现，coredns ping本地的dns超时

[root@crm-server ~]# kubectl logs --namespace=kube-system coredns-6d444f4677-sfrfh 
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 152539192193979321.8927508496407462216. HINFO: read udp 10.233.71.3:34216->10.21.9.241:53: i/o timeout
[root@crm-server ~]# 
[root@crm-server ~]# 
[root@crm-server ~]# kubectl logs --namespace=kube-system coredns-6d444f4677-rnzxc 
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 509654373290296709.787190718157698380. HINFO: read udp 10.233.71.2:41470->10.21.9.241:53: i/o timeout

检查nodeloacaldns，连接coredns svc ip被拒：

[root@crm-server ~]# kubectl logs --namespace=kube-system nodelocaldns-kcwgr
2020/10/13 06:43:20 [INFO] Using Corefile /etc/coredns/Corefile
2020/10/13 06:43:20 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2020/10/13 06:43:20 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter INPUT [-p tcp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter INPUT [-p udp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 06:43:20 [INFO] Added interface - nodelocaldns
.:53 on 169.254.25.10
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[ERROR] plugin/errors: 2 4616685356003502933.393542240170391009.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 3954823728226110956.2439121849018967759.cluster.local. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 7817138522425820044.931794010996272835.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 3954823728226110956.2439121849018967759.cluster.local. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 4616685356003502933.393542240170391009.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused
[ERROR] plugin/errors: 2 7817138522425820044.931794010996272835.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: connect: connection refused

主机上的防火墙都已经关闭，selinux也置为disabled。

yuswift

10.21.9.241是你自己的dns server吗如果没用可以临时禁用掉然后重启coredns pod
另外你集群是怎么安装的方便的话可以用kubekey重装试试

ethan_x

yuswift 就算禁掉dns server，重启coredns后，nodelocaldns与coredns间仍然不通

[root@crm-server ~]# kubectl logs --namespace=kube-system nodelocaldns-jkd9l -f
2020/10/13 07:48:12 [INFO] Using Corefile /etc/coredns/Corefile
2020/10/13 07:48:12 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2020/10/13 07:48:12 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw PREROUTING [-p tcp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw PREROUTING [-p udp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter INPUT [-p tcp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter INPUT [-p udp -d 169.254.25.10 --dport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p tcp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {filter OUTPUT [-p udp -s 169.254.25.10 --sport 53 -j ACCEPT]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p udp -d 169.254.25.10 --dport 53 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -d 169.254.25.10 --dport 8080 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added back nodelocaldns rule - {raw OUTPUT [-p tcp -s 169.254.25.10 --sport 8080 -j NOTRACK]}
2020/10/13 07:48:12 [INFO] Added interface - nodelocaldns
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
.:53 on 169.254.25.10
[ERROR] Recovered from panic in server: "dns://169.254.25.10:53"
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[ERROR] plugin/errors: 2 7843590585922501453.7235693862240383215.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 7843590585922501453.7235693862240383215.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. A: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. A: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 minio.kubesphere-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout

ethan_x

10.21.9.241是我们自己的dns server。集群全部使用kk命令部署的（k8s+KubeSphere）

yuswift

ethan_x
你去coredns 所在的节点用 telenet试试感觉是网络问题

ethan_x

集群节点都检查过了，与dns server连接都是通的：

[root@perf-db-0001 ~]# telnet 10.21.9.241 53
Trying 10.21.9.241...
Connected to 10.21.9.241.
Escape character is '^]'.

^]
telnet> quit
Connection closed.

[root@crm-server ~]# telnet 10.21.9.241 53
Trying 10.21.9.241...
Connected to 10.21.9.241.
Escape character is '^]'.

^]
telnet> quit
Connection closed.

yuswift

你去nodelocaldns所在节点手动连接一下cordons试试呢

ethan_x

在节点上直接ping codedns ip没问题，但Telnet就没反应了：

[root@crm-server ~]# kubectl get svc --all-namespaces 
NAMESPACE           NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default             kubernetes       ClusterIP   10.233.0.1      <none>        443/TCP                  118m
kube-system         coredns          ClusterIP   10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP   117m
kube-system         metrics-server   ClusterIP   10.233.0.239    <none>        443/TCP                  114m
kubesphere-system   minio            ClusterIP   10.233.34.202   <none>        9000/TCP                 111m
kubesphere-system   openldap         ClusterIP   None            <none>        389/TCP                  111m
kubesphere-system   redis            ClusterIP   10.233.51.70    <none>        6379/TCP                 112m

[root@crm-server ~]# ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3) 56(84) bytes of data.
64 bytes from 10.233.0.3: icmp_seq=1 ttl=64 time=0.075 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from 10.233.0.3: icmp_seq=3 ttl=64 time=0.046 ms
^C
--- 10.233.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.046/0.060/0.075/0.014 ms
[root@crm-server ~]# 
[root@crm-server ~]# telnet 10.233.0.3 53
Trying 10.233.0.3...
^C

[root@perf-db-0001 ~]# ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3) 56(84) bytes of data.
64 bytes from 10.233.0.3: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=64 time=0.054 ms
^C
--- 10.233.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.054/0.060/0.066/0.006 ms
[root@perf-db-0001 ~]# 
[root@perf-db-0001 ~]# 
[root@perf-db-0001 ~]# telnet 10.233.0.3 53
Trying 10.233.0.3...
^C

yuswift

那还是网络问题啊相当于localdns连不上coredns 这个得自行排查一下

xxm13504577723

是网络问题，建议你查看一下网络组件是否正常运行，ping ip肯定是没问题的，ping 一下svc name试一试，检查一下各个节点的网卡是否选择正确，我遇到过由于某个节点的网卡名字不同导致的解析失败问题。

pyfs

这种问题有没有解决思路；

看了几个帖子，都是烂尾的问题；

yuswift

pyfs 把你遇到的问题详细描述清楚

pyfs

yuswift

跟楼主一样， minio 安装报错；看日志提示说：

[root@crm-server storage]# kubectl logs --namespace=kubesphere-system minio-make-bucket-job-nkstp 
Connecting to Minio server: http://minio:9000
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-fs5ok5gjghb1/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 1"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-gtt6ktoe4jsz/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 2"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-6yibhu1w9vjs/?location=: dial tcp 10.233.18.29:9000: i/o timeout.
"Failed attempts: 3"

Jeff

pyfs https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ 你跟着k8s官方文档检查一遍，90%的网络问题看过这个文档都能解决

pyfs

Jeff

收到，先排查排查；多谢

lhq8620

解决了吗，方便提供一下解决方案吗？

pyfs

lhq8620

一直没解决，后来释放掉所有资源，重新申请一套虚拟机就可以了；

感觉是网络的问题；