V3.0.0 多节点安装报错

tzghost

发现30880端口没有起来

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 127.0.0.1:15106         0.0.0.0:*               LISTEN      24369/kubelet       

tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1698/zabbix_agentd  

tcp        0      0 169.254.25.10:9254      0.0.0.0:*               LISTEN      25248/node-cache    

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      24369/kubelet       

tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      24956/kube-proxy    

tcp        0      0 192.168.0.55:2379       0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 127.0.0.1:9099          0.0.0.0:*               LISTEN      26148/calico-node   

tcp        0      0 192.168.0.55:2380       0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 0.0.0.0:179             0.0.0.0:*               LISTEN      26268/bird          

tcp        0      0 169.254.25.10:53        0.0.0.0:*               LISTEN      25248/node-cache    

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      7054/sshd           

tcp6       0      0 :::10050                :::*                    LISTEN      1698/zabbix_agentd  

tcp6       0      0 :::9253                 :::*                    LISTEN      25248/node-cache    

tcp6       0      0 :::9353                 :::*                    LISTEN      25248/node-cache    

tcp6       0      0 :::10250                :::*                    LISTEN      24369/kubelet       

tcp6       0      0 :::10251                :::*                    LISTEN      4810/kube-scheduler 
tcp6       0      0 :::6443                 :::*                    LISTEN      24064/kube-apiserve 
tcp6       0      0 :::10252                :::*                    LISTEN      4803/kube-controlle 
tcp6       0      0 :::10256                :::*                    LISTEN      24956/kube-proxy    

tcp6       0      0 :::10257                :::*                    LISTEN      4803/kube-controlle 
tcp6       0      0 :::10259                :::*                    LISTEN      4810/kube-scheduler 
tcp6       0      0 :::22                   :::*                    LISTEN      7054/sshd

tzghost

[root@BG-003-KVM004-VMS003 ~]# kubectl get pods --all-namespaces 
NAMESPACE           NAME                                       READY   STATUS              RESTARTS   AGE
kube-system         calico-kube-controllers-578894d4cd-5scsq   1/1     Running             0          59m
kube-system         calico-node-6hdrt                          1/1     Running             0          58m
kube-system         calico-node-78gdl                          1/1     Running             0          58m
kube-system         calico-node-dpdvf                          1/1     Running             0          59m
kube-system         calico-node-k99wz                          1/1     Running             0          58m
kube-system         calico-node-mz2tf                          1/1     Running             0          58m
kube-system         calico-node-x8d9s                          1/1     Running             0          58m
kube-system         coredns-6b55b6764d-4xccb                   1/1     Running             0          59m
kube-system         coredns-6b55b6764d-mrv99                   1/1     Running             0          59m
kube-system         kube-apiserver-master1                     1/1     Running             0          59m
kube-system         kube-apiserver-master2                     1/1     Running             0          58m
kube-system         kube-apiserver-master3                     1/1     Running             0          58m
kube-system         kube-controller-manager-master1            1/1     Running             1          59m
kube-system         kube-controller-manager-master2            1/1     Running             0          58m
kube-system         kube-controller-manager-master3            1/1     Running             0          58m
kube-system         kube-proxy-4gzww                           1/1     Running             0          58m
kube-system         kube-proxy-74z5f                           1/1     Running             0          58m
kube-system         kube-proxy-7zgqv                           1/1     Running             0          59m
kube-system         kube-proxy-96mpk                           1/1     Running             0          58m
kube-system         kube-proxy-nscqh                           1/1     Running             0          58m
kube-system         kube-proxy-qt767                           1/1     Running             0          58m
kube-system         kube-scheduler-master1                     1/1     Running             1          59m
kube-system         kube-scheduler-master2                     1/1     Running             0          58m
kube-system         kube-scheduler-master3                     1/1     Running             0          58m
kube-system         metrics-server-f47758d8f-f5csr             1/1     Running             0          20m
kube-system         nodelocaldns-ddx49                         1/1     Running             0          59m
kube-system         nodelocaldns-nn2ts                         1/1     Running             0          58m
kube-system         nodelocaldns-rzfqs                         1/1     Running             0          58m
kube-system         nodelocaldns-w48jc                         1/1     Running             0          58m
kube-system         nodelocaldns-wxqbg                         1/1     Running             0          58m
kube-system         nodelocaldns-xnw5s                         1/1     Running             0          58m
kube-system         snapshot-controller-0                      1/1     Running             0          18m
kubesphere-system   ks-installer-74757f9779-bxj4z              1/1     Running             0          26m
kubesphere-system   minio-7bfdb5968b-vmb2k                     0/1     ContainerCreating   0          17m
kubesphere-system   openldap-0                                 1/1     Running             0          17m
kubesphere-system   openldap-1                                 0/1     ContainerCreating   0          8m20s
kubesphere-system   redis-ha-haproxy-5c6559d588-2mwz7          1/1     Running             0          17m
kubesphere-system   redis-ha-haproxy-5c6559d588-ff8jv          1/1     Running             3          17m
kubesphere-system   redis-ha-haproxy-5c6559d588-htbxd          0/1     Init:0/1            0          17m
kubesphere-system   redis-ha-server-0                          2/2     Running             0          17m
kubesphere-system   redis-ha-server-1                          0/2     Init:0/1            0          13m

Cauchy

describe看看那个minio为啥还在创建中，是不是网络不好，还在下镜像

tzghost

显示卷挂载失败，请问这种问题要怎么处理？Cauchy

Cauchy

节点上装glusterfs的client了么？

tzghost

Cauchy 都装上了的

tzghost

我的glusterfs是4个节点，但是删除sc，改成4后，PVC的状态就是Pending

volumetype: replicate:2 后，PVC就正常BOUND

在glusterfs这边能正常看到状态

secret不是base64的话，PVC也能正常BOUND吗？bmingithub Cauchy

bmingithub

看起来有两个问题：
1 . secret要base64的，应该填123456的base64后的，MTIzNDU2

volumetype: replicate:2 要根据你的集群情况填写，replicate:2表示2个glusterfs节点。
你可以就不急着重装，直接把原先的sc删掉，然后新建sc（因为sc的parameter不能edit），然后创建一个pvc试试看
先把sc的参数调正确了。

tzghost

配置了volumetype: replicate:2 后，采用的是分布式复制模式

tzghost

删除集群环境重新安装也是一样的问题

bmingithub

tzghost 环境方便发我看看么？
你的peer是3个，replicate:3和replicate:2应该都可以。但是4应该不对。

secret 应该是attach的时候用

tzghost

bmingithub 配置成replicate:2了，但看日志还是如下
[2020-09-03 08:15:30.037401] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists

节点：
glusterFS: 4个节点
master: 3个节点
work：3个节点
系统版本：CentOS Linux release 7.7

tzghost

kubernetes 1.18.6和1.17.9两个版本都是一样的问题，麻烦帮忙看下bmingithub

bmingithub

看下这个log里面报什么错

tzghost

bmingithub

tzghost

查找了相关的资料，看起来这个问题比较常见，不过没找到合适的解决方案。我用的是glusterfs 7.7，你们测试用的是哪个版本，有没有遇到过这个问题？bmingithub @Cauchy
https://github.com/gluster/gluster-kubernetes/issues/641
https://stackoverflow.com/questions/60683873/0-glusterfs-failed-to-set-volfile-server-file-exists

bmingithub

tzghost 我看了下，我用的版本比较老，glusterfs 3.12.15。

tzghost

已解决，节点间网络问题
https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md

kingsleyzhou

我也遇到了同样的问题，volfile server:file exists，请问是怎么处理的