V3.0.0 多节点安装报错

Cauchy

[ERROR FileExisting-conntrack]: conntrack not found in system path
机器上把conntrack装下。

https://github.com/kubesphere/kubekey#requirements-and-recommendations

tzghost

存储是使用addons的方式，这样有问题吗？ Cauchy

  addons: 
  - name: glusterfs
    namespace: kube-system
    sources:
      yaml:
        path:
        - /root/ks/glusterfs.yaml  # or https://raw.githubusercontent.com/xxx/glusterfs.yaml

tzghost

安装环境后，glusterfs这块失败了

INFO[16:14:35 CST] Refreshing etcd configuration                

INFO[16:14:36 CST] Get cluster status                           

[master1 192.168.0.55] MSG:
Cluster already exists.
[master1 192.168.0.55] MSG:
v1.18.6
[master1 192.168.0.55] MSG:
I0902 16:14:37.972377   11514 version.go:252] remote version is much newer: v1.19.0; falling back to: stable-1.18
W0902 16:14:38.570091   11514 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
9ae8cea2c2c21340426086c215ab4da35f9ab8c40dba8edc543ac9e1c75b91c4
[master1 192.168.0.55] MSG:
secret/kubeadm-certs patched
[master1 192.168.0.55] MSG:
secret/kubeadm-certs patched
[master1 192.168.0.55] MSG:
secret/kubeadm-certs patched
[master1 192.168.0.55] MSG:
W0902 16:14:40.543906   11768 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join lb.kubesphere.local:6443 --token rcsqqh.t2ejxp9u7vsuuk2x     --discovery-token-ca-cert-hash sha256:ebb5cbf4873f17ebe5aa5b238f1b1f2a52290e2ae754b198e037ab6e7bbbaa4e
[master1 192.168.0.55] MSG:
NAME      STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
master1   Ready    master   13m   v1.18.6   192.168.0.55   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.8
master2   Ready    master   12m   v1.18.6   192.168.0.58   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.8
master3   Ready    master   12m   v1.18.6   192.168.0.59   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://1.13.1
node1     Ready    worker   12m   v1.18.6   192.168.0.52   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.8
node2     Ready    worker   12m   v1.18.6   192.168.0.60   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.8
node3     Ready    worker   12m   v1.18.6   192.168.0.61   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64   docker://19.3.8
INFO[16:14:41 CST] Installing kube binaries                     

INFO[16:14:41 CST] Initializing kubernetes cluster              

INFO[16:14:41 CST] Deploying network plugin ...                 

[master1 192.168.0.55] MSG:
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged
INFO[16:14:44 CST] Joining nodes to cluster                     

INFO[16:14:44 CST] Installing addon [1-1]: glusterfs            

secret/heketi-secret serverside-applied
WARN[16:14:44 CST] Task failed ...                              

WARN[16:14:44 CST] error: failed to create typed patch object: errors:
  .parameters.gidMax: expected string, got &value.valueUnstructured{Value:50000}
  .parameters.gidMin: expected string, got &value.valueUnstructured{Value:40000}
  .parameters.restauthenabled: expected string, got &value.valueUnstructured{Value:true} 
Error: Failed to deploy addons: failed to create typed patch object: errors:
  .parameters.gidMax: expected string, got &value.valueUnstructured{Value:50000}
  .parameters.gidMin: expected string, got &value.valueUnstructured{Value:40000}
  .parameters.restauthenabled: expected string, got &value.valueUnstructured{Value:true}
Usage:
  kk create cluster [flags]

Flags:
  -f, --filename string          Path to a configuration file
  -h, --help                     help for cluster
      --skip-pull-images         Skip pre pull images
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-kubesphere          Deploy a specific version of kubesphere (default v3.0.0)
  -y, --yes                      Skip pre-check of the installation

Global Flags:
      --debug   Print detailed information (default true)

Failed to deploy addons: failed to create typed patch object: errors:
  .parameters.gidMax: expected string, got &value.valueUnstructured{Value:50000}
  .parameters.gidMin: expected string, got &value.valueUnstructured{Value:40000}
  .parameters.restauthenabled: expected string, got &value.valueUnstructured{Value:true}

Cauchy

tzghost 格式不对哦，k8s对sc的参数类型有要求，下边那些非字符串的参数加上引号试试

parameters:
  resturl: http://192.168.0.4:8080
  clusterid: 6a6792ed25405eaa6302da99f2f5e24b
  restauthenabled: “true”
  restuser: admin
  secretNamespace: "kube-system"
  secretName: heketi-secret
  gidMin: “40000“
  gidMax: ”50000”
  volumetype: “replicate:2”
allowVolumeExpansion: true

tzghost

---
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: kube-system
type: kubernetes.io/glusterfs
data:
  # e.g. jwt_admin_key: 123456
  key: 123456

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name:  glusterfs
  annotations:
    storageclass.kubesphere.io/supported-access-modes: '["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]'
    # default storage
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
parameters:
  # e.g. glusterfs_provisioner_resturl: http://192.168.0.4:8080
  resturl: http://192.168.0.50:8080
  # e.g. glusterfs_provisioner_clusterid: 6a6792ed25405eaa6302da99f2f5e24b
  clusterid: 55058919a7641bcc64d1b8763200843a
  restauthenabled: true
  restuser: admin
  secretNamespace: "kube-system"
  secretName: heketi-secret
  gidMin: 40000
  gidMax: 50000
  volumetype: replicate:2
allowVolumeExpansion: true

tzghost

Cauchy 安装正常了，初始密码在哪里？

clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged
INFO[16:33:58 CST] Joining nodes to cluster                     

INFO[16:33:58 CST] Installing addon [1-1]: glusterfs            

secret/heketi-secret serverside-applied
storageclass.storage.k8s.io/glusterfs serverside-applied
INFO[16:33:59 CST] Deploying KubeSphere ...                     

v3.0.0
[master1 192.168.0.55] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system created
[master1 192.168.0.55] MSG:
secret/kube-etcd-client-certs created
[master1 192.168.0.55] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created


INFO[16:39:18 CST] Installation is complete.

Please check the result using the command:

       kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

tzghost

显示卷挂载失败，请问这种问题要怎么处理？Cauchy

tzghost

发现30880端口没有起来

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

tcp        0      0 127.0.0.1:15106         0.0.0.0:*               LISTEN      24369/kubelet       

tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1698/zabbix_agentd  

tcp        0      0 169.254.25.10:9254      0.0.0.0:*               LISTEN      25248/node-cache    

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      24369/kubelet       

tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      24956/kube-proxy    

tcp        0      0 192.168.0.55:2379       0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 127.0.0.1:9099          0.0.0.0:*               LISTEN      26148/calico-node   

tcp        0      0 192.168.0.55:2380       0.0.0.0:*               LISTEN      4176/etcd           

tcp        0      0 0.0.0.0:179             0.0.0.0:*               LISTEN      26268/bird          

tcp        0      0 169.254.25.10:53        0.0.0.0:*               LISTEN      25248/node-cache    

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      7054/sshd           

tcp6       0      0 :::10050                :::*                    LISTEN      1698/zabbix_agentd  

tcp6       0      0 :::9253                 :::*                    LISTEN      25248/node-cache    

tcp6       0      0 :::9353                 :::*                    LISTEN      25248/node-cache    

tcp6       0      0 :::10250                :::*                    LISTEN      24369/kubelet       

tcp6       0      0 :::10251                :::*                    LISTEN      4810/kube-scheduler 
tcp6       0      0 :::6443                 :::*                    LISTEN      24064/kube-apiserve 
tcp6       0      0 :::10252                :::*                    LISTEN      4803/kube-controlle 
tcp6       0      0 :::10256                :::*                    LISTEN      24956/kube-proxy    

tcp6       0      0 :::10257                :::*                    LISTEN      4803/kube-controlle 
tcp6       0      0 :::10259                :::*                    LISTEN      4810/kube-scheduler 
tcp6       0      0 :::22                   :::*                    LISTEN      7054/sshd

tzghost

[root@BG-003-KVM004-VMS003 ~]# kubectl get pods --all-namespaces 
NAMESPACE           NAME                                       READY   STATUS              RESTARTS   AGE
kube-system         calico-kube-controllers-578894d4cd-5scsq   1/1     Running             0          59m
kube-system         calico-node-6hdrt                          1/1     Running             0          58m
kube-system         calico-node-78gdl                          1/1     Running             0          58m
kube-system         calico-node-dpdvf                          1/1     Running             0          59m
kube-system         calico-node-k99wz                          1/1     Running             0          58m
kube-system         calico-node-mz2tf                          1/1     Running             0          58m
kube-system         calico-node-x8d9s                          1/1     Running             0          58m
kube-system         coredns-6b55b6764d-4xccb                   1/1     Running             0          59m
kube-system         coredns-6b55b6764d-mrv99                   1/1     Running             0          59m
kube-system         kube-apiserver-master1                     1/1     Running             0          59m
kube-system         kube-apiserver-master2                     1/1     Running             0          58m
kube-system         kube-apiserver-master3                     1/1     Running             0          58m
kube-system         kube-controller-manager-master1            1/1     Running             1          59m
kube-system         kube-controller-manager-master2            1/1     Running             0          58m
kube-system         kube-controller-manager-master3            1/1     Running             0          58m
kube-system         kube-proxy-4gzww                           1/1     Running             0          58m
kube-system         kube-proxy-74z5f                           1/1     Running             0          58m
kube-system         kube-proxy-7zgqv                           1/1     Running             0          59m
kube-system         kube-proxy-96mpk                           1/1     Running             0          58m
kube-system         kube-proxy-nscqh                           1/1     Running             0          58m
kube-system         kube-proxy-qt767                           1/1     Running             0          58m
kube-system         kube-scheduler-master1                     1/1     Running             1          59m
kube-system         kube-scheduler-master2                     1/1     Running             0          58m
kube-system         kube-scheduler-master3                     1/1     Running             0          58m
kube-system         metrics-server-f47758d8f-f5csr             1/1     Running             0          20m
kube-system         nodelocaldns-ddx49                         1/1     Running             0          59m
kube-system         nodelocaldns-nn2ts                         1/1     Running             0          58m
kube-system         nodelocaldns-rzfqs                         1/1     Running             0          58m
kube-system         nodelocaldns-w48jc                         1/1     Running             0          58m
kube-system         nodelocaldns-wxqbg                         1/1     Running             0          58m
kube-system         nodelocaldns-xnw5s                         1/1     Running             0          58m
kube-system         snapshot-controller-0                      1/1     Running             0          18m
kubesphere-system   ks-installer-74757f9779-bxj4z              1/1     Running             0          26m
kubesphere-system   minio-7bfdb5968b-vmb2k                     0/1     ContainerCreating   0          17m
kubesphere-system   openldap-0                                 1/1     Running             0          17m
kubesphere-system   openldap-1                                 0/1     ContainerCreating   0          8m20s
kubesphere-system   redis-ha-haproxy-5c6559d588-2mwz7          1/1     Running             0          17m
kubesphere-system   redis-ha-haproxy-5c6559d588-ff8jv          1/1     Running             3          17m
kubesphere-system   redis-ha-haproxy-5c6559d588-htbxd          0/1     Init:0/1            0          17m
kubesphere-system   redis-ha-server-0                          2/2     Running             0          17m
kubesphere-system   redis-ha-server-1                          0/2     Init:0/1            0          13m

Cauchy

describe看看那个minio为啥还在创建中，是不是网络不好，还在下镜像

Cauchy

节点上装glusterfs的client了么？

tzghost

Cauchy 都装上了的

tzghost

我的glusterfs是4个节点，但是删除sc，改成4后，PVC的状态就是Pending

volumetype: replicate:2 后，PVC就正常BOUND

在glusterfs这边能正常看到状态

secret不是base64的话，PVC也能正常BOUND吗？bmingithub Cauchy

bmingithub

看起来有两个问题：
1 . secret要base64的，应该填123456的base64后的，MTIzNDU2

volumetype: replicate:2 要根据你的集群情况填写，replicate:2表示2个glusterfs节点。
你可以就不急着重装，直接把原先的sc删掉，然后新建sc（因为sc的parameter不能edit），然后创建一个pvc试试看
先把sc的参数调正确了。

tzghost

配置了volumetype: replicate:2 后，采用的是分布式复制模式

tzghost

删除集群环境重新安装也是一样的问题

bmingithub

tzghost 环境方便发我看看么？
你的peer是3个，replicate:3和replicate:2应该都可以。但是4应该不对。

secret 应该是attach的时候用

tzghost

bmingithub 配置成replicate:2了，但看日志还是如下
[2020-09-03 08:15:30.037401] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists

节点：
glusterFS: 4个节点
master: 3个节点
work：3个节点
系统版本：CentOS Linux release 7.7

tzghost

kubernetes 1.18.6和1.17.9两个版本都是一样的问题，麻烦帮忙看下bmingithub

bmingithub

看下这个log里面报什么错

tzghost

bmingithub

tzghost

查找了相关的资料，看起来这个问题比较常见，不过没找到合适的解决方案。我用的是glusterfs 7.7，你们测试用的是哪个版本，有没有遇到过这个问题？bmingithub @Cauchy
https://github.com/gluster/gluster-kubernetes/issues/641
https://stackoverflow.com/questions/60683873/0-glusterfs-failed-to-set-volfile-server-file-exists