多节点安装问题,卡在etcd报错

88fantasy · 2022年4月9日

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息
例如：虚拟机/物理机，Centos7.5/Ubuntu18.04，4C/8G

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

KubeSphere版本信息
例如：v3.2.1。离线安装还是在线安装。在已有K8s上安装还是使用kk安装。

问题是

error: Pipeline[CreateClusterPipeline] execute failed: Module[ETCDConfigureModule] exec failed:

failed: [master1] [ExistETCDHealthCheck] exec failed after 20 retires: etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCD

CTL_CERT_FILE=‘/etc/ssl/etcd/ssl/admin-master1.pem’;export ETCDCTL_KEY_FILE=‘/etc/ssl/etcd/ssl/admin-master1-key.pem’;export ETCDCTL_CA_FILE=‘/etc/ssl/etcd/ssl/ca.pem’;/usr/lo

cal/bin/etcdctl –endpoints=https://10.81.17.164:2379 cluster-health | grep -q ‘cluster is healthy’"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

详细日志为

[root@master1 k8s]# ./kk create cluster -f config-sample.yaml

_ __ _ _ __

| | / / | | | | / /

| |/ / _ _| |__ ___| |/ / ___ _ _

| \| | | | '_ \ / _ \ \ / _ \ | | |

| |\ \ |_| | |_) | __/ |\ \ __/ |_| |

\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |

__/ |

|___/

19:09:26 CST [NodePreCheckModule] A pre-check on nodes

19:09:27 CST success: [node2]

19:09:27 CST success: [master1]

19:09:27 CST success: [node1]

19:09:27 CST [ConfirmModule] Display confirmation form

+---------+------+------+---------+----------+-------+-------+-----------+--------+---------+------------+-------------+------------------+--------------+

+---------+------+------+---------+----------+-------+-------+-----------+--------+---------+------------+-------------+------------------+--------------+

| master1 | y | y | y | y | y | y | y | y | 20.10.8 | | | | CST 19:09:27 |

| node1 | y | y | y | y | y | y | y | y | 20.10.8 | | | | CST 19:09:22 |

| node2 | y | y | y | y | y | y | y | y | 20.10.8 | | | | CST 19:09:22 |

+---------+------+------+---------+----------+-------+-------+-----------+--------+---------+------------+-------------+------------------+--------------+

This is a simple check of your environment.

Before installation, you should ensure that your machines meet all requirements specified at

https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes

19:09:29 CST success: [LocalHost]

19:09:29 CST [NodeBinariesModule] Download installation binaries

19:09:29 CST message: [localhost]

downloading amd64 kubeadm v1.21.5 ...

19:09:29 CST message: [localhost]

kubeadm is existed

19:09:29 CST message: [localhost]

downloading amd64 kubelet v1.21.5 ...

19:09:30 CST message: [localhost]

kubelet is existed

19:09:30 CST message: [localhost]

downloading amd64 kubectl v1.21.5 ...

19:09:30 CST message: [localhost]

kubectl is existed

19:09:30 CST message: [localhost]

downloading amd64 helm v3.6.3 ...

19:09:30 CST message: [localhost]

helm is existed

19:09:30 CST message: [localhost]

downloading amd64 kubecni v0.9.1 ...

19:09:31 CST message: [localhost]

kubecni is existed

19:09:31 CST message: [localhost]

downloading amd64 docker 20.10.8 ...

19:09:31 CST message: [localhost]

docker is existed

19:09:31 CST message: [localhost]

downloading amd64 crictl v1.22.0 ...

19:09:31 CST message: [localhost]

crictl is existed

19:09:31 CST message: [localhost]

downloading amd64 etcd v3.4.13 ...

19:09:31 CST message: [localhost]

etcd is existed

19:09:31 CST success: [LocalHost]

19:09:31 CST [ConfigureOSModule] Prepare to init OS

19:09:32 CST success: [node1]

19:09:32 CST success: [node2]

19:09:32 CST success: [master1]

19:09:32 CST [ConfigureOSModule] Generate init os script

19:09:32 CST success: [master1]

19:09:32 CST success: [node2]

19:09:32 CST success: [node1]

19:09:32 CST [ConfigureOSModule] Exec init os script

19:09:33 CST stdout: [master1]

setenforce: SELinux is disabled

Disabled

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

19:09:33 CST stdout: [node1]

setenforce: SELinux is disabled

Disabled

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

19:09:33 CST stdout: [node2]

setenforce: SELinux is disabled

Disabled

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

19:09:33 CST success: [master1]

19:09:33 CST success: [node1]

19:09:33 CST success: [node2]

19:09:33 CST [ConfigureOSModule] configure the ntp server for each node

19:09:33 CST skipped: [node2]

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node1]

19:09:33 CST [KubernetesStatusModule] Get kubernetes cluster status

19:09:33 CST success: [master1]

19:09:33 CST [InstallContainerModule] Sync docker binaries

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node2]

19:09:33 CST skipped: [node1]

19:09:33 CST [InstallContainerModule] Generate containerd service

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node1]

19:09:33 CST skipped: [node2]

19:09:33 CST [InstallContainerModule] Enable containerd

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node2]

19:09:33 CST skipped: [node1]

19:09:33 CST [InstallContainerModule] Generate docker service

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node1]

19:09:33 CST skipped: [node2]

19:09:33 CST [InstallContainerModule] Generate docker config

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node2]

19:09:33 CST skipped: [node1]

19:09:33 CST [InstallContainerModule] Enable docker

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node2]

19:09:33 CST skipped: [node1]

19:09:33 CST [InstallContainerModule] Add auths to container runtime

19:09:33 CST skipped: [master1]

19:09:33 CST skipped: [node1]

19:09:33 CST skipped: [node2]

19:09:33 CST [PullModule] Start to pull images on all nodes

19:09:33 CST message: [master1]

downloading image: kubesphere/pause:3.4.1

19:09:33 CST message: [node2]

downloading image: kubesphere/pause:3.4.1

19:09:33 CST message: [node1]

downloading image: kubesphere/pause:3.4.1

19:09:36 CST message: [master1]

downloading image: kubesphere/kube-apiserver:v1.21.5

19:09:36 CST message: [node2]

downloading image: kubesphere/kube-proxy:v1.21.5

19:09:37 CST message: [node1]

downloading image: kubesphere/kube-proxy:v1.21.5

19:09:40 CST message: [master1]

downloading image: kubesphere/kube-controller-manager:v1.21.5

19:09:40 CST message: [node2]

downloading image: coredns/coredns:1.8.0

19:09:40 CST message: [node1]

downloading image: coredns/coredns:1.8.0

19:09:43 CST message: [master1]

downloading image: kubesphere/kube-scheduler:v1.21.5

19:09:43 CST message: [node1]

downloading image: kubesphere/k8s-dns-node-cache:1.15.12

19:09:43 CST message: [node2]

downloading image: kubesphere/k8s-dns-node-cache:1.15.12

19:09:46 CST message: [master1]

downloading image: kubesphere/kube-proxy:v1.21.5

19:09:46 CST message: [node2]

downloading image: calico/kube-controllers:v3.20.0

19:09:46 CST message: [node1]

downloading image: calico/kube-controllers:v3.20.0

19:09:50 CST message: [master1]

downloading image: coredns/coredns:1.8.0

19:09:50 CST message: [node2]

downloading image: calico/cni:v3.20.0

19:09:50 CST message: [node1]

downloading image: calico/cni:v3.20.0

19:09:53 CST message: [master1]

downloading image: kubesphere/k8s-dns-node-cache:1.15.12

19:09:53 CST message: [node2]

downloading image: calico/node:v3.20.0

19:09:53 CST message: [node1]

downloading image: calico/node:v3.20.0

19:09:56 CST message: [node2]

downloading image: calico/pod2daemon-flexvol:v3.20.0

19:09:56 CST message: [master1]

downloading image: calico/kube-controllers:v3.20.0

19:09:56 CST message: [node1]

downloading image: calico/pod2daemon-flexvol:v3.20.0

19:09:59 CST message: [master1]

downloading image: calico/cni:v3.20.0

19:10:03 CST message: [master1]

downloading image: calico/node:v3.20.0

19:10:06 CST message: [master1]

downloading image: calico/pod2daemon-flexvol:v3.20.0

19:10:09 CST success: [node2]

19:10:09 CST success: [node1]

19:10:09 CST success: [master1]

19:10:09 CST [ETCDPreCheckModule] Get etcd status

19:10:09 CST stdout: [master1]

ETCD_NAME=etcd-master1

19:10:09 CST success: [master1]

19:10:09 CST [CertsModule] Fetcd etcd certs

19:10:10 CST success: [master1]

19:10:10 CST [CertsModule] Generate etcd Certs

[certs] Using existing ca certificate authority

[certs] Using existing admin-master1 certificate and key on disk

[certs] Using existing member-master1 certificate and key on disk

[certs] Using existing node-master1 certificate and key on disk

19:10:10 CST success: [LocalHost]

19:10:10 CST [CertsModule] Synchronize certs file

19:10:11 CST success: [master1]

19:10:11 CST [CertsModule] Synchronize certs file to master

19:10:11 CST skipped: [master1]

19:10:11 CST [InstallETCDBinaryModule] Install etcd using binary

19:10:12 CST success: [master1]

19:10:12 CST [InstallETCDBinaryModule] Generate etcd service

19:10:12 CST success: [master1]

19:10:12 CST [InstallETCDBinaryModule] Generate access address

19:10:12 CST success: [master1]

19:10:12 CST [ETCDConfigureModule] Health check on exist etcd

19:10:12 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:12 CST retry: [master1]

19:10:18 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:18 CST retry: [master1]

19:10:23 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:23 CST retry: [master1]

19:10:28 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:28 CST retry: [master1]

19:10:33 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:33 CST retry: [master1]

19:10:38 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:38 CST retry: [master1]

19:10:43 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:43 CST retry: [master1]

19:10:48 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:48 CST retry: [master1]

19:10:53 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:53 CST retry: [master1]

19:10:58 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:10:58 CST retry: [master1]

19:11:03 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:03 CST retry: [master1]

19:11:08 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:08 CST retry: [master1]

19:11:13 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:13 CST retry: [master1]

19:11:18 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:18 CST retry: [master1]

19:11:23 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:23 CST retry: [master1]

19:11:28 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:28 CST retry: [master1]

19:11:33 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:33 CST retry: [master1]

19:11:39 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:39 CST retry: [master1]

19:11:44 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:44 CST retry: [master1]

19:11:49 CST message: [master1]

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_

FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health |

grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

19:11:49 CST failed: [master1]

error: Pipeline[CreateClusterPipeline] execute failed: Module[ETCDConfigureModule] exec failed:

failed: [master1] [ExistETCDHealthCheck] exec failed after 20 retires: etcd health check failed: Failed to exec command: sudo -E /bin/bash -c "export ETCDCTL_API=2;export ETCD

CTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master1.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-master1-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/lo

cal/bin/etcdctl --endpoints=https://10.81.17.164:2379 cluster-health | grep -q 'cluster is healthy'"

Error: client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

error #0: Forbidden: Process exited with status 1

24sama · 2022年4月11日

节点间是不是开了防火墙

gwinfo · 2022年4月19日

试试执行: ./kk delete cluster 然后重新安装：

export KKZONE=cn

./kk create cluster -f config-sample.yaml

doudecheng · 2022年6月1日

24sama

节点之间没有开防火墙，遇到同样的错误。centOS 7.8

doudecheng · 2022年6月1日

gwinfo

卸载之后重新安装，依然同样的报错

doudecheng · 2022年6月1日

centOS7.8 3个master节点，3个worker节点，etcd部署在master节点上，遇到同样的问题。

24sama · 2022年6月1日

doudecheng

如果是用的kk 2.1之后的版本，试试在config文件中，配置.etcd.type为kubeadm。

https://github.com/kubesphere/kubekey/blob/master/docs/config-example.md

doudecheng · 2022年6月2日

24sama

使用的kubekey是2.0.0版本的。我这边尝试了将etcd type 改为kubeadm 后依然报同样的错误。

具体错误：

etcd health check failed: Failed to exec command: sudo -E /bin/bash -c “export ETCDCTL_API=2;export ETCDCTL_CERT_FILE=‘/etc/ssl/etcd/ssl/admin-test-k8s-master3.pem’;export ETCDCTL_KEY_FILE=‘/etc/ssl/etcd/ssl/admin-test-k8s-master3-key.pem’;export ETCDCTL_CA_FILE=‘/etc/ssl/etcd/ssl/ca.pem’;/usr/local/bin/etcdctl –endpoints=https://ip:2379,https://ip:2379,https://ip:2379 cluster-health | grep -q ‘cluster is healthy’”

24sama · 2022年6月2日

doudecheng
2.1之后的版本才支持了这个配置。2.0版本不行

doudecheng · 2022年6月2日

24sama

可知道这个错误是什么原因导致的，我们开发环境也是按照同样的配置同样的安装环境装的，没有问题，执行kk的安装很顺利，现在按照同样的步骤安装测试环境却报这个错误。

24sama · 2022年6月2日

doudecheng
可能是防火墙，安全组，网络之类的问题。
可以手动把上面kk执行的命令复制下来，去掉 grep 执行一下，看一下命令返回的内容。
也可以到对应机器上用systemctl status etcd看一下具体节点上的etcd日志

epiphyllum · 2022年6月13日

我也遇到类似问题

这5台机器都能登录的。为啥kk 报不能访问呢？

24sama · 2022年6月13日

epiphyllum
hadoop100这台节点ssh到其他节点的端口是默认的22吗？

epiphyllum · 2022年6月14日

24sama 换2.0.0 就可以了。说明肯定是2.2.0有问题。

24sama · 2022年6月14日

epiphyllum
2.1.0之后的版本都加上了连接超时的pre-check，你应该是建立ssh的连接时间超时了（默认值：10秒）。

  - {name: master1, address: 192.168.101.128, internalAddress: 192.168.101.128, user: root, password: "123", timeout: 30}

可以参考上面的内容配置timeout，单位是秒

epiphyllum · 2022年6月14日

我机器都是虚拟机，不太可能30s超时，而且我都验证了，每台之间无密码登录都非常流畅。

下次我改下参数试试看看。

asdadasdad · 2022年7月8日

请问问题解决了嘛

epiphyllum

Iipbear · 2022年7月14日

你好，你的问题解决了吗？

Iipbear · 2022年7月14日

asdadasdad 就是将kubekey的版本更换为2.0.0就可以了

Xxulang · 2023年3月24日

这个最后怎么解决啊