ks3.0安装3master方式，第一遍安装后 openldap无法启动：启动后端口超时（此次etcd集群状态是正常的），解决无果后
，于是开始重置安装：
使用kk delete -f xx.yaml 重置集群后，第二次安装后却无法在master2和master3上启动etcd，

WARN[13:49:30 CST] Task failed ...
WARN[13:49:30 CST] error: Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master2.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-master2-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.22.215.70:2379,https://172.22.215.71:2379,https://172.22.215.72:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  tls: failed to find any PEM data in certificate input: Process exited with status 1
Error: Failed to start etcd cluster: Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master2.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-master2-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.22.215.70:2379,https://172.22.215.71:2379,https://172.22.215.72:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  tls: failed to find any PEM data in certificate input: Process exited with status 1
Usage:
  kk create cluster [flags]

Flags:
  -f, --filename string          Path to a configuration file
  -h, --help                     help for cluster
      --skip-pull-images         Skip pre pull images
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-kubesphere          Deploy a specific version of kubesphere (default v3.0.0)
  -y, --yes                      Skip pre-check of the installation

Global Flags:
      --debug   Print detailed information (default true)

Failed to start etcd cluster: Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-master2.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-master2-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.22.215.70:2379,https://172.22.215.71:2379,https://172.22.215.72:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  tls: failed to find any PEM data in certificate input: Process exited with status 1

查看master2系统日志报错：

Sep 10 11:00:12 0002 systemd: Started etcd docker wrapper.
Sep 10 11:00:12 0002 dockerd: time="2020-09-10T11:00:12.715170562+08:00" level=warning msg="Your kernel does not support Block I/O weight or the cgroup is not mounted. Weight discarded."
Sep 10 11:00:12 0002 etcd: WARNING: Your kernel does not support Block I/O weight or the cgroup is not mounted. Weight discarded.
Sep 10 11:00:12 0002 containerd: time="2020-09-10T11:00:12.753300735+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/160cccd59de17ca7f77b73550f27d0bda3416cce051b51b89176659ccf10da54/shim.sock" debug=false pid=2837
Sep 10 11:00:12 0002 systemd: Started libcontainer container 160cccd59de17ca7f77b73550f27d0bda3416cce051b51b89176659ccf10da54.
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850325 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://172.22.215.71:2379
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850389 I | pkg/flags: recognized and used environment variable ETCD_AUTO_COMPACTION_RETENTION=8
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850396 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-master2.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850403 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850412 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850420 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=5000
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850435 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=250
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850441 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.22.215.71:2380
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850461 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd1=https://172.22.215.70:2380,etcd2=https://172.22.215.71:2380,etcd3=https://172.22.215.72:2380
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850468 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850472 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850477 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-master2-key.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850492 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://172.22.215.71:2379,https://127.0.0.1:2379
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850514 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://172.22.215.71:2380
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850524 I | pkg/flags: recognized and used environment variable ETCD_METRICS=basic
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850531 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd2
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850547 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-master2.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850552 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=True
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850561 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-master2-key.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850565 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850569 I | pkg/flags: recognized and used environment variable ETCD_PROXY=off
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850578 I | pkg/flags: recognized and used environment variable ETCD_SNAPSHOT_COUNT=10000
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850584 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850625 I | etcdmain: etcd Version: 3.3.12
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850632 I | etcdmain: Git SHA: d57e8b8
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850636 I | etcdmain: Go Version: go1.10.8
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850639 I | etcdmain: Go OS/Arch: linux/amd64
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850644 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850695 I | embed: peerTLS: cert = /etc/ssl/etcd/ssl/member-master2.pem, key = /etc/ssl/etcd/ssl/member-master2-key.pem, ca = , trusted-ca = /etc/ssl/etcd/ssl/ca.pem, client-cert-auth = true, crl-file =
Sep 10 11:00:12 0002 etcd: 2020-09-10 03:00:12.850780 C | etcdmain: tls: failed to find any PEM data in certificate input
Sep 10 11:00:12 0002 containerd: time="2020-09-10T11:00:12.918680707+08:00" level=info msg="shim reaped" id=160cccd59de17ca7f77b73550f27d0bda3416cce051b51b89176659ccf10da54
Sep 10 11:00:12 0002 dockerd: time="2020-09-10T11:00:12.928728144+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 10 11:00:12 0002 systemd: etcd.service: main process exited, code=exited, status=1/FAILURE
Sep 10 11:00:13 0002 docker: etcd2
Sep 10 11:00:13 0002 systemd: Unit etcd.service entered failed state.
Sep 10 11:00:13 0002 systemd: etcd.service failed.

错误：tls: failed to find any PEM data in certificate input

错误2：
kubelet找不到(可能跟etcd的状态没启动有关)?
Failed at step EXEC spawning /usr/local/bin/kubelet: No such file or directory -- Subject: Process /usr/local/bin/kubelet could not be executed

[root@master2 ~]# ll /usr/local/bin/
total 54884
-rwxr-xr-x 1 kube root      347 Sep 10 13:47 etcd
-rwxr-xr-x 1 root root 15817472 Sep 10 13:47 etcdctl
-rwxr-xr-x 1 kube root 40378368 Sep  8 15:44 helm
drwxr-xr-x 2 kube root       23 Sep  8 15:40 kube-scripts

#journalctl -xe
Sep 10 13:57:11 master2 systemd[1]: etcd.service failed.
Sep 10 13:57:16 master2 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Sep 10 13:57:16 master2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.
Sep 10 13:57:16 master2 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Sep 10 13:57:16 master2 systemd[20376]: Failed at step EXEC spawning /usr/local/bin/kubelet: No such file or directory
-- Subject: Process /usr/local/bin/kubelet could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/local/bin/kubelet could not be executed and failed.
--
-- The error number returned by this process is 2.
Sep 10 13:57:16 master2 systemd[1]: kubelet.service: main process exited, code=exited, status=203/EXEC
Sep 10 13:57:16 master2 systemd[1]: Unit kubelet.service entered failed state.
Sep 10 13:57:16 master2 systemd[1]: kubelet.service failed.

请教大佬如何调整呢