集群安装报error: Failed to patch kubeadm secret

fcemails

集群安装过程中报以下错误，大神们帮看看要怎么处理好

[upload-certs] Using certificate key:
e45861790f393d35a970c872dc6e9103c4d7305d5cf1fdf377c821c06
WARN[17:36:29 CST] Task failed ...                              
WARN[17:36:29 CST] error: Failed to patch kubeadm secret: Failed to exec command: sudo -E /bin/sh -c "/usr/local/bin/kubectl patch -n kube-system secret kubeadm-certs -p '{\"data\": {\"external-etcd-ca.crt\": \"\"}}'" 
The connection to the server localhost:8080 was refused - did you specify the right host or port?: Process exited with status 1 
Error: Failed to get cluster status: Failed to patch kubeadm secret: Failed to exec command: sudo -E /bin/sh -c "/usr/local/bin/kubectl patch -n kube-system secret kubeadm-certs -p '{\"data\": {\"external-etcd-ca.crt\": \"\"}}'" 
The connection to the server localhost:8080 was refused - did you specify the right host or port?: Process exited with status 1
Usage:
  kk create cluster [flags]

Flags:
  -f, --filename string          Path to a configuration file
  -h, --help                     help for cluster
      --skip-pull-images         Skip pre pull images
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-kubesphere          Deploy a specific version of kubesphere (default v3.0.0)
  -y, --yes                      Skip pre-check of the installation

Global Flags:
      --debug   Print detailed information (default true)

Failed to get cluster status: Failed to patch kubeadm secret: Failed to exec command: sudo -E /bin/sh -c "/usr/local/bin/kubectl patch -n kube-system secret kubeadm-certs -p '{\"data\": {\"external-etcd-ca.crt\": \"\"}}'" 
The connection to the server localhost:8080 was refused - did you specify the right host or port?: Process exited with status 1

Cauchy

fcemails

这个看起来是由于没有kubeconfig导致的，可以执行卸载之后尝试重新执行安装。

卸载命令：

./kk delete cluster -f xxx.yaml

fcemails

Cauchy 缷载之后重新安装，到了这一步又出错了，麻烦再帮忙看看如何解决，谢谢

INFO[13:49:13 CST] Initializing kubernetes cluster              
[master1 172.20.253.190] MSG:
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1209 13:53:17.237038    2421 reset.go:99] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://lb.kubesphere.local:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s: EOF
[preflight] Running pre-flight checks
W1209 13:53:17.237158    2421 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[master1 172.20.253.190] MSG:
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1209 13:55:38.300141    4187 config.go:72] Warning: No kube-proxy config is loaded. Continuing without it: configmaps "kube-proxy" not found
W1209 13:55:38.321502    4187 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
[preflight] Running pre-flight checks
[reset] Removing info for node "master1" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
W1209 13:55:38.853935    4187 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
ERRO[13:59:43 CST] Failed to init kubernetes cluster: Failed to exec command: sudo -E /bin/sh -c "/usr/local/bin/kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml" 
W1209 13:55:39.921708    4417 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
W1209 13:55:39.921903    4417 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1209 13:55:39.921916    4417 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.9
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet

Cauchy

fcemails

这个报错信息没有发完整吧？
[kubelet-start] Starting the kubelet 这行下边应该还有内容。

fcemails

Cauchy 是的，补充下上图后面没发完整的内容

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local lb.kubesphere.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost lb.kubesphere.local master1 master1.cluster.local master2 master2.cluster.local master3 master3.cluster.local node1 node1.cluster.local node2 node2.cluster.local node3 node3.cluster.local] and IPs [10.233.0.1 172.20.253.190 127.0.0.1 172.20.253.183 172.20.253.190 172.20.253.191 172.20.253.192 172.20.253.193 172.20.253.194 172.20.253.195 10.233.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[controlplane] Adding extra host path mount "host-time" to "kube-controller-manager"
W1209 13:55:43.653742    4417 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[controlplane] Adding extra host path mount "host-time" to "kube-controller-manager"
W1209 13:55:43.660792    4417 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[controlplane] Adding extra host path mount "host-time" to "kube-controller-manager"
W1209 13:55:43.662516    4417 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1  node=172.20.253.190
WARN[13:59:43 CST] Task failed ...                              
WARN[13:59:43 CST] error: interrupted by error                  
Error: Failed to init kubernetes cluster: interrupted by error
Usage:
  kk create cluster [flags]

Flags:
  -f, --filename string          Path to a configuration file
  -h, --help                     help for cluster
      --skip-pull-images         Skip pre pull images
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-kubesphere          Deploy a specific version of kubesphere (default v3.0.0)
  -y, --yes                      Skip pre-check of the installation

Global Flags:
      --debug   Print detailed information (default true)

Failed to init kubernetes cluster: interrupted by error

Cauchy

fcemails

看报错应该kubelet没有启动，可以systemctl status kubelet看下kubelet状态，然后执行journalctl -f -u kubelet 看下kubelet 日志

fcemails

Cauchy
[root@master1 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 三 2020-12-09 13:55:51 CST; 1 day 1h ago
Docs: http://kubernetes.io/docs/
Main PID: 4680 (kubelet)
Tasks: 20
Memory: 43.6M
CGroup: /system.slice/kubelet.service
└─4680 /usr/local/bin/kubelet –bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf –kubeconfig=/etc/kubernetes/kubelet.conf –config=/var/lib/kubelet/config.yaml –cgroup-driver=systemd –network-plugin=cni –pod-infra-container-image=kubesphere/…

12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.114183 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.119170 12月 10 15:01:46 master1 kubelet[4680]: I1210 15:01:46.120823 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.124320 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.128881 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.133425 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.138104 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.142894 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.144632 12月 10 15:01:46 master1 kubelet[4680]: E1210 15:01:46.381953 Hint: Some lines were ellipsized, use -l to show in full.
[root@master1 ~]# journalctl -f -u kubelet
– Logs begin at 四 2020-12-10 09:44:04 CST. –
12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.393818 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.398364 12月 10 15:02:56 master1 kubelet[4680]: I1210 15:02:56.400043 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.403649 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.412536 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.417582 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.422247 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.426969 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.428650 12月 10 15:02:56 master1 kubelet[4680]: E1210 15:02:56.524419 12月 10 15:03:00 master1 kubelet[4680]: W1210 15:03:00.325542 12月 10 15:03:01 master1 kubelet[4680]: E1210 15:03:01.534162 4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:105] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update node lease
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node…
4680 controller.go:113] failed 5 attempts to update node lease, will retry after 10s
4680 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:105] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update node lease
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io “master1”: StorageError: invalid object, Code: 4, Key: /registry/leases/kube-node-lease/master1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aaa3416a-ba43-433d-bf51-3ed2e4d954fc, UID in object meta:
4680 controller.go:113] failed 5 attempts to update node lease, will retry after 10s
4680 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
4680 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
4680 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized