V3.1.0 添加节点报错

bjdaijun

./kk add nodes -f sample.yaml

报错信息如下：

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
ERRO[21:13:50 CST] Failed to add worker to cluster: Failed to exec command: sudo env PATH=$PATH /bin/sh -c "/usr/local/bin/kubeadm join  lb.kubesphere.local:6443 --token imm2la.aypw53popwr00o2l     --discovery-token-ca-cert-hash sha256:79dd25eac4df250b7e739ceb1d6fd083b57319a45508d1b1cf7f91cd67cda720"
[preflight] Running pre-flight checks
        [WARNING FileExisting-ethtool]: ethtool not found in system path
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.2. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0701 21:11:55.359893   24030 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
error execution phase kubelet-start: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1  node=192.168.1.83
WARN[21:13:50 CST] Task failed ...
WARN[21:13:50 CST] error: interrupted by error
Error: Failed to join node: interrupted by error

bjdaijun

在192.168.1.83执行 journalctl -xeu kubelet，报错信息如下：

7月 01 21:15:13 k8s3 kubelet[25957]: W0701 21:15:13.395716   25957 feature_gate.go:235] Setting GA feature gate CSINodeInfo=true. It will be removed in a future release.
7月 01 21:15:13 k8s3 kubelet[25957]: W0701 21:15:13.395904   25957 feature_gate.go:235] Setting GA feature gate RotateKubeletClientCertificate=true. It will be removed in a future release.
7月 01 21:15:13 k8s3 kubelet[25957]: W0701 21:15:13.396095   25957 feature_gate.go:235] Setting GA feature gate VolumeSnapshotDataSource=true. It will be removed in a future release.
7月 01 21:15:13 k8s3 kubelet[25957]: I0701 21:15:13.396552   25957 server.go:837] Client rotation is on, will bootstrap in background
7月 01 21:15:13 k8s3 kubelet[25957]: E0701 21:15:13.398215   25957 bootstrap.go:240] unable to read existing bootstrap client config: invalid configuration: [unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem for
7月 01 21:15:13 k8s3 kubelet[25957]: F0701 21:15:13.398471   25957 server.go:269] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
7月 01 21:15:13 k8s3 kubelet[25957]: goroutine 1 [running]:
7月 01 21:15:13 k8s3 kubelet[25957]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000010001, 0xc000c7e000, 0xb0, 0x102)
7月 01 21:15:13 k8s3 kubelet[25957]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
7月 01 21:15:13 k8s3 kubelet[25957]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x70e94e0, 0xc000000003, 0x0, 0x0, 0xc000c46230, 0x6f53cea, 0x9, 0x10d, 0x411b00)
7月 01 21:15:13 k8s3 kubelet[25957]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x19b
7月 01 21:15:13 k8s3 kubelet[25957]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x70e94e0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc000c606d0, 0x1, 0x1)
7月 01 21:15:13 k8s3 kubelet[25957]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:732 +0x16f
7月 01 21:15:13 k8s3 kubelet[25957]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).print(...)
7月 01 21:15:13 k8s3 kubelet[25957]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:714

kevendeng

bjdaijun 虽然通过更换操作系统解决了，但是要注意的是你在报错后手动执行kubelet时的第二份报错日志是无效信息：kubeadm在添加节点失败后会进行清理操作，删除该节点上kubelet的配置文件，而你手动运行kubelet的报错正是找不到这些文件，还没有到root cause那一步。

bjdaijun

换了操作系统ubuntu1804 server ，重新安装，问题解决。