v4.1.2 離線安裝報錯 etcd health check failed

Cauchy
kubelet沒有起來，現在etcd正常執行

h00283@coverity-ms:~$ sudo systemctl status kubelet
Warning: The unit file, source configuration file or drop-ins of kubelet.service changed on disk. Run 'systemctl daemon-reload' to reload units.
○ kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: http://kubernetes.io/docs/

Feb 10 14:13:29 coverity-ms kubelet[9858]: E0210 14:13:29.319649    9858 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://lb.kubesp>
Feb 10 14:13:31 coverity-ms kubelet[9858]: E0210 14:13:31.987813    9858 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, Obje>
Feb 10 14:13:32 coverity-ms kubelet[9858]: E0210 14:13:32.614103    9858 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node i>
Feb 10 14:13:33 coverity-ms kubelet[9858]: W0210 14:13:33.068103    9858 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Ge>
Feb 10 14:13:33 coverity-ms kubelet[9858]: E0210 14:13:33.068169    9858 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: f>
Feb 10 14:13:33 coverity-ms systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Feb 10 14:13:33 coverity-ms kubelet[9858]: I0210 14:13:33.123346    9858 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/>
Feb 10 14:13:33 coverity-ms systemd[1]: kubelet.service: Deactivated successfully.
Feb 10 14:13:33 coverity-ms systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 10 14:13:33 coverity-ms systemd[1]: kubelet.service: Consumed 1.825s CPU time.

h00283@coverity-ms:~/ks$ sudo journalctl -u kubelet -f
[sudo] password for h00283: 
Feb 10 15:07:16 coverity-ms kubelet[16512]: E0210 15:07:16.938712   16512 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Feb 10 15:07:16 coverity-ms systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 15:07:16 coverity-ms systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 10 15:07:27 coverity-ms systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 90.
Feb 10 15:07:27 coverity-ms systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 10 15:07:27 coverity-ms systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 10 15:07:27 coverity-ms kubelet[16541]: E0210 15:07:27.189671   16541 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Feb 10 15:07:27 coverity-ms systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 15:07:27 coverity-ms systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 10 15:07:33 coverity-ms systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

h00283@coverity-ms:~/ks$ kubeadm config images list
W0210 16:27:02.652703   20003 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0210 16:27:02.652766   20003 version.go:105] falling back to the local client version: v1.28.0
registry.k8s.io/kube-apiserver:v1.28.0
registry.k8s.io/kube-controller-manager:v1.28.0
registry.k8s.io/kube-scheduler:v1.28.0
registry.k8s.io/kube-proxy:v1.28.0

Cauchy
我用預設的沒改其他東西

h00283@coverity-ms:~/ks$ sudo journalctl -u kubelet -f
Feb 10 16:33:37 coverity-ms kubelet[20129]: E0210 16:33:37.612226   20129 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://lb.kubesphere.local:6443/api/v1/nodes\": dial tcp 172.1.30.21:6443: connect: connection refused" node="coverity-ms"
Feb 10 16:33:37 coverity-ms kubelet[20129]: E0210 16:33:37.974136   20129 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp 34.96.108.209:443: i/o timeout"
Feb 10 16:33:37 coverity-ms kubelet[20129]: E0210 16:33:37.974191   20129 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp 34.96.108.209:443: i/o timeout" pod="kube-system/kube-scheduler-coverity-ms"
Feb 10 16:33:37 coverity-ms kubelet[20129]: E0210 16:33:37.974223   20129 kuberuntime_manager.go:1119] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp 34.96.108.209:443: i/o timeout" pod="kube-system/kube-scheduler-coverity-ms"
Feb 10 16:33:37 coverity-ms kubelet[20129]: E0210 16:33:37.974304   20129 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-scheduler-coverity-ms_kube-system(b68b9e35fcab51848c5f2ecaf37ba14d)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-scheduler-coverity-ms_kube-system(b68b9e35fcab51848c5f2ecaf37ba14d)\\\": rpc error: code = DeadlineExceeded desc = failed to get sandbox image \\\"registry.k8s.io/pause:3.8\\\": failed to pull image \\\"registry.k8s.io/pause:3.8\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.8\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.8\\\": failed to do request: Head \\\"https://registry.k8s.io/v2/pause/manifests/3.8\\\": dial tcp 34.96.108.209:443: i/o timeout\"" pod="kube-system/kube-scheduler-coverity-ms" podUID="b68b9e35fcab51848c5f2ecaf37ba14d"
Feb 10 16:33:38 coverity-ms kubelet[20129]: I0210 16:33:38.768345   20129 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Feb 10 16:33:38 coverity-ms systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Feb 10 16:33:38 coverity-ms systemd[1]: kubelet.service: Deactivated successfully.
Feb 10 16:33:38 coverity-ms systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 10 16:33:38 coverity-ms systemd[1]: kubelet.service: Consumed 1.265s CPU time.

init時查的log

Cauchy 我發現docker image很多沒拉下來成功導致kubelet kubeadm containerd等等的沒在運作

已經成功了!

—

過程：
把相關安裝的工具都先移除，資料夾也是

發現

1. etcd 跟 containerd 本來就系統安裝過，導致版本有問題 => 移除安裝

2. kubelet、kubectl 未安裝完成，資料夾裡缺少相關配置 => 資料夾刪除乾淨

3. sudo ./kk delete cluster -f config-sample.yaml

4. 移除 kubekey 資料夾

5. sudo ./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-local-storage

6. 執行到 [init cluster using kubeadm] 的時候:
   (1) 確認 etcd 狀態 sudo systemctl status etcd => 確定沒有錯誤訊息

   (2) 確認 containerd 狀態 sudo systemctl status containerd

   　　<遇到鏡像拉取失敗問題>
   　　檢查有沒有這份文件 /etc/containerd/config.toml => 沒有的話從有的節點複製過去，重複上面 delete cluster 動作，還是有報錯的話繼續解決，直到沒有錯的時候應該就能順利成功 create cluster 了

7. create cluster 完成後就可以安裝 KubeSphere

我的 /etc/containerd/config.toml 沒有自己建立起來，加進去之後也順利create cluste，你也能檢查一下