操作系统信息
实体机器，108核128G
操作系统：

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12", GitCommit:"b058e1760c79f46a834ba59bd7a3486ecf28237d", GitTreeState:"clean", BuildDate:"2022-07-13T14:59:18Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12", GitCommit:"b058e1760c79f46a834ba59bd7a3486ecf28237d", GitTreeState:"clean", BuildDate:"2022-07-13T14:53:39Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

Client:
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:50:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:55:09 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b638
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

crictl version
没有这个命令
nerdctl version
没有这个命令

KubeSphere版本信息
例如：v2.1.1/v3.0.0。离线安装还是在线安装。在已有K8s上安装还是使用kk安装。

kk version: &version.Info{Major:"3", Minor:"0", GitVersion:"v3.0.7", GitCommit:"e755baf67198d565689d7207378174f429b508ba", GitTreeState:"clean", BuildDate:"2023-01-18T01:57:24Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

离线安装

问题是什么
已经离线安装成功了，重启了整个集群之后，web页面登录不了，提示“request to http://ks-apiserver/oauth/token failed, reason: getaddrinfo EAI_AGAIN ks-apiserver”

NAMESPACE                      NAME                                               READY   STATUS             RESTARTS        AGE
kube-system                    calico-kube-controllers-846ddd49bc-6szkv           1/1     Running            5 (64s ago)     28h
kube-system                    calico-node-8tkmg                                  1/1     Running            6 (112m ago)    28h
kube-system                    calico-node-mbh7m                                  1/1     Running            2 (116m ago)    28h
kube-system                    calico-node-mpvtq                                  1/1     Running            3 (116m ago)    28h
kube-system                    calico-node-sz7qm                                  1/1     Running            2 (116m ago)    28h
kube-system                    coredns-558b97598-fwhww                            1/1     Running            3 (116m ago)    28h
kube-system                    coredns-558b97598-s64mn                            1/1     Running            3 (116m ago)    28h
kube-system                    haproxy-worker-216-01                              1/1     Running            2 (116m ago)    28h
kube-system                    kube-apiserver-master-213-01                       1/1     Running            37 (115m ago)   28h
kube-system                    kube-apiserver-master-214-02                       1/1     Running            31 (116m ago)   28h
kube-system                    kube-apiserver-master-215-03                       1/1     Running            31 (116m ago)   28h
kube-system                    kube-controller-manager-master-213-01              1/1     Running            7 (116m ago)    28h
kube-system                    kube-controller-manager-master-214-02              1/1     Running            4 (116m ago)    28h
kube-system                    kube-controller-manager-master-215-03              1/1     Running            3 (116m ago)    28h
kube-system                    kube-proxy-4dpjk                                   1/1     Running            2 (116m ago)    28h
kube-system                    kube-proxy-dlfjk                                   1/1     Running            7 (112m ago)    28h
kube-system                    kube-proxy-mj6fp                                   1/1     Running            4 (116m ago)    28h
kube-system                    kube-proxy-x9nwz                                   1/1     Running            2 (116m ago)    28h
kube-system                    kube-scheduler-master-213-01                       1/1     Running            7 (116m ago)    28h
kube-system                    kube-scheduler-master-214-02                       1/1     Running            3 (116m ago)    28h
kube-system                    kube-scheduler-master-215-03                       1/1     Running            3 (116m ago)    28h
kube-system                    nodelocaldns-9tvcv                                 0/1     CrashLoopBackOff   80 (25s ago)    28h
kube-system                    nodelocaldns-ddvg2                                 0/1     CrashLoopBackOff   95 (35s ago)    28h
kube-system                    nodelocaldns-fmmrm                                 0/1     CrashLoopBackOff   61 (32s ago)    28h
kube-system                    nodelocaldns-g4f4x                                 0/1     CrashLoopBackOff   84 (48s ago)    28h
kube-system                    openebs-localpv-provisioner-6f54869bc7-6mn6b       0/1     Error              29              8h
kube-system                    snapshot-controller-0                              1/1     Running            1 (116m ago)    6h45m
kubesphere-controls-system     default-http-backend-59d5cf569f-4gsjb              0/1     Error              0               8h
kubesphere-controls-system     kubectl-admin-7ffdf4596b-82rfv                     1/1     Running            1 (116m ago)    8h
kubesphere-monitoring-system   alertmanager-main-0                                1/2     Running            2 (116m ago)    6h45m
kubesphere-monitoring-system   alertmanager-main-1                                0/2     Completed          0               6h45m
kubesphere-monitoring-system   alertmanager-main-2                                0/2     Completed          0               6h45m
kubesphere-monitoring-system   kube-state-metrics-5474f8f7b-sfjfc                 0/3     Completed          1               8h
kubesphere-monitoring-system   node-exporter-cq78v                                2/2     Running            4 (116m ago)    28h
kubesphere-monitoring-system   node-exporter-fs6lh                                2/2     Running            10 (112m ago)   28h
kubesphere-monitoring-system   node-exporter-svwtp                                2/2     Running            4 (116m ago)    28h
kubesphere-monitoring-system   node-exporter-wtbgp                                2/2     Running            8 (116m ago)    28h
kubesphere-monitoring-system   notification-manager-deployment-7b586bd8fb-j4g86   0/2     Error              2               8h
kubesphere-monitoring-system   notification-manager-deployment-7b586bd8fb-ljfz4   2/2     Running            4 (113m ago)    8h
kubesphere-monitoring-system   notification-manager-operator-64ff97cb98-j2tzf     0/2     Completed          30              8h
kubesphere-monitoring-system   prometheus-k8s-0                                   0/2     Error              0               6h45m
kubesphere-monitoring-system   prometheus-k8s-1                                   0/2     Error              0               6h45m
kubesphere-monitoring-system   prometheus-operator-64b7b4db85-qhhbn               0/2     Completed          1               8h
kubesphere-system              ks-apiserver-848bfd75fd-4tnbz                      0/1     CrashLoopBackOff   58 (23s ago)    28h
kubesphere-system              ks-apiserver-848bfd75fd-cjbwv                      0/1     Error              47 (57s ago)    6h49m
kubesphere-system              ks-apiserver-848bfd75fd-k52lc                      0/1     CrashLoopBackOff   59 (18s ago)    28h
kubesphere-system              ks-console-868887c49f-9ltmd                        1/1     Running            3 (116m ago)    28h
kubesphere-system              ks-console-868887c49f-lql7m                        1/1     Running            1 (112m ago)    6h50m
kubesphere-system              ks-console-868887c49f-vsn57                        1/1     Running            2 (116m ago)    28h
kubesphere-system              ks-controller-manager-67b896bb6d-2rrrb             1/1     Running            5 (43s ago)     28h
kubesphere-system              ks-controller-manager-67b896bb6d-9dx65             1/1     Running            1 (112m ago)    6h49m
kubesphere-system              ks-controller-manager-67b896bb6d-tlj8h             1/1     Running            3 (116m ago)    28h
kubesphere-system              ks-installer-5655f896fb-5k28b                      0/1     Completed          1               8h
kubesphere-system              redis-7cc8746478-g2p9c                             1/1     Running            5 (112m ago)    28h
pic-distribute                 mysql-v1-0                                         0/1     Completed          0               6h44m

没有启动成功的过滤出来如下

kubectl get pods -A|grep -v Running
NAMESPACE                      NAME                                               READY   STATUS             RESTARTS         AGE
kube-system                    nodelocaldns-9tvcv                                 0/1     CrashLoopBackOff   86 (35s ago)     28h
kube-system                    nodelocaldns-ddvg2                                 0/1     CrashLoopBackOff   99 (3m4s ago)    28h
kube-system                    nodelocaldns-fmmrm                                 0/1     CrashLoopBackOff   66 (2m23s ago)   28h
kube-system                    nodelocaldns-g4f4x                                 0/1     CrashLoopBackOff   87 (4m43s ago)   28h
kubesphere-system              ks-apiserver-848bfd75fd-4tnbz                      0/1     CrashLoopBackOff   64 (61s ago)     28h
kubesphere-system              ks-apiserver-848bfd75fd-cjbwv                      0/1     CrashLoopBackOff   51 (2m25s ago)   7h6m
kubesphere-system              ks-apiserver-848bfd75fd-k52lc                      0/1     CrashLoopBackOff   65 (56s ago)     28h

nodelocaldns-9tvcv -n kube-system的描述如下

kubectl describe nodelocaldns-9tvcv -n kube-system
error: the server doesn't have a resource type "nodelocaldns-9tvcv"
root@master-214-02:~# kubectl describe pod nodelocaldns-9tvcv -n kube-system
Name:                 nodelocaldns-9tvcv
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 worker-216-01/192.168.50.216
Start Time:           Thu, 16 Nov 2023 06:26:21 +0000
Labels:               controller-revision-hash=5855b6bfd
                      k8s-app=nodelocaldns
                      pod-template-generation=1
Annotations:          prometheus.io/port: 9253
                      prometheus.io/scrape: true
Status:               Running
IP:                   192.168.50.216
IPs:
  IP:           192.168.50.216
Controlled By:  DaemonSet/nodelocaldns
Containers:
  node-cache:
    Container ID:  docker://bcdd43f3ea3658b7cd4aabc9afd5fc9897ab77a6835744d916abaf95275a20f8
    Image:         dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.15.12
    Image ID:      docker-pullable://dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache@sha256:b6b9dc5cb4ab54aea6905ceeceb61e54791bc2acadecbea65db3641d99c7fe69
    Ports:         53/UDP, 53/TCP, 9253/TCP
    Host Ports:    53/UDP, 53/TCP, 9253/TCP
    Args:
      -localip
      169.254.25.10
      -conf
      /etc/coredns/Corefile
      -upstreamsvc
      coredns
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 17 Nov 2023 11:04:09 +0000
      Finished:     Fri, 17 Nov 2023 11:04:09 +0000
    Ready:          False
    Restart Count:  86
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://169.254.25.10:9254/health delay=0s timeout=5s period=10s #success=1 #failure=10
    Readiness:    http-get http://169.254.25.10:9254/health delay=0s timeout=5s period=10s #success=1 #failure=10
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vlnth (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nodelocaldns
    Optional:  false
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  kube-api-access-vlnth:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                   From     Message
  ----     ------          ----                  ----     -------
  Normal   SandboxChanged  18m                   kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy       18m                   kubelet  Readiness probe failed: Get "http://169.254.25.10:9254/health": dial tcp 169.254.25.10:9254: connect: connection refused
  Normal   Started         17m (x3 over 18m)     kubelet  Started container node-cache
  Normal   Pulled          16m (x4 over 18m)     kubelet  Container image "dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.15.12" already present on machine
  Normal   Created         16m (x4 over 18m)     kubelet  Created container node-cache
  Warning  BackOff         3m54s (x78 over 18m)  kubelet  Back-off restarting failed container

nodelocaldns-9tvcv日志

kubectl logs -f nodelocaldns-9tvcv -n kube-system
2023/11/17 11:04:09 [INFO] Using Corefile /etc/coredns/Corefile
2023/11/17 11:04:09 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2023/11/17 11:04:09 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
.:53 on 169.254.25.10
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[FATAL] plugin/loop: Loop (169.254.25.10:54381 -> 169.254.25.10:53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 2309482293177081995.1716274136328598708."

ks-apiserver-848bfd75fd-4tnbz的日志

kubectl describe pod -n kubesphere-system              ks-apiserver-848bfd75fd-4tnbz  
Name:         ks-apiserver-848bfd75fd-4tnbz
Namespace:    kubesphere-system
Priority:     0
Node:         master-214-02/192.168.50.214
Start Time:   Thu, 16 Nov 2023 06:28:52 +0000
Labels:       app=ks-apiserver
              pod-template-hash=848bfd75fd
              tier=backend
Annotations:  cni.projectcalico.org/containerID: 4e2209e4b5c0d712d66cab6b82f276cc3f5eabb2451ddd5ad37526995963239d
              cni.projectcalico.org/podIP: 10.233.109.20/32
              cni.projectcalico.org/podIPs: 10.233.109.20/32
Status:       Running
IP:           10.233.109.20
IPs:
  IP:           10.233.109.20
Controlled By:  ReplicaSet/ks-apiserver-848bfd75fd
Containers:
  ks-apiserver:
    Container ID:  docker://a29abea9b50633e849ba87955f72e71d093be684691c3f1ab279aade83dacacc
    Image:         dockerhub.kubekey.local/kubesphereio/ks-apiserver:v3.3.2
    Image ID:      docker-pullable://dockerhub.kubekey.local/kubesphereio/ks-apiserver@sha256:78d856f371d0981f9acef156da3869cf8b0a609bedf93f7d6a6d98d77d40ecd8
    Port:          9090/TCP
    Host Port:     0/TCP
    Command:
      ks-apiserver
      --logtostderr=true
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 17 Nov 2023 11:08:56 +0000
      Finished:     Fri, 17 Nov 2023 11:09:01 +0000
    Ready:          False
    Restart Count:  65
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     20m
      memory:  100Mi
    Liveness:  http-get http://:9090/kapis/version delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:
      KUBESPHERE_CACHE_OPTIONS_PASSWORD:  <set to the key 'auth' in secret 'redis-secret'>  Optional: false
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/kubesphere/ingress-controller from ks-router-config (rw)
      /etc/localtime from host-time (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6lhcp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ks-router-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ks-router-config
    Optional:  false
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  host-time:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:  
  kube-api-access-6lhcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason          Age                   From     Message
  ----     ------          ----                  ----     -------
  Normal   SandboxChanged  24m                   kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy       23m (x3 over 23m)     kubelet  Liveness probe failed: Get "http://10.233.109.20:9090/kapis/version": dial tcp 10.233.109.20:9090: connect: connection refused
  Normal   Pulled          21m (x4 over 24m)     kubelet  Container image "dockerhub.kubekey.local/kubesphereio/ks-apiserver:v3.3.2" already present on machine
  Normal   Created         21m (x4 over 24m)     kubelet  Created container ks-apiserver
  Normal   Started         21m (x4 over 24m)     kubelet  Started container ks-apiserver
  Warning  BackOff         4m13s (x92 over 23m)  kubelet  Back-off restarting failed container

ks-apiserver-848bfd75fd-4tnbz的日志

kubectl logs -f -n kubesphere-system              ks-apiserver-848bfd75fd-4tnbz  
W1117 11:08:56.216916       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1117 11:08:56.219836       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1117 11:08:56.236662       1 metricsserver.go:238] Metrics API not available.
E1117 11:09:01.237069       1 cache.go:76] failed to create cache, error: dial tcp: i/o timeout
Error: failed to create cache, error: dial tcp: i/o timeout
2023/11/17 11:09:01 failed to create cache, error: dial tcp: i/o timeout