kubesphere4.2.1社区版在海光CPU的服务器部署失败

zhu

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息
物理机，Ubuntu24.04，32C/128G

Kubernetes版本信息

root@node1:/data# kubectl version
Client Version: v1.33.4
Kustomize Version: v5.6.0
Server Version: v1.33.4

容器运行时

root@node1:/data# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.13
RuntimeApiVersion:  v1
root@node1:/data#

KubeSphere版本信息
v4.2.1。在线安装。使用kk安装。

问题是什么
报错信息

root@node1:/data# kubectl get pod -A
NAMESPACE           NAME                                          READY   STATUS              RESTARTS      AGE
kube-system         calico-kube-controllers-678fc69664-2hsql      0/1     ContainerCreating   0             60m
kube-system         calico-node-q8bph                             0/1     CrashLoopBackOff    1 (2s ago)    8s
kube-system         calico-node-t9t4d                             0/1     Error               1 (3s ago)    8s
kube-system         coredns-7cf6479957-gzp7h                      0/1     ContainerCreating   0             60m
kube-system         coredns-7cf6479957-vp2kp                      0/1     ContainerCreating   0             60m
kube-system         kube-apiserver-node1                          1/1     Running             1 (39m ago)   60m
kube-system         kube-controller-manager-node1                 1/1     Running             1 (39m ago)   60m
kube-system         kube-proxy-rmp9x                              1/1     Running             1 (39m ago)   60m
kube-system         kube-proxy-wc49q                              1/1     Running             1 (39m ago)   60m
kube-system         kube-scheduler-node1                          1/1     Running             1 (39m ago)   60m
kube-system         nodelocaldns-2bxgn                            1/1     Running             1 (39m ago)   60m
kube-system         nodelocaldns-m649j                            1/1     Running             1 (39m ago)   60m
kube-system         openebs-localpv-provisioner-9644bcfd5-hc78n   0/1     ContainerCreating   0             60m
kubesphere-system   extensions-museum-7f7599d5dc-fpjvz            0/1     Init:0/1            0             16m
kubesphere-system   ks-apiserver-85fd4c486f-fgzvq                 0/1     ContainerCreating   0             16m
kubesphere-system   ks-console-9dd5784-j4lgz                      0/1     ContainerCreating   0             16m
kubesphere-system   ks-controller-manager-6d9f76f468-sddvb        0/1     ContainerCreating   0             16m
kubesphere-system   ks-posthog-7c85f658dc-h4ckk                   0/1     ContainerCreating   0             16m

pod日志

root@node1:/data# kubectl logs -n kube-system calico-node-q8bph 
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
root@node1:/data# kubectl logs -n kube-system calico-node-t9t4d 
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)

pod详情

root@node1:/data# kubectl describe pod -n kube-system calico-node-q8bph 
Name:                 calico-node-q8bph
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      calico-node
Node:                 node2/172.17.9.4
Start Time:           Mon, 18 May 2026 09:03:10 +0000
Labels:               controller-revision-hash=cf56dffc
                      k8s-app=calico-node
                      pod-template-generation=7
Annotations:          <none>
Status:               Running
IP:                   172.17.9.4
IPs:
  IP:           172.17.9.4
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  containerd://c47252476b1662b674406d843233ca2072124d902f76d95c060aedb98ba8c423
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
    Image ID:      registry.cn-beijing.aliyuncs.com/kubesphereio/cni@sha256:cddc08395c5b0bea0daaa7650f4bd7e2a75a57db0cfb82c973100d3d6b897463
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 18 May 2026 09:03:11 +0000
      Finished:     Mon, 18 May 2026 09:03:11 +0000
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      IP_AUTODETECTION_METHOD:    interface=en.*
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj89d (ro)
  install-cni:
    Container ID:  containerd://818d89cb444bd93f1b29ae772952b3d1b62aad2d97d9aa70fb560665c1f8579b
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
    Image ID:      registry.cn-beijing.aliyuncs.com/kubesphereio/cni@sha256:cddc08395c5b0bea0daaa7650f4bd7e2a75a57db0cfb82c973100d3d6b897463
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 18 May 2026 09:03:12 +0000
      Finished:     Mon, 18 May 2026 09:03:12 +0000
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:            10-calico.conflist
      CNI_NETWORK_CONFIG:       <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:      (v1:spec.nodeName)
      CNI_MTU:                  <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                    false
      IP_AUTODETECTION_METHOD:  interface=en.*
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj89d (ro)
  mount-bpffs:
    Container ID:  containerd://d98a37abb506624151b22f1dad8cb29170c7c744b3eac2cec1e3afd65b842dab
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
    Image ID:      registry.cn-beijing.aliyuncs.com/kubesphereio/node@sha256:9bc7a4101f08f70d9554fadb784ae833e8088a4d24b208340a371d47806cc435
    Port:          <none>
    Host Port:     <none>
    Command:
      calico-node
      -init
      -best-effort
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 18 May 2026 09:03:14 +0000
      Finished:     Mon, 18 May 2026 09:03:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      IP_AUTODETECTION_METHOD:  interface=en.*
    Mounts:
      /nodeproc from nodeproc (ro)
      /sys/fs from sys-fs (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj89d (ro)
Containers:
  calico-node:
    Container ID:   containerd://5baae46b64413348603666aabbe5b323d83f07945f7747021d6d48059ada49f0
    Image:          registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
    Image ID:       registry.cn-beijing.aliyuncs.com/kubesphereio/node@sha256:9bc7a4101f08f70d9554fadb784ae833e8088a4d24b208340a371d47806cc435
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Mon, 18 May 2026 09:06:18 +0000
      Finished:     Mon, 18 May 2026 09:06:18 +0000
    Ready:          False
    Restart Count:  5
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      NODEIP:                              (v1:status.hostIP)
      IP_AUTODETECTION_METHOD:            interface=en.*
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      CALICO_IPV4POOL_NAT_OUTGOING:       true
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               10.233.64.0/18
      CALICO_IPV4POOL_BLOCK_SIZE:         24
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
      FELIX_DEVICEROUTESOURCEADDRESS:      (v1:status.hostIP)
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpffs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj89d (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sys-fs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  bpffs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  Directory
  nodeproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-nj89d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  3m56s                  default-scheduler  Successfully assigned kube-system/calico-node-q8bph to node2
  Normal   Pulled     3m56s                  kubelet            Container image "registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4" already present on machine
  Normal   Created    3m56s                  kubelet            Created container: upgrade-ipam
  Normal   Started    3m55s                  kubelet            Started container upgrade-ipam
  Normal   Pulled     3m55s                  kubelet            Container image "registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4" already present on machine
  Normal   Created    3m55s                  kubelet            Created container: install-cni
  Normal   Started    3m54s                  kubelet            Started container install-cni
  Normal   Pulled     3m53s                  kubelet            Container image "registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4" already present on machine
  Normal   Created    3m53s                  kubelet            Created container: mount-bpffs
  Normal   Started    3m52s                  kubelet            Started container mount-bpffs
  Normal   Created    2m14s (x5 over 3m51s)  kubelet            Created container: calico-node
  Normal   Started    2m14s (x5 over 3m51s)  kubelet            Started container calico-node
  Normal   Pulled     48s (x6 over 3m52s)    kubelet            Container image "registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4" already present on machine
  Warning  Unhealthy  48s                    kubelet            Readiness probe errored and resulted in unknown state: rpc error: code = NotFound desc = failed to exec in container: failed to load task: no running task found: task 5baae46b64413348603666aabbe5b323d83f07945f7747021d6d48059ada49f0 not found: not found
  Warning  BackOff    17s (x24 over 3m50s)   kubelet            Back-off restarting failed container calico-node in pod calico-node-q8bph_kube-system(6cdd347a-e1d5-494b-ba11-156d8b2e09dc)
root@node1:/data#

根据AI指导，排查内存异常

root@node1:/data# dmesg -T | grep -i segfault
[Mon May 18 08:24:35 2026] start_runit[4256]: segfault at 63bac96aa320 ip 00007a2aef7d4ead sp 00007ffdbbb59818 error 4 in libc-2.28.so[7a2aef7b0000+1cd000] likely on CPU 75 (core 11, socket 0)
[Mon May 18 08:24:58 2026] start_runit[4385]: segfault at 5745b6d6d320 ip 000079f314acfead sp 00007ffc87be2ab8 error 4 in libc-2.28.so[79f314aab000+1cd000] likely on CPU 57 (core 25, socket 1)
[Mon May 18 08:25:32 2026] start_runit[4729]: segfault at 55baecee5320 ip 0000758f4f50bead sp 00007ffcff8c6a18 error 4 in libc-2.28.so[758f4f4e7000+1cd000] likely on CPU 51 (core 19, socket 1)
[Mon May 18 08:26:24 2026] start_runit[4935]: segfault at 5f1e71032320 ip 00007f68355f5ead sp 00007ffc3b224828 error 4 in libc-2.28.so[7f68355d1000+1cd000] likely on CPU 127 (core 31, socket 1)
[Mon May 18 08:27:46 2026] start_runit[5368]: segfault at 57554b8bc320 ip 00007e7e7051dead sp 00007ffd92016008 error 4 in libc-2.28.so[7e7e704f9000+1cd000] likely on CPU 48 (core 16, socket 1)
[Mon May 18 08:27:57 2026] start_runit[5770]: segfault at 579943c30320 ip 00007e4f7a9d5ead sp 00007ffd0c7153c8 error 4 in libc-2.28.so[7e4f7a9b1000+1cd000] likely on CPU 119 (core 23, socket 1)
[Mon May 18 08:27:58 2026] start_runit[5831]: segfault at 5d5d8eef7320 ip 00007b2924cbdead sp 00007ffdba15c658 error 4 in libc-2.28.so[7b2924c99000+1cd000] likely on CPU 28 (core 28, socket 0)
[Mon May 18 08:28:03 2026] start_runit[6345]: segfault at 5d546715c320 ip 000079210589cead sp 00007ffc8f0ac108 error 4 in libc-2.28.so[792105878000+1cd000] likely on CPU 31 (core 31, socket 0)
[Mon May 18 08:28:04 2026] start_runit[6401]: segfault at 56d7feddd320 ip 0000703fc1c1fead sp 00007ffd03eb1f88 error 4 in libc-2.28.so[703fc1bfb000+1cd000] likely on CPU 109 (core 13, socket 1)
[Mon May 18 08:28:21 2026] start_runit[6560]: segfault at 58a8410c9320 ip 00007d78b6d7aead sp 00007ffe12785308 error 4 in libc-2.28.so[7d78b6d56000+1cd000] likely on CPU 107 (core 11, socket 1)
[Mon May 18 08:28:44 2026] start_runit[6764]: segfault at 5bc157e6d320 ip 000074a860680ead sp 00007ffe393e9968 error 4 in libc-2.28.so[74a86065c000+1cd000] likely on CPU 41 (core 9, socket 1)
[Mon May 18 08:29:29 2026] start_runit[6978]: segfault at 5d20a31a6320 ip 00007a144c3d0ead sp 00007ffc57ab34f8 error 4 in libc-2.28.so[7a144c3ac000+1cd000] likely on CPU 62 (core 30, socket 1)
[Mon May 18 08:30:54 2026] start_runit[7413]: segfault at 57f2ed7ce320 ip 00007b8b741b5ead sp 00007fffb231bb18 error 4 in libc-2.28.so[7b8b74191000+1cd000] likely on CPU 72 (core 8, socket 0)
[Mon May 18 08:33:38 2026] start_runit[8254]: segfault at 58a1f143d320 ip 000076b02e3a4ead sp 00007ffc7d71f508 error 4 in libc-2.28.so[76b02e380000+1cd000] likely on CPU 116 (core 20, socket 1)
[Mon May 18 08:35:08 2026] start_runit[9030]: segfault at 6399b2b87320 ip 00007a5afab14ead sp 00007ffd6e7dd918 error 4 in libc-2.28.so[7a5afaaf0000+1cd000] likely on CPU 33 (core 1, socket 1)
[Mon May 18 08:35:09 2026] start_runit[9090]: segfault at 609f382c9320 ip 0000750866390ead sp 00007ffc4b1e71f8 error 4 in libc-2.28.so[75086636c000+1cd000] likely on CPU 94 (core 30, socket 0)
[Mon May 18 08:35:26 2026] start_runit[9352]: segfault at 5bf83742c320 ip 0000715b1df12ead sp 00007ffeea763038 error 4 in libc-2.28.so[715b1deee000+1cd000] likely on CPU 68 (core 4, socket 0)
[Mon May 18 08:35:49 2026] start_runit[9664]: segfault at 61fbc2c71320 ip 000073d82bfb6ead sp 00007ffe5b696928 error 4 in libc-2.28.so[73d82bf92000+1cd000] likely on CPU 97 (core 1, socket 1)
[Mon May 18 08:36:34 2026] start_runit[9916]: segfault at 578d80c88320 ip 00007091ebc67ead sp 00007ffd0c264e18 error 4 in libc-2.28.so[7091ebc43000+1cd000] likely on CPU 45 (core 13, socket 1)
[Mon May 18 08:37:59 2026] start_runit[10711]: segfault at 5c574e36f320 ip 0000749dd848fead sp 00007ffefb44b4e8 error 4 in libc-2.28.so[749dd846b000+1cd000] likely on CPU 93 (core 29, socket 0)
[Mon May 18 08:38:00 2026] start_runit[10791]: segfault at 5902e64cb320 ip 00007292737a2ead sp 00007fff912a18b8 error 4 in libc-2.28.so[72927377e000+1cd000] likely on CPU 66 (core 2, socket 0)
[Mon May 18 08:38:17 2026] start_runit[11065]: segfault at 5a2850d2c320 ip 000077bd9d915ead sp 00007ffdb4b75878 error 4 in libc-2.28.so[77bd9d8f1000+1cd000] likely on CPU 48 (core 16, socket 1)
[Mon May 18 08:38:50 2026] start_runit[11253]: segfault at 56dc13af8320 ip 000076bcc6b7bead sp 00007fffe362e1b8 error 4 in libc-2.28.so[76bcc6b57000+1cd000] likely on CPU 23 (core 23, socket 0)
[Mon May 18 08:39:33 2026] start_runit[11441]: segfault at 64fc86473320 ip 00007583d5398ead sp 00007fffd2bc7eb8 error 4 in libc-2.28.so[7583d5374000+1cd000] likely on CPU 91 (core 27, socket 0)
[Mon May 18 08:40:56 2026] start_runit[11855]: segfault at 5e90d1a86320 ip 00007ba560295ead sp 00007ffc9931c2e8 error 4 in libc-2.28.so[7ba560271000+1cd000] likely on CPU 96 (core 0, socket 1)
[Mon May 18 08:43:41 2026] start_runit[12722]: segfault at 6231e13f4320 ip 00007c9782ffaead sp 00007ffd3414f308 error 4 in libc-2.28.so[7c9782fd6000+1cd000] likely on CPU 28 (core 28, socket 0)
[Mon May 18 08:48:50 2026] start_runit[16091]: segfault at 611b358d2320 ip 00007cf276d38ead sp 00007fffe68b4948 error 4 in libc-2.28.so[7cf276d14000+1cd000] likely on CPU 22 (core 22, socket 0)
[Mon May 18 08:53:59 2026] start_runit[20758]: segfault at 5f390e61b320 ip 00007999ee25aead sp 00007fff946170e8 error 4 in libc-2.28.so[7999ee236000+1cd000] likely on CPU 10 (core 10, socket 0)
[Mon May 18 08:59:05 2026] start_runit[25001]: segfault at 5ef50fdfa320 ip 000079ff5906aead sp 00007ffe167ec178 error 4 in libc-2.28.so[79ff59046000+1cd000] likely on CPU 79 (core 15, socket 0)
[Mon May 18 09:03:14 2026] start_runit[29308]: segfault at 5c10705e6320 ip 00007c3dd6c89ead sp 00007ffc5146b538 error 4 in libc-2.28.so[7c3dd6c65000+1cd000] likely on CPU 103 (core 7, socket 1)
[Mon May 18 09:03:15 2026] start_runit[29397]: segfault at 6297edd05320 ip 00007dc38bea1ead sp 00007ffc58ddb488 error 4 in libc-2.28.so[7dc38be7d000+1cd000] likely on CPU 45 (core 13, socket 1)
[Mon May 18 09:03:34 2026] start_runit[29763]: segfault at 5710410a0320 ip 00007eac1554aead sp 00007fff36bd0b58 error 4 in libc-2.28.so[7eac15526000+1cd000] likely on CPU 35 (core 3, socket 1)
[Mon May 18 09:04:05 2026] start_runit[30285]: segfault at 59cf9a72e320 ip 000077829d3dcead sp 00007ffcf2b7e268 error 4 in libc-2.28.so[77829d3b8000+1cd000] likely on CPU 80 (core 16, socket 0)
[Mon May 18 09:04:50 2026] start_runit[30973]: segfault at 5c2f207a2320 ip 000073b99d805ead sp 00007ffcc098b378 error 4 in libc-2.28.so[73b99d7e1000+1cd000] likely on CPU 11 (core 11, socket 0)
[Mon May 18 09:06:16 2026] start_runit[32229]: segfault at 5ab14e177320 ip 0000702d0fc10ead sp 00007ffe84c7c048 error 4 in libc-2.28.so[702d0fbec000+1cd000] likely on CPU 13 (core 13, socket 0)
root@node1:/data#

AI指导的排查

root@node1:/data# journalctl -k --since "1 hour ago" | grep -i segfault
May 18 08:24:35 node1 kernel: start_runit[4256]: segfault at 63bac96aa320 ip 00007a2aef7d4ead sp 00007ffdbbb59818 error 4 in libc-2.28.so[7a2aef7b0000+1cd000] likely on CPU 75 (core 11, socket 0)
May 18 08:24:58 node1 kernel: start_runit[4385]: segfault at 5745b6d6d320 ip 000079f314acfead sp 00007ffc87be2ab8 error 4 in libc-2.28.so[79f314aab000+1cd000] likely on CPU 57 (core 25, socket 1)
May 18 08:25:32 node1 kernel: start_runit[4729]: segfault at 55baecee5320 ip 0000758f4f50bead sp 00007ffcff8c6a18 error 4 in libc-2.28.so[758f4f4e7000+1cd000] likely on CPU 51 (core 19, socket 1)
May 18 08:26:24 node1 kernel: start_runit[4935]: segfault at 5f1e71032320 ip 00007f68355f5ead sp 00007ffc3b224828 error 4 in libc-2.28.so[7f68355d1000+1cd000] likely on CPU 127 (core 31, socket 1)
May 18 08:27:46 node1 kernel: start_runit[5368]: segfault at 57554b8bc320 ip 00007e7e7051dead sp 00007ffd92016008 error 4 in libc-2.28.so[7e7e704f9000+1cd000] likely on CPU 48 (core 16, socket 1)
May 18 08:27:57 node1 kernel: start_runit[5770]: segfault at 579943c30320 ip 00007e4f7a9d5ead sp 00007ffd0c7153c8 error 4 in libc-2.28.so[7e4f7a9b1000+1cd000] likely on CPU 119 (core 23, socket 1)
May 18 08:27:58 node1 kernel: start_runit[5831]: segfault at 5d5d8eef7320 ip 00007b2924cbdead sp 00007ffdba15c658 error 4 in libc-2.28.so[7b2924c99000+1cd000] likely on CPU 28 (core 28, socket 0)
May 18 08:28:03 node1 kernel: start_runit[6345]: segfault at 5d546715c320 ip 000079210589cead sp 00007ffc8f0ac108 error 4 in libc-2.28.so[792105878000+1cd000] likely on CPU 31 (core 31, socket 0)
May 18 08:28:04 node1 kernel: start_runit[6401]: segfault at 56d7feddd320 ip 0000703fc1c1fead sp 00007ffd03eb1f88 error 4 in libc-2.28.so[703fc1bfb000+1cd000] likely on CPU 109 (core 13, socket 1)
May 18 08:28:21 node1 kernel: start_runit[6560]: segfault at 58a8410c9320 ip 00007d78b6d7aead sp 00007ffe12785308 error 4 in libc-2.28.so[7d78b6d56000+1cd000] likely on CPU 107 (core 11, socket 1)
May 18 08:28:44 node1 kernel: start_runit[6764]: segfault at 5bc157e6d320 ip 000074a860680ead sp 00007ffe393e9968 error 4 in libc-2.28.so[74a86065c000+1cd000] likely on CPU 41 (core 9, socket 1)
May 18 08:29:29 node1 kernel: start_runit[6978]: segfault at 5d20a31a6320 ip 00007a144c3d0ead sp 00007ffc57ab34f8 error 4 in libc-2.28.so[7a144c3ac000+1cd000] likely on CPU 62 (core 30, socket 1)
May 18 08:30:54 node1 kernel: start_runit[7413]: segfault at 57f2ed7ce320 ip 00007b8b741b5ead sp 00007fffb231bb18 error 4 in libc-2.28.so[7b8b74191000+1cd000] likely on CPU 72 (core 8, socket 0)
May 18 08:33:38 node1 kernel: start_runit[8254]: segfault at 58a1f143d320 ip 000076b02e3a4ead sp 00007ffc7d71f508 error 4 in libc-2.28.so[76b02e380000+1cd000] likely on CPU 116 (core 20, socket 1)
May 18 08:35:08 node1 kernel: start_runit[9030]: segfault at 6399b2b87320 ip 00007a5afab14ead sp 00007ffd6e7dd918 error 4 in libc-2.28.so[7a5afaaf0000+1cd000] likely on CPU 33 (core 1, socket 1)
May 18 08:35:09 node1 kernel: start_runit[9090]: segfault at 609f382c9320 ip 0000750866390ead sp 00007ffc4b1e71f8 error 4 in libc-2.28.so[75086636c000+1cd000] likely on CPU 94 (core 30, socket 0)
May 18 08:35:26 node1 kernel: start_runit[9352]: segfault at 5bf83742c320 ip 0000715b1df12ead sp 00007ffeea763038 error 4 in libc-2.28.so[715b1deee000+1cd000] likely on CPU 68 (core 4, socket 0)
May 18 08:35:49 node1 kernel: start_runit[9664]: segfault at 61fbc2c71320 ip 000073d82bfb6ead sp 00007ffe5b696928 error 4 in libc-2.28.so[73d82bf92000+1cd000] likely on CPU 97 (core 1, socket 1)
May 18 08:36:34 node1 kernel: start_runit[9916]: segfault at 578d80c88320 ip 00007091ebc67ead sp 00007ffd0c264e18 error 4 in libc-2.28.so[7091ebc43000+1cd000] likely on CPU 45 (core 13, socket 1)
May 18 08:37:59 node1 kernel: start_runit[10711]: segfault at 5c574e36f320 ip 0000749dd848fead sp 00007ffefb44b4e8 error 4 in libc-2.28.so[749dd846b000+1cd000] likely on CPU 93 (core 29, socket 0)
May 18 08:38:00 node1 kernel: start_runit[10791]: segfault at 5902e64cb320 ip 00007292737a2ead sp 00007fff912a18b8 error 4 in libc-2.28.so[72927377e000+1cd000] likely on CPU 66 (core 2, socket 0)
May 18 08:38:17 node1 kernel: start_runit[11065]: segfault at 5a2850d2c320 ip 000077bd9d915ead sp 00007ffdb4b75878 error 4 in libc-2.28.so[77bd9d8f1000+1cd000] likely on CPU 48 (core 16, socket 1)
May 18 08:38:50 node1 kernel: start_runit[11253]: segfault at 56dc13af8320 ip 000076bcc6b7bead sp 00007fffe362e1b8 error 4 in libc-2.28.so[76bcc6b57000+1cd000] likely on CPU 23 (core 23, socket 0)
May 18 08:39:33 node1 kernel: start_runit[11441]: segfault at 64fc86473320 ip 00007583d5398ead sp 00007fffd2bc7eb8 error 4 in libc-2.28.so[7583d5374000+1cd000] likely on CPU 91 (core 27, socket 0)
May 18 08:40:56 node1 kernel: start_runit[11855]: segfault at 5e90d1a86320 ip 00007ba560295ead sp 00007ffc9931c2e8 error 4 in libc-2.28.so[7ba560271000+1cd000] likely on CPU 96 (core 0, socket 1)
May 18 08:43:41 node1 kernel: start_runit[12722]: segfault at 6231e13f4320 ip 00007c9782ffaead sp 00007ffd3414f308 error 4 in libc-2.28.so[7c9782fd6000+1cd000] likely on CPU 28 (core 28, socket 0)
May 18 08:48:50 node1 kernel: start_runit[16091]: segfault at 611b358d2320 ip 00007cf276d38ead sp 00007fffe68b4948 error 4 in libc-2.28.so[7cf276d14000+1cd000] likely on CPU 22 (core 22, socket 0)
May 18 08:53:59 node1 kernel: start_runit[20758]: segfault at 5f390e61b320 ip 00007999ee25aead sp 00007fff946170e8 error 4 in libc-2.28.so[7999ee236000+1cd000] likely on CPU 10 (core 10, socket 0)
May 18 08:59:05 node1 kernel: start_runit[25001]: segfault at 5ef50fdfa320 ip 000079ff5906aead sp 00007ffe167ec178 error 4 in libc-2.28.so[79ff59046000+1cd000] likely on CPU 79 (core 15, socket 0)
May 18 09:03:14 node1 kernel: start_runit[29308]: segfault at 5c10705e6320 ip 00007c3dd6c89ead sp 00007ffc5146b538 error 4 in libc-2.28.so[7c3dd6c65000+1cd000] likely on CPU 103 (core 7, socket 1)
May 18 09:03:15 node1 kernel: start_runit[29397]: segfault at 6297edd05320 ip 00007dc38bea1ead sp 00007ffc58ddb488 error 4 in libc-2.28.so[7dc38be7d000+1cd000] likely on CPU 45 (core 13, socket 1)
May 18 09:03:34 node1 kernel: start_runit[29763]: segfault at 5710410a0320 ip 00007eac1554aead sp 00007fff36bd0b58 error 4 in libc-2.28.so[7eac15526000+1cd000] likely on CPU 35 (core 3, socket 1)
May 18 09:04:05 node1 kernel: start_runit[30285]: segfault at 59cf9a72e320 ip 000077829d3dcead sp 00007ffcf2b7e268 error 4 in libc-2.28.so[77829d3b8000+1cd000] likely on CPU 80 (core 16, socket 0)
May 18 09:04:50 node1 kernel: start_runit[30973]: segfault at 5c2f207a2320 ip 000073b99d805ead sp 00007ffcc098b378 error 4 in libc-2.28.so[73b99d7e1000+1cd000] likely on CPU 11 (core 11, socket 0)
May 18 09:06:16 node1 kernel: start_runit[32229]: segfault at 5ab14e177320 ip 0000702d0fc10ead sp 00007ffe84c7c048 error 4 in libc-2.28.so[702d0fbec000+1cd000] likely on CPU 13 (core 13, socket 0)
root@node1:/data#

CPU信息

root@node1:/data# cat /proc/cpuinfo | head -n 20
processor	: 0
vendor_id	: HygonGenuine
cpu family	: 24
model		: 2
model name	: Hygon C86-3G 7380 32-core Processor
stepping	: 2
microcode	: 0x80901050
cpu MHz		: 1200.000
cache size	: 512 KB
physical id	: 0
siblings	: 64
core id		: 0
cpu cores	: 32
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev sev_es
root@node1:/data#

系统环境

root@node1:/data# uname -a
Linux node1 6.8.0-53-generic #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:37:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@node1:/data#

AI分析是calico不支持当前在用的海光CPU导致的问题。

zhu

节点信息

root@node1:/data/kubesphere# kubectl get node -A -o wide
NAME    STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
node1   Ready    control-plane,worker   73m   v1.33.4   172.17.9.2    <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.13
node2   Ready    worker                 73m   v1.33.4   172.17.9.4    <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.13

zhu

补充描述

两台测试用的机器是同硬件配置。都是海光的CPU

root@node1:/data/kubesphere# lscpu 
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   128
  On-line CPU(s) list:    0-127
Vendor ID:                HygonGenuine
  BIOS Vendor ID:         Chengdu Hygon
  Model name:             Hygon C86-3G 7380 32-core Processor
    BIOS Model name:      Hygon C86-3G 7380 32-core Processor   CPU @ 2.2GHz
    BIOS CPU family:      1
    CPU family:           24
    Model:                2
    Thread(s) per core:   2
    Core(s) per socket:   32
    Socket(s):            2
    Stepping:             2
    Frequency boost:      disabled
    CPU(s) scaling MHz:   55%
    CPU max MHz:          2200.0000
    CPU min MHz:          1200.0000
    BogoMIPS:             4391.75
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good 
                          nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy 
                          svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall 
                          fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_c
                          lean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev sev_es
Virtualization features:  
  Virtualization:         AMD-V
Caches (sum of all):      
  L1d:                    2 MiB (64 instances)
  L1i:                    4 MiB (64 instances)
  L2:                     32 MiB (64 instances)
  L3:                     128 MiB (16 instances)
NUMA:                     
  NUMA node(s):           8
  NUMA node0 CPU(s):      0-7,64-71
  NUMA node1 CPU(s):      8-15,72-79
  NUMA node2 CPU(s):      16-23,80-87
  NUMA node3 CPU(s):      24-31,88-95
  NUMA node4 CPU(s):      32-39,96-103
  NUMA node5 CPU(s):      40-47,104-111
  NUMA node6 CPU(s):      48-55,112-119
  NUMA node7 CPU(s):      56-63,120-127
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; untrained return thunk; SMT vulnerable
  Spec rstack overflow:   Vulnerable: Safe RET, no microcode
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
root@node1:/data/kubesphere#

zhu

节点状态如下

root@node1:~# kubectl get node -A -o wide
NAME    STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
node1   Ready    control-plane,worker   16h   v1.33.4   172.17.9.2    <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.13
node2   Ready    worker                 16h   v1.33.4   172.17.9.4    <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.13
root@node1:~# kubectl get pod -A -o wide
NAMESPACE           NAME                                          READY   STATUS              RESTARTS          AGE   IP           NODE    NOMINATED NODE   READINESS GATES
kube-system         calico-kube-controllers-678fc69664-2hsql      0/1     ContainerCreating   0                 16h   <none>       node2   <none>           <none>
kube-system         calico-node-q8bph                             0/1     CrashLoopBackOff    185 (4m9s ago)    15h   172.17.9.4   node2   <none>           <none>
kube-system         calico-node-t9t4d                             0/1     CrashLoopBackOff    185 (4m35s ago)   15h   172.17.9.2   node1   <none>           <none>
kube-system         coredns-7cf6479957-gzp7h                      0/1     ContainerCreating   0                 16h   <none>       node2   <none>           <none>
kube-system         coredns-7cf6479957-vp2kp                      0/1     ContainerCreating   0                 16h   <none>       node2   <none>           <none>
kube-system         kube-apiserver-node1                          1/1     Running             1 (16h ago)       16h   172.17.9.2   node1   <none>           <none>
kube-system         kube-controller-manager-node1                 1/1     Running             1 (16h ago)       16h   172.17.9.2   node1   <none>           <none>
kube-system         kube-proxy-rmp9x                              1/1     Running             1 (16h ago)       16h   172.17.9.4   node2   <none>           <none>
kube-system         kube-proxy-wc49q                              1/1     Running             1 (16h ago)       16h   172.17.9.2   node1   <none>           <none>
kube-system         kube-scheduler-node1                          1/1     Running             1 (16h ago)       16h   172.17.9.2   node1   <none>           <none>
kube-system         nodelocaldns-2bxgn                            1/1     Running             1 (16h ago)       16h   172.17.9.4   node2   <none>           <none>
kube-system         nodelocaldns-m649j                            1/1     Running             1 (16h ago)       16h   172.17.9.2   node1   <none>           <none>
kube-system         openebs-localpv-provisioner-9644bcfd5-hc78n   0/1     ContainerCreating   0                 16h   <none>       node2   <none>           <none>
kubesphere-system   extensions-museum-7f7599d5dc-fpjvz            0/1     Init:0/1            0                 15h   <none>       node2   <none>           <none>
kubesphere-system   ks-apiserver-85fd4c486f-fgzvq                 0/1     ContainerCreating   0                 15h   <none>       node1   <none>           <none>
kubesphere-system   ks-console-9dd5784-j4lgz                      0/1     ContainerCreating   0                 15h   <none>       node1   <none>           <none>
kubesphere-system   ks-controller-manager-6d9f76f468-sddvb        0/1     ContainerCreating   0                 15h   <none>       node1   <none>           <none>
kubesphere-system   ks-posthog-7c85f658dc-h4ckk                   0/1     ContainerCreating   0                 15h   <none>       node1   <none>           <none>

zhu

目前根据AI排查判断，是安装时使用的calico版本较低会产生该问题，可以把calico版本更新到最新适配的稳定版

zhu

已解决，解决办法是，完整删掉calico，清空资源。部署原生calico新版



# 1. 停止kubelet，防止反复创建失败的容器
systemctl stop kubelet

# 2. 彻底删除Calico所有残留文件
rm -rf /var/lib/calico
rm -rf /etc/cni/net.d/10-calico.conflist
rm -rf /opt/cni/bin/calico*

# 3. 清理网络接口
ip link delete cali* 2>/dev/null
ip link delete tunl0 2>/dev/null
ip link delete vxlan.calico 2>/dev/null

# 4. 重启容器运行时
systemctl restart containerd

# 5. 启动kubelet
systemctl start kubelet

删掉已部署的calico资源，
kubectl delete -f calico.yaml
删除资源的方法还没确认。

部署新版
https://github.com/projectcalico/calico/blob/v3.28.0/manifests/calico.yaml

看到calico可以正常运行