kubeedge部署不成功

系统环境及软件版本

基础环境-1

| 系统及版本信息 | 角色 | IP | DNS | 架构 | Docker版本 |

|:-😐:-😐:-😐:-😐:-😐:-😐

|Ubuntu 23.10|Master|—-|168.95.1.1|x86_64|v20.10

|Ubuntu 23.10|Node1|—-|168.95.1.1|x86_64|v20.10

|Ubuntu 23.10|Node2|—-|168.95.1.1|x86_64|v.20.10

|Ubuntu 23.10|edge|—-|168.95.1.1|x86_64|v26.0.0

|Ubuntu 22.04|edge|—-|168.95.1.1|arm64|

kubeedge

最初在瑞芯微RK3588开发板上部署kubeedge，一直无法成功，以为是开发板的系统内核是被裁切过。后来陆续在x86的Ubuntu虚拟机和树莓派3B上部署，发现都遇到同样的问题。

基础环境-2

|软件名称|版本|

|:-😐:-😐-😐

KubeKey|v3.0.7|

Kubesphere|v3.3.0|

Kubernetes|v1.21.1|

kubeedge|v1.9|

安装遇到的问题

1. cgroup驱动不匹配，经查询得知，Docker默认的cgroup驱动为systemd，kubeedge配置文件指定驱动为cgroupfs，直接单独安装完docker，接着安装kubeedge会报错，提示驱动不匹配，手动在/etc/docker路径下添加daemon.json文件，配置cgroup驱动为cgroupfs，与kubeedge配置文件一致，容器组正常开始配置

2. 部署kubeedge时会自动配置三个容器组，其中 kube-proxy能正常初始化，另两个容器组nodelocaldns与calico-node无法正常初始化。

3. 在kubesphere的Web端查看边缘节点的状态时，无法使用日志功能，提示10350端口无法连接。

无法初始化容器组的日志

nodelocaldns

# 通过portainer查看容器状态，日志如下：

2024/04/15 01:36:55 [INFO] Using Corefile /etc/coredns/Corefile

2024/04/15 01:36:55 [FATAL] Failed to obtain CacheApp instance, err Unable to lookup IP address of Upstream service coredns, env $COREDNS_SERVICE_HOST \`\`

容器状态中有一条命令

CMD -localip 169.254.25.10 -conf /etc/coredns/Corefile -upstreamsvc coredns

在kubeedge边缘设备上无法ping通这个IP，回到Master节点可以ping通这个IP

calico-node

# 使用 journalctl -u edgecore.service 查看运行日志如下：

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: E0412 09:21:27.160519    2441 metaclient.go:112] send sync message kube-system/configmap/kubernetes-services-endpoint failed, error:timeout to get response, retries: 2

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: I0412 09:21:27.160615    2441 record.go:19] Warning Failed Error: get configmap from metaManager failed, err: timed out waiting for the condition

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: I0412 09:21:27.160497    2441 process.go:282] ########## process get: req[{Header:{ID:3d549606-6469-4e62-a397-43bcde5ba1e3 ParentID: Timestamp:1712913567149 ResourceVersion: Sync:true MessageType:} Router:{Source:edged Destination: Group:meta Operation:query Resource:kube-system/configmap/kubernetes-services-endpoint} Content:<nil>}], resp[{Header:{ID: ParentID: Timestamp:0 ResourceVersion: Sync:false MessageType:} Router:{Source: Destination: Group: Operation: Resource:} Content:<nil>}], err[timeout to get response]

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: E0412 09:21:27.160730    2441 process.go:284] remote query failed: timeout to get response

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: E0412 09:21:27.160772    2441 kuberuntime_manager.go:864] init container &Container{Name:upgrade-ipam,Image:calico/cni:v3.23.2,Command:[/opt/cni/bin/calico-ipam -upgrade],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:KUBERNETES_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:CALICO_NETWORKING_BACKEND,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:calico-config,},Key:calico_backend,Optional:nil,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:host-local-net-dir,ReadOnly:false,MountPath:/var/lib/cni/networks,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-bin-dir,ReadOnly:false,MountPath:/host/opt/cni/bin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-g5ghn,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:\*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{EnvFromSource{Prefix:,ConfigMapRef:&ConfigMapEnvSource{LocalObjectReference:LocalObjectReference{Name:kubernetes-services-endpoint,},Optional:\*true,},SecretRef:nil,},},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod calico-node-xrkwm_kube-system(49983c1d-6c81-44ce-a5a9-9a45286a1052): CreateContainerConfigError: get configmap from metaManager failed, err: timed out waiting for the condition

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: I0412 09:21:27.160845    2441 edged.go:990] worker [2] get pod addition item [calico-node-xrkwm]

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: E0412 09:21:27.160898    2441 edged.go:993] consume pod addition backoff: Back-off consume pod [calico-node-xrkwm] addition  error, backoff: [5m0s]

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: I0412 09:21:27.160938    2441 edged.go:995] worker [2] backoff pod addition item [calico-node-xrkwm] failed, re-add to queue

Apr 12 09:21:27 k8s-kubeedge1 edgecore[2441]: E0412 09:21:27.160945    2441 edged.go:1009] worker [2] handle pod addition item [calico-node-xrkwm] failed: failed to "StartContainer" for "upgrade-ipam" with CreateContainerConfigError: "get configmap from metaManager failed, err: timed out waiting for the condition", re-add to queue

Apr 12 09:21:37 k8s-kubeedge1 edgecore[2441]: I0412 09:21:37.161965    2441 edged.go:990] worker [0] get pod addition item [calico-node-xrkwm]

Apr 12 09:21:37 k8s-kubeedge1 edgecore[2441]: E0412 09:21:37.162024    2441 edged.go:993] consume pod addition backoff: Back-off consume pod [calico-node-xrkwm] addition  error, backoff: [5m0s]

Apr 12 09:21:37 k8s-kubeedge1 edgecore[2441]: I0412 09:21:37.162070    2441 edged.go:995] worker [0] backoff pod addition item [calico-node-xrkwm] failed, re-add to queue

Apr 12 09:22:03 k8s-kubeedge1 edgecore[2441]: I0412 09:22:03.907300    2441 process.go:329] start to sync pod status in edge-store to cloud

# 这里说找不到指定的configmap，不知道是不是跟calico-node有关。

# 日志中提到的找不到configmap，以下是在Master上的查询记录

root@k8s-master01:/etc# kubectl get configmap -n kube-system

NAME                                 DATA   AGE

calico-config                        4      73d

coredns                              1      73d

extension-apiserver-authentication   6      73d

kube-proxy                           2      73d

kube-root-ca.crt                     1      73d

kubeadm-config                       2      73d

kubelet-config-1.21                  1      73d

nodelocaldns                         1      73d

root@k8s-master01:/etc# kubectl get configmap kubernetes-services-endpoint -n kube-system

Error from server (NotFound): configmaps "kubernetes-services-endpoint" not found

root@k8s-master01:/etc#