由于文章被限制在 65535 个字符，无法详细描述，文章内容会精简一部分；
设备环境相关信息
物理环境信息
- 一台服务器虚拟出来的五台虚拟机：
- Master 节点两台，4C/8G；
- Worker 节点三台，8C/16G；
操作系统信息
- CentOS 7.9
系统安装相关信息
Docker 版本
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
containerd:
Version: 1.4.12
runc:
Version: 1.0.2Kubernetes 版本
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}KubeSphere 版本、安装方式
版本：v3.2.1
安装方式：
在纯净的 CentOS 上，使用 kk All in one 方式安装，kk 配置如下：
apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata: name: sample spec: hosts: - {name: k8s-test-master1, address: 192.168.10.10, internalAddress: 192.168.10.10, user: root, password: "root"} - {name: k8s-test-master2, address: 192.168.10.11, internalAddress: 192.168.10.11, user: root, password: "root"} - {name: k8s-test-worker1, address: 192.168.10.12, internalAddress: 192.168.10.12, user: root, password: "root"} - {name: k8s-test-worker2, address: 192.168.10.13, internalAddress: 192.168.10.13, user: root, password: "root"} - {name: k8s-test-worker3, address: 192.168.10.14, internalAddress: 192.168.10.14, user: root, password: "root"} roleGroups: etcd: - k8s-test-master1 - k8s-test-master2 control-plane: - k8s-test-master1 - k8s-test-master2 worker: - k8s-test-worker1 - k8s-test-worker2 - k8s-test-worker3 controlPlaneEndpoint: internalLoadbalancer: haproxy domain: lb.kubesphere.local address: "" port: 6443 kubernetes: version: v1.21.5 clusterName: cluster.local autoRenewCerts: true etcd: type: kubekey network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 multusCNI: enabled: false registry: plainHTTP: false privateRegistry: "" namespaceOverride: "" registryMirrors: ["https://registry.docker-cn.com", "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com"] addons: [] --- apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: name: ks-installer namespace: kubesphere-system labels: version: v3.2.1 spec: persistence: storageClass: "" authentication: jwtSecret: "" zone: "" local_registry: "" namespace_override: "" etcd: monitoring: true endpointIps: localhost port: 2379 tlsEnable: true common: core: console: enableMultiLogin: true port: 30880 type: NodePort redis: enabled: false volumeSize: 2Gi openldap: enabled: false volumeSize: 2Gi minio: volumeSize: 20Gi monitoring: endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090 GPUMonitoring: enabled: false gpu: kinds: - resourceName: "nvidia.com/gpu" resourceType: "GPU" default: true es: logMaxAge: 7 elkPrefix: logstash basicAuth: enabled: false username: "" password: "" externalElasticsearchHost: "" externalElasticsearchPort: "" alerting: enabled: true auditing: enabled: true devops: enabled: true jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsJavaOpts_MaxRAM: 2g events: enabled: true logging: enabled: true containerruntime: docker logsidecar: enabled: true replicas: 2 metrics_server: enabled: true monitoring: storageClass: "" gpu: nvidia_dcgm_exporter: enabled: false # resources: {} multicluster: clusterRole: none network: networkpolicy: enabled: false ippool: type: calico topology: type: weave scope openpitrix: store: enabled: true servicemesh: enabled: true kubeedge: enabled: true cloudCore: nodeSelector: {"node-role.kubernetes.io/worker": ""} tolerations: [] cloudhubPort: "10000" cloudhubQuicPort: "10001" cloudhubHttpsPort: "10002" cloudstreamPort: "10003" tunnelPort: "10004" cloudHub: advertiseAddress: - "192.168.10.10" - "192.168.10.11" nodeLimit: "800" service: cloudhubNodePort: "30000" cloudhubQuicNodePort: "30001" cloudhubHttpsNodePort: "30002" cloudstreamNodePort: "30003" tunnelNodePort: "30004" edgeWatcher: nodeSelector: {"node-role.kubernetes.io/worker": ""} tolerations: [] edgeWatcherAgent: nodeSelector: {"node-role.kubernetes.io/worker": ""} tolerations: []
KubeEdge 版本
v1.7.2
问题描述
背景
边缘节点也是服务器下的一台虚拟机，用于作为边缘节点测试；
使用容忍的方式，使得 iptables 可以运行在 master 节点上，解决了无法在 KubeSphere 上对边缘节点进行 log、exec；
kubectl edit iptables -n kubeedge
在 master 节点执行了以下脚本，用于对边缘节点进行了 patch，使得容忍度较强的 calico、kubeproxy 等 pod 不会被调度到边缘节点上：
#!/bin/bash NodeSelectorPatchJson='{"spec":{"template":{"spec":{"nodeSelector":{"node-role.kubernetes.io/master": "","node-role.kubernetes.io/worker": ""}}}}}' NoShedulePatchJson='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}}}}}}' edgenode="edge1" if [ $1 ]; then edgenode="$1" fi namespaces=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $1}' )) pods=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $2}' )) length=${#namespaces[@]} for((i=0;i<$length;i++)); do ns=${namespaces[$i]} pod=${pods[$i]} resources=$(kubectl -n $ns describe pod $pod | grep "Controlled By" |awk '{print $3}') echo "Patching for ns: $ns, resources: $resources" kubectl -n $ns patch $resources --type merge --patch "$NoShedulePatchJson" sleep 1 done目前已经可以成功对边缘节点进行 log、exec、metric；
问题一：安装完 KubeSphere 后，cloudcore 的报错问题
此时还没加入边缘节点，不知道该报错是否有影响，以下为 cloudcore 的日志：
W0527 10:09:10.897412 1 validation.go:168] TLSTunnelPrivateKeyFile does not exist in /etc/kubeedge/certs/server.key, will load from secret W0527 10:09:10.897547 1 validation.go:171] TLSTunnelCertFile does not exist in /etc/kubeedge/certs/server.crt, will load from secret W0527 10:09:10.897556 1 validation.go:174] TLSTunnelCAFile does not exist in /etc/kubeedge/ca/rootCA.crt, will load from secret I0527 10:09:10.897566 1 server.go:73] Version: v1.7.2 W0527 10:09:10.897580 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0527 10:09:11.910684 1 module.go:34] Module cloudhub registered successfully I0527 10:09:11.922923 1 module.go:34] Module edgecontroller registered successfully I0527 10:09:11.923136 1 module.go:34] Module devicecontroller registered successfully I0527 10:09:11.923769 1 module.go:34] Module synccontroller registered successfully I0527 10:09:11.924170 1 module.go:34] Module cloudStream registered successfully W0527 10:09:11.924183 1 module.go:37] Module router is disabled, do not register W0527 10:09:11.924188 1 module.go:37] Module dynamiccontroller is disabled, do not register I0527 10:09:11.924299 1 core.go:24] Starting module cloudhub I0527 10:09:11.924344 1 core.go:24] Starting module edgecontroller I0527 10:09:11.924389 1 core.go:24] Starting module devicecontroller I0527 10:09:11.924408 1 upstream.go:121] start upstream controller I0527 10:09:11.924428 1 core.go:24] Starting module synccontroller I0527 10:09:11.924451 1 core.go:24] Starting module cloudStream I0527 10:09:11.924497 1 downstream.go:870] Start downstream devicecontroller I0527 10:09:11.925257 1 downstream.go:566] start downstream controller E0527 10:09:11.998958 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io) ***E0527 10:09:12.096627 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)*** I0527 10:09:12.124506 1 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret I0527 10:09:12.126786 1 server.go:247] Ca and CaKey don't exist in the secret, and will be created by CloudCore I0527 10:09:12.201854 1 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret I0527 10:09:12.203132 1 server.go:292] CloudCoreCert and key don't exist in the secret, and will be signed by CA I0527 10:09:12.207185 1 tunnelserver.go:136] Succeed in loading TunnelCA from CloudHub I0527 10:09:12.207531 1 tunnelserver.go:149] Succeed in loading TunnelCert and Key from CloudHub I0527 10:09:12.207700 1 tunnelserver.go:169] Prepare to start tunnel server ... I0527 10:09:12.209109 1 streamserver.go:280] Prepare to start stream server ... I0527 10:09:12.210027 1 signcerts.go:100] Succeed to creating token I0527 10:09:12.210065 1 server.go:44] start unix domain socket server I0527 10:09:12.210225 1 uds.go:71] listening on: //var/lib/kubeedge/kubeedge.sock I0527 10:09:12.210407 1 server.go:64] Starting cloudhub websocket server ***E0527 10:09:13.124710 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io) E0527 10:09:13.450656 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)*** I0527 10:09:13.924726 1 upstream.go:63] Start upstream devicecontroller ***E0527 10:09:15.639730 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io) E0527 10:09:16.432358 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)***
问题二：边缘节点加入时的报错问题
使用 KubeSphere 加入边缘节点时生成的命令，并修改为内网 IP ，在边缘节点上执行以下命令来加入集群：
arch=$(uname -m); curl -LO https://kubeedge.pek3b.qingstor.com/bin/v1.7.2/$arch/keadm-v1.7.2-linux-$arch.tar.gz && tar xvf keadm-v1.7.2-linux-$arch.tar.gz && chmod +x keadm && ./keadm join --kubeedge-version=1.7.2 --region=zh --cloudcore-ipport=192.168.10.40:30000 --quicport 30001 --certport 30002 --tunnelport 30004 --edgenode-name edge1 --edgenode-ip 192.168.10.141 --token e467ec90405bd002fcbda2594c62683f0e2ed3694ccd6e07439fb1d8be94572e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTM3MDM3NTJ9.nhjRfMZdj17wzLd8zsnHto9aEqGESTEwM2BWT-mQxxk --with-edge-taint执行以上命令后，边缘节点可以成功加入集群，但查看边缘节点的日志有几处报错，不知道是否有影响，报错如下：
报错1：
May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998159 6149 core.go:24] Starting module edgemesh May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998240 6149 core.go:24] Starting module metaManager May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998307 6149 core.go:24] Starting module edgestream May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998390 6149 core.go:24] Starting module twin May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998480 6149 core.go:24] Starting module edged May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998537 6149 edged.go:290] Starting edged... May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998642 6149 http.go:40] tlsConfig InsecureSkipVerify true May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998502 6149 process.go:113] Begin to sync sqlite May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998554 6149 core.go:24] Starting module websocket May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.999086 6149 core.go:24] Starting module eventbus ***May 27 02:27:33 k8s-test-edge2 edgecore[6149]: E0527 02:27:33.998678 6149 csi_plugin.go:226] kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost*** May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.999471 6149 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000315 6149 client.go:86] parsed scheme: "unix" May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000332 6149 client.go:86] scheme "unix" not registered, fallback to default scheme May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000382 6149 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000389 6149 clientconn.go:948] ClientConn switching balancer to "pick_first" May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000823 6149 common.go:96] start connect to mqtt server with client id: hub-client-sub-1653618454 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000855 6149 common.go:98] client hub-client-sub-1653618454 isconnected: false May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998396 6149 log.go:181] DEBUG: Installed strategy plugin: [RoundRobin]. May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001331 6149 log.go:181] DEBUG: ConfigurationFactory Initiated May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001354 6149 log.go:181] INFO: Configuration files: [] May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001403 6149 log.go:181] WARN: empty configurtion from [FileSource] May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001420 6149 log.go:181] INFO: invoke dynamic handler:FileSource May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001463 6149 log.go:181] INFO: archaius init success May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001847 6149 log.go:181] INFO: create new watcher May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003303 6149 client.go:150] finish hub-client sub May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003342 6149 common.go:96] start connect to mqtt server with client id: hub-client-pub-1653618454 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003357 6149 common.go:98] client hub-client-pub-1653618454 isconnected: false May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010648 6149 client.go:166] finish hub-client pub May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010671 6149 eventbus.go:63] Init Sub And Pub Client for externel mqtt broker tcp://127.0.0.1:1883 successfully May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010707 6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/upload/# May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010924 6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/state/update May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011051 6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/twin/+ May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011163 6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/node/+/membership/get May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011272 6149 client.go:91] edge-hub-cli subscribe topic to SYS/dis/upload_records May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011406 6149 client.go:91] edge-hub-cli subscribe topic to +/user/# May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011506 6149 client.go:99] list edge-hub-cli-topics status, no record, skip sync ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.014687 6149 proxy.go:143] [EdgeMesh] open file /run/edgemesh-iptables err: open /run/edgemesh-iptables: no such file or directory*** May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.018287 6149 proxy.go:95] [EdgeMesh] chain EDGE-MESH not exists May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.024132 6149 proxy.go:103] [EdgeMesh] inbound rule -p tcp -d 9.251.0.0/16 -i docker0 -j EDGE-MESH not exists May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.034342 6149 certmanager.go:159] Certificate rotation is enabled. May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.034366 6149 websocket.go:51] Websocket start to connect Access May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.035274 6149 proxy.go:111] [EdgeMesh] outbound rule -p tcp -d 9.251.0.0/16 -o docker0 -j EDGE-MESH not exists May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.058096 6149 proxy.go:119] [EdgeMesh] dnat rule -p tcp -j DNAT --to-destination 172.17.0.1:40001 not exists May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.058842 6149 ws.go:46] dial wss://192.168.10.40:30000/e632aba927ea4ac2b575ec1603d56f10/edge1/events successfully May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059074 6149 websocket.go:93] Websocket connect to cloud access successful May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059368 6149 process.go:513] node connection event occur: cloud_connected May 27 02:27:34 k8s-test-edge2 edgecore[6149]: W0527 02:27:34.059464 6149 eventbus.go:148] Action not found May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059541 6149 process.go:513] node connection event occur: cloud_connected May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059601 6149 process.go:282] DeviceTwin receive msg May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059671 6149 process.go:66] Send msg to the CommModule module in twin May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111086 6149 cpu_manager.go:184] [cpumanager] starting with none policy May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111469 6149 cpu_manager.go:185] [cpumanager] reconciling every 0s May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111640 6149 state_mem.go:36] [cpumanager] initializing new in-memory state store May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.113127 6149 policy_none.go:43] [cpumanager] none policy: Start May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.139178 6149 record.go:19] Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.139511 6149 volume_manager.go:265] Starting Kubelet Volume Manager May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.141891 6149 desired_state_of_world_populator.go:139] Desired state populator starts to run ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.153820 6149 imitator.go:222] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in '{"metadata":{"name":"edge1","creationTimestamp":null,"labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"edge1","kubernetes.io/os":"linux","node-role.kubernetes.io/agent":"","node-role.kubernetes.io/edge":""}},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}'***报错2：
May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156121 6149 status_manager.go:53] Starting to sync pod status with apiserver May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156307 6149 edged.go:890] start pod addition queue work 0 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156592 6149 edged.go:890] start pod addition queue work 1 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156692 6149 edged.go:890] start pod addition queue work 2 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156755 6149 edged.go:890] start pod addition queue work 3 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156819 6149 edged.go:890] start pod addition queue work 4 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156895 6149 edged.go:356] starting plugin manager May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156896 6149 server.go:35] starting to listen read-only on 127.0.0.1:10350 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.157240 6149 plugin_manager.go:114] Starting Kubelet Plugin Manager May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.158731 6149 server.go:425] Adding debug handlers to kubelet server. May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.159897 6149 edged_status.go:390] Attempting to register node edge1 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160798 6149 cpu_manager.go:184] [cpumanager] starting with none policy May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160809 6149 cpu_manager.go:185] [cpumanager] reconciling every 1s May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160823 6149 state_mem.go:36] [cpumanager] initializing new in-memory state store May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160924 6149 state_mem.go:88] [cpumanager] updated default cpuset: "" May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160931 6149 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]" May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160941 6149 policy_none.go:43] [cpumanager] none policy: Start May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160957 6149 edged.go:368] starting syncPod May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.170464 6149 edged_status.go:409] Successfully registered node edge1 May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.179008 6149 edged_status.go:198] Sync VolumesInUse: [] ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.211687 6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {}*** May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.258100 6149 listener.go:316] [EdgeMesh] update services: 50 resource: namespace/servicelist/service May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.258392 6149 listener.go:327] [EdgeMesh] update svc kubesphere-logging-system.ks-events-ruler in cache ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.259566 6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {} May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.263035 6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {} May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.359786 6149 imitator.go:222] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in 'null'***
问题三：在边缘节点部署 pod 时出现问题
问题描述：
该应用在集群内部创建可以成功运行；
将该应用调度到边缘节点创建，则无法成功运行：
cloudcore 日志：
I0527 10:27:37.900475 1 session.go:125] Add a new apiserver connection APIServer_MetricsConnection MessageID 3 in to Tunnel session [edge1]
I0527 10:27:37.908200 1 containermetrics_connection.go:117] APIServer_MetricsConnection MessageID 3 find edge peer done, so stop this connection
I0527 10:27:37.908216 1 containermetrics_connection.go:93] APIServer_MetricsConnection MessageID 3 end successful
I0527 10:27:37.908223 1 session.go:133] Delete a apiserver connection APIServer_MetricsConnection MessageID 3 from Tunnel session [edge1]
I0527 10:27:37.908227 1 streamserver.go:189] Delete APIServer_MetricsConnection MessageID 3 from Tunnel session [edge1]
I0527 10:27:40.126371 1 upstream.go:88] Dispatch message: 5f765cde-43e3-4fad-b4a9-4dbe0d4fa21f
W0527 10:27:40.127293 1 upstream.go:92] Parse message: 5f765cde-43e3-4fad-b4a9-4dbe0d4fa21f resource type with error: unknown resource
***E0527 10:27:49.999931 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)
E0527 10:28:02.091135 1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)***
I0527 10:28:27.573267 1 session.go:125] Add a new apiserver connection APIServer_LogsConnection MessageID 4 in to Tunnel session [edge1]
I0527 10:28:27.577977 1 containerlog_connection.go:116] APIServer_LogsConnection MessageID 4 find edge peer done, so stop this connection
I0527 10:28:27.577988 1 containerlog_connection.go:92] APIServer_LogsConnection MessageID 4 end successful
I0527 10:28:27.577995 1 session.go:133] Delete a apiserver connection APIServer_LogsConnection MessageID 4 from Tunnel session [edge1]
I0527 10:28:27.577999 1 streamserver.go:139] Delete APIServer_LogsConnection MessageID 4 from Tunnel session [edge1]
***E0527 10:28:29.251566 1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:default,name:pvc-a1e452be-a4a5-451a-8d1f-3569a8289835), default "pvc-a1e452be-a4a5-451a-8d1f-3569a8289835" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "default" in API group "" at the cluster scope
E0527 10:28:29.257808 1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:consul-config), edge-edgex1 "consul-config" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.262112 1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:consul-data), edge-edgex1 "consul-data" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.274941 1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:db-data), edge-edgex1 "db-data" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.303132 1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:default,name:pvc-ec504cbf-a4b7-4a04-9b38-932c50bf2607), default "pvc-ec504cbf-a4b7-4a04-9b38-932c50bf2607" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "default" in API group "" at the cluster scope***边缘节点 edgecore 日志：
May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184467 6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184817 6149 record.go:24] Warning MissingClusterDNS pod: "edgex-core-metadata-78788c8c48-2cmkz_edge-edgex1(eb4a0914-f8c4-4a23-9ea4-b8c077dc9a7d)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184953 6149 edged.go:1015] consume added pod [edgex-core-metadata-78788c8c48-2cmkz] successfully May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.189078 6149 edged.go:900] worker [2] get pod addition item [edgex-support-notifications-598f7f85d-q6bfw] ***May 27 03:03:09 k8s-test-edge2 edgecore[6149]: E0527 03:03:09.189313 6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-support-notifications-598f7f85d-q6bfw] addition error, backoff: [20s]*** May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.189413 6149 edged.go:905] worker [2] backoff pod addition item [edgex-support-notifications-598f7f85d-q6bfw] failed, re-add to queue May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.344129 6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-5pbcc May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.007517 6149 edged.go:900] worker [3] get pod addition item [edgex-support-scheduler-5f9c499574-4jj64] May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.007606 6149 edged.go:968] start to consume added pod [edgex-support-scheduler-5f9c499574-4jj64] May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.008306 6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.008359 6149 record.go:24] Warning MissingClusterDNS pod: "edgex-support-scheduler-5f9c499574-4jj64_edge-edgex1(da0167dc-a3cb-4760-842e-ee4601a139f7)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.009382 6149 record.go:24] Warning BackOff Back-off restarting failed container ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.009558 6149 edged.go:919] worker [3] handle pod addition item [edgex-support-scheduler-5f9c499574-4jj64] failed: failed to "StartContainer" for "edgex-support-scheduler" with CrashLoopBackOff: "back-off 5m0s restarting failed container=edgex-support-scheduler pod=edgex-support-scheduler-5f9c499574-4jj64_edge-edgex1(da0167dc-a3cb-4760-842e-ee4601a139f7)", re-add to queue*** May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.146028 6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-zv6l9 May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.720984 6149 edged.go:900] worker [1] get pod addition item [edgex-core-data-5bb8bcc584-95zzk] ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.721058 6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-core-data-5bb8bcc584-95zzk] addition error, backoff: [1m20s]*** May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.721118 6149 edged.go:905] worker [1] backoff pod addition item [edgex-core-data-5bb8bcc584-95zzk] failed, re-add to queue May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.727149 6149 edged.go:900] worker [4] get pod addition item [edgex-core-command-6fb8d849bc-q9qvr] ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.727166 6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-core-command-6fb8d849bc-q9qvr] addition error, backoff: [20s]*** May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.727184 6149 edged.go:905] worker [4] backoff pod addition item [edgex-core-command-6fb8d849bc-q9qvr] failed, re-add to queue May 27 03:03:11 k8s-test-edge2 edgecore[6149]: I0527 03:03:11.327897 6149 edged_status.go:198] Sync VolumesInUse: [] May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.945156 6149 edged.go:900] worker [0] get pod addition item [edgex-sys-mgmt-agent-76698f698-g2n7w] May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.945856 6149 edged.go:968] start to consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w] May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.946594 6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.946878 6149 record.go:24] Warning MissingClusterDNS pod: "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949592 6149 record.go:19] Normal Pulled Container image "edgexfoundry/sys-mgmt-agent:2.1.0" already present on machine May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949665 6149 edged_pods.go:321] container: edge-edgex1/edgex-sys-mgmt-agent-76698f698-g2n7w/edgex-sys-mgmt-agent podIP: "172.17.0.13" creating hosts mount: true May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949693 6149 edged_pods.go:403] Pod "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)" container "edgex-sys-mgmt-agent" mount "system-claim0" has propagation "PROPAGATION_HOST_TO_CONTAINER" May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949721 6149 edged_pods.go:403] Pod "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)" container "edgex-sys-mgmt-agent" mount "kube-api-access-f7s2p" has propagation "PROPAGATION_HOST_TO_CONTAINER" May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.978162 6149 record.go:19] Normal Created Created container edgex-sys-mgmt-agent May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.053832 6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.058828 6149 record.go:19] Normal Started Started container edgex-sys-mgmt-agent May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.058926 6149 edged.go:1015] consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w] successfully May 27 03:03:19 k8s-test-edge2 edgecore[6149]: E0527 03:03:19.067279 6149 dns.go:290] [EdgeMesh] service edgex-core-consul is not found in this cluster May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.067297 6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.067311 6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.069355 6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-f7s2p ***May 27 03:03:19 k8s-test-edge2 edgecore[6149]: E0527 03:03:19.078343 6149 dns.go:290] [EdgeMesh] service edgex-core-consul is not found in this cluster*** May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.078520 6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns May 27 03:03:20 k8s-test-edge2 edgecore[6149]: I0527 03:03:20.721558 6149 edged.go:900] worker [1] get pod addition item [edgex-sys-mgmt-agent-76698f698-g2n7w] May 27 03:03:20 k8s-test-edge2 edgecore[6149]: I0527 03:03:20.721595 6149 edged.go:968] start to consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w]
问题四：物理环境重启后无法对边缘节点进行 log、exec、metric
- 将运行着 KubeSphere 、边缘节点的虚拟机所在的服务器重启后，在 KubeSphere 上查看边缘节点信息，此时能看到边缘节点，但无法 log、exec、metric，包括 CPU 、内存的信息都无法显示，对边缘节点的 pod 进行 log 、exec 的时候出现报错： ip:端口 connect refused；
