由于文章被限制在 65535 个字符，无法详细描述，文章内容会精简一部分；

设备环境相关信息

物理环境信息

一台服务器虚拟出来的五台虚拟机：
Master 节点两台，4C/8G；
Worker 节点三台，8C/16G；

操作系统信息

CentOS 7.9

系统安装相关信息

Docker 版本

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
 containerd:
  Version:          1.4.12
 runc:
  Version:          1.0.2

Kubernetes 版本

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

KubeSphere 版本、安装方式

版本：v3.2.1

安装方式：

在纯净的 CentOS 上，使用 kk All in one 方式安装，kk 配置如下：

  apiVersion: kubekey.kubesphere.io/v1alpha2
  kind: Cluster
  metadata:
    name: sample
  spec:
    hosts:
    - {name: k8s-test-master1, address: 192.168.10.10, internalAddress: 192.168.10.10, user: root, password: "root"}
    - {name: k8s-test-master2, address: 192.168.10.11, internalAddress: 192.168.10.11, user: root, password: "root"}
    - {name: k8s-test-worker1, address: 192.168.10.12, internalAddress: 192.168.10.12, user: root, password: "root"}
    - {name: k8s-test-worker2, address: 192.168.10.13, internalAddress: 192.168.10.13, user: root, password: "root"}
    - {name: k8s-test-worker3, address: 192.168.10.14, internalAddress: 192.168.10.14, user: root, password: "root"}
    roleGroups:
      etcd:
      - k8s-test-master1
      - k8s-test-master2   
      control-plane: 
      - k8s-test-master1
      - k8s-test-master2   
      worker:
      - k8s-test-worker1
      - k8s-test-worker2
      - k8s-test-worker3
    controlPlaneEndpoint:
      internalLoadbalancer: haproxy
      domain: lb.kubesphere.local
      address: ""
      port: 6443
    kubernetes:
      version: v1.21.5
      clusterName: cluster.local
      autoRenewCerts: true
    etcd:
      type: kubekey
    network:
      plugin: calico
      kubePodsCIDR: 10.233.64.0/18
      kubeServiceCIDR: 10.233.0.0/18
      multusCNI:
        enabled: false
    registry:
      plainHTTP: false
      privateRegistry: ""
      namespaceOverride: ""
      registryMirrors: ["https://registry.docker-cn.com", "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com"]
    addons: []
  
  ---
  apiVersion: installer.kubesphere.io/v1alpha1
  kind: ClusterConfiguration
  metadata:
    name: ks-installer
    namespace: kubesphere-system
    labels:
      version: v3.2.1
  spec:
    persistence:
      storageClass: ""
    authentication:
      jwtSecret: ""
    zone: ""
    local_registry: ""
    namespace_override: ""
    etcd:
      monitoring: true
      endpointIps: localhost
      port: 2379
      tlsEnable: true
    common:
      core:
        console:
          enableMultiLogin: true
          port: 30880
          type: NodePort
      redis:
        enabled: false
        volumeSize: 2Gi
      openldap:
        enabled: false
        volumeSize: 2Gi
      minio:
        volumeSize: 20Gi
      monitoring:
        endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
        GPUMonitoring:
          enabled: false
      gpu:
        kinds:
        - resourceName: "nvidia.com/gpu"
          resourceType: "GPU"
          default: true
      es:
        logMaxAge: 7
        elkPrefix: logstash
        basicAuth:
          enabled: false
          username: ""
          password: ""
        externalElasticsearchHost: ""
        externalElasticsearchPort: ""
    alerting:
      enabled: true
    auditing:
      enabled: true
    devops:
      enabled: true
      jenkinsMemoryLim: 2Gi
      jenkinsMemoryReq: 1500Mi
      jenkinsVolumeSize: 8Gi
      jenkinsJavaOpts_Xms: 512m
      jenkinsJavaOpts_Xmx: 512m
      jenkinsJavaOpts_MaxRAM: 2g
    events:
      enabled: true
    logging:
      enabled: true
      containerruntime: docker
      logsidecar:
        enabled: true
        replicas: 2
    metrics_server:
      enabled: true
    monitoring:
      storageClass: ""
      gpu:
        nvidia_dcgm_exporter:
          enabled: false
          # resources: {}
    multicluster:
      clusterRole: none
    network:
      networkpolicy:
        enabled: false
      ippool:
        type: calico
      topology:
        type: weave scope
    openpitrix:
      store:
        enabled: true
    servicemesh:
      enabled: true
    kubeedge:
      enabled: true   
      cloudCore:
        nodeSelector: {"node-role.kubernetes.io/worker": ""}
        tolerations: []
        cloudhubPort: "10000"
        cloudhubQuicPort: "10001"
        cloudhubHttpsPort: "10002"
        cloudstreamPort: "10003"
        tunnelPort: "10004"
        cloudHub:
          advertiseAddress:
            - "192.168.10.10"
            - "192.168.10.11"
          nodeLimit: "800"
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
      edgeWatcher:
        nodeSelector: {"node-role.kubernetes.io/worker": ""}
        tolerations: []
        edgeWatcherAgent:
          nodeSelector: {"node-role.kubernetes.io/worker": ""}
          tolerations: []

KubeEdge 版本

v1.7.2

问题描述

背景

边缘节点也是服务器下的一台虚拟机，用于作为边缘节点测试；

使用容忍的方式，使得 iptables 可以运行在 master 节点上，解决了无法在 KubeSphere 上对边缘节点进行 log、exec；

  kubectl edit iptables -n kubeedge

在 master 节点执行了以下脚本，用于对边缘节点进行了 patch，使得容忍度较强的 calico、kubeproxy 等 pod 不会被调度到边缘节点上：

  #!/bin/bash
  
  NodeSelectorPatchJson='{"spec":{"template":{"spec":{"nodeSelector":{"node-role.kubernetes.io/master": "","node-role.kubernetes.io/worker": ""}}}}}'
  NoShedulePatchJson='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}}}}}}'
  
  edgenode="edge1"
  if [ $1 ]; then
          edgenode="$1"
  fi
  
  namespaces=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $1}' ))
  pods=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $2}' ))
  length=${#namespaces[@]}
  
  for((i=0;i<$length;i++));  
  do
          ns=${namespaces[$i]}
          pod=${pods[$i]}
          resources=$(kubectl -n $ns describe pod $pod | grep "Controlled By" |awk '{print $3}')
          echo "Patching for ns: $ns, resources: $resources"
          kubectl -n $ns patch $resources --type merge --patch "$NoShedulePatchJson"
          sleep 1
  done

目前已经可以成功对边缘节点进行 log、exec、metric；

问题一：安装完 KubeSphere 后，cloudcore 的报错问题

此时还没加入边缘节点，不知道该报错是否有影响，以下为 cloudcore 的日志：

  W0527 10:09:10.897412       1 validation.go:168] TLSTunnelPrivateKeyFile does not exist in /etc/kubeedge/certs/server.key, will load from secret
  W0527 10:09:10.897547       1 validation.go:171] TLSTunnelCertFile does not exist in /etc/kubeedge/certs/server.crt, will load from secret
  W0527 10:09:10.897556       1 validation.go:174] TLSTunnelCAFile does not exist in /etc/kubeedge/ca/rootCA.crt, will load from secret
  I0527 10:09:10.897566       1 server.go:73] Version: v1.7.2
  W0527 10:09:10.897580       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  I0527 10:09:11.910684       1 module.go:34] Module cloudhub registered successfully
  I0527 10:09:11.922923       1 module.go:34] Module edgecontroller registered successfully
  I0527 10:09:11.923136       1 module.go:34] Module devicecontroller registered successfully
  I0527 10:09:11.923769       1 module.go:34] Module synccontroller registered successfully
  I0527 10:09:11.924170       1 module.go:34] Module cloudStream registered successfully
  W0527 10:09:11.924183       1 module.go:37] Module router is disabled, do not register
  W0527 10:09:11.924188       1 module.go:37] Module dynamiccontroller is disabled, do not register
  I0527 10:09:11.924299       1 core.go:24] Starting module cloudhub
  I0527 10:09:11.924344       1 core.go:24] Starting module edgecontroller
  I0527 10:09:11.924389       1 core.go:24] Starting module devicecontroller
  I0527 10:09:11.924408       1 upstream.go:121] start upstream controller
  I0527 10:09:11.924428       1 core.go:24] Starting module synccontroller
  I0527 10:09:11.924451       1 core.go:24] Starting module cloudStream
  I0527 10:09:11.924497       1 downstream.go:870] Start downstream devicecontroller
  I0527 10:09:11.925257       1 downstream.go:566] start downstream controller
  E0527 10:09:11.998958       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)
  ***E0527 10:09:12.096627       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)***
  I0527 10:09:12.124506       1 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret
  I0527 10:09:12.126786       1 server.go:247] Ca and CaKey don't exist in the secret, and will be created by CloudCore
  I0527 10:09:12.201854       1 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret
  I0527 10:09:12.203132       1 server.go:292] CloudCoreCert and key don't exist in the secret, and will be signed by CA
  I0527 10:09:12.207185       1 tunnelserver.go:136] Succeed in loading TunnelCA from CloudHub
  I0527 10:09:12.207531       1 tunnelserver.go:149] Succeed in loading TunnelCert and Key from CloudHub
  I0527 10:09:12.207700       1 tunnelserver.go:169] Prepare to start tunnel server ...
  I0527 10:09:12.209109       1 streamserver.go:280] Prepare to start stream server ...
  I0527 10:09:12.210027       1 signcerts.go:100] Succeed to creating token
  I0527 10:09:12.210065       1 server.go:44] start unix domain socket server
  I0527 10:09:12.210225       1 uds.go:71] listening on: //var/lib/kubeedge/kubeedge.sock
  I0527 10:09:12.210407       1 server.go:64] Starting cloudhub websocket server
  ***E0527 10:09:13.124710       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)
  E0527 10:09:13.450656       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)***
  I0527 10:09:13.924726       1 upstream.go:63] Start upstream devicecontroller
  ***E0527 10:09:15.639730       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)
  E0527 10:09:16.432358       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)***

问题二：边缘节点加入时的报错问题

使用 KubeSphere 加入边缘节点时生成的命令，并修改为内网 IP ，在边缘节点上执行以下命令来加入集群：

  arch=$(uname -m); curl -LO https://kubeedge.pek3b.qingstor.com/bin/v1.7.2/$arch/keadm-v1.7.2-linux-$arch.tar.gz  && tar xvf keadm-v1.7.2-linux-$arch.tar.gz && chmod +x keadm && ./keadm join --kubeedge-version=1.7.2 --region=zh --cloudcore-ipport=192.168.10.40:30000 --quicport 30001 --certport 30002 --tunnelport 30004 --edgenode-name edge1 --edgenode-ip 192.168.10.141 --token e467ec90405bd002fcbda2594c62683f0e2ed3694ccd6e07439fb1d8be94572e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTM3MDM3NTJ9.nhjRfMZdj17wzLd8zsnHto9aEqGESTEwM2BWT-mQxxk --with-edge-taint

执行以上命令后，边缘节点可以成功加入集群，但查看边缘节点的日志有几处报错，不知道是否有影响，报错如下：

报错1：

    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998159    6149 core.go:24] Starting module edgemesh
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998240    6149 core.go:24] Starting module metaManager
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998307    6149 core.go:24] Starting module edgestream
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998390    6149 core.go:24] Starting module twin
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998480    6149 core.go:24] Starting module edged
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998537    6149 edged.go:290] Starting edged...
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998642    6149 http.go:40] tlsConfig InsecureSkipVerify true
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998502    6149 process.go:113] Begin to sync sqlite
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998554    6149 core.go:24] Starting module websocket
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.999086    6149 core.go:24] Starting module eventbus
    ***May 27 02:27:33 k8s-test-edge2 edgecore[6149]: E0527 02:27:33.998678    6149 csi_plugin.go:226] kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost***
    May 27 02:27:33 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.999471    6149 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000315    6149 client.go:86] parsed scheme: "unix"
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000332    6149 client.go:86] scheme "unix" not registered, fallback to default scheme
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000382    6149 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000389    6149 clientconn.go:948] ClientConn switching balancer to "pick_first"
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000823    6149 common.go:96] start connect to mqtt server with client id: hub-client-sub-1653618454
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.000855    6149 common.go:98] client hub-client-sub-1653618454 isconnected: false
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:33.998396    6149 log.go:181] DEBUG: Installed strategy plugin: [RoundRobin].
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001331    6149 log.go:181] DEBUG: ConfigurationFactory Initiated
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001354    6149 log.go:181] INFO: Configuration files: []
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001403    6149 log.go:181] WARN: empty configurtion from [FileSource]
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001420    6149 log.go:181] INFO: invoke dynamic handler:FileSource
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001463    6149 log.go:181] INFO: archaius init success
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.001847    6149 log.go:181] INFO: create new watcher
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003303    6149 client.go:150] finish hub-client sub
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003342    6149 common.go:96] start connect to mqtt server with client id: hub-client-pub-1653618454
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.003357    6149 common.go:98] client hub-client-pub-1653618454 isconnected: false
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010648    6149 client.go:166] finish hub-client pub
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010671    6149 eventbus.go:63] Init Sub And Pub Client for externel mqtt broker tcp://127.0.0.1:1883 successfully
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010707    6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/upload/#
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.010924    6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/state/update
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011051    6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/twin/+
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011163    6149 client.go:91] edge-hub-cli subscribe topic to $hw/events/node/+/membership/get
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011272    6149 client.go:91] edge-hub-cli subscribe topic to SYS/dis/upload_records
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011406    6149 client.go:91] edge-hub-cli subscribe topic to +/user/#
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.011506    6149 client.go:99] list edge-hub-cli-topics status, no record, skip sync
    ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.014687    6149 proxy.go:143] [EdgeMesh] open file /run/edgemesh-iptables err: open /run/edgemesh-iptables: no such file or directory***
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.018287    6149 proxy.go:95] [EdgeMesh] chain EDGE-MESH not exists
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.024132    6149 proxy.go:103] [EdgeMesh] inbound rule -p tcp -d 9.251.0.0/16 -i docker0 -j EDGE-MESH not exists
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.034342    6149 certmanager.go:159] Certificate rotation is enabled.
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.034366    6149 websocket.go:51] Websocket start to connect Access
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.035274    6149 proxy.go:111] [EdgeMesh] outbound rule -p tcp -d 9.251.0.0/16 -o docker0 -j EDGE-MESH not exists
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.058096    6149 proxy.go:119] [EdgeMesh] dnat rule -p tcp -j DNAT --to-destination 172.17.0.1:40001 not exists
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.058842    6149 ws.go:46] dial wss://192.168.10.40:30000/e632aba927ea4ac2b575ec1603d56f10/edge1/events successfully
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059074    6149 websocket.go:93] Websocket connect to cloud access successful
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059368    6149 process.go:513] node connection event occur: cloud_connected
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: W0527 02:27:34.059464    6149 eventbus.go:148] Action not found
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059541    6149 process.go:513] node connection event occur: cloud_connected
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059601    6149 process.go:282] DeviceTwin receive msg
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.059671    6149 process.go:66] Send msg to the CommModule module in twin
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111086    6149 cpu_manager.go:184] [cpumanager] starting with none policy
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111469    6149 cpu_manager.go:185] [cpumanager] reconciling every 0s
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.111640    6149 state_mem.go:36] [cpumanager] initializing new in-memory state store
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.113127    6149 policy_none.go:43] [cpumanager] none policy: Start
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.139178    6149 record.go:19] Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.139511    6149 volume_manager.go:265] Starting Kubelet Volume Manager
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.141891    6149 desired_state_of_world_populator.go:139] Desired state populator starts to run
    ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.153820    6149 imitator.go:222] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in '{"metadata":{"name":"edge1","creationTimestamp":null,"labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"edge1","kubernetes.io/os":"linux","node-role.kubernetes.io/agent":"","node-role.kubernetes.io/edge":""}},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}'***

报错2：

    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156121    6149 status_manager.go:53] Starting to sync pod status with apiserver
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156307    6149 edged.go:890] start pod addition queue work 0
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156592    6149 edged.go:890] start pod addition queue work 1
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156692    6149 edged.go:890] start pod addition queue work 2
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156755    6149 edged.go:890] start pod addition queue work 3
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156819    6149 edged.go:890] start pod addition queue work 4
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156895    6149 edged.go:356] starting plugin manager
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.156896    6149 server.go:35] starting to listen read-only on 127.0.0.1:10350
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.157240    6149 plugin_manager.go:114] Starting Kubelet Plugin Manager
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.158731    6149 server.go:425] Adding debug handlers to kubelet server.
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.159897    6149 edged_status.go:390] Attempting to register node edge1
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160798    6149 cpu_manager.go:184] [cpumanager] starting with none policy
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160809    6149 cpu_manager.go:185] [cpumanager] reconciling every 1s
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160823    6149 state_mem.go:36] [cpumanager] initializing new in-memory state store
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160924    6149 state_mem.go:88] [cpumanager] updated default cpuset: ""
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160931    6149 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]"
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160941    6149 policy_none.go:43] [cpumanager] none policy: Start
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.160957    6149 edged.go:368] starting syncPod
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.170464    6149 edged_status.go:409] Successfully registered node edge1
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.179008    6149 edged_status.go:198] Sync VolumesInUse: []
    ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.211687    6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {}***
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.258100    6149 listener.go:316] [EdgeMesh] update services: 50 resource: namespace/servicelist/service
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: I0527 02:27:34.258392    6149 listener.go:327] [EdgeMesh] update svc kubesphere-logging-system.ks-events-ruler in cache
    ***May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.259566    6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {}
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.263035    6149 imitator.go:222] failed to unmarshal message content to unstructured obj: json: cannot unmarshal array into Go value of type map[string]interface {}
    May 27 02:27:34 k8s-test-edge2 edgecore[6149]: E0527 02:27:34.359786    6149 imitator.go:222] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in 'null'***

问题三：在边缘节点部署 pod 时出现问题

问题描述：
- 该应用在集群内部创建可以成功运行；
- 将该应用调度到边缘节点创建，则无法成功运行：
cloudcore 日志：

I0527 10:27:37.900475       1 session.go:125] Add a new apiserver connection APIServer_MetricsConnection MessageID 3 in to Tunnel session [edge1]
I0527 10:27:37.908200       1 containermetrics_connection.go:117] APIServer_MetricsConnection MessageID 3 find edge peer done, so stop this connection
I0527 10:27:37.908216       1 containermetrics_connection.go:93] APIServer_MetricsConnection MessageID 3 end successful
I0527 10:27:37.908223       1 session.go:133] Delete a apiserver connection APIServer_MetricsConnection MessageID 3 from Tunnel session [edge1]
I0527 10:27:37.908227       1 streamserver.go:189] Delete APIServer_MetricsConnection MessageID 3 from Tunnel session [edge1]
I0527 10:27:40.126371       1 upstream.go:88] Dispatch message: 5f765cde-43e3-4fad-b4a9-4dbe0d4fa21f
W0527 10:27:40.127293       1 upstream.go:92] Parse message: 5f765cde-43e3-4fad-b4a9-4dbe0d4fa21f resource type with error: unknown resource
***E0527 10:27:49.999931       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.DeviceModel: failed to list *v1alpha2.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)
E0527 10:28:02.091135       1 reflector.go:127] github.com/kubeedge/kubeedge/cloud/pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1alpha2.Device: failed to list *v1alpha2.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)***
I0527 10:28:27.573267       1 session.go:125] Add a new apiserver connection APIServer_LogsConnection MessageID 4 in to Tunnel session [edge1]
I0527 10:28:27.577977       1 containerlog_connection.go:116] APIServer_LogsConnection MessageID 4 find edge peer done, so stop this connection
I0527 10:28:27.577988       1 containerlog_connection.go:92] APIServer_LogsConnection MessageID 4 end successful
I0527 10:28:27.577995       1 session.go:133] Delete a apiserver connection APIServer_LogsConnection MessageID 4 from Tunnel session [edge1]
I0527 10:28:27.577999       1 streamserver.go:139] Delete APIServer_LogsConnection MessageID 4 from Tunnel session [edge1]
***E0527 10:28:29.251566       1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:default,name:pvc-a1e452be-a4a5-451a-8d1f-3569a8289835), default "pvc-a1e452be-a4a5-451a-8d1f-3569a8289835" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "default" in API group "" at the cluster scope
E0527 10:28:29.257808       1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:consul-config), edge-edgex1 "consul-config" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.262112       1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:consul-data), edge-edgex1 "consul-data" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.274941       1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:edge-edgex1,name:db-data), edge-edgex1 "db-data" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "edge-edgex1" in API group "" at the cluster scope
E0527 10:28:29.303132       1 objectsync.go:38] failed to get obj(gvr:/, Resource=,namespace:default,name:pvc-ec504cbf-a4b7-4a04-9b38-932c50bf2607), default "pvc-ec504cbf-a4b7-4a04-9b38-932c50bf2607" is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot get resource "default" in API group "" at the cluster scope***

边缘节点 edgecore 日志：

  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184467    6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184817    6149 record.go:24] Warning MissingClusterDNS pod: "edgex-core-metadata-78788c8c48-2cmkz_edge-edgex1(eb4a0914-f8c4-4a23-9ea4-b8c077dc9a7d)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.184953    6149 edged.go:1015] consume added pod [edgex-core-metadata-78788c8c48-2cmkz] successfully
  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.189078    6149 edged.go:900] worker [2] get pod addition item [edgex-support-notifications-598f7f85d-q6bfw]
  ***May 27 03:03:09 k8s-test-edge2 edgecore[6149]: E0527 03:03:09.189313    6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-support-notifications-598f7f85d-q6bfw] addition  error, backoff: [20s]***
  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.189413    6149 edged.go:905] worker [2] backoff pod addition item [edgex-support-notifications-598f7f85d-q6bfw] failed, re-add to queue
  May 27 03:03:09 k8s-test-edge2 edgecore[6149]: I0527 03:03:09.344129    6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-5pbcc
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.007517    6149 edged.go:900] worker [3] get pod addition item [edgex-support-scheduler-5f9c499574-4jj64]
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.007606    6149 edged.go:968] start to consume added pod [edgex-support-scheduler-5f9c499574-4jj64]
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.008306    6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.008359    6149 record.go:24] Warning MissingClusterDNS pod: "edgex-support-scheduler-5f9c499574-4jj64_edge-edgex1(da0167dc-a3cb-4760-842e-ee4601a139f7)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.009382    6149 record.go:24] Warning BackOff Back-off restarting failed container
  ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.009558    6149 edged.go:919] worker [3] handle pod addition item [edgex-support-scheduler-5f9c499574-4jj64] failed: failed to "StartContainer" for "edgex-support-scheduler" with CrashLoopBackOff: "back-off 5m0s restarting failed container=edgex-support-scheduler pod=edgex-support-scheduler-5f9c499574-4jj64_edge-edgex1(da0167dc-a3cb-4760-842e-ee4601a139f7)", re-add to queue***
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.146028    6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-zv6l9
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.720984    6149 edged.go:900] worker [1] get pod addition item [edgex-core-data-5bb8bcc584-95zzk]
  ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.721058    6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-core-data-5bb8bcc584-95zzk] addition  error, backoff: [1m20s]***
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.721118    6149 edged.go:905] worker [1] backoff pod addition item [edgex-core-data-5bb8bcc584-95zzk] failed, re-add to queue
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.727149    6149 edged.go:900] worker [4] get pod addition item [edgex-core-command-6fb8d849bc-q9qvr]
  ***May 27 03:03:10 k8s-test-edge2 edgecore[6149]: E0527 03:03:10.727166    6149 edged.go:903] consume pod addition backoff: Back-off consume pod [edgex-core-command-6fb8d849bc-q9qvr] addition  error, backoff: [20s]***
  May 27 03:03:10 k8s-test-edge2 edgecore[6149]: I0527 03:03:10.727184    6149 edged.go:905] worker [4] backoff pod addition item [edgex-core-command-6fb8d849bc-q9qvr] failed, re-add to queue
  May 27 03:03:11 k8s-test-edge2 edgecore[6149]: I0527 03:03:11.327897    6149 edged_status.go:198] Sync VolumesInUse: []
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.945156    6149 edged.go:900] worker [0] get pod addition item [edgex-sys-mgmt-agent-76698f698-g2n7w]
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.945856    6149 edged.go:968] start to consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w]
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.946594    6149 record.go:24] Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.946878    6149 record.go:24] Warning MissingClusterDNS pod: "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949592    6149 record.go:19] Normal Pulled Container image "edgexfoundry/sys-mgmt-agent:2.1.0" already present on machine
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949665    6149 edged_pods.go:321] container: edge-edgex1/edgex-sys-mgmt-agent-76698f698-g2n7w/edgex-sys-mgmt-agent podIP: "172.17.0.13" creating hosts mount: true
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949693    6149 edged_pods.go:403] Pod "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)" container "edgex-sys-mgmt-agent" mount "system-claim0" has propagation "PROPAGATION_HOST_TO_CONTAINER"
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.949721    6149 edged_pods.go:403] Pod "edgex-sys-mgmt-agent-76698f698-g2n7w_edge-edgex1(5e5563dd-1d8e-48eb-aab6-708f4c12d1d9)" container "edgex-sys-mgmt-agent" mount "kube-api-access-f7s2p" has propagation "PROPAGATION_HOST_TO_CONTAINER"
  May 27 03:03:18 k8s-test-edge2 edgecore[6149]: I0527 03:03:18.978162    6149 record.go:19] Normal Created Created container edgex-sys-mgmt-agent
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.053832    6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.058828    6149 record.go:19] Normal Started Started container edgex-sys-mgmt-agent
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.058926    6149 edged.go:1015] consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w] successfully
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: E0527 03:03:19.067279    6149 dns.go:290] [EdgeMesh] service edgex-core-consul is not found in this cluster
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.067297    6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.067311    6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: I0527 03:03:19.069355    6149 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-api-access-f7s2p
  ***May 27 03:03:19 k8s-test-edge2 edgecore[6149]: E0527 03:03:19.078343    6149 dns.go:290] [EdgeMesh] service edgex-core-consul is not found in this cluster***
  May 27 03:03:19 k8s-test-edge2 edgecore[6149]: W0527 03:03:19.078520    6149 dns.go:125] [EdgeMesh] failed to resolve dns: get from real dns
  May 27 03:03:20 k8s-test-edge2 edgecore[6149]: I0527 03:03:20.721558    6149 edged.go:900] worker [1] get pod addition item [edgex-sys-mgmt-agent-76698f698-g2n7w]
  May 27 03:03:20 k8s-test-edge2 edgecore[6149]: I0527 03:03:20.721595    6149 edged.go:968] start to consume added pod [edgex-sys-mgmt-agent-76698f698-g2n7w]

问题四：物理环境重启后无法对边缘节点进行 log、exec、metric

将运行着 KubeSphere 、边缘节点的虚拟机所在的服务器重启后，在 KubeSphere 上查看边缘节点信息，此时能看到边缘节点，但无法 log、exec、metric，包括 CPU 、内存的信息都无法显示，对边缘节点的 pod 进行 log 、exec 的时候出现报错： ip:端口 connect refused；