KubeSphere 3.1 KubeEdge边缘节点的服务间通信问题

liamylian

安装方式：
All-in-One 3.1（./kk create cluster –with-kubernetes v1.20.4 –with-kubesphere v3.1.0）
KubeEdge配置方式：
【自定义资源CRD】【ClusterConfiguration】【ks-installer】配置文件中，设置KubeEdge.enable=true，及advertiseAddress为master节点外网IP。
集群服务器
阿里云4C8G，Ubuntu20.04
边缘节点
阿里云2C2G, Ubuntu20.04

按KubeSphere3.1及KubeEdge官网添加边缘节点后，在Master节点部署了httpd，边缘节点A部署了nginx，边缘节点B部署了caddy。

在边缘节点A上测试发现连续访问这些地址非常不稳定（在边缘节点B上也一样）：http://nginx-svc.default.svc.cluster.local:12345，http://caddy-svc.default.svc.cluster.local:12346，即一下访问正常，一下访问不通，一下访问Nginx的页面却访问到了Caddy的页面。

访问Nginx页面正常情况：

访问Nginx页面却访问到了Caddy：

访问Nginx页面失败情况一：

访问Nginx页面失败情况二：

访问Caddy页面正常情况：

访问Caddy页面却访问到了Nginx：

访问Caddy页面失败情况一：

访问Caddy页面失败情况二：

在边缘节点A上测试访问（在边缘节点B上也一样）http://httpd-svc.default.svc.cluster.local:12347，一般都正常，但耗时较久。

附服务配置文件：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
              hostPort: 80
      nodeSelector:
        node-role.kubernetes.io/edge: ""
      tolerations:
        - key: node-role.kubernetes.io/edge
          operator: Exists
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
    - name: http-0
      port: 12345
      protocol: TCP
      targetPort: 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: caddy-deployment
  labels:
    app: caddy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: caddy
  template:
    metadata:
      labels:
        app: caddy
    spec:
      hostNetwork: true
      containers:
        - name: caddy
          image: caddy:2.3.0-alpine
          ports:
            - containerPort: 80
              hostPort: 80
      nodeSelector:
        node-role.kubernetes.io/edge: ""
      tolerations:
        - key: node-role.kubernetes.io/edge
          operator: Exists
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: caddy-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: caddy
  ports:
    - name: http-0
      port: 12346
      protocol: TCP
      targetPort: 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment
  labels:
    app: httpd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
        - name: httpd
          image: httpd:2.4-alpine
          ports:
            - containerPort: 80
              hostPort: 8030
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: httpd-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: httpd
  ports:
    - name: http-0
      port: 12347
      protocol: TCP
      targetPort: 80

liamylian

今天使用KubeEdge官网的安装方式安装KubeEdge，而不是使用KubeSphere3.1的安装方式，仍然有服务页面串掉的问题。应该是KubeEdge引起的问题！

另外，今天尝试部署不同端口的服务，才不会有服务页面串掉的问题。。。

附：
部署在主节点的Nginx（8050端口）

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx8050-configmap
data:
  nginx_conf: |-
    user  nginx;
    worker_processes  1;
    events {
        worker_connections  1024;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;
        sendfile        on;
        keepalive_timeout  65;
        server {
            listen       8050;
            listen  [::]:8050;
            server_name  localhost;
            location / {
                root   /usr/share/nginx/html;
                index  index.html index.htm;
            }
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   /usr/share/nginx/html;
            }
        }
    }
  index_page: |-
    <html><body><h1>It works! Nginx@8050. </h1></body></html>

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx8050-deployment
  labels:
    app: nginx8050
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx8050
  template:
    metadata:
      labels:
        app: nginx8050
    spec:
      hostNetwork: true
      containers:
        - name: nginx8050
          image: nginx
          ports:
            - containerPort: 8050
              hostPort: 8050
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              name: nginx-volume
              subPath: nginx.conf
            - mountPath: /usr/share/nginx/html/index.html
              name: nginx-volume
              subPath: index.html
      volumes:
        - name: nginx-volume
          configMap:
            name: nginx8050-configmap
            items:
              - key: nginx_conf
                path: nginx.conf
              - key: index_page
                path: index.html
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      tolerations:
        - key: node-role.kubernetes.io/edge
          operator: Exists
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: nginx8050-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx8050
  ports:
    - name: http-0
      port: 18050
      protocol: TCP
      targetPort: 8050

部署在边缘节点A的Nginx（8060端口）：

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx8060-configmap
data:
  nginx_conf: |-
    user  nginx;
    worker_processes  1;
    events {
        worker_connections  1024;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;
        sendfile        on;
        keepalive_timeout  65;
        server {
            listen       8060;
            listen  [::]:8060;
            server_name  localhost;
            location / {
                root   /usr/share/nginx/html;
                index  index.html index.htm;
            }
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   /usr/share/nginx/html;
            }
        }
    }
  index_page: |-
    <html><body><h1>It works! Nginx@8060. </h1></body></html>

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx8060-deployment
  labels:
    app: nginx8060
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx8060
  template:
    metadata:
      labels:
        app: nginx8060
    spec:
      hostNetwork: true
      containers:
        - name: nginx8060
          image: nginx
          ports:
            - containerPort: 8060
              hostPort: 8060
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              name: nginx-volume
              subPath: nginx.conf
            - mountPath: /usr/share/nginx/html/index.html
              name: nginx-volume
              subPath: index.html
      volumes:
        - name: nginx-volume
          configMap:
            name: nginx8060-configmap
            items:
              - key: nginx_conf
                path: nginx.conf
              - key: index_page
                path: index.html
      nodeSelector:
        node-role.kubernetes.io/edge: ""
      tolerations:
        - key: node-role.kubernetes.io/edge
          operator: Exists
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: nginx8060-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx8060
  ports:
    - name: http-0
      port: 18060
      protocol: TCP
      targetPort: 8060

部署在边缘节点B的Nginx（8070端口）：

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx8070-configmap
data:
  nginx_conf: |-
    user  nginx;
    worker_processes  1;
    events {
        worker_connections  1024;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;
        sendfile        on;
        keepalive_timeout  65;
        server {
            listen       8070;
            listen  [::]:8070;
            server_name  localhost;
            location / {
                root   /usr/share/nginx/html;
                index  index.html index.htm;
            }
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   /usr/share/nginx/html;
            }
        }
    }
  index_page: |-
    <html><body><h1>It works! Nginx@8070. </h1></body></html>

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx8070-deployment
  labels:
    app: nginx8070
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx8070
  template:
    metadata:
      labels:
        app: nginx8070
    spec:
      hostNetwork: true
      containers:
        - name: nginx8070
          image: nginx
          ports:
            - containerPort: 8070
              hostPort: 8070
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              name: nginx-volume
              subPath: nginx.conf
            - mountPath: /usr/share/nginx/html/index.html
              name: nginx-volume
              subPath: index.html
      volumes:
        - name: nginx-volume
          configMap:
            name: nginx8070-configmap
            items:
              - key: nginx_conf
                path: nginx.conf
              - key: index_page
                path: index.html
      nodeSelector:
        node-role.kubernetes.io/edge: ""
      tolerations:
        - key: node-role.kubernetes.io/edge
          operator: Exists
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

---

apiVersion: v1
kind: Service
metadata:
  name: nginx8070-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx8070
  ports:
    - name: http-0
      port: 18070
      protocol: TCP
      targetPort: 8070

liamylian

已提交 KubeEdge Issue kubeedge/kubeedge#2845

lynxcat

liamylian 你好，请问一下如何在kubesphere上自定义安装kubeedge呢，自行安装的话还能用kubesphere的操作界面吗