kubesphere3.3.0网关如何保留客户端ip?

yuwen

操作系统信息
CentOS Linux release 7.5.1804

k8s版本

[root@k8s-node-01-22 haproxy]# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

容器运行时

[root@k8s-node-01-22 haproxy]# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

KubeSphere版本信息

v3.3.0,在线安装

问题是什么
在kubesphere3.3.0中需要保留客户端ip，按照官方操作，获取的ip为127.0.0.6,不通过ingress可以正确的获取客户端ip，具体操作如下

增加在网关参数转发请求头等配置

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  compute-full-forwarded-for: "true"
  enable-real-ip: "true"
  enable-underscores-in-headers: "true"
  forwarded-for-header: X_FORWARDED_FOR
  log-format-upstream: '{"time": "$time_iso8601","status":$status,"path": "$uri",
    "proxy_protocol_addr": "$proxy_protocol_addr","proxy_add_x_forwarded_for": "$proxy_add_x_forwarded_for",
    "request_id": "$req_id",   "remote_user":"$remote_user", "bytes_sent": $bytes_sent,
    "request_time": $request_time,, "vhost": "$host", "request_proto": "$server_protocol","request_query":
    "$args", "request_length": $request_length, "duration": $request_time,"method":
    "$request_method", "http_referrer": "$http_referer",   "http_user_agent":"$http_user_agent",
    "remote_addr":"$remote_addr", "remote_port": "$remote_port","x_forwarded_for":
    "$http_x_forwarded_for"}'
  proxy-real-ip-cidr: 127.0.0.6
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
    meta.helm.sh/release-namespace: kubesphere-controls-system
  creationTimestamp: "2022-07-01T07:48:49Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: kubesphere-router-kubesphere-system-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    helm.sh/chart: ingress-nginx-4.0.13
  name: kubesphere-router-kubesphere-system
  namespace: kubesphere-controls-system
  ownerReferences:
  - apiVersion: gateway.kubesphere.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Nginx
    name: kubesphere-router-kubesphere-system-ingress
    uid: 861e7337-7cda-42c6-a5e1-b2965b727446
  resourceVersion: "11113940"
  uid: 22df94a2-d3ca-4b81-981b-545c7575b668

更改ingress的svc的externalTrafficPolicy为Local

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-07-01T07:48:49Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: kubesphere-router-kubesphere-system-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    helm.sh/chart: ingress-nginx-4.0.13
  name: kubesphere-router-kubesphere-system
  namespace: kubesphere-controls-system
  ownerReferences:
  - apiVersion: gateway.kubesphere.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Nginx
    name: kubesphere-router-kubesphere-system-ingress
    uid: 861e7337-7cda-42c6-a5e1-b2965b727446
  resourceVersion: "11163184"
  uid: 7ee3e1cc-edf8-4de0-af58-a23a7caf8664
spec:
  clusterIP: 10.233.60.74
  clusterIPs:
  - 10.233.60.74
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 31990
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 31670
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: kubesphere-router-kubesphere-system-ingress
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

使用whoami工具测试

Hostname: whoami-bdd68f9-4p8rd
IP: 127.0.0.1
IP: 10.233.86.204
RemoteAddr: 10.233.87.45:59558
GET / HTTP/1.1
Host: xxx.xxx.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,zh-TW;q=0.8,zh-HK;q=0.7
Cache-Control: max-age=0
Cookie: SECKEY_ABVK=EPfiE4PGZQJOgKbAlQUIEpClZypn4ndE7F/sErGi++c%3D; BMAP_SECKEY=ASHIliu6bLjnpHB4CG0V40rk96cq3SvmO5JiKcGkrk8dL7jX4BRWB_TPN7397FSdEwFpJVkRQXCDXqYUiLS8i_ZNHmJbhnXvF3Fa7zAC9ihe5W54meLcmTCBkw6_vCseOwPha_k-Mwv719GRivyNU9aIcFx_6JJv2tmUHS8V_7g-eXx65DE2h1nSdBXLv8iY
Sec-Ch-Ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "macOS"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-B3-Sampled: 0
X-B3-Spanid: 5daff7c301f0366b
X-B3-Traceid: d45c419924e3fc745daff7c301f0366b
X-Envoy-Attempt-Count: 1
X-Envoy-Peer-Metadata: Ch4KDkFQUF9DT05UQUlORVJTEgwaCmNvbnRyb2xsZXIKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xMS4xCp0DCgZMQUJFTFMSkgMqjwMKKwobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgwaCmNvbnRyb2xsZXIKSwoaYXBwLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2USLRora3ViZXNwaGVyZS1yb3V0ZXIta3ViZXNwaGVyZS1zeXN0ZW0taW5ncmVzcwopChZhcHAua3ViZXJuZXRlcy5pby9uYW1lEg8aDWluZ3Jlc3MtbmdpbngKOgohaXBwb29sLm5ldHdvcmsua3ViZXNwaGVyZS5pby9uYW1lEhUaE2RlZmF1bHQtaXB2NC1pcHBvb2wKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU5OWNkZmJkNwokChlzZWN1cml0eS5pc3Rpby5pby90bHNNb2RlEgcaBWlzdGlvCjIKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSDxoNaW5ncmVzcy1uZ2lueAovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCj4KBE5BTUUSNho0a3ViZXNwaGVyZS1yb3V0ZXIta3ViZXNwaGVyZS1zeXN0ZW0tNjU5OWNkZmJkNy1yd3BibAopCglOQU1FU1BBQ0USHBoaa3ViZXNwaGVyZS1jb250cm9scy1zeXN0ZW0KegoFT1dORVIScRpva3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2t1YmVzcGhlcmUtY29udHJvbHMtc3lzdGVtL2RlcGxveW1lbnRzL2t1YmVzcGhlcmUtcm91dGVyLWt1YmVzcGhlcmUtc3lzdGVtChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAo2Cg1XT1JLTE9BRF9OQU1FEiUaI2t1YmVzcGhlcmUtcm91dGVyLWt1YmVzcGhlcmUtc3lzdGVt
X-Envoy-Peer-Metadata-Id: sidecar~10.233.87.45~kubesphere-router-kubesphere-system-6599cdfbd7-rwpbl.kubesphere-controls-system~kubesphere-controls-system.svc.cluster.local
X-Forwarded-For: 127.0.0.6
X-Forwarded-Host: xxx.xxx.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Real-Ip: 127.0.0.6
X-Request-Id: c566e61d0841620561e713ce8c249c39
X-Scheme: https

测试结果

不能正确的获取客户端ip地址，客户端ip获取为127.0.0.6

ingress网关日志

"time": "2022-07-19T12:30:11+00:00","status":200,"path": "/xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "c707dd649f0d4a7f9bd6c9ec795277cb",   "remote_user":"-", "bytes_sent": 2672, "request_time": 0.001,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1420, "duration": 0.001,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "43529","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "/xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "b4ccf187a9a16e276e40c415012f6d1d",   "remote_user":"-", "bytes_sent": 2562, "request_time": 0.002,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1418, "duration": 0.002,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "56861","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "/xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "c569e34ece0312e9eb39efbe8ceeee2f",   "remote_user":"-", "bytes_sent": 3403, "request_time": 0.002,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1418, "duration": 0.002,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "42595","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "/xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "1fa43dc1dbd189b8839aee9fa8904f29",   "remote_user":"-", "bytes_sent": 1993, "request_time": 0.003,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1421, "duration": 0.003,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "40931","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "223.147.79.48, 127.0.0.6", "request_id": "59c30c043e5114b431aefbf1a1aff48e",   "remote_user":"-", "bytes_sent": 286260, "request_time": 0.019,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 936, "duration": 0.019,"method": "GET", "http_referrer": "https://www.baidu.com/",   "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.25(0x18001927) NetType/4G Language/zh_CN", "remote_addr":"127.0.0.6", "remote_port": "53829","x_forwarded_for": "223.147.79.48"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "951a70855951a6d4a7a2e1179c556f00",   "remote_user":"-", "bytes_sent": 3096, "request_time": 0.002,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1421, "duration": 0.002,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "56861","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "8bd21d81f4b3789a5b821d628e23dc7c",   "remote_user":"-", "bytes_sent": 2113, "request_time": 0.004,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1419, "duration": 0.004,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "43529","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "d5c0d6206add76e33e374cf5ca23fd83",   "remote_user":"-", "bytes_sent": 1854, "request_time": 0.006,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1424, "duration": 0.006,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "42595","x_forwarded_for": "111.22.182.104"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "223.147.79.48, 127.0.0.6", "request_id": "4574852dab2d51531a540656b00679a9",   "remote_user":"-", "bytes_sent": 210284, "request_time": 0.013,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 936, "duration": 0.013,"method": "GET", "http_referrer": "https://www.baidu.com/",   "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.25(0x18001927) NetType/4G Language/zh_CN", "remote_addr":"127.0.0.6", "remote_port": "48413","x_forwarded_for": "223.147.79.48"}

 {"time": "2022-07-19T12:30:11+00:00","status":200,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "223.147.79.48, 127.0.0.6", "request_id": "8addd54b8a198e481174a2c8873e2319",   "remote_user":"-", "bytes_sent": 28575, "request_time": 0.002,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 996, "duration": 0.002,"method": "GET", "http_referrer": "https://www.baidu.com/",   "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.25(0x18001927) NetType/4G Language/zh_CN", "remote_addr":"127.0.0.6", "remote_port": "49157","x_forwarded_for": "223.147.79.48"}

 {"time": "2022-07-19T12:30:11+00:00","status":404,"path": "xxx", "proxy_protocol_addr": "-","proxy_add_x_forwarded_for": "111.22.182.104, 127.0.0.6", "request_id": "e6cf5acbc3e598fc5e9d4f2d7bfbc1e7",   "remote_user":"-", "bytes_sent": 752, "request_time": 0.000,, "vhost": "m.cs96111.com", "request_proto": "HTTP/1.1","request_query": "-", "request_length": 1401, "duration": 0.000,"method": "GET", "http_referrer": "https://www.baidu.com/net/",   "http_user_agent":"Mozilla/5.0 (Linux; Android 10; STK-AL00 Build/HUAWEISTK-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/86.0.4240.99 XWEB/4267 MMWEBSDK/20220604 Mobile Safari/537.36 MMWEBID/556 MicroMessenger/8.0.24.2180(0x2800183F) WeChat/arm64 Weixin NetType/WIFI Language/zh_CN ABI/arm64", "remote_addr":"127.0.0.6", "remote_port": "42595","x_forwarded_for": "111.22.182.104"}

通过查看日志在请求头中的proxy_add_x_forwarded_for存在客户端的ip，但是不能正确的被转发到后台服务，后台服务不能获取到，请各位大佬只招，不知道是哪里做错了，请赐教 :

yuwen

@hongzhouzi 大佬能抽空给看一下嘛？

JoeWhatUWant

yuwen 请问解决了吗？我获取到的也是127.0.0.6

yuwen

网络链路

阿里云负载均衡 -> ingress网关 -> whomi中的svc

chengjun

JoeWhatUWant 127.0.0.6 的问题解决了吗？

AntSworD

一样这个问题用户-> ngress网关 -> whomi中的svc

JoeWhatUWant

chengjun 没有。我是在前面加一层nginx。使用七层协议转发，才可以。参考文档里面提供的方式都不太适合