ingress-nginx-controller將ks-console用https訪問，502bad gateway

Ccici · 10 3月

me:~/kubesphere$ kubectl describe pods -n ingress-nginx ingress-nginx-controller-ccbf44649-8c6bt
Name:             ingress-nginx-controller-ccbf44649-8c6bt
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             coverity-ms/172.1.30.21
Start Time:       Mon, 10 Mar 2025 15:29:45 +0800
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.12.0
                  pod-template-hash=ccbf44649
Annotations:      cni.projectcalico.org/containerID: b30e7d604943de7a0029f8fc5872682af3d35f8212b9e24527e05ab868700557
                  cni.projectcalico.org/podIP: 10.233.127.66/32
                  cni.projectcalico.org/podIPs: 10.233.127.66/32
                  kubectl.kubernetes.io/restartedAt: 2025-03-10T15:29:45+08:00
Status:           Running
IP:               10.233.127.66
IPs:
  IP:           10.233.127.66
Controlled By:  ReplicaSet/ingress-nginx-controller-ccbf44649
Containers:
  controller:
    Container ID:    containerd://f81ad29dd86a4935b60684ae1a4950f2730aca669499af4a9078dc68a212e2d4
    Image:           ficms-ms-cns.myfiinet.com/kubesphere/ingress-nginx/controller:v1.12.0
    Image ID:        ficms-ms-cns.myfiinet.com/kubesphere/ingress-nginx/controller@sha256:5c06930ada97e5df24a3ec60c55723acf78d3177086922ba6352b4f863a39649
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Mon, 10 Mar 2025 15:29:45 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-ccbf44649-8c6bt (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pxsdh (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-pxsdh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age                From                      Message
  ----    ------     ----               ----                      -------
  Normal  Scheduled  51m                default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-ccbf44649-8c6bt to coverity-ms
  Normal  Pulled     51m                kubelet                   Container image "ficms-ms-cns.myfiinet.com/kubesphere/ingress-nginx/controller:v1.12.0" already present on machine
  Normal  Created    51m                kubelet                   Created container controller
  Normal  Started    51m                kubelet                   Started container controller
  Normal  RELOAD     14m (x3 over 51m)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

me:~/kubesphere$ kubectl logs -n ingress-nginx ingress-nginx-controller-ccbf44649-8c6bt
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.0
  Build:         ba73b2c24d355f1cdcf4b31ef7c5574059f12118
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

W0310 07:29:45.822476       7 client_config.go:667] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0310 07:29:45.822606       7 main.go:205] "Creating API client" host="https://10.233.0.1:443"
I0310 07:29:45.846641       7 main.go:248] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.0" state="clean" commit="855e7c48de7388eb330da0f8d9d2394ee818fb8d" platform="linux/amd64"
I0310 07:29:45.971372       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0310 07:29:45.977163       7 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0310 07:29:45.984138       7 nginx.go:271] "Starting NGINX Ingress controller"
I0310 07:29:45.986924       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"c24c810f-f4e9-406d-bcfc-1feb4c67cada", APIVersion:"v1", ResourceVersion:"766415", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0310 07:29:47.085897       7 store.go:440] "Found valid IngressClass" ingress="kubesphere-system/ks-console" ingressclass="nginx"
I0310 07:29:47.086267       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubesphere-system", Name:"ks-console", UID:"3ae8b8ab-b21d-42c8-b1af-0baf06fa1272", APIVersion:"networking.k8s.io/v1", ResourceVersion:"755951", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0310 07:29:47.086708       7 backend_ssl.go:67] "Adding secret to local store" name="kubesphere-system/kubesphere-tls-certs"
I0310 07:29:47.185438       7 leaderelection.go:257] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0310 07:29:47.185516       7 nginx.go:317] "Starting NGINX process"
I0310 07:29:47.185768       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0310 07:29:47.185944       7 controller.go:196] "Configuration changes detected, backend reload required"
I0310 07:29:47.186966       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-d5c666948-vzxqz"
I0310 07:29:47.217930       7 controller.go:216] "Backend successfully reloaded"
I0310 07:29:47.217979       7 controller.go:227] "Initial sync, sleeping for 1 second"
I0310 07:29:47.218019       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ccbf44649-8c6bt", UID:"a789a08f-7889-42fa-b1f2-bed4470590a6", APIVersion:"v1", ResourceVersion:"766463", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0310 07:30:36.089339       7 leaderelection.go:271] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0310 07:30:36.089409       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-ccbf44649-8c6bt"
172.1.30.21 - - [10/Mar/2025:07:31:46 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - 5178bf7c7302551fcb2c47d63d64904a
172.1.30.21 - - [10/Mar/2025:07:32:48 +0000] "\x16\x03\x01\x01\x97\x01\x00\x01\x93\x03\x03o;\xFF+\x069\xD3\x1E\x16\xB8\x0B\xB4\xE7\xCBH\x9FZ\x09\xF1\xCF\x5C\xD7\xB2\xA5ay\xCA\x98P\xBB\xB6# i'\xD4E\xF2h\x16\xB6\xA6w\xD4|\x07Y\xEE\xBB\x92\xBF_\xE3j\x8A5a\xBBO\x87i\xFE:\x1B^\x00\x96\x13\x02\x13\x03\x13\x01\xC0,\xC00\x00\xA3\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0\xAF\xC0\xAD\xC0\xA3\xC0\x9F\xC0]\xC0a\xC0W\xC0S\xC0+\xC0/\x00\xA2\x00\x9E\xC0\xAE\xC0\xAC\xC0\xA2\xC0\x9E\xC0\x5C\xC0`\xC0V\xC0R\xC0$\xC0(\x00k\x00j\xC0s\xC0w\x00\xC4\x00\xC3\xC0#\xC0'\x00g\x00@\xC0r\xC0v\x00\xBE\x00\xBD\xC0" 400 150 "-" "-" 0 0.000 [] [] - - - - 991e77adab91842af8f853fb45ece64e
172.1.30.21 - - [10/Mar/2025:07:33:01 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - 49c8db1c0050f27bb7760ca30c5b1441
172.1.30.21 - - [10/Mar/2025:07:33:07 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - 69b6f9c8a514f577bfe7b0f7936a8e5a
2025/03/10 07:33:12 [error] 37#37: *3666 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:33:31 +0000] "GET / HTTP/1.1" 499 0 "-" "Wget/1.21.2" 139 4.314 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000 0 4.314 - e4f385a3f21eb0196dde8483f609330d
172.1.30.21 - - [10/Mar/2025:07:34:26 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 88 0.000 [kubesphere-system-ks-console-80] [] - - - - a7ec39edebe0cdff97b36da641d4b53b
2025/03/10 07:35:23 [error] 37#37: *3666 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:35:44 [error] 41#41: *10957 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:36:34 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 88 0.000 [kubesphere-system-ks-console-80] [] - - - - 9baf3537bd1a6236cfe8c8a787721b17
2025/03/10 07:37:34 [error] 37#37: *3666 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:37:34 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/7.81.0" 37 391.353 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.210, 131.072, 131.072 502, 502, 502 82e3a157a7680c3a2f6f2fa5d2372a24
2025/03/10 07:37:55 [error] 41#41: *10957 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:40:06 [error] 41#41: *10957 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:40:06 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 139 391.952 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.805, 131.072, 131.076 502, 502, 502 4a5dc0304169f42705c5e43d85223aef
I0310 07:45:15.378005       7 main.go:107] "successfully validated configuration, accepting" ingress="kubesphere-system/ks-console"
I0310 07:45:15.382808       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubesphere-system", Name:"ks-console", UID:"3ae8b8ab-b21d-42c8-b1af-0baf06fa1272", APIVersion:"networking.k8s.io/v1", ResourceVersion:"768597", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
172.1.30.21 - - [10/Mar/2025:07:45:43 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 88 0.000 [kubesphere-system-ks-console-80] [] - - - - 73ba6667d5a7b3d76c89192678264e92
2025/03/10 07:47:30 [error] 34#34: *45028 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:47:40 [error] 38#38: *45480 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:48:03 [error] 39#39: *46592 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:49:41 [error] 34#34: *45028 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:49:51 [error] 38#38: *45480 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:50:14 [error] 39#39: *46592 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
I0310 07:50:52.004352       7 main.go:107] "successfully validated configuration, accepting" ingress="kubesphere-system/ks-console"
I0310 07:50:52.006207       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubesphere-system", Name:"ks-console", UID:"3ae8b8ab-b21d-42c8-b1af-0baf06fa1272", APIVersion:"networking.k8s.io/v1", ResourceVersion:"769363", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
172.1.30.21 - - [10/Mar/2025:07:50:55 +0000] "GET / HTTP/2.0" 499 0 "-" "curl/7.81.0" 37 333.930 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.464, 131.072, 73.393 502, 502, - 0b115c4887424da7bf54d2bc4d5500b3
172.1.30.21 - - [10/Mar/2025:07:50:58 +0000] "GET / HTTP/2.0" 499 0 "-" "curl/7.81.0" 37 304.972 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 130.126, 131.072, 43.773 502, 502, - cc6802f092acfd2b623a2c785af3bbf7
172.1.30.21 - - [10/Mar/2025:07:51:01 +0000] "GET / HTTP/2.0" 499 0 "-" "curl/7.81.0" 37 1.005 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000 0 1.005 - c7b0a18afef2270e12f0e196e4c7173c
172.1.30.21 - - [10/Mar/2025:07:51:06 +0000] "GET / HTTP/1.1" 499 0 "-" "Wget/1.21.2" 139 336.025 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 130.404, 131.072, 74.549 502, 502, - 8b2807599168418b2f5c27a61a8e0119
172.1.30.21 - - [10/Mar/2025:07:51:41 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - 4daf04a0f1b6ac57091e68d4de2637e9
2025/03/10 07:53:06 [error] 38#38: *61207 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:53:18 [error] 43#43: *61791 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:53:43 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - 66e96d9e2d4d9d258be7563fc96ed395
172.1.30.21 - - [10/Mar/2025:07:53:50 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - c6a6ed0eaa366e83162dc989c815446b
172.1.30.21 - - [10/Mar/2025:07:53:53 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - b571253d19a3c8a5935b3388705eeb0c
2025/03/10 07:55:17 [error] 38#38: *61207 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:55:29 [error] 43#43: *61791 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:57:28 [error] 38#38: *61207 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:57:28 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/7.81.0" 37 391.958 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.815, 131.072, 131.072 502, 502, 502 6b8a6373d8c9546f5cb575d8fda88341
2025/03/10 07:57:40 [error] 43#43: *61791 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:07:57:40 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 139 391.637 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.494, 131.072, 131.072 502, 502, 502 9531e7780fc8b7f1cbfca1ff516779ff
I0310 07:58:51.003448       7 main.go:107] "successfully validated configuration, accepting" ingress="kubesphere-system/ks-console"
I0310 07:58:51.005389       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubesphere-system", Name:"ks-console", UID:"3ae8b8ab-b21d-42c8-b1af-0baf06fa1272", APIVersion:"networking.k8s.io/v1", ResourceVersion:"770449", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0310 07:58:51.006080       7 controller.go:196] "Configuration changes detected, backend reload required"
I0310 07:58:51.037637       7 controller.go:216] "Backend successfully reloaded"
I0310 07:58:51.037781       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ccbf44649-8c6bt", UID:"a789a08f-7889-42fa-b1f2-bed4470590a6", APIVersion:"v1", ResourceVersion:"766463", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2025/03/10 08:01:05 [error] 585#585: *84409 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31661"
2025/03/10 08:01:11 [error] 585#585: *84651 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:01:15 [error] 583#583: *84846 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:03:16 [error] 585#585: *84409 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31661"
2025/03/10 08:03:22 [error] 585#585: *84651 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:03:26 [error] 583#583: *84846 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:05:27 [error] 585#585: *84409 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31661"
172.1.30.21 - - [10/Mar/2025:08:05:27 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 139 391.747 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.603, 131.072, 131.072 502, 502, 502 d6257e8aa4208e7205a9619cff9786ee
2025/03/10 08:05:33 [error] 585#585: *84651 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:08:05:33 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 139 392.293 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 130.154, 131.068, 131.072 502, 502, 502 3cb2e1c3892b777d596a74a39e1020e6
2025/03/10 08:05:38 [error] 583#583: *84846 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:08:05:38 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/7.81.0" 37 392.680 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 130.537, 131.072, 131.072 502, 502, 502 c428c14ff66d996f41c7664890bc665a
I0310 08:06:22.329323       7 main.go:107] "successfully validated configuration, accepting" ingress="kubesphere-system/ks-console"
I0310 08:06:22.331735       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubesphere-system", Name:"ks-console", UID:"3ae8b8ab-b21d-42c8-b1af-0baf06fa1272", APIVersion:"networking.k8s.io/v1", ResourceVersion:"771475", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0310 08:06:22.332387       7 controller.go:196] "Configuration changes detected, backend reload required"
I0310 08:06:22.363885       7 controller.go:216] "Backend successfully reloaded"
I0310 08:06:22.364031       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ccbf44649-8c6bt", UID:"a789a08f-7889-42fa-b1f2-bed4470590a6", APIVersion:"v1", ResourceVersion:"766463", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
172.1.30.21 - - [10/Mar/2025:08:06:44 +0000] "GET / HTTP/1.1" 308 164 "-" "Wget/1.21.2" 139 0.000 [kubesphere-system-ks-console-80] [] - - - - c981a4fb80ae9049e71e30a1b22b3887
2025/03/10 08:08:42 [error] 1199#1199: *106402 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:08:46 [error] 1114#1114: *106695 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:08:09:57 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 88 0.000 [kubesphere-system-ks-console-80] [] - - - - 18b37ae2576bd508c22d00637558f32c
2025/03/10 08:10:53 [error] 1199#1199: *106402 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:10:57 [error] 1114#1114: *106695 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 08:13:04 [error] 1199#1199: *106402 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:08:13:04 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 139 393.240 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 131.097, 131.072, 131.072 502, 502, 502 62fbe5ab01c2a358b458eab262d18e87
2025/03/10 08:13:08 [error] 1114#1114: *106695 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
172.1.30.21 - - [10/Mar/2025:08:13:08 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/7.81.0" 37 391.725 [kubesphere-system-ks-console-80] [] 10.233.127.31:8000, 10.233.127.31:8000, 10.233.127.31:8000 0, 0, 0 129.585, 131.068, 131.072 502, 502, 502 7a93ec926a1c49bbb552bf758f96c0f9
172.1.30.21 - - [10/Mar/2025:08:18:14 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 88 0.000 [kubesphere-system-ks-console-80] [] - - - - 0cccd4576f71a61b270ad2dafee119f6

klj890 · 10 3月

2025/03/10 07:47:30 [error] 34#34: *45028 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:47:40 [error] 38#38: *45480 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:48:03 [error] 39#39: *46592 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:49:41 [error] 34#34: *45028 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:49:51 [error] 38#38: *45480 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/1.1", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"
2025/03/10 07:50:14 [error] 39#39: *46592 connect() failed (110: Operation timed out) while connecting to upstream, client: 172.1.30.21, server: 172.1.30.21.nip.io, request: "GET / HTTP/2.0", upstream: "http://10.233.127.31:8000/", host: "172.1.30.21.nip.io:31199"

这里说了，你的upstream连接 timeout了

Ccici · 11 3月

klj890
我已經在ingress的annotation加入這些配置仍然是一樣的錯誤

    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"

klj890 · 11 3月

klj890 · 11 3月

为什么是8000端口？默认不是80端口吗？

Ccici · 11 3月

klj890 我其實也不太知道這個端口是怎麼跑出來的，不過我ingress內配置確實是用80

Ccici · 11 3月

但我現在是在內網環境，外部確實有被防火牆擋

Ccici · 11 3月

me:~/kubesphere$ curl -v http://10.233.127.31:8000
*   Trying 10.233.127.31:8000...
* Connected to 10.233.127.31 (10.233.127.31) port 8000 (#0)
> GET / HTTP/1.1
> Host: 10.233.127.31:8000
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Vary: Accept-Encoding
< Location: /login
< Content-Type: text/html; charset=utf-8
< Content-Length: 43
< Date: Tue, 11 Mar 2025 03:26:16 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host 10.233.127.31 left intact

klj890 · 11 3月

cici
进入ingress pod里面，看下nginx的配置。找到代理ks-console的域名，找个pod，在里面curl/nslookup这个域名看看后端IP是不是ks-console的ip

klj890 · 11 3月

cici 这是在哪里curl的？

Ccici · 11 3月

klj890 抱歉前面沒修改到回復我後來檢查到這個10.233.127.31是ks-console pod IP

klj890 · 11 3月

cici 但是端口看着不对啊。。。

Ccici · 11 3月

klj890 Ingress-controller pod 裡面curl 10.233.127.31:8000

h00283@coverity-ms:~/kubesphere$ kubectl exec -it ingress-nginx-controller-79746f9d79-2j5b8 -n ingress-nginx -- curl -v http://10.233.127.31:8000
*   Trying 10.233.127.31:8000...
* connect to 10.233.127.31 port 8000 from 10.233.127.73 port 58146 failed: Operation timed out
* Failed to connect to 10.233.127.31 port 8000 after 130821 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to 10.233.127.31 port 8000 after 130821 ms: Could not connect to server
command terminated with exit code 28


#改用80試看看
h00283@coverity-ms:~/kubesphere$ kubectl exec -it ingress-nginx-controller-79746f9d79-2j5b8 -n ingress-nginx -- curl -v http://10.233.127.31:80
*   Trying 10.233.127.31:80...
* connect to 10.233.127.31 port 80 from 10.233.127.73 port 50282 failed: Operation timed out
* Failed to connect to 10.233.127.31 port 80 after 130403 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to 10.233.127.31 port 80 after 130403 ms: Could not connect to server
command terminated with exit code 28

Ccici · 11 3月

我用另一台一樣配置安裝的機器試了一下，確定是ks-console pod IP:8000 port ，curl 出來應該會是

me2:~/kubesphere$ kubectl exec -it ingress-nginx-controller-d6f759d4d-8fflw -n ingress-nginx -- curl -v http://10.233.120.30:8000
*   Trying 10.233.120.30:8000...
* Connected to 10.233.120.30 (10.233.120.30) port 8000
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 10.233.120.30:8000
> User-Agent: curl/8.11.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 302 Found
< Vary: Accept-Encoding
< Location: /login
< Content-Type: text/html; charset=utf-8
< Content-Length: 43
< Date: Tue, 11 Mar 2025 06:08:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host 10.233.120.30 left intact
Redirecting to <a href="/login">/login</a>.

不過就不知道為什麼這台不行

klj890 · 11 3月

是我错了。pod监听的就是8000端口，所以路由是对的

那你还是检查ks-console pod本身的健康吧

klj890 · 11 3月

如果pod本身没有问题，检查你的网络。

klj890 · 11 3月

可以问下AI

Ccici · 11 3月

klj890 看起來確實是兩個pod互聯都time out，有可能因為我外部防火牆影響嗎

klj890 · 11 3月

cici
怀疑它，证明它

Ccici · 11 3月

klj890 我進ks-console跟ingress的pod分別做了都time out

me:~/kubesphere$ kubectl exec -it ks-console-5675bf657b-kzplx -n kubesphere-system -- nc -zv 10.233.127.77 80
nc: 10.233.127.77 (10.233.127.77:80): Operation timed out
command terminated with exit code 1

me:~/kubesphere$ kubectl exec -it ingress-nginx-controller-d5c666948-8fg6h -n ingress-nginx -- nc -zv 10.233.127.31 8000
nc: 10.233.127.31 (10.233.127.31:8000): Operation timed out
command terminated with exit code 1

ingress-nginx-controller將ks-console用https訪問，502bad gateway

Ccici

klj890K零S

Ccici

klj890K零S

klj890K零S

Ccici

Ccici

Ccici

klj890K零S

klj890K零S

Ccici

klj890K零S

Ccici

Ccici

klj890K零S

klj890K零S

klj890K零S

Ccici

klj890K零S

Ccici