突然无法登录，报拒绝连接

tanjunjie

看api日志
kubectl logs -f ks-apiserver-7bb5c66f9-gq7lh -n kubesphere-system
E1103 10:50:06.658504 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
E1103 10:50:06.660814 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
E1103 10:50:20.899320 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
E1103 10:50:26.456772 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused

root@node1:~# E1103 10:59:23.824278 E1103 10:59:23.825455 E1103 10:59:25.277513 E1103 10:59:25.278494 E1103 10:59:26.282932 E1103 10:59:26.283942 E1103 10:59:27.329460 E1103 10:59:27.330243 E1103 10:59:28.370657 E1103 10:59:28.371528 root@node1:~# I1103 02:22:38.758793 I1103 02:22:38.758910 I1103 02:22:59.115864 I1103 02:22:59.116308 2021/11/03 02:28:03 2021/11/03 02:28:33 ^[[A^[[A^C root@node1:~# 2021-11-03T10:22:46.714+0800 2021-11-03T10:22:46.715+0800 I1103 10:22:46.716076 2021-11-03T10:22:46.716+0800 I1103 10:23:04.037012 2021-11-03T10:23:04.037+0800 2021-11-03T10:23:04.037+0800 2021-11-03T10:23:04.337+0800 2021-11-03T10:23:04.338+0800 2021-11-03T10:23:04.338+0800 2021-11-03T10:23:04.927+0800 2021-11-03T10:23:05.046+0800 kubectl logs –tail=10 notification-manager-deployment-7bd887ffb4-rmmf7
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: dial tcp 10.233.56.198:443: connect: connection refused
kubectl logs -f notification-manager-operator-85797b9f87-4lnfs -c kube-rbac-proxy
1 main.go:190] Valid token audiences:
1 main.go:262] Generating self signed cert as no cert is provided
1 main.go:311] Starting TCP socket on 0.0.0.0:8443
1 main.go:318] Listening securely on 0.0.0.0:8443
http: TLS handshake error from 127.0.0.1:44686: EOF
http: TLS handshake error from 127.0.0.1:47128: local error: tls: bad record MAC
这个TLS错误是我用端口转发测试连通性的，端口转发可以连接上
kubectl logs -f notification-manager-operator-85797b9f87-4lnfs -c notification-manager-operator
INFO controller-runtime.metrics metrics server is starting to listen {“addr”: “127.0.0.1:8080”}
INFO setup starting manager
1 leaderelection.go:242] attempting to acquire leader lease kubesphere-monitoring-system/7b8d27e6.kubesphere.io…
INFO controller-runtime.manager starting metrics server {“path”: “/metrics”}
1 leaderelection.go:252] successfully acquired lease kubesphere-monitoring-system/7b8d27e6.kubesphere.io
DEBUG controller-runtime.manager.events Normal {“object”: {“kind”:“ConfigMap”,“namespace”:“kubesphere-monitoring-system”,“name”:“7b8d27e6.kubesphere.io”,“uid”:“7a9957c6-7a85-4287-9df4-0e5b8c94e155”,“apiVersion”:“v1”,“resourceVersion”:“96504950”}, “reason”: “LeaderElection”, “message”: “notification-manager-operator-85797b9f87-4lnfs_adc6de01-1ebe-4bb0-8a21-d5157ea1a19d became leader”}
INFO controller-runtime.controller Starting EventSource {“controller”: “notificationmanager”, “source”: “kind source: /, Kind=”}
INFO controller-runtime.controller Starting EventSource {“controller”: “notificationmanager”, “source”: “kind source: /, Kind=”}
INFO controller-runtime.controller Starting Controller {“controller”: “notificationmanager”}
INFO controller-runtime.controller Starting workers {“controller”: “notificationmanager”, “worker count”: 1}
DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “notificationmanager”, “request”: “/notification-manager”}
DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “notificationmanager”, “request”: “/notification-manager”}

看不出是哪里的问题，是我的K8S哪里不对了吗？

tanjunjie

无人关注啊。
root@node1:~# kubectl describe svc notification-manager-webhook
Name: notification-manager-webhook
Namespace: kubesphere-monitoring-system
Labels: <none>
Annotations: <none>
Selector: control-plane=controller-manager
Type: ClusterIP
IP Families: <none>
IP: 10.233.2.97
IPs: 10.233.2.97
Port: <unset> 443/TCP
TargetPort: 9443/TCP
Endpoints: 10.233.90.155:9443
Session Affinity: None
Events: <none>

实际上这个10.233.90.155 POD 上根本不存在9443端口，莫名其妙啊，只有8443端口，如果配置成8443，会报证书不匹配。
这到底该如何处理呢？

wanjunlei

你的notification manager operator的版本不对，你把镜像换成kubesphere/notification-manager-operator:v1.4.0试试

tanjunjie

wanjunlei
更换版本后，无法启动，报错如下
root@node1:~# kubectl logs -p notification-manager-operator-5d5fb5fff8-8rmrc -c notification-manager-operator
2021-11-03T17:52:48.618+0800 INFO controller-runtime.metrics metrics server is starting to listen {“addr”: “127.0.0.1:8080”}
2021-11-03T17:52:48.714+0800 INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {“GVK”: “notification.kubesphere.io/v2beta2, Kind=Config”}
2021-11-03T17:52:48.714+0800 INFO controller-runtime.builder Registering a validating webhook {“GVK”: “notification.kubesphere.io/v2beta2, Kind=Config”, “path”: “/validate-notification-kubesphere-io-v2beta2-config”}
2021-11-03T17:52:48.714+0800 INFO controller-runtime.webhook registering webhook {“path”: “/validate-notification-kubesphere-io-v2beta2-config”}
2021-11-03T17:52:48.714+0800 INFO controller-runtime.webhook registering webhook {“path”: “/convert”}
2021-11-03T17:52:48.714+0800 INFO controller-runtime.builder conversion webhook enabled {“object”: {“metadata”:{“creationTimestamp”:null},“spec”:{},“status”:{}}}
2021-11-03T17:52:48.715+0800 INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {“GVK”: “notification.kubesphere.io/v2beta2, Kind=Receiver”}
2021-11-03T17:52:48.715+0800 INFO controller-runtime.builder Registering a validating webhook {“GVK”: “notification.kubesphere.io/v2beta2, Kind=Receiver”, “path”: “/validate-notification-kubesphere-io-v2beta2-receiver”}
2021-11-03T17:52:48.715+0800 INFO controller-runtime.webhook registering webhook {“path”: “/validate-notification-kubesphere-io-v2beta2-receiver”}
2021-11-03T17:52:48.715+0800 INFO controller-runtime.builder conversion webhook enabled {“object”: {“metadata”:{“creationTimestamp”:null},“spec”:{},“status”:{}}}
2021-11-03T17:52:48.716+0800 INFO setup starting manager
I1103 17:52:48.716424 1 leaderelection.go:242] attempting to acquire leader lease kubesphere-monitoring-system/7b8d27e6.kubesphere.io…
2021-11-03T17:52:48.716+0800 INFO controller-runtime.manager starting metrics server {“path”: “/metrics”}
2021-11-03T17:52:50.716+0800 INFO controller-runtime.webhook.webhooks starting webhook server
2021-11-03T17:52:50.717+0800 DEBUG controller-runtime.manager non-leader-election runnable finished {“runnable type”: “webhook.Server”}
2021-11-03T17:52:50.717+0800 ERROR setup problem running manager {“error”: “open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory”}
github.com/go-logr/zapr.(zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
main.main
/workspace/main.go:101
runtime.main
/usr/local/go/src/runtime/proc.go:203

添加秘钥后，可以了，多谢大佬指点！！！
就是有些疑惑，为啥莫名其妙的就这样子了呢，是KUBESPHERE会自己悄悄升级吗，还是会自己变动的？从而导致哪里对不上号了？我没有动过里面的东西啊。

cqwang9

wanjunlei

大佬帮我也看看把，我也是这个问题，不过我按照的是最新的版本kubesphere3.4.1的，突然就出现了登陆不上，我看是因为ks-apiserver启动不了，调用notification-manager-webhook（kubesphere/notification-manager-operator:v2.3.0）总是报证书问题。

wanjunlei

kubectl apply -f https://raw.githubusercontent.com/kubesphere/notification-manager/release-1.4/config/bundle.yaml
把notification manager整个升级下吧，应该是你之前手动升级notification manage，但是只升级了部分造成的。

rufei

wanjunlei 我的就是升级这个出的问题，感谢。