- 已编辑
操作系统信息
虚拟机，Centos7.9，8C/16G
Kubernetes版本信息
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}容器运行时
[root@master1 ~]# docker version
Client:
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:50:40 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:55:09 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b638
docker-init:
Version: 0.19.0
GitCommit: de40ad0KubeSphere版本信息
v3.3.1 在线安装，使用kk安装。
问题是什么
kubernetes集群在一次证书将要过期的节点，使用kubeadm传统方式更新了证书，但是告警还是一直频繁出现7天以及24小时过期，持续了个把月了
想不出解决方案了，发现api-server中有kubersphere相关的日志。，检查所有证书都是更新过后的状态，并且集群使用也正常
以下是api-server内的报错日志。
E0723 02:11:46.057215 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:47.060183 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:48.063586 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
I0723 02:11:48.696878 1 client.go:360] parsed scheme: "passthrough"
I0723 02:11:48.696920 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.1.3.75:2379 <nil> 0 <nil>}] <nil> <nil>}
I0723 02:11:48.696930 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
E0723 02:11:49.066390 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:50.070041 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:51.072887 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:52.075356 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:52.639454 1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:52Z is after 2024-05-14T02:28:45Z, verifying certificate SN=2131681115118581220, SKID=, AKID=5D:CE:2C:E4:49:73:25:CA:C4:84:8C:A0:BB:7E:8F:15:36:33:BD:85 failed: x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:52Z is after 2024-05-14T02:28:45Z]"
E0723 02:11:53.078359 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:54.073947 1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:54Z is after 2024-05-14T02:28:45Z, verifying certificate SN=2131681115118581220, SKID=, AKID=5D:CE:2C:E4:49:73:25:CA:C4:84:8C:A0:BB:7E:8F:15:36:33:BD:85 failed: x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:54Z is after 2024-05-14T02:28:45Z]"
E0723 02:11:54.081472 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:55.024945 1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:55Z is after 2024-05-14T02:28:45Z, verifying certificate SN=2131681115118581220, SKID=, AKID=5D:CE:2C:E4:49:73:25:CA:C4:84:8C:A0:BB:7E:8F:15:36:33:BD:85 failed: x509: certificate has expired or is not yet valid: current time 2024-07-23T02:11:55Z is after 2024-05-14T02:28:45Z]"
E0723 02:11:55.083596 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...
E0723 02:11:56.086879 1 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list notification.kubesphere.io/v2beta1, Kind=Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": service "notification-manager-webhook" not found; reinitializing...希望能得到官方人员的帮助，谢谢。
