• 安装部署
  • 虚拟机重启后kube-apiserver-master/kube-scheduler-master都起不来

操作系统信息
虚拟机,Centos7.9/,4C/16G

Kubernetes版本信息
v1.28.15

容器运行时

KubeSphere版本信息
v4.1.2。使用kk安装,在线安装。

问题是什么
虚拟机重启后kube-apiserver-master/kube-controller-manager-master/kube-scheduler-master都起不来,报错:Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]

1、describe查询事件都没有

2、logs看kube-apiserver-master日志报错如下:

W0218 06:21:01.156365       1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:01.156447       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0218 06:21:01.157066       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
E0218 06:21:01.455651       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
W0218 06:21:02.112856       1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:02.112956       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0218 06:21:06.565066       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:08.141753       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:17.249336       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
W0218 06:21:18.154250       1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:18.154337       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0218 06:21:18.154346       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
W0218 06:21:18.208498       1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:18.208581       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
I0218 06:21:18.208598       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0218 06:21:32.576820       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0218 06:21:32.761313       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
E0218 06:21:34.417991       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:34.830833       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:43.545862       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:58.207352       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"

logs查看kube-scheduler-master日志报错如下

[root@master ~]# kubectl logs -n kube-system kube-scheduler-master
I0218 09:40:58.789680       1 serving.go:348] Generated self-signed cert in-memory
W0218 09:41:00.291461       1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0218 09:41:00.291522       1 authentication.go:368] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0218 09:41:00.291564       1 authentication.go:369] Continuing without authentication configuration. This may treat all requests as anonymous.
W0218 09:41:00.291569       1 authentication.go:370] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0218 09:41:00.305762       1 server.go:154] "Starting Kubernetes Scheduler" version="v1.28.15"
I0218 09:41:00.305835       1 server.go:156] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0218 09:41:00.318909       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0218 09:41:00.320706       1 secure_serving.go:213] Serving securely on [::]:10259
I0218 09:41:00.320777       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0218 09:41:00.321340       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0218 09:41:00.422208       1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-scheduler...
I0218 09:41:00.422584       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0218 09:41:16.758498       1 leaderelection.go:260] successfully acquired lease kube-system/kube-scheduler
E0218 01:46:03.303045       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:47:03.302323       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:48:03.304486       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:49:03.302893       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:50:03.304108       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:51:03.301668       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:52:03.302627       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:53:03.302932       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:54:03.302129       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:55:03.302563       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:56:03.302652       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"

时间正常,证书正常,这个要怎么处理一下,不能卸载重装