操作系统信息
虚拟机,Centos7.9/,4C/16G
Kubernetes版本信息
v1.28.15
容器运行时
KubeSphere版本信息
v4.1.2。使用kk安装,在线安装。
问题是什么
虚拟机重启后kube-apiserver-master/kube-controller-manager-master/kube-scheduler-master都起不来,报错:Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]
1、describe查询事件都没有
2、logs看kube-apiserver-master日志报错如下:
W0218 06:21:01.156365 1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:01.156447 1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0218 06:21:01.157066 1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
E0218 06:21:01.455651 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
W0218 06:21:02.112856 1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:02.112956 1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0218 06:21:06.565066 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:08.141753 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:17.249336 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
W0218 06:21:18.154250 1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:18.154337 1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0218 06:21:18.154346 1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
W0218 06:21:18.208498 1 handler_proxy.go:93] no RequestInfo found in the context
E0218 06:21:18.208581 1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
I0218 06:21:18.208598 1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0218 06:21:32.576820 1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0218 06:21:32.761313 1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
E0218 06:21:34.417991 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:34.830833 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:43.545862 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
E0218 06:21:58.207352 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
logs查看kube-scheduler-master日志报错如下
[root@master ~]# kubectl logs -n kube-system kube-scheduler-master
I0218 09:40:58.789680 1 serving.go:348] Generated self-signed cert in-memory
W0218 09:41:00.291461 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0218 09:41:00.291522 1 authentication.go:368] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0218 09:41:00.291564 1 authentication.go:369] Continuing without authentication configuration. This may treat all requests as anonymous.
W0218 09:41:00.291569 1 authentication.go:370] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0218 09:41:00.305762 1 server.go:154] "Starting Kubernetes Scheduler" version="v1.28.15"
I0218 09:41:00.305835 1 server.go:156] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0218 09:41:00.318909 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0218 09:41:00.320706 1 secure_serving.go:213] Serving securely on [::]:10259
I0218 09:41:00.320777 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0218 09:41:00.321340 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0218 09:41:00.422208 1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-scheduler...
I0218 09:41:00.422584 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0218 09:41:16.758498 1 leaderelection.go:260] successfully acquired lease kube-system/kube-scheduler
E0218 01:46:03.303045 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:47:03.302323 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:48:03.304486 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:49:03.302893 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:50:03.304108 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:51:03.301668 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:52:03.302627 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:53:03.302932 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:54:03.302129 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:55:03.302563 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
E0218 01:56:03.302652 1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, service account token is not valid yet]]"
时间正常,证书正常,这个要怎么处理一下,不能卸载重装