集群安装完成后,调度器监控显示有20多个失败数。查看日志如下:

/ # kubectl logs -f kube-scheduler-master165 -n kube-system
I0914 10:38:11.682835       1 registry.go:173] Registering SelectorSpread plugin
I0914 10:38:11.682950       1 registry.go:173] Registering SelectorSpread plugin
W0914 10:38:11.683913       1 feature_gate.go:235] Setting GA feature gate CSINodeInfo=true. It will be removed in a future release.
W0914 10:38:11.683957       1 feature_gate.go:235] Setting GA feature gate RotateKubeletClientCertificate=true. It will be removed in a future release.
I0914 10:38:12.741865       1 serving.go:331] Generated self-signed cert in-memory
W0914 10:38:13.950240       1 authentication.go:294] Error looking up in-cluster authentication configuration: Get "https://172.22.151.165:6443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 172.22.151.165:6443: connect: connection refused
W0914 10:38:13.950370       1 authentication.go:295] Continuing without authentication configuration. This may treat all requests as anonymous.
W0914 10:38:13.950387       1 authentication.go:296] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0914 10:38:13.957906       1 registry.go:173] Registering SelectorSpread plugin
I0914 10:38:13.957972       1 registry.go:173] Registering SelectorSpread plugin
I0914 10:38:13.963635       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0914 10:38:13.963705       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0914 10:38:13.964029       1 secure_serving.go:197] Serving securely on [::]:10259
I0914 10:38:13.964122       1 tlsconfig.go:240] Starting DynamicServingCertificateController
E0914 10:38:13.965462       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://172.22.151.165:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.965576       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://172.22.151.165:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.965466       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://172.22.151.165:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.965882       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://172.22.151.165:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.965889       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.965888       1 reflector.go:127] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://172.22.151.165:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966065       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966075       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://172.22.151.165:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966121       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966489       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://172.22.151.165:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966521       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://172.22.151.165:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966521       1 reflector.go:127] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:188: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:13.966783       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://172.22.151.165:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:14.832646       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:14.833930       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://172.22.151.165:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:14.874105       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://172.22.151.165:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:14.888285       1 reflector.go:127] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:188: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:14.918944       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.106246       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://172.22.151.165:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.113122       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://172.22.151.165:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.156621       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://172.22.151.165:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.159556       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://172.22.151.165:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.361185       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://172.22.151.165:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.418550       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://172.22.151.165:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.457854       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:15.525986       1 reflector.go:127] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://172.22.151.165:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:16.807119       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://172.22.151.165:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:16.985205       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://172.22.151.165:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.017253       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://172.22.151.165:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.037042       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://172.22.151.165:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.186606       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.475481       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.600782       1 reflector.go:127] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:188: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.659827       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://172.22.151.165:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.769937       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:17.779420       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://172.22.151.165:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:18.105723       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: Get "https://172.22.151.165:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:18.107774       1 reflector.go:127] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://172.22.151.165:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:18.429218       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://172.22.151.165:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:20.303233       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://172.22.151.165:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:20.663624       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.22.151.165:6443/api/v1/pods?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:21.678147       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://172.22.151.165:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:21.809804       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://172.22.151.165:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 172.22.151.165:6443: connect: connection refused
E0914 10:38:30.519069       1 reflector.go:127] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0914 10:38:30.520914       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0914 10:38:30.521246       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0914 10:38:30.521294       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets"in API group "apps" at the cluster scope
E0914 10:38:30.521821       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0914 10:38:30.521877       1 reflector.go:127] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:188: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0914 10:38:30.523144       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannotlist resource "persistentvolumeclaims" in API group "" at the cluster scope
E0914 10:38:30.523183       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0914 10:38:30.523603       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0914 10:38:30.523616       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0914 10:38:30.523768       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0914 10:38:31.288057       1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannotlist resource "replicationcontrollers" in API group "" at the cluster scope
I0914 10:38:42.763958       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0914 10:38:54.164710       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/kube-scheduler...
I0914 10:38:54.193656       1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

    mars

    fukang0611 E0914 10:38:13.965462 1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get “https://172.22.151.165:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0”: dial tcp 172.22.151.165:6443: connect: connection refused

    这些错误是因为kube-apiserver还没有启动完成并开始监听端口,kube-scheduler连接kube-apiserver失败。

    fukang0611 E0914 10:38:30.519069 1 reflector.go:127] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps “extension-apiserver-authentication” is forbidden: User “system:kube-scheduler” cannot list resource “configmaps” in API group "" in the namespace “kube-system”

    这些错误是因为kube-apiserver还没有初始化完成本地鉴权所需的数据,kube-scheduler的请求因为权限不足失败。

    在这之后,kube-scheduler应该已经正常运行了。

      7 个月 后

      kevendeng 请问第二种类型的错误如何解决呢,怎么让kube-apiserver初始化鉴权数据