root@node1:/etc/ssl/etcd/ssl# kubectl logs -n kube-system kube-apiserver-node1

W1215 08:28:24.743834 1 feature_gate.go:237] Setting GA feature gate TTLAfterFinished=true. It will be removed in a future release.

I1215 08:28:24.744410 1 server.go:565] external host was not specified, using 10.2.100.18

I1215 08:28:24.745265 1 server.go:172] Version: v1.23.17

I1215 08:28:25.055077 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer

I1215 08:28:25.056101 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:25.056124 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

I1215 08:28:25.057049 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:25.057059 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

W1215 08:28:25.145049 1 genericapiserver.go:538] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.

I1215 08:28:25.146077 1 instance.go:274] Using reconciler: lease

W1215 08:28:26.127422 1 genericapiserver.go:538] Skipping API authentication.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.139499 1 genericapiserver.go:538] Skipping API authorization.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.160882 1 genericapiserver.go:538] Skipping API certificates.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.163025 1 genericapiserver.go:538] Skipping API coordination.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.170233 1 genericapiserver.go:538] Skipping API networking.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.174109 1 genericapiserver.go:538] Skipping API node.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.198346 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.198373 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.201297 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.201313 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.208110 1 genericapiserver.go:538] Skipping API storage.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.214277 1 genericapiserver.go:538] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.219755 1 genericapiserver.go:538] Skipping API apps/v1beta2 because it has no resources.

W1215 08:28:26.219772 1 genericapiserver.go:538] Skipping API apps/v1beta1 because it has no resources.

W1215 08:28:26.222367 1 genericapiserver.go:538] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.

I1215 08:28:26.227891 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:26.227913 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

W1215 08:28:26.294657 1 genericapiserver.go:538] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.

I1215 08:28:27.593642 1 dynamic_cafile_content.go:156] “Starting controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:28:27.593646 1 dynamic_cafile_content.go:156] “Starting controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

I1215 08:28:27.594039 1 dynamic_serving_content.go:131] “Starting controller” name=“serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key”

I1215 08:28:27.594088 1 secure_serving.go:267] Serving securely on [::]:6443

I1215 08:28:27.594142 1 tlsconfig.go:240] “Starting DynamicServingCertificateController”

I1215 08:28:27.594235 1 apf_controller.go:317] Starting API Priority and Fairness config controller

I1215 08:28:27.594330 1 available_controller.go:491] Starting AvailableConditionController

I1215 08:28:27.594359 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller

I1215 08:28:27.594428 1 customresource_discovery_controller.go:209] Starting DiscoveryController

I1215 08:28:27.594451 1 controller.go:83] Starting OpenAPI AggregationController

I1215 08:28:27.594495 1 establishing_controller.go:76] Starting EstablishingController

I1215 08:28:27.594500 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller

I1215 08:28:27.594512 1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller

I1215 08:28:27.594519 1 naming_controller.go:291] Starting NamingConditionController

I1215 08:28:27.594585 1 autoregister_controller.go:141] Starting autoregister controller

I1215 08:28:27.594601 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController

I1215 08:28:27.594606 1 cache.go:32] Waiting for caches to sync for autoregister controller

I1215 08:28:27.594639 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController

I1215 08:28:27.594656 1 controller.go:85] Starting OpenAPI controller

I1215 08:28:27.594676 1 crd_finalizer.go:266] Starting CRDFinalizer

I1215 08:28:27.594701 1 dynamic_serving_content.go:131] “Starting controller” name=“aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key”

I1215 08:28:27.594702 1 dynamic_cafile_content.go:156] “Starting controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:28:27.594981 1 dynamic_cafile_content.go:156] “Starting controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

I1215 08:28:27.595025 1 crdregistration_controller.go:111] Starting crd-autoregister controller

I1215 08:28:27.595060 1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister

I1215 08:28:27.601292 1 apiservice_controller.go:97] Starting APIServiceRegistrationController

I1215 08:28:27.603571 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller

I1215 08:28:27.636993 1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io

I1215 08:28:27.656162 1 shared_informer.go:247] Caches are synced for node_authorizer

I1215 08:28:27.694496 1 apf_controller.go:322] Running API Priority and Fairness config worker

I1215 08:28:27.694549 1 cache.go:39] Caches are synced for AvailableConditionController controller

I1215 08:28:27.694739 1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller

I1215 08:28:27.694945 1 cache.go:39] Caches are synced for autoregister controller

I1215 08:28:27.695205 1 shared_informer.go:247] Caches are synced for crd-autoregister

I1215 08:28:27.703791 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller

I1215 08:28:28.594008 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).

I1215 08:28:28.598375 1 storage_scheduling.go:109] all system priority classes are created successfully or already exist.

root@node1:/etc/ssl/etcd/ssl# kubectl logs -n kube-system kube-apiserver-node1 -f

W1215 08:28:24.743834 1 feature_gate.go:237] Setting GA feature gate TTLAfterFinished=true. It will be removed in a future release.

I1215 08:28:24.744410 1 server.go:565] external host was not specified, using 10.2.100.18

I1215 08:28:24.745265 1 server.go:172] Version: v1.23.17

I1215 08:28:25.055077 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer

I1215 08:28:25.056101 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:25.056124 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

I1215 08:28:25.057049 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:25.057059 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

W1215 08:28:25.145049 1 genericapiserver.go:538] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.

I1215 08:28:25.146077 1 instance.go:274] Using reconciler: lease

W1215 08:28:26.127422 1 genericapiserver.go:538] Skipping API authentication.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.139499 1 genericapiserver.go:538] Skipping API authorization.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.160882 1 genericapiserver.go:538] Skipping API certificates.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.163025 1 genericapiserver.go:538] Skipping API coordination.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.170233 1 genericapiserver.go:538] Skipping API networking.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.174109 1 genericapiserver.go:538] Skipping API node.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.198346 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.198373 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.201297 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.

W1215 08:28:26.201313 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.208110 1 genericapiserver.go:538] Skipping API storage.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.214277 1 genericapiserver.go:538] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.

W1215 08:28:26.219755 1 genericapiserver.go:538] Skipping API apps/v1beta2 because it has no resources.

W1215 08:28:26.219772 1 genericapiserver.go:538] Skipping API apps/v1beta1 because it has no resources.

W1215 08:28:26.222367 1 genericapiserver.go:538] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.

I1215 08:28:26.227891 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.

I1215 08:28:26.227913 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

W1215 08:28:26.294657 1 genericapiserver.go:538] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.

I1215 08:28:27.593642 1 dynamic_cafile_content.go:156] “Starting controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:28:27.593646 1 dynamic_cafile_content.go:156] “Starting controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

I1215 08:28:27.594039 1 dynamic_serving_content.go:131] “Starting controller” name=“serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key”

I1215 08:28:27.594088 1 secure_serving.go:267] Serving securely on [::]:6443

I1215 08:28:27.594142 1 tlsconfig.go:240] “Starting DynamicServingCertificateController”

I1215 08:28:27.594235 1 apf_controller.go:317] Starting API Priority and Fairness config controller

I1215 08:28:27.594330 1 available_controller.go:491] Starting AvailableConditionController

I1215 08:28:27.594359 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller

I1215 08:28:27.594428 1 customresource_discovery_controller.go:209] Starting DiscoveryController

I1215 08:28:27.594451 1 controller.go:83] Starting OpenAPI AggregationController

I1215 08:28:27.594495 1 establishing_controller.go:76] Starting EstablishingController

I1215 08:28:27.594500 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller

I1215 08:28:27.594512 1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller

I1215 08:28:27.594519 1 naming_controller.go:291] Starting NamingConditionController

I1215 08:28:27.594585 1 autoregister_controller.go:141] Starting autoregister controller

I1215 08:28:27.594601 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController

I1215 08:28:27.594606 1 cache.go:32] Waiting for caches to sync for autoregister controller

I1215 08:28:27.594639 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController

I1215 08:28:27.594656 1 controller.go:85] Starting OpenAPI controller

I1215 08:28:27.594676 1 crd_finalizer.go:266] Starting CRDFinalizer

I1215 08:28:27.594701 1 dynamic_serving_content.go:131] “Starting controller” name=“aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key”

I1215 08:28:27.594702 1 dynamic_cafile_content.go:156] “Starting controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:28:27.594981 1 dynamic_cafile_content.go:156] “Starting controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

I1215 08:28:27.595025 1 crdregistration_controller.go:111] Starting crd-autoregister controller

I1215 08:28:27.595060 1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister

I1215 08:28:27.601292 1 apiservice_controller.go:97] Starting APIServiceRegistrationController

I1215 08:28:27.603571 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller

I1215 08:28:27.636993 1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io

I1215 08:28:27.656162 1 shared_informer.go:247] Caches are synced for node_authorizer

I1215 08:28:27.694496 1 apf_controller.go:322] Running API Priority and Fairness config worker

I1215 08:28:27.694549 1 cache.go:39] Caches are synced for AvailableConditionController controller

I1215 08:28:27.694739 1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller

I1215 08:28:27.694945 1 cache.go:39] Caches are synced for autoregister controller

I1215 08:28:27.695205 1 shared_informer.go:247] Caches are synced for crd-autoregister

I1215 08:28:27.703791 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller

I1215 08:28:28.594008 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).

I1215 08:28:28.598375 1 storage_scheduling.go:109] all system priority classes are created successfully or already exist.

I1215 08:30:21.507448 1 controller.go:611] quota admission added evaluator for: events.events.k8s.io

I1215 08:31:19.468110 1 controller.go:186] Shutting down kubernetes service endpoint reconciler

I1215 08:31:19.468280 1 object_count_tracker.go:84] “StorageObjectCountTracker pruner is exiting”

I1215 08:31:19.468406 1 nonstructuralschema_controller.go:204] Shutting down NonStructuralSchemaConditionController

I1215 08:31:19.468436 1 controller.go:122] Shutting down OpenAPI controller

I1215 08:31:19.468468 1 crdregistration_controller.go:142] Shutting down crd-autoregister controller

I1215 08:31:19.468490 1 naming_controller.go:302] Shutting down NamingConditionController

I1215 08:31:19.468495 1 autoregister_controller.go:165] Shutting down autoregister controller

I1215 08:31:19.468454 1 crd_finalizer.go:278] Shutting down CRDFinalizer

I1215 08:31:19.468525 1 available_controller.go:503] Shutting down AvailableConditionController

I1215 08:31:19.468561 1 apf_controller.go:326] Shutting down API Priority and Fairness config worker

I1215 08:31:19.468572 1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller

I1215 08:31:19.468602 1 establishing_controller.go:87] Shutting down EstablishingController

I1215 08:31:19.468624 1 apiapproval_controller.go:198] Shutting down KubernetesAPIApprovalPolicyConformantConditionController

I1215 08:31:19.468673 1 storage_flowcontrol.go:150] APF bootstrap ensurer is exiting

I1215 08:31:19.468771 1 apiservice_controller.go:131] Shutting down APIServiceRegistrationController

I1215 08:31:19.468417 1 customresource_discovery_controller.go:245] Shutting down DiscoveryController

I1215 08:31:19.469133 1 dynamic_cafile_content.go:170] “Shutting down controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:31:19.469177 1 dynamic_cafile_content.go:170] “Shutting down controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

I1215 08:31:19.469230 1 controller.go:89] Shutting down OpenAPI AggregationController

I1215 08:31:19.469276 1 dynamic_cafile_content.go:170] “Shutting down controller” name=“client-ca-bundle::/etc/kubernetes/pki/ca.crt”

I1215 08:31:19.469334 1 dynamic_serving_content.go:145] “Shutting down controller” name=“serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key”

I1215 08:31:19.469377 1 dynamic_serving_content.go:145] “Shutting down controller” name=“aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key”

I1215 08:31:19.469399 1 secure_serving.go:312] Stopped listening on [::]:6443

I1215 08:31:19.469451 1 tlsconfig.go:255] “Shutting down DynamicServingCertificateController”

I1215 08:31:19.469487 1 dynamic_cafile_content.go:170] “Shutting down controller” name=“request-header::/etc/kubernetes/pki/front-proxy-ca.crt”

E1215 08:31:19.472229 1 controller.go:189] Unable to remove endpoints from kubernetes service: Get “https://[::1]:6443/api/v1/namespaces/default/endpoints/kubernetes”: dial tcp [::1]:6443: connect: connection refused

error: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""

这是我的apiserver的日志，一直在无限重启，不是很清楚为什么会出现这个情况