创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。
操作系统信息
物理机，Centos7.9，56H/128G
Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方
Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.10”, GitCommit:“7e54d50d3012cf3389e43b096ba35300f36e0817”, GitTreeState:“clean”, BuildDate:“2022-08-17T18:32:54Z”, GoVersion:“go1.17.13”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.10”, GitCommit:“7e54d50d3012cf3389e43b096ba35300f36e0817”, GitTreeState:“clean”, BuildDate:“2022-08-17T18:26:59Z”, GoVersion:“go1.17.13”, Compiler:“gc”, Platform:“linux/amd64”}
容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方
KubeSphere版本信息
V3.4.1
问题是什么
正式环境遇到了异常，导致了系统无法访问，求助各位官方人员帮忙分析一下问题所在：
情况是这样的：
我们有5个节点node1 node2 node3 node4 node5
其中node1为主节点，node5为后续添加的主节点。在发生异常之前停止了node1的调度。并且停止了很长一段时间，其间都能正常访问。
在我重启了openelb-manager和coredns并且恢复node1的调度之后，访问就正常了。以下是一些报错时的日志：
openelb：
{“level”:“error”,“ts”:1715677393.75362,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“gp/im-push”,“error”:“node node1 has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
{“level”:“info”,“ts”:1715677394.753864,“msg”:“setup openelb service”,“service”:“kubesphere-controls-system/kubesphere-router-k8s-console”}
{“level”:“info”,“ts”:1715677394.754201,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kubesphere-controls-system/kubesphere-router-k8s-console”,“Addr”:"",“Eip”:“k8s-eip”,“Protocol”:“layer2”,“Unalloc”:false},“peek”:true,“result”:{“Addr”:“192.168.1.120”,“Eip”:“k8s-eip”,“Protocol”:“layer2”,“Sp”:{}},“err”:null}
{“level”:“error”,“ts”:1715677394.7545292,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“kubesphere-controls-system/kubesphere-router-k8s-console”,“error”:“node node2 has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
{“level”:“info”,“ts”:1715677395.7546456,“msg”:“setup openelb service”,“service”:“tms/gps-play-service”}
{“level”:“info”,“ts”:1715677395.754832,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“tms/gps-play-service”,“Addr”:"",“Eip”:“gps-play-eip”,“Protocol”:“layer2”,“Unalloc”:false},“peek”:true,“result”:{“Addr”:“192.168.1.169”,“Eip”:“gps-play-eip”,“Protocol”:“layer2”,“Sp”:{}},“err”:null}
{“level”:“error”,“ts”:1715677395.7550762,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“tms/gps-play-service”,“error”:“node node2 has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
{“level”:“info”,“ts”:1715677396.7552729,“msg”:“setup openelb service”,“service”:“kubesphere-controls-system/kubesphere-router-gp”}
{“level”:“info”,“ts”:1715677396.7555099,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kubesphere-controls-system/kubesphere-router-gp”,“Addr”:"",“Eip”:“gp-eip”,“Protocol”:“layer2”,“Unalloc”:false},“peek”:true,“result”:{“Addr”:“192.168.1.127”,“Eip”:“gp-eip”,“Protocol”:“layer2”,“Sp”:{}},“err”:null}
{“level”:“error”,“ts”:1715677396.7557223,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“kubesphere-controls-system/kubesphere-router-gp”,“error”:“node node2 has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
{“level”:“info”,“ts”:1715677415.2347229,“msg”:“setup openelb service”,“service”:“kubesphere-controls-system/kubesphere-router-k8s-console”}
{“level”:“info”,“ts”:1715677415.2349832,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kubesphere-controls-system/kubesphere-router-k8s-console”,“Addr”:"",“Eip”:“k8s-eip”,“Protocol”:“layer2”,“Unalloc”:false},“peek”:true,“result”:{“Addr”:“192.168.1.120”,“Eip”:“k8s-eip”,“Protocol”:“layer2”,“Sp”:{}},“err”:null}
{“level”:“error”,“ts”:1715677415.2352688,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“kubesphere-controls-system/kubesphere-router-k8s-console”,“error”:“node node2 has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
{“level”:“info”,“ts”:1715677423.5271025,“msg”:“setup openelb service”,“service”:“kubesphere-controls-system/kubesphere-router-paytest”}
另外一些不一样的日志：
{“level”:“error”,“ts”:1715676493.6489966,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“kubesphere-controls-system/kubesphere-router-gpdev”,“error”:“no avliable eip”,“stacktrace”:“github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}
网关日志：
E0514 02:42:48.133320 7 event.go:276] Unable to write event: ‘&v1.Event{TypeMeta:v1.TypeMeta{Kind:““, APIVersion:””}, ObjectMeta:v1.ObjectMeta{Name:“kubesphere-router-gptest-8477bc6f7-6q6fm.17cf3a2f8aec9758”, GenerateName:"“, Namespace:“kubesphere-controls-system”, SelfLink:”“, UID:”“, ResourceVersion:”“, Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:“Pod”, Namespace:“kubesphere-controls-system”, Name:“kubesphere-router-gptest-8477bc6f7-6q6fm”, UID:“7147ecf9-f771-40df-98e3-1a70c153a505”, APIVersion:“v1”, ResourceVersion:“37821695”, FieldPath:”"}, Reason:“RELOAD”, Message:“Error reloading NGINX: \n——————————————————————————-\nError: exit status 1\n2024/05/14 02:32:38 [warn] 4787#4787: the \“http2_max_field_size\” directive is obsolete, use the \“large_client_header_buffers\” directive instead in /tmp/nginx/nginx-cfg3180120453:144\nnginx: [warn] the \“http2_max_field_size\” directive is obsolete, use the \“large_client_header_buffers\” directive instead in /tmp/nginx/nginx-cfg3180120453:144\n2024/05/14 02:32:38 [warn] 4787#4787: the \“http2_max_header_size\” directive is obsolete, use the \“large_client_header_buffers\” directive instead in /tmp/nginx/nginx-cfg3180120453:145\nnginx: [warn] the \“http2_max_header_size\” directive is obsolete, use the \“large_client_header_buffers\” directive instead in /tmp/nginx/nginx-cfg3180120453:145\n2024/05/14 02:32:38 [warn] 4787#4787: the \“http2_max_requests\” directive is obsolete, use the \“keepalive_requests\” directive instead in /tmp/nginx/nginx-cfg3180120453:146\nnginx: [warn] the \“http2_max_requests\” directive is obsolete, use the \“keepalive_requests\” directive instead in /tmp/nginx/nginx-cfg3180120453:146\n2024/05/14 02:32:38 [emerg] 4787#4787: \“underscores_in_headers\” directive is not allowed here in /tmp/nginx/nginx-cfg3180120453:684\nnginx: [emerg] \“underscores_in_headers\” directive is not allowed here in /tmp/nginx/nginx-cfg3180120453:684\nnginx: configuration file /tmp/nginx/nginx-cfg3180120453 test failed\n\n——————————————————————————-\n”, Source:v1.EventSource{Component:“nginx-ingress-controller”, Host:"“}, FirstTimestamp:time.Date(2024, time.May, 14, 2, 32, 38, 920083288, time.Local), LastTimestamp:time.Date(2024, time.May, 14, 2, 32, 38, 920083288, time.Local), Count:1, Type:“Warning”, EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series*v1.EventSeries)(nil), Action:”“, Related*v1.ObjectReference)(nil), ReportingController:”“, ReportingInstance:”"}’: ’can’t create an event with namespace ‘kubesphere-controls-system’ in namespace ‘gptest’'(may retry after sleeping)
