创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
操作系统信息
虚拟机Centos7
KubeSphere版本信息
在centos7上基于kk部署的1.18.3版本的k8s,安装V3.1.1版的kubusphere,调试可用。
在集群管理界面多次刷新会出现内部error,导致刷新失败,界面显示not found。
该问题的复现操作就是不停刷新【集群节点】界面,有时候需要刷新大几十次、有时候只需要十几次甚至打开界面时会出现,只是刷新界面这个常规操作下出现该问题,对软件来说相当于bug。
登录https://demo.kubesphere.io/clusters/dev/nodes进行刷新也会出现该异常(v3.1.0版本,k8s版本v1.19.8),但无法获取日志,因为ks-apiserver会很快重启恢复正常。

在自己本地环境下获取的日志输出如下:
W0105 08:23:36.256505 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0105 08:23:36.289706 1 apiserver.go:300] Start cache objects
W0105 08:23:37.060849 1 apiserver.go:433] resource types.kubefed.io/v1beta1, Resource=federatedresourcequotas not exists in the cluster
W0105 08:23:37.461486 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotclasses not exists in the cluster
W0105 08:23:37.461522 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshots not exists in the cluster
W0105 08:23:37.461541 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotcontents not exists in the cluster
I0105 08:23:37.761742 1 apiserver.go:502] Finished caching objects
I0105 08:23:37.761778 1 apiserver.go:232] Start listening on :9090
fatal error: concurrent map writes
goroutine 3090 [running]:
runtime.throw(0x28ff724, 0x15)
/usr/local/go/src/runtime/panic.go:1116 +0x54 fp=0x40039fc530 sp=0x40039fc500 pc=0x42274
runtime.mapassign_faststr(0x2402e40, 0x400378f7a0, 0x29153d0, 0x1d, 0x4000463a88)
/usr/local/go/src/runtime/map_faststr.go:211 +0x3c4 fp=0x40039fc5a0 sp=0x40039fc530 pc=0x220c4
kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node.nodesGetter.annotateNode(0x2ff4300, 0x4000149590, 0x4001a5a600, 0x4002e1be00, 0x3f, 0x40)
/src/kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node/nodes.go:178 +0x430 fp=0x40039fcb20 sp=0x40039fc5a0 pc=0x117b530
kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node.nodesGetter.List(0x2ff4300, 0x4000149590, 0x0, 0x0, 0x40038d1f80, 0x5030004, 0x2, 0xffffffffffffffff)
/src/kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node/nodes.go:120 +0x5f8 fp=0x40039fccc0 sp=0x40039fcb20 pc=0x117ac88
kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node.(*nodesGetter).List(0x40004a8930, 0x0, 0x0, 0x40038d1f80, 0x40004a8930, 0x2000107, 0x400398f7a8)
<autogenerated>:1 +0x58 fp=0x40039fcd10 sp=0x40039fccc0 pc=0x117cef8
kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/resource.(*ResourceGetter).List(0x4000680150, 0x40017523b8, 0x5, 0x0, 0x0, 0x40038d1f80, 0x0, 0x0, 0x40039fce68)
/src/kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/resource/resource.go:155 +0x68 fp=0x40039fcd60 sp=0x40039fcd10 pc=0x1188b48
kubesphere.io/kubesphere/pkg/kapis/resources/v1alpha3.(*Handler).handleListResources(0x4000264a40, 0x4003993440, 0x400078f030)
/src/kubesphere.io/kubesphere/pkg/kapis/resources/v1alpha3/handler.go:83 +0xf0 fp=0x40039fce30 sp=0x40039fcd60 pc=0x1f6c7e0
kubesphere.io/kubesphere/pkg/kapis/resources/v1alpha3.(*Handler).handleListResources-fm(0x4003993440, 0x400078f030)
/src/kubesphere.io/kubesphere/pkg/kapis/resources/v1alpha3/handler.go:78 +0x38 fp=0x40039fce60 sp=0x40039fce30 pc=0x1f70978
github.com/emicklei/go-restful.(*Container).dispatch.func5(0x4003993440, 0x400078f030)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:283 +0x40 fp=0x40039fce80 sp=0x40039fce60 pc=0x3b8640
github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0x40039934d0, 0x4003993440, 0x400078f030)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/filter.go:21 +0x90 fp=0x40039fceb0 sp=0x40039fce80 pc=0x3afdb0
kubesphere.io/kubesphere/pkg/apiserver.logRequestAndResponse(0x4003993440, 0x400078f030, 0x40039934d0)
/src/kubesphere.io/kubesphere/pkg/apiserver/apiserver.go:531 +0x5c fp=0x40039fd130 sp=0x40039fceb0 pc=0x20743dc
github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0x40039934d0, 0x4003993440, 0x400078f030)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/filter.go:19 +0x68 fp=0x40039fd160 sp=0x40039fd130 pc=0x3afd88
github.com/emicklei/go-restful.(*Container).dispatch(0x4000990000, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:285 +0x6d8 fp=0x40039fd300 sp=0x40039fd160 pc=0x3acee8
github.com/emicklei/go-restful.(*Container).dispatch-fm(0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:203 +0x40 fp=0x40039fd330 sp=0x40039fd300 pc=0x3b9a60
net/http.HandlerFunc.ServeHTTP(0x40004a8fc0, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fd360 sp=0x40039fd330 pc=0x297be0
net/http.(*ServeMux).ServeHTTP(0x400071e500, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/usr/local/go/src/net/http/server.go:2387 +0x190 fp=0x40039fd3c0 sp=0x40039fd360 pc=0x2995d0
github.com/emicklei/go-restful.(*Container).ServeHTTP(0x4000990000, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/root/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:303 +0x44 fp=0x40039fd3f0 sp=0x40039fd3c0 pc=0x3ad2d4
kubesphere.io/kubesphere/pkg/apiserver/filters.WithKubeAPIServer.func1(0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/src/kubesphere.io/kubesphere/pkg/apiserver/filters/kubeapiserver.go:71 +0x31c fp=0x40039fd530 sp=0x40039fd3f0 pc=0x1c7072c
net/http.HandlerFunc.ServeHTTP(0x40006b3b40, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fd560 sp=0x40039fd530 pc=0x297be0
kubesphere.io/kubesphere/pkg/apiserver/filters.WithAuthorization.func1(0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/src/kubesphere.io/kubesphere/pkg/apiserver/filters/authorization.go:50 +0x27c fp=0x40039fd660 sp=0x40039fd560 pc=0x1c7020c
net/http.HandlerFunc.ServeHTTP(0x40006b3bc0, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fd690 sp=0x40039fd660 pc=0x297be0
kubesphere.io/kubesphere/pkg/apiserver/dispatch.(*clusterDispatch).Dispatch(0x40006b3c00, 0x2f9c840, 0x4000efe2a0, 0x4002620b00, 0x2f39da0, 0x40006b3bc0)
/src/kubesphere.io/kubesphere/pkg/apiserver/dispatch/dispatch.go:117 +0x2ec fp=0x40039fd820 sp=0x40039fd690 pc=0x1c6cbcc
kubesphere.io/kubesphere/pkg/apiserver/filters.WithMultipleClusterDispatcher.func1(0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/src/kubesphere.io/kubesphere/pkg/apiserver/filters/dispatch.go:45 +0x108 fp=0x40039fd890 sp=0x40039fd820 pc=0x1c70388
net/http.HandlerFunc.ServeHTTP(0x4000bc7dd0, 0x2f9c840, 0x4000efe2a0, 0x4002620b00)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fd8c0 sp=0x40039fd890 pc=0x297be0
kubesphere.io/kubesphere/pkg/apiserver/filters.WithAuthentication.func1(0x2f9c840, 0x4000efe2a0, 0x4002620a00)
/src/kubesphere.io/kubesphere/pkg/apiserver/filters/authentication.go:95 +0x8c4 fp=0x40039fda80 sp=0x40039fd8c0 pc=0x1c6fd74
net/http.HandlerFunc.ServeHTTP(0x4000bb2640, 0x2f9c840, 0x4000efe2a0, 0x4002620a00)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fdab0 sp=0x40039fda80 pc=0x297be0
kubesphere.io/kubesphere/pkg/apiserver/filters.WithRequestInfo.func1(0x2f9c840, 0x4000efe2a0, 0x4000b74800)
/src/kubesphere.io/kubesphere/pkg/apiserver/filters/requestinfo.go:67 +0x31c fp=0x40039fdbb0 sp=0x40039fdab0 pc=0x1c70b5c
net/http.HandlerFunc.ServeHTTP(0x4000f04570, 0x2f9c840, 0x4000efe2a0, 0x4000b74800)
/usr/local/go/src/net/http/server.go:2012 +0x40 fp=0x40039fdbe0 sp=0x40039fdbb0 pc=0x297be0
net/http.serverHandler.ServeHTTP(0x400027e000, 0x2f9c840, 0x4000efe2a0, 0x4000b74800)
/usr/local/go/src/net/http/server.go:2807 +0xbc fp=0x40039fdc10 sp=0x40039fdbe0 pc=0x29a88c
net/http.(*conn).serve(0x4000d78140, 0x2faa7c0, 0x400368ec80)
/usr/local/go/src/net/http/server.go:1895 +0x70c fp=0x40039fdfc0 sp=0x40039fdc10 pc=0x296bcc
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1148 +0x4 fp=0x40039fdfc0 sp=0x40039fdfc0 pc=0x71ac4
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2933 +0x2e8
goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0xffff861241c8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:203 +0x44
internal/poll.(*pollDesc).wait(0x4002f24098, 0x72, 0x0, 0x0, 0x28dcc91)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0x4002f24080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x184
net.(*netFD).accept(0x4002f24080, 0xe819746e, 0x400166fa88, 0x29b69c)
/usr/local/go/src/net/fd_unix.go:238 +0x2c
net.(*TCPListener).accept(0x4000c73d20, 0x61d4e6e2, 0x400166fab8, 0x4ccf4)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x2c
net.(*TCPListener).Accept(0x4000c73d20, 0x400166fb20, 0x400166fb28, 0x18, 0x4000000180)
/usr/local/go/src/net/tcpsock.go:261 +0x5c
net/http.(*Server).Serve(0x400027e000, 0x2f9c580, 0x4000c73d20, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2901 +0x218
net/http.(*Server).ListenAndServe(0x400027e000, 0x0, 0x28ff0a9)
/usr/local/go/src/net/http/server.go:2830 +0xac
kubesphere.io/kubesphere/pkg/apiserver.(*APIServer).Run(0x40006260d0, 0x4000203800, 0x0, 0x0)
/src/kubesphere.io/kubesphere/pkg/apiserver/apiserver.go:236 +0x158
kubesphere.io/kubesphere/cmd/ks-apiserver/app.Run(0x4000972450, 0x4000203800, 0x0, 0x0)
/src/kubesphere.io/kubesphere/cmd/ks-apiserver/app/server.go:89 +0x8c
kubesphere.io/kubesphere/cmd/ks-apiserver/app.NewAPIServerCommand.func1(0x40001f0c80, 0x40003eb880, 0x0, 0x1, 0x0, 0x0)
/src/kubesphere.io/kubesphere/cmd/ks-apiserver/app/server.go:55 +0xa0
github.com/spf13/cobra.(*Command).execute(0x40001f0c80, 0x400000c070, 0x1, 0x1, 0x40001f0c80, 0x400000c070)
/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826 +0x30c
github.com/spf13/cobra.(*Command).ExecuteC(0x40001f0c80, 0x0, 0x4000104768, 0x44764)
/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x22c
github.com/spf13/cobra.(*Command).Execute(...)
/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main()
/src/kubesphere.io/kubesphere/cmd/ks-apiserver/apiserver.go:28 +0x24
查找相关异常如:kubesphere/kubesphere#4357
kubesphere/kubesphere#3469
大多意为并发访问map是不安全的,会出现未定义行为,导致程序退出。所以如果希望在多协程中并发访问map,必须提供某种同步机制,一般情况下通过读写锁sync.RWMutex实现对map的并发访问控制,将map和sync.RWMutex封装一下,可以实现对map的安全并发访问。
https://github.com/zryfish/kubesphere/tree/fix_concurrent_map_writes连接中查看/src/kubesphere.io/kubesphere/pkg/models/resources/v1alpha3/node/nodes.go中已经引用。
对比v3.1.1版本中也有相关引用,但还是会出现该问题,希望能够给予解答。