告警策略和告警消息出现服务内部错误

FlywinNing

告警策略和告警消息突然出现服务内部错误，相关的pod，svc都是正常的，请指导下
eks: 1.17
kubesphere: 3.0.0

hongming

FlywinNing 可以看一下 kubesphere-system 下 ks-apiserver 的日志

FlywinNing

hongming 你好，日志有这个错误

2021/02/08 06:12:58 http: panic serving 10.59.34.43:34448: runtime error: invalid memory address or nil pointer dereference
goroutine 691252 [running]:
net/http.(*conn).serve.func1(0xc000c7c8c0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:1795 +0x139
panic(0x2f8d800, 0x5973d20)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/panic.go:679 +0x1b2
kubesphere.io/kubesphere/pkg/models/alerting.(*operator).listCustomAlertingRules(0xc000bc23f0, 0x3c59b80, 0xc004048930, 0xc0005a0d70, 0x34be6db, 0x7, 0xc0042a5380, 0xc0042a53b0, 0x0, 0x0, ...)
        /home/runner/work/kubesphere/kubesphere/pkg/models/alerting/alerting.go:288 +0x1b1
kubesphere.io/kubesphere/pkg/models/alerting.(*operator).ListCustomRulesAlerts(0xc000bc23f0, 0x3c59b80, 0xc004048930, 0x34f52ee, 0x1c, 0xc0042a5290, 0xc000b85110, 0x1, 0x1)
        /home/runner/work/kubesphere/kubesphere/pkg/models/alerting/alerting.go:159 +0xf8
kubesphere.io/kubesphere/pkg/kapis/alerting/v2alpha1.(*handler).handleListCustomRulesAlerts(0xc000b850b0, 0xc0042a51d0, 0xc000416b60)
        /home/runner/work/kubesphere/kubesphere/pkg/kapis/alerting/v2alpha1/handler.go:75 +0x1c2
github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc0042a5260, 0xc0042a51d0, 0xc000416b60)
        /home/runner/go/pkg/mod/github.com/emicklei/go-restful@v2.14.3+incompatible/filter.go:21 +0x8b
kubesphere.io/kubesphere/pkg/apiserver.monitorRequest(0xc0042a51d0, 0xc000416b60, 0xc0042a5260)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/apiserver.go:187 +0x7c
github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc0042a5260, 0xc0042a51d0, 0xc000416b60)
        /home/runner/go/pkg/mod/github.com/emicklei/go-restful@v2.14.3+incompatible/filter.go:19 +0x65
kubesphere.io/kubesphere/pkg/apiserver.logRequestAndResponse(0xc0042a51d0, 0xc000416b60, 0xc0042a5260)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/apiserver.go:592 +0x82
github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc0042a5260, 0xc0042a51d0, 0xc000416b60)
        /home/runner/go/pkg/mod/github.com/emicklei/go-restful@v2.14.3+incompatible/filter.go:19 +0x65
github.com/emicklei/go-restful.(*Container).dispatch(0xc000c82510, 0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /home/runner/go/pkg/mod/github.com/emicklei/go-restful@v2.14.3+incompatible/container.go:291 +0x48d
net/http.HandlerFunc.ServeHTTP(0xc0007820c0, 0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0007b0e40, 0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2416 +0x1bd
github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc000c82510, 0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /home/runner/go/pkg/mod/github.com/emicklei/go-restful@v2.14.3+incompatible/container.go:311 +0x2c9
kubesphere.io/kubesphere/pkg/apiserver/filters.WithKubeAPIServer.func1(0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/filters/kubeapiserver.go:62 +0x37f
net/http.HandlerFunc.ServeHTTP(0xc0008df280, 0x3c45b00, 0xc0042a5170, 0xc003c76900)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
kubesphere.io/kubesphere/pkg/apiserver/filters.WithAuditing.func1(0x3c461c0, 0xc003ff8380, 0xc003c76900)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/filters/auditing.go:54 +0x1b8
net/http.HandlerFunc.ServeHTTP(0xc000b19e60, 0x3c461c0, 0xc003ff8380, 0xc003c76900)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
kubesphere.io/kubesphere/pkg/apiserver/filters.WithAuthorization.func1(0x3c461c0, 0xc003ff8380, 0xc003c76900)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/filters/authorization.go:50 +0x37c
net/http.HandlerFunc.ServeHTTP(0xc0008df300, 0x3c461c0, 0xc003ff8380, 0xc003c76900)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
kubesphere.io/kubesphere/pkg/apiserver/filters.WithAuthentication.func1(0x3c461c0, 0xc003ff8380, 0xc003c76900)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/filters/authentication.go:68 +0x5ce
net/http.HandlerFunc.ServeHTTP(0xc0008df400, 0x3c461c0, 0xc003ff8380, 0xc003c76800)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
kubesphere.io/kubesphere/pkg/apiserver/filters.WithRequestInfo.func1(0x3c461c0, 0xc003ff8380, 0xc003c76700)
        /home/runner/work/kubesphere/kubesphere/pkg/apiserver/filters/requestinfo.go:67 +0x3c5
net/http.HandlerFunc.ServeHTTP(0xc000b36fc0, 0x3c461c0, 0xc003ff8380, 0xc003c76700)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2036 +0x44
net/http.serverHandler.ServeHTTP(0xc0002ba380, 0x3c461c0, 0xc003ff8380, 0xc003c76700)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2831 +0xa4
net/http.(*conn).serve(0xc000c7c8c0, 0x3c59ac0, 0xc003c86380)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:1919 +0x875
created by net/http.(*Server).Serve
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2957 +0x384

FlywinNing

hongming 我发现有个minio的pvc没有创建出来

xulai

贴一下kubesphere-system下ks-apiserver所使用的镜像版本，这个报错不是3.0.0版本的异常

FlywinNing

xulai 我删除了重装，我强行删了ns：kubesphere-logging-system，现在安装说这个ns是terminated

TASK [common : KubeSphere | Create ImagePullSecrets] ***************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
failed: [localhost] (item=kubesphere-logging-system) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: "cat <<EOF | kubectl apply -f -\n apiVersion: v1\n kind: Secret\n metadata:\n name: qingcloud\n namespace: \“kubesphere-logging-system\”\n data:\n .dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXJodWIucWluZ2Nsb3VkLmNvbSI6eyJhdXRoIjoiWjNWbGMzUTZaM1ZsYzNRPSJ9fX0=\n type: kubernetes.io/dockerconfigjson\nEOF\n", “delta”: “0:00:00.476683”, “end”: “2021-02-08 09:23:33.285294”, “item”: “kubesphere-logging-system”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2021-02-08 09:23:32.808611”, “stderr”: "Error from server (Forbidden): error when creating \“STDIN\”: secrets \“qingcloud\” is forbidden: unable to create new content in namespace kubesphere-logging-system because it is being terminated", “stderr_lines”: ["Error from server (Forbidden): error when creating \“STDIN\”: secrets \“qingcloud\” is forbidden: unable to create new content in namespace kubesphere-logging-system because it is being terminated"], “stdout”: "", “stdout_lines”: []}
changed: [localhost] => (item=kubesphere-alerting-system)

PLAY RECAP *********************************************************************
localhost : ok=7 changed=3 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0