碰到过，我之前开启防火墙，按文档开放了端口，就出现了这个问题。关闭防火墙后恢复

    wnxn 三个master节点，一个slave节点，参与工作的节点有2个master+1个slave

    • wnxn 回复了此帖

        lvelvis 参与工作的节点有2个master，还有1个master是不能工作吗？

            5 天 后

            5ea6c16e conn=1367 op=7 BIND anonymous mech=implicit ssf=0
            5ea6c16e conn=1367 op=7 BIND dn="cn=admin,dc=kubesphere,dc=io" method=128
            5ea6c16e conn=1367 op=7 BIND dn="cn=admin,dc=kubesphere,dc=io" mech=SIMPLE ssf=0
            5ea6c16e conn=1367 op=7 RESULT tag=97 err=0 text=
            5ea6c16e conn=1367 op=8 SRCH base="ou=Users,dc=kubesphere,dc=io" scope=2 deref=0 filter="(&(objectClass=inetOrgPerson)(|(uid=admin)(mail=admin)))"
            5ea6c16e conn=1367 op=8 SRCH attr=uid mail
            5ea6c16e conn=1367 op=8 SEARCH RESULT tag=101 err=0 nentries=1 text=
            5ea6c16e conn=1367 op=9 BIND anonymous mech=implicit ssf=0
            5ea6c16e conn=1367 op=9 BIND dn="uid=admin,ou=Users,dc=kubesphere,dc=io" method=128
            5ea6c16e conn=1367 op=9 BIND dn="uid=admin,ou=Users,dc=kubesphere,dc=io" mech=SIMPLE ssf=0
            5ea6c16e conn=1367 op=9 RESULT tag=97 err=0 text=
            5ea6c173 conn=1452 fd=19 ACCEPT from IP=172.28.161.107:52250 (IP=0.0.0.0:389)
            5ea6c173 conn=1452 fd=19 closed (connection lost)
            5ea6c17d conn=1453 fd=19 ACCEPT from IP=172.28.161.107:52596 (IP=0.0.0.0:389)
            5ea6c17d conn=1453 fd=19 closed (connection lost)
            怀疑是不是连接自动断开了，以上是openldap日志

              lvelvis 像三master部署情况，ks核心模块只调度到master节点上。不建议master节点部署业务应用。

              [root@master0 ~]# kubectl get pod -n kubesphere-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
etcd-f988bdb6f-c2klg                   1/1     Running   0          2d14h   10.233.96.5     node2     <none>           <none>
ks-account-7bf75c8b6-2qk5t             1/1     Running   0          2d14h   10.233.98.6     master2   <none>           <none>
ks-account-7bf75c8b6-8glfh             1/1     Running   0          2d14h   10.233.101.9    master0   <none>           <none>
ks-account-7bf75c8b6-czlfk             1/1     Running   0          2d14h   10.233.97.7     master1   <none>           <none>
ks-apigateway-5664c4b76f-gz57c         1/1     Running   0          2d14h   10.233.101.7    master0   <none>           <none>
ks-apigateway-5664c4b76f-hgm5m         1/1     Running   0          2d14h   10.233.97.5     master1   <none>           <none>
ks-apigateway-5664c4b76f-z8gp6         1/1     Running   0          2d14h   10.233.98.4     master2   <none>           <none>
ks-apiserver-75f468d48b-nzr7p          1/1     Running   0          2d14h   10.233.101.8    master0   <none>           <none>
ks-apiserver-75f468d48b-rp9dh          1/1     Running   0          2d14h   10.233.98.5     master2   <none>           <none>
ks-apiserver-75f468d48b-wzgb2          1/1     Running   0          2d14h   10.233.97.6     master1   <none>           <none>
ks-console-78bddc5bfb-7m95c            1/1     Running   0          2d14h   10.233.98.8     master2   <none>           <none>
ks-console-78bddc5bfb-8sg6b            1/1     Running   0          2d14h   10.233.101.11   master0   <none>           <none>
ks-console-78bddc5bfb-vzgzb            1/1     Running   0          2d14h   10.233.97.9     master1   <none>           <none>
ks-controller-manager-d4788677-84lmt   1/1     Running   1          2d14h   10.233.98.7     master2   <none>           <none>
ks-controller-manager-d4788677-lpln7   1/1     Running   0          2d14h   10.233.97.8     master1   <none>           <none>
ks-controller-manager-d4788677-rfbt4   1/1     Running   0          2d14h   10.233.101.10   master0   <none>           <none>

                kubectl -n kubesphere-system logs -l app=ks-account
                kubectl -n kubesphere-system logs -l app=ks-apigateway

                看看这两个组件的日志

                8 天 后

                kubectl -n kubesphere-system logs -l app=ks-account
                W0504 05:56:30.792621 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
                I0504 05:56:31.412132 1 server.go:113] Server listening on 0.0.0.0:9090
                W0506 02:11:47.383782 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
                I0506 02:11:48.019245 1 server.go:113] Server listening on 0.0.0.0:9090
                E0506 13:57:22.452061 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                E0506 13:57:22.452061 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                E0506 13:57:22.452141 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                E0506 13:57:22.452080 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                E0506 13:57:22.452104 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                E0506 13:57:22.452344 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out
                W0506 15:47:19.855660 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
                I0506 15:47:20.482187 1 server.go:113] Server listening on 0.0.0.0:9090

                kubectl -n kubesphere-system logs -l app=ks-apigateway
                E0506 16:03:34.569208 1 authenticate.go:170] signature is invalid
                2020/05/06 16:03:34 Unauthorized,signature is invalid
                172.28.94.161 06/May/2020:16:03:34 +0000 GET /kapis/monitoring.kubesphere.io/v1alpha2/nodes?type=rank&metrics_filter=node_cpu_utilisation%7Cnode_cpu_usage%7Cnode_cpu_total%7Cnode_memory_utilisation%7Cnode_memory_usage_wo_cache%7Cnode_memory_total%7Cnode_disk_size_utilisation%7Cnode_disk_size_usage%7Cnode_disk_size_capacity%7Cnode_pod_utilisation%7Cnode_pod_running_count%7Cnode_pod_quota%7Cnode_disk_inode_utilisation%7Cnode_disk_inode_total%7Cnode_disk_inode_usage%7Cnode_load1%24&page=1&limit=5&sort_type=desc&sort_metric=node_cpu_utilisation HTTP/1.1 401 17 0ms
                E0506 16:03:34.617070 1 authenticate.go:170] signature is invalid
                2020/05/06 16:03:34 Unauthorized,signature is invalid
                172.28.92.19 06/May/2020:16:03:46 +0000 POST /kapis/iam.kubesphere.io/v1alpha2/login HTTP/1.1 200 194 5ms
                172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/v1alpha1/configz HTTP/1.1 200 251 1ms
                E0506 16:03:51.837591 1 authenticate.go:170] signature is invalid
                2020/05/06 16:03:51 Unauthorized,signature is invalid
                172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms
                172.28.88.139 06/May/2020:15:52:05 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms
                172.28.92.19 06/May/2020:16:03:33 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 200 1137 15ms
                172.28.92.19 06/May/2020:16:03:34 +0000 GET /kapis/monitoring.kubesphere.io/v1alpha2/cluster?type=statistics HTTP/1.1 200 904 27ms
                E0506 16:03:46.918451 1 authenticate.go:170] signature is invalid
                2020/05/06 16:03:46 Unauthorized,signature is invalid
                172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 401 17 0ms
                172.28.92.19 06/May/2020:16:03:51 +0000 POST /kapis/iam.kubesphere.io/v1alpha2/login HTTP/1.1 200 194 9ms
                172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/v1alpha1/configz HTTP/1.1 200 251 3ms
                2020/05/06 16:11:08 [INFO][cache:0xc00012bb80] Scanning for stale OCSP staples
                2020/05/06 16:11:08 [INFO][cache:0xc00012bb80] Done checking OCSP staples
                2020/05/06 15:56:07 [INFO][cache:0xc00013b860] Scanning for stale OCSP staples
                2020/05/06 15:56:07 [INFO][cache:0xc00013b860] Done checking OCSP staples
                2020/05/06 15:57:01 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/ce64b5fd-2b91-4e0f-91c2-93342b835dad: dial tcp: lookup telemetry.caddyserver.com on 172.28.96.2:53: no such host
                172.28.92.19 06/May/2020:16:03:33 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 200 1590 12ms
                172.28.94.161 06/May/2020:16:03:34 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/devopscount/ HTTP/1.1 200 15 5ms
                172.28.92.19 06/May/2020:16:03:34 +0000 GET /kapis/resources.kubesphere.io/v1alpha2/componenthealth HTTP/1.1 200 14896 5ms
                172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms
                E0506 16:03:46.917688 1 authenticate.go:170] signature is invalid
                2020/05/06 16:03:46 Unauthorized,signature is invalid
                172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 200 1137 6ms

                  lvelvis

                  172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms
E0506 16:03:46.917688 1 authenticate.go:170] signature is invalid
2020/05/06 16:03:46 Unauthorized,signature is invalid

                  可以看到是因为签名校验失败导致登出，是不是ks-console前还有代理呢？ 直接用 30880 这个node port 试试

                    现在用的就是nodeport方式直接访问的
                    ks-console NodePort 172.28.96.39 <none> 80:30880/TCP 2d19h

                      @leoendless 这个问题需要看看，我也遇到过，而且有的时候点几次这个弹出窗口。

                      lvelvis 我看 ks-apigateway 和 ks-account有过更新，但是还处于pending状态， 可以检查一下是什么原因，确保这两个组件的pod中挂载的环境变量JWT_SECRET 是一致的，可以把副本数调整为1，快速的定位一下问题。

                        我已经重新安装了下，暂时恢复了，但是这个问题过不了多久又会重现 基本一周出现2次 稍后再出现我再调下副本数

                        最好是找到问题出在哪个环节，日志中已经有明确的错误， 再往下排查就很快了

                        1 个月 后

                        遇到同样的问题， 我是单节点部署的k8s，这5个pod每隔几分钟总是重启

                        求大神指点