登录验证成功后，反复提示“会话超时或此账户在其它地方登录，请重新登录”

lvelvis

k8s集群从v1.15.10升级到v1.16.9后，登录提示如上，kube-apiserver已经开启了参数–enable-aggregator-routing=true，登录还是一样
组件状态正常
NAME READY STATUS RESTARTS AGE etcd-b7959ccd7-2xbsw 1/1 Running 1 21h ks-account-77bcff7cd6-rpmpc 0/1 Pending 0 10h ks-account-845d86f776-rkr97 1/1 Running 1 21h ks-account-c575d8996-qf5bg 1/1 Running 1 21h ks-account-c575d8996-r7vlx 1/1 Running 1 21h ks-apigateway-596b6c5f7-kdmqb 1/1 Running 1 21h ks-apigateway-7584f9f645-s6p2q 1/1 Running 5 21h ks-apigateway-779d95c6b5-pgtzd 1/1 Running 2 21h ks-apigateway-7d46fb6c86-qshg4 0/1 Pending 0 10h ks-apiserver-55588b79cb-9grfj 1/1 Running 1 21h ks-apiserver-56cb4b45d4-2jlv4 1/1 Running 1 41h ks-apiserver-5dff5c594d-2lwlz 1/1 Running 1 22h ks-apiserver-84c8878c99-g5kl9 0/1 Pending 0 10h ks-console-59fb8cfc59-78kpx 1/1 Running 1 21h ks-console-5c666c644-gp496 1/1 Running 1 22h ks-console-655877bd76-rrq68 0/1 Pending 0 10h ks-console-7df8d5b947-75ts9 1/1 Running 0 11h ks-controller-manager-66cbf8f97c-lhljn 0/1 Pending 0 10h ks-controller-manager-6cf6466594-vvmpd 1/1 Running 1 21h ks-controller-manager-bbb7b649b-l9qlz 1/1 Running 1 41h ks-controller-manager-d84487545-g7cl2 1/1 Running 1 21h ks-installer-75d9d66745-75jxm 1/1 Running 4 21h minio-8cd46c8d9-bwhpd 1/1 Running 1 21h mysql-b5597d996-kwh5r 1/1 Running 3 21h openldap-0 1/1 Running 0 12m openldap-1 1/1 Running 0 12m redis-ha-haproxy-75776f44c4-c5bvg 1/1 Running 1 21h redis-ha-haproxy-75776f44c4-t274v 1/1 Running 1 21h redis-ha-haproxy-75776f44c4-xwn7x 1/1 Running 1 21h redis-ha-server-0 2/2 Running 2 21h redis-ha-server-1 2/2 Running 2 21h redis-ha-server-2 2/2 Running 2 41h

wnxn

lvelvis 你好，你的集群是几个 master 的呢？

lvelvis

wnxn 三个master节点，一个slave节点，参与工作的节点有2个master+1个slave

hudcan

+1，碰到过。。。

tzghost

碰到过，我之前开启防火墙，按文档开放了端口，就出现了这个问题。关闭防火墙后恢复

lvelvis

tzghost 我的节点iptables firewall都关闭掉了

wnxn

lvelvis 参与工作的节点有2个master，还有1个master是不能工作吗？

lvelvis

wnxn 那个master设置了不参与pods调度

lvelvis

5ea6c16e conn=1367 op=7 BIND anonymous mech=implicit ssf=0 5ea6c16e conn=1367 op=7 BIND dn="cn=admin,dc=kubesphere,dc=io" method=128 5ea6c16e conn=1367 op=7 BIND dn="cn=admin,dc=kubesphere,dc=io" mech=SIMPLE ssf=0 5ea6c16e conn=1367 op=7 RESULT tag=97 err=0 text= 5ea6c16e conn=1367 op=8 SRCH base="ou=Users,dc=kubesphere,dc=io" scope=2 deref=0 filter="(&(objectClass=inetOrgPerson)(|(uid=admin)(mail=admin)))" 5ea6c16e conn=1367 op=8 SRCH attr=uid mail 5ea6c16e conn=1367 op=8 SEARCH RESULT tag=101 err=0 nentries=1 text= 5ea6c16e conn=1367 op=9 BIND anonymous mech=implicit ssf=0 5ea6c16e conn=1367 op=9 BIND dn="uid=admin,ou=Users,dc=kubesphere,dc=io" method=128 5ea6c16e conn=1367 op=9 BIND dn="uid=admin,ou=Users,dc=kubesphere,dc=io" mech=SIMPLE ssf=0 5ea6c16e conn=1367 op=9 RESULT tag=97 err=0 text= 5ea6c173 conn=1452 fd=19 ACCEPT from IP=172.28.161.107:52250 (IP=0.0.0.0:389) 5ea6c173 conn=1452 fd=19 closed (connection lost) 5ea6c17d conn=1453 fd=19 ACCEPT from IP=172.28.161.107:52596 (IP=0.0.0.0:389) 5ea6c17d conn=1453 fd=19 closed (connection lost)
怀疑是不是连接自动断开了，以上是openldap日志

Forest-L

lvelvis pending状态的pod可以删掉嘛？

lvelvis

Forest-L 那个没有影响的只是有一个master节点设置了禁止分配pods运行

Forest-L

lvelvis 像三master部署情况，ks核心模块只调度到master节点上。不建议master节点部署业务应用。

[root@master0 ~]# kubectl get pod -n kubesphere-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
etcd-f988bdb6f-c2klg                   1/1     Running   0          2d14h   10.233.96.5     node2     <none>           <none>
ks-account-7bf75c8b6-2qk5t             1/1     Running   0          2d14h   10.233.98.6     master2   <none>           <none>
ks-account-7bf75c8b6-8glfh             1/1     Running   0          2d14h   10.233.101.9    master0   <none>           <none>
ks-account-7bf75c8b6-czlfk             1/1     Running   0          2d14h   10.233.97.7     master1   <none>           <none>
ks-apigateway-5664c4b76f-gz57c         1/1     Running   0          2d14h   10.233.101.7    master0   <none>           <none>
ks-apigateway-5664c4b76f-hgm5m         1/1     Running   0          2d14h   10.233.97.5     master1   <none>           <none>
ks-apigateway-5664c4b76f-z8gp6         1/1     Running   0          2d14h   10.233.98.4     master2   <none>           <none>
ks-apiserver-75f468d48b-nzr7p          1/1     Running   0          2d14h   10.233.101.8    master0   <none>           <none>
ks-apiserver-75f468d48b-rp9dh          1/1     Running   0          2d14h   10.233.98.5     master2   <none>           <none>
ks-apiserver-75f468d48b-wzgb2          1/1     Running   0          2d14h   10.233.97.6     master1   <none>           <none>
ks-console-78bddc5bfb-7m95c            1/1     Running   0          2d14h   10.233.98.8     master2   <none>           <none>
ks-console-78bddc5bfb-8sg6b            1/1     Running   0          2d14h   10.233.101.11   master0   <none>           <none>
ks-console-78bddc5bfb-vzgzb            1/1     Running   0          2d14h   10.233.97.9     master1   <none>           <none>
ks-controller-manager-d4788677-84lmt   1/1     Running   1          2d14h   10.233.98.7     master2   <none>           <none>
ks-controller-manager-d4788677-lpln7   1/1     Running   0          2d14h   10.233.97.8     master1   <none>           <none>
ks-controller-manager-d4788677-rfbt4   1/1     Running   0          2d14h   10.233.101.10   master0   <none>           <none>

hongming

kubectl -n kubesphere-system logs -l app=ks-account
kubectl -n kubesphere-system logs -l app=ks-apigateway

看看这两个组件的日志

lvelvis

kubectl -n kubesphere-system logs -l app=ks-account
W0504 05:56:30.792621 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0504 05:56:31.412132 1 server.go:113] Server listening on 0.0.0.0:9090 W0506 02:11:47.383782 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0506 02:11:48.019245 1 server.go:113] Server listening on 0.0.0.0:9090 E0506 13:57:22.452061 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out E0506 13:57:22.452061 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out E0506 13:57:22.452141 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out E0506 13:57:22.452080 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out E0506 13:57:22.452104 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out E0506 13:57:22.452344 1 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 172.28.94.158:48528->172.28.96.1:443: read: connection timed out W0506 15:47:19.855660 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0506 15:47:20.482187 1 server.go:113] Server listening on 0.0.0.0:9090

kubectl -n kubesphere-system logs -l app=ks-apigateway
E0506 16:03:34.569208 1 authenticate.go:170] signature is invalid 2020/05/06 16:03:34 Unauthorized,signature is invalid 172.28.94.161 06/May/2020:16:03:34 +0000 GET /kapis/monitoring.kubesphere.io/v1alpha2/nodes?type=rank&metrics_filter=node_cpu_utilisation%7Cnode_cpu_usage%7Cnode_cpu_total%7Cnode_memory_utilisation%7Cnode_memory_usage_wo_cache%7Cnode_memory_total%7Cnode_disk_size_utilisation%7Cnode_disk_size_usage%7Cnode_disk_size_capacity%7Cnode_pod_utilisation%7Cnode_pod_running_count%7Cnode_pod_quota%7Cnode_disk_inode_utilisation%7Cnode_disk_inode_total%7Cnode_disk_inode_usage%7Cnode_load1%24&page=1&limit=5&sort_type=desc&sort_metric=node_cpu_utilisation HTTP/1.1 401 17 0ms E0506 16:03:34.617070 1 authenticate.go:170] signature is invalid 2020/05/06 16:03:34 Unauthorized,signature is invalid 172.28.92.19 06/May/2020:16:03:46 +0000 POST /kapis/iam.kubesphere.io/v1alpha2/login HTTP/1.1 200 194 5ms 172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/v1alpha1/configz HTTP/1.1 200 251 1ms E0506 16:03:51.837591 1 authenticate.go:170] signature is invalid 2020/05/06 16:03:51 Unauthorized,signature is invalid 172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms 172.28.88.139 06/May/2020:15:52:05 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms 172.28.92.19 06/May/2020:16:03:33 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 200 1137 15ms 172.28.92.19 06/May/2020:16:03:34 +0000 GET /kapis/monitoring.kubesphere.io/v1alpha2/cluster?type=statistics HTTP/1.1 200 904 27ms E0506 16:03:46.918451 1 authenticate.go:170] signature is invalid 2020/05/06 16:03:46 Unauthorized,signature is invalid 172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 401 17 0ms 172.28.92.19 06/May/2020:16:03:51 +0000 POST /kapis/iam.kubesphere.io/v1alpha2/login HTTP/1.1 200 194 9ms 172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/v1alpha1/configz HTTP/1.1 200 251 3ms 2020/05/06 16:11:08 [INFO][cache:0xc00012bb80] Scanning for stale OCSP staples 2020/05/06 16:11:08 [INFO][cache:0xc00012bb80] Done checking OCSP staples 2020/05/06 15:56:07 [INFO][cache:0xc00013b860] Scanning for stale OCSP staples 2020/05/06 15:56:07 [INFO][cache:0xc00013b860] Done checking OCSP staples 2020/05/06 15:57:01 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/ce64b5fd-2b91-4e0f-91c2-93342b835dad: dial tcp: lookup telemetry.caddyserver.com on 172.28.96.2:53: no such host 172.28.92.19 06/May/2020:16:03:33 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 200 1590 12ms 172.28.94.161 06/May/2020:16:03:34 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/devopscount/ HTTP/1.1 200 15 5ms 172.28.92.19 06/May/2020:16:03:34 +0000 GET /kapis/resources.kubesphere.io/v1alpha2/componenthealth HTTP/1.1 200 14896 5ms 172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms E0506 16:03:46.917688 1 authenticate.go:170] signature is invalid 2020/05/06 16:03:46 Unauthorized,signature is invalid 172.28.92.19 06/May/2020:16:03:51 +0000 GET /kapis/iam.kubesphere.io/v1alpha2/users/admin HTTP/1.1 200 1137 6ms

hongming

lvelvis

172.28.92.19 06/May/2020:16:03:46 +0000 GET /kapis/tenant.kubesphere.io/v1alpha2/workspaces HTTP/1.1 401 17 0ms
E0506 16:03:46.917688 1 authenticate.go:170] signature is invalid
2020/05/06 16:03:46 Unauthorized,signature is invalid

可以看到是因为签名校验失败导致登出，是不是ks-console前还有代理呢？直接用 30880 这个node port 试试

lvelvis

现在用的就是nodeport方式直接访问的
ks-console NodePort 172.28.96.39 <none> 80:30880/TCP 2d19h

hongming

lvelvis 我看 ks-apigateway 和 ks-account有过更新，但是还处于pending状态，可以检查一下是什么原因，确保这两个组件的pod中挂载的环境变量JWT_SECRET 是一致的，可以把副本数调整为1，快速的定位一下问题。

rayzhou2017

@leoendless 这个问题需要看看，我也遇到过，而且有的时候点几次这个弹出窗口。

lvelvis

我已经重新安装了下，暂时恢复了，但是这个问题过不了多久又会重现基本一周出现2次稍后再出现我再调下副本数

hongming

最好是找到问题出在哪个环节，日志中已经有明确的错误，再往下排查就很快了