ks v3.0.0 服务暴露突然Internal Server Error

Mm18684686660 · 2020年10月27日

Jeff kube apiserver logs

E1026 19:15:21.573985 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 2h4m52s

E1026 19:15:21.642471 1 jwt.go:51] token is expired by 2h4m52s

E1026 19:15:21.642510 1 token.go:57] token is expired by 2h4m52s

E1026 19:15:21.642520 1 jwt_token.go:45] token is expired by 2h4m52s

E1026 19:15:21.642560 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 2h4m52s

E1026 19:15:21.659974 1 jwt.go:51] token is expired by 2h4m52s

E1026 19:15:21.660003 1 token.go:57] token is expired by 2h4m52s

E1026 19:15:21.660011 1 jwt_token.go:45] token is expired by 2h4m52s

E1026 19:15:21.660022 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 2h4m52s

I1026 19:15:32.277977 1 apiserver.go:539] 192.168.150.109 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-devfk7fl/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

I1026 19:17:03.768523 1 apiserver.go:539] 192.168.150.109 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-devfk7fl/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

I1026 19:55:01.810525 1 apiserver.go:539] 192.168.150.109 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-devfk7fl/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

I1026 20:11:10.427621 1 apiserver.go:539] 10.233.69.183 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-prod47jz4/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

E1026 20:19:38.547659 1 jwt.go:51] token is expired by 1h55m49s

E1026 20:19:38.547681 1 token.go:57] token is expired by 1h55m49s

E1026 20:19:38.547686 1 jwt_token.go:45] token is expired by 1h55m49s

E1026 20:19:38.547693 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 1h55m49s

E1026 20:19:38.550826 1 jwt.go:51] token is expired by 1h55m49s

E1026 20:19:38.550842 1 token.go:57] token is expired by 1h55m49s

E1026 20:19:38.550846 1 jwt_token.go:45] token is expired by 1h55m49s

E1026 20:19:38.550851 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 1h55m49s

E1026 20:19:38.562989 1 jwt.go:51] token is expired by 1h55m49s

E1026 20:19:38.563023 1 token.go:57] token is expired by 1h55m49s

E1026 20:19:38.563031 1 jwt_token.go:45] token is expired by 1h55m49s

E1026 20:19:38.563039 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 1h55m49s

E1026 20:19:38.603592 1 jwt.go:51] token is expired by 1h55m49s

E1026 20:19:38.603623 1 token.go:57] token is expired by 1h55m49s

E1026 20:19:38.603632 1 jwt_token.go:45] token is expired by 1h55m49s

E1026 20:19:38.603645 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 1h55m49s

E1026 20:19:38.612797 1 jwt.go:51] token is expired by 1h55m49s

E1026 20:19:38.612830 1 token.go:57] token is expired by 1h55m49s

E1026 20:19:38.612840 1 jwt_token.go:45] token is expired by 1h55m49s

E1026 20:19:38.612853 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 1h55m49s

I1026 20:19:43.771935 1 apiserver.go:539] 10.233.69.183 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-devfk7fl/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

E1026 20:38:13.025157 1 jwt.go:51] token is expired by 22m59s

E1026 20:38:13.025196 1 token.go:57] token is expired by 22m59s

E1026 20:38:13.025207 1 jwt_token.go:45] token is expired by 22m59s

E1026 20:38:13.025220 1 authentication.go:60] Unable to authenticate the request due to error: token is expired by 22m59s

I1026 20:38:24.034595 1 apiserver.go:539] 192.168.150.109 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/sino-platform-devfk7fl/pipelines/sino-service/sonarstatus HTTP/1.1" 404 19 0ms

Mm18684686660 · 2020年10月27日

kube-apiserver-kspmaster logs
W1026 19:53:44.114199 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 20:15:18.046992 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 20:33:45.088126 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 20:53:40.080873 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 21:14:16.984523 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

I1026 21:40:28.482287 1 trace.go:116] Trace[707912963]: “Create” url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:manager/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.233.69.180 (started: 2020-10-26 21:40:26.866801779 +0000 UTC m=+468141.862376848) (total time: 1.615437424s):

Trace[707912963]: [1.60112893s] [1.60112893s] About to convert to expected version

I1026 21:40:37.072139 1 trace.go:116] Trace[2115158917]: “Update” url:/apis/apps/v1/namespaces/kubesphere-logging-system/daemonsets/fluent-bit,user-agent:manager/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.233.69.180 (started: 2020-10-26 21:40:35.465206249 +0000 UTC m=+468150.460781392) (total time: 1.606869249s):

Trace[2115158917]: [1.600460047s] [1.600460047s] About to convert to expected version

I1026 21:40:51.377238 1 trace.go:116] Trace[103179630]: “Update” url:/apis/apps/v1/namespaces/kubesphere-logging-system/daemonsets/fluent-bit,user-agent:manager/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.233.69.180 (started: 2020-10-26 21:40:50.663471605 +0000 UTC m=+468165.659046700) (total time: 713.711484ms):

Trace[103179630]: [707.787554ms] [707.787554ms] About to convert to expected version

I1026 21:42:43.780075 1 trace.go:116] Trace[1929312456]: “Create” url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:manager/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.233.69.180 (started: 2020-10-26 21:42:40.366607561 +0000 UTC m=+468275.362182648) (total time: 3.413399356s):

Trace[1929312456]: [3.399308974s] [3.399308974s] About to convert to expected version

I1026 21:42:57.973364 1 trace.go:116] Trace[1374841154]: “Update” url:/apis/apps/v1/namespaces/kubesphere-logging-system/daemonsets/fluent-bit,user-agent:manager/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.233.69.180 (started: 2020-10-26 21:42:56.868337965 +0000 UTC m=+468291.863913067) (total time: 1.104965172s):

Trace[1374841154]: [1.098875204s] [1.098875204s] About to convert to expected version

W1026 22:05:27.504349 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 22:26:09.756407 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 22:33:36.697370 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 22:43:21.941685 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 23:35:54.109192 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 23:44:38.700362 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1026 23:53:46.071298 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1027 00:15:22.980192 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1027 00:33:57.509020 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

E1027 00:44:40.554923 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:“unexpected EOF”}

W1027 00:46:31.490665 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

E1027 00:48:32.954185 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:“unexpected EOF”}

W1027 00:55:00.946833 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

W1027 01:15:20.279682 1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

yuswift · 2020年10月27日

看看etcd日志

Mm18684686660 · 2020年10月27日

yuswift

2020-10-21 11:39:51.474773 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.233.21.227

2020-10-21 11:39:51.478891 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.233.21.227:2379

2020-10-21 11:39:51.478903 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.233.21.227

2020-10-21 11:39:51.478924 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379

2020-10-21 11:39:51.478937 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379

2020-10-21 11:39:51.478960 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.233.21.227:2379

2020-10-21 11:39:51.478978 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_TCP=2379

2020-10-21 11:39:51.478995 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp

2020-10-21 11:39:51.479041 I | etcdmain: etcd Version: 3.2.18

2020-10-21 11:39:51.479056 I | etcdmain: Git SHA: eddf599c6

2020-10-21 11:39:51.479064 I | etcdmain: Go Version: go1.8.7

2020-10-21 11:39:51.479073 I | etcdmain: Go OS/Arch: linux/amd64

2020-10-21 11:39:51.479084 I | etcdmain: setting maximum number of CPUs to 16, total number of available CPUs is 16

2020-10-21 11:39:51.479147 N | etcdmain: the server is already initialized as member before, starting as etcd member…

2020-10-21 11:39:51.480257 I | embed: listening for peers on http://localhost:2380

2020-10-21 11:39:51.480354 I | embed: listening for client requests on 0.0.0.0:2379

2020-10-21 11:39:51.536601 I | etcdserver: name = default

2020-10-21 11:39:51.536625 I | etcdserver: data dir = /data

2020-10-21 11:39:51.536631 I | etcdserver: member dir = /data/member

2020-10-21 11:39:51.536637 I | etcdserver: heartbeat = 100ms

2020-10-21 11:39:51.536641 I | etcdserver: election = 1000ms

2020-10-21 11:39:51.536646 I | etcdserver: snapshot count = 100000

2020-10-21 11:39:51.536663 I | etcdserver: advertise client URLs = http://etcd.kubesphere-system.svc:2379

2020-10-21 11:39:51.578160 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 2623

2020-10-21 11:39:51.578410 I | raft: 8e9e05c52164694d became follower at term 5

2020-10-21 11:39:51.578435 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 5, commit: 2623, applied: 0, lastindex: 2623, lastterm: 5]

2020-10-21 11:39:51.583326 I | mvcc: restore compact to 1316

2020-10-21 11:39:51.585168 W | auth: simple token is not cryptographically signed

2020-10-21 11:39:51.586404 I | etcdserver: starting server… [version: 3.2.18, cluster version: to_be_decided]

2020-10-21 11:39:51.587195 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32

2020-10-21 11:39:51.587357 N | etcdserver/membership: set the initial cluster version to 3.2

2020-10-21 11:39:51.587413 I | etcdserver/api: enabled capabilities for version 3.2

2020-10-21 11:39:53.278874 I | raft: 8e9e05c52164694d is starting a new election at term 5

2020-10-21 11:39:53.278957 I | raft: 8e9e05c52164694d became candidate at term 6

2020-10-21 11:39:53.278981 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 6

2020-10-21 11:39:53.278996 I | raft: 8e9e05c52164694d became leader at term 6

2020-10-21 11:39:53.279004 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 6

2020-10-21 11:39:53.279412 I | etcdserver: published {Name:default ClientURLs:[http://etcd.kubesphere-system.svc:2379]} to cluster cdf818194e3a8c32

2020-10-21 11:39:53.279505 I | embed: ready to serve client requests

2020-10-21 11:39:53.279864 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!

2020-10-23 02:31:25.512767 W | etcdserver: apply entries took too long [782.482764ms for 1 entries]

2020-10-23 02:31:25.512804 W | etcdserver: avoid queries with large range/delete range!

2020-10-23 02:31:26.197609 W | etcdserver: apply entries took too long [570.467887ms for 1 entries]

2020-10-23 02:31:26.197647 W | etcdserver: avoid queries with large range/delete range!

Jeff · 2020年10月27日

m18684686660 你看下节点时间是不是有差异

Mm18684686660 · 2020年10月27日

Jeff
pod时间和节点系统时间确实有8小时的差异，应该是时区设置问题

Jeff · 2020年10月27日

m18684686660 不是指pod和节点时间，是节点之间的时间，你看下集群每个节点的时间是不是一样的

Mm18684686660 · 2020年10月27日

Jeff 我这个是allinone

Jeff · 2020年10月27日

m18684686660 OK，你的Kubernetes版本是多少的，这个问题和和kube-apiserver连接有问题，你的环境怎么安装的呢。

Mm18684686660 · 2020年10月27日

Jeff
角色:
master, worker
操作系统:
CentOS Linux 7 (Core)
操作系统类型:
linux
内核版本:
3.10.0-1127.19.1.el7.x86_64
容器版本:
docker://19.3.8
Kubelet 版本:
v1.18.6
Kube-Proxy 版本:
v1.18.6
采用官方allinone在线安装脚本:

Jeff · 2020年10月27日

m18684686660 内核版本有点低，你升级下内核，然后重启下集群试下

Mm18684686660 · 2020年10月27日

Jeff 但是但是在此之前服务暴露都是可以的

ks v3.0.0 服务暴露突然Internal Server Error

Mm18684686660K零S

Mm18684686660K零S

yuswiftK零S

Mm18684686660K零S

JeffK零SK壹S

Mm18684686660K零S

JeffK零SK壹S

Mm18684686660K零S

JeffK零SK壹S

Mm18684686660K零S

JeffK零SK壹S

Mm18684686660K零S