创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
操作系统信息
4台实体服务器,版本 UBUNTU 20.04 Server
Kubernetes版本信息
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
容器运行时
Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:02:56 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:02 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0
KubeSphere版本信息
3.3.0
问题是什么
我们的 kubesphere 集群版本为 3.3.0 一直正常运转了几个月,今天为了添加一个节点,通过使用命令:
./kk add nodes -f sample.yaml 出现一下错误,导致部署没法进行下去,不知道如何去查找这个问题的根本原因。目前 ETCD 是三个master 节点。k8s 里的服务目前工作正常。只是 kubesphere 名称空间的下的服务都不在了。我们如何定位和解决这个问题?
downloading image: calico/pod2daemon-flexvol:v3.20.0
11:06:45 CST success: [k8s04]
11:06:45 CST success: [k8s03]
11:06:45 CST success: [k8s02]
11:06:45 CST success: [k8s01]
11:06:45 CST [ETCDPreCheckModule] Get etcd status
11:06:45 CST success: [k8s01]
11:06:45 CST success: [k8s02]
11:06:45 CST success: [k8s03]
11:06:45 CST [CertsModule] Fetch etcd certs
11:06:45 CST success: [k8s01]
11:06:45 CST skipped: [k8s02]
11:06:45 CST skipped: [k8s03]
11:06:45 CST [CertsModule] Generate etcd Certs
[certs] Using existing ca certificate authority
[certs] Using existing admin-k8s01 certificate and key on disk
[certs] Using existing member-k8s01 certificate and key on disk
[certs] Using existing node-k8s01 certificate and key on disk
[certs] Using existing admin-k8s02 certificate and key on disk
[certs] Using existing member-k8s02 certificate and key on disk
11:06:45 CST message: [LocalHost]
[certs] certificate node-k8s02 not signed by CA certificate ca: x509: certificate signed by unknown authority
11:06:45 CST failed: [LocalHost]
error: Pipeline[AddNodesPipeline] execute failed: Module[CertsModule] exec failed:
failed: [LocalHost] [GenerateETCDCerts] exec failed after 1 retires: [certs] certificate node-k8s02 not signed by CA certificate ca: x509: certificate signed by unknown authority
Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:02:56 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:02 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0