kubesphere3.1.1安装失败

dongoewang

操作系统信息：
1 主机名：worker1
硬件配置：cpu：8核，内存：16G，硬盘：500G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.109

2 主机名：worker2
硬件配置：cpu：8核，内存：16G，硬盘：500G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.110

3 主机名：worker3
硬件配置：cpu：8核，内存：16G，硬盘：500G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.111

4 主机名：worker4
硬件配置：cpu：8核，内存：16G，硬盘：200G

系统：CentOS Linux release 7.7.1908 (Core)

  ip：192.168.126.112

5 主机名：worker5
硬件配置：cpu：8核，内存：16G，硬盘：200G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.113

6 主机名：worker6
硬件配置：cpu：8核，内存：16G，硬盘：200G
ip：192.168.126.114

7 主机名：worker7
硬件配置：cpu：8核，内存：16G，硬盘：500G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.115

8 主机名：master1
硬件配置：cpu：4核，内存：8G，硬盘：100G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.131
9 主机名：master2
硬件配置：cpu：4核，内存：8G，硬盘：100G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.132
10 主机名：master3
硬件配置：cpu：4核，内存：8G，硬盘：100G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.133
11 主机名：ha1(vip:192.168.126.146)
硬件配置：cpu：4核，内存：8G，硬盘：100G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.134
12 主机名：ha2
硬件配置：cpu：4核，内存：8G，硬盘：100G

系统：CentOS Linux release 7.7.1908 (Core)

ip：192.168.126.135

*Kubernetes版本信息：v1.20.6。多节点。

KubeSphere版本信息：v3.1.1。在线安装，全套安装。

安装提示日志：请参考安装提示日志

kubelet日志：请参考kubelet日志

系统监控日志：请参考系统监控日志

config-sample.yml
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:

{name: worker1, address: 192.168.126.109, internalAddress: 192.168.126.109, user: root, password: P@ssw0rd}
{name: worker2, address: 192.168.126.110, internalAddress: 192.168.126.110, user: root, password: P@ssw0rd}
{name: worker3, address: 192.168.126.111, internalAddress: 192.168.126.111, user: root, password: P@ssw0rd}
{name: worker4, address: 192.168.126.112, internalAddress: 192.168.126.112, user: root, password: P@ssw0rd}
{name: worker5, address: 192.168.126.113, internalAddress: 192.168.126.113, user: root, password: P@ssw0rd}
{name: worker6, address: 192.168.126.114, internalAddress: 192.168.126.114, user: root, password: P@ssw0rd}
{name: worker7, address: 192.168.126.115, internalAddress: 192.168.126.115, user: root, password: P@ssw0rd}
{name: master1, address: 192.168.126.131, internalAddress: 192.168.126.131, user: root, password: P@ssw0rd}
{name: master2, address: 192.168.126.132, internalAddress: 192.168.126.132, user: root, password: P@ssw0rd}
{name: master3, address: 192.168.126.133, internalAddress: 192.168.126.133, user: root, password: P@ssw0rd}
roleGroups:
etcd:
master1
master2
master3
master:
master1
master2
master3
worker:
worker1
worker2
worker3
worker4
worker5
worker6
worker7
controlPlaneEndpoint:
domain: lb.kubesphere.local
address: 192.168.126.146
port: 6443
kubernetes:
version: v1.20.4
imageRepo: kubesphere
clusterName: gyhl.k8s.com
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
registry:
registryMirrors: []
insecureRegistries: []
addons: []
#- name: glusterfs
namespace: kube-system
sources:
yaml:
path:

/root/glusterfs-sc.yaml
—
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.1.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
zone: ""
local_registry: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
redis:
enabled: false
redisVolumSize: 2Gi
openldap:
enabled: false
openldapVolumeSize: 2Gi
minioVolumeSize: 20Gi
monitoring:
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
es:
elasticsearchMasterVolumeSize: 4Gi
elasticsearchDataVolumeSize: 20Gi
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchUrl: ""
externalElasticsearchPort: ""
console:
enableMultiLogin: true
port: 30880
alerting:
enabled: false

thanosruler:
replicas: 1
resources: {}
auditing:
enabled: false
devops:
enabled: false
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: false
ruler:
enabled: true
replicas: 2
logging:
enabled: false
logsidecar:
enabled: true
replicas: 2
metrics_server:
enabled: false
monitoring:
storageClass: ""
prometheusMemoryRequest: 400Mi
prometheusVolumeSize: 20Gi
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: false
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
cloudhubPort: “10000”
cloudhubQuicPort: “10001”
cloudhubHttpsPort: “10002”
cloudstreamPort: “10003”
tunnelPort: “10004”
cloudHub:
advertiseAddress:

""
nodeLimit: “100”
service:
cloudhubNodePort: “30000”
cloudhubQuicNodePort: “30001”
cloudhubHttpsNodePort: “30002”
cloudstreamNodePort: “30003”
tunnelNodePort: “30004”
edgeWatcher:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []

平台部署前的检查信息

dongoewang

[root@master1 k8ssoft]# ./kk create cluster -f config-sample.yaml
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| master1 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker5 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker2 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker7 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker4 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker3 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker6 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
| worker1 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 14:15:26 |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+

This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes
INFO[14:15:29 CST] Downloading Installation Files
INFO[14:15:29 CST] Downloading kubeadm …
INFO[14:15:29 CST] Downloading kubelet …
INFO[14:15:29 CST] Downloading kubectl …
INFO[14:15:29 CST] Downloading helm …
INFO[14:15:30 CST] Downloading kubecni …
INFO[14:15:30 CST] Configuring operating system …
[worker6 192.168.126.114] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker2 192.168.126.110] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker5 192.168.126.113] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker1 192.168.126.109] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker3 192.168.126.111] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker7 192.168.126.115] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker4 192.168.126.112] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master1 192.168.126.131] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
INFO[14:15:31 CST] Installing docker …
INFO[14:15:31 CST] Start to download images on all nodes
[master1] Downloading image: kubesphere/etcd:v3.4.13
[worker1] Downloading image: kubesphere/pause:3.1
[worker2] Downloading image: kubesphere/pause:3.1
[worker3] Downloading image: kubesphere/pause:3.1
[worker4] Downloading image: kubesphere/pause:3.1
[worker5] Downloading image: kubesphere/pause:3.1
[worker6] Downloading image: kubesphere/pause:3.1
[worker7] Downloading image: kubesphere/pause:3.1
[worker4] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker1] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker7] Downloading image: kubesphere/kube-proxy:v1.17.9
[master1] Downloading image: kubesphere/pause:3.1
[worker2] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker5] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker3] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker6] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker1] Downloading image: coredns/coredns:1.6.9
[worker7] Downloading image: coredns/coredns:1.6.9
[worker4] Downloading image: coredns/coredns:1.6.9
[master1] Downloading image: kubesphere/kube-apiserver:v1.17.9
[worker2] Downloading image: coredns/coredns:1.6.9
[worker5] Downloading image: coredns/coredns:1.6.9
[worker3] Downloading image: coredns/coredns:1.6.9
[worker6] Downloading image: coredns/coredns:1.6.9
[worker4] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker1] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker7] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[master1] Downloading image: kubesphere/kube-controller-manager:v1.17.9
[worker2] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker5] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker3] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker6] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker1] Downloading image: calico/kube-controllers:v3.16.3
[master1] Downloading image: kubesphere/kube-scheduler:v1.17.9
[worker4] Downloading image: calico/kube-controllers:v3.16.3
[worker5] Downloading image: calico/kube-controllers:v3.16.3
[worker2] Downloading image: calico/kube-controllers:v3.16.3
[worker7] Downloading image: calico/kube-controllers:v3.16.3
[worker3] Downloading image: calico/kube-controllers:v3.16.3
[worker6] Downloading image: calico/kube-controllers:v3.16.3
[worker1] Downloading image: calico/cni:v3.16.3
[master1] Downloading image: kubesphere/kube-proxy:v1.17.9
[worker4] Downloading image: calico/cni:v3.16.3
[worker5] Downloading image: calico/cni:v3.16.3
[worker7] Downloading image: calico/cni:v3.16.3
[worker2] Downloading image: calico/cni:v3.16.3
[worker3] Downloading image: calico/cni:v3.16.3
[worker6] Downloading image: calico/cni:v3.16.3
[worker1] Downloading image: calico/node:v3.16.3
[worker4] Downloading image: calico/node:v3.16.3
[worker7] Downloading image: calico/node:v3.16.3
[master1] Downloading image: coredns/coredns:1.6.9
[worker5] Downloading image: calico/node:v3.16.3
[worker2] Downloading image: calico/node:v3.16.3
[worker3] Downloading image: calico/node:v3.16.3
[worker6] Downloading image: calico/node:v3.16.3
[worker1] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[worker4] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[worker7] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[master1] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[worker5] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[worker2] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[worker3] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[worker6] Downloading image: calico/pod2daemon-flexvol:v3.16.3
[master1] Downloading image: calico/kube-controllers:v3.16.3
[master1] Downloading image: calico/cni:v3.16.3
[master1] Downloading image: calico/node:v3.16.3
[master1] Downloading image: calico/pod2daemon-flexvol:v3.16.3
INFO[14:16:07 CST] Generating etcd certs
INFO[14:16:08 CST] Synchronizing etcd certs
INFO[14:16:08 CST] Creating etcd service
[master1 192.168.126.131] MSG:
etcd will be installed
INFO[14:16:09 CST] Starting etcd cluster
[master1 192.168.126.131] MSG:
Configuration file will be created
INFO[14:16:09 CST] Refreshing etcd configuration
Waiting for etcd to start
INFO[14:16:14 CST] Backup etcd data regularly
INFO[14:16:21 CST] Get cluster status
[master1 192.168.126.131] MSG:
Cluster will be created.
INFO[14:16:21 CST] Installing kube binaries
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.112:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.131:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.109:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.111:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.110:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.115:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.113:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubeadm to 192.168.126.114:/tmp/kubekey/kubeadm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.112:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.109:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.131:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.112:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.131:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.112:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.131:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.109:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.131:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.112:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.111:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.109:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.115:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.114:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.113:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubelet to 192.168.126.110:/tmp/kubekey/kubelet Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.109:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.111:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.115:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.113:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.114:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/kubectl to 192.168.126.110:/tmp/kubekey/kubectl Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.111:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.115:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.113:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.114:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/helm to 192.168.126.110:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.111:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.115:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.113:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.110:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /home/k8ssoft/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.114:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
INFO[14:16:25 CST] Initializing kubernetes cluster
[master1 192.168.126.131] MSG:
W0929 14:16:25.330839 32754 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0929 14:16:25.330976 32754 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0929 14:16:25.330980 32754 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.9
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local lb.kubesphere.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost lb.kubesphere.local worker1 worker1.cluster.local worker2 worker2.cluster.local worker3 worker3.cluster.local worker4 worker4.cluster.local worker5 worker5.cluster.local worker6 worker6.cluster.local worker7 worker7.cluster.local master1 master1.cluster.local] and IPs [10.233.0.1 192.168.126.131 127.0.0.1 192.168.126.131 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115 10.233.0.1]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[controlplane] Adding extra host path mount “host-time” to “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[controlplane] Adding extra host path mount “host-time” to “kube-controller-manager”
W0929 14:16:28.009868 32754 manifests.go:214] the default kube-apiserver authorization-mode is “Node,RBAC”; using “Node,RBAC”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[controlplane] Adding extra host path mount “host-time” to “kube-controller-manager”
W0929 14:16:28.010922 32754 manifests.go:214] the default kube-apiserver authorization-mode is “Node,RBAC”; using “Node,RBAC”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[apiclient] All control plane components are healthy after 34.501710 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config-1.17” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see –upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the label “node-role.kubernetes.io/master=''”
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: g6qwsy.qb1304k8adaobkdc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join lb.kubesphere.local:6443 –token g6qwsy.qb1304k8adaobkdc \
–discovery-token-ca-cert-hash sha256:4ac009087300dfe9cb9d02a89ecb16899952285f7670c27f0eb977574d3fb8bf \
–control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join lb.kubesphere.local:6443 –token g6qwsy.qb1304k8adaobkdc \
–discovery-token-ca-cert-hash sha256:4ac009087300dfe9cb9d02a89ecb16899952285f7670c27f0eb977574d3fb8bf
[master1 192.168.126.131] MSG:
service “kube-dns” deleted
[master1 192.168.126.131] MSG:
service/coredns created
[master1 192.168.126.131] MSG:
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
[master1 192.168.126.131] MSG:
configmap/nodelocaldns created
[master1 192.168.126.131] MSG:
I0929 14:17:25.823229 2321 version.go:251] remote version is much newer: v1.22.2; falling back to: stable-1.17
W0929 14:17:26.462848 2321 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0929 14:17:26.462860 2321 validation.go:28] Cannot validate kubelet config - no validator is available
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[upload-certs] Using certificate key:
8e48ec2794807714a370e63e7b002952b6922062ab15813a96ac4e0ebb471aab
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
W0929 14:17:26.796108 2900 validation.go:28] Cannot validate kubelet config - no validator is available
W0929 14:17:26.796147 2900 validation.go:28] Cannot validate kube-proxy config - no validator is available
kubeadm join lb.kubesphere.local:6443 –token 77f6dg.oi55azko4eg72ajs –discovery-token-ca-cert-hash sha256:4ac009087300dfe9cb9d02a89ecb16899952285f7670c27f0eb977574d3fb8bf
[master1 192.168.126.131] MSG:
master1 v1.17.9 [map[address:192.168.126.131 type:InternalIP] map[address:master1 type:Hostname]]
INFO[14:17:26 CST] Joining nodes to cluster
[worker4 192.168.126.112] MSG:
W0929 14:17:27.066553 4357 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.491951 4357 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker2 192.168.126.110] MSG:
W0929 14:17:27.110966 2613 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.555827 2613 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker3 192.168.126.111] MSG:
W0929 14:17:27.094385 6717 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.568640 6717 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker5 192.168.126.113] MSG:
W0929 14:17:27.162779 884 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.614047 884 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker6 192.168.126.114] MSG:
W0929 14:17:27.187209 21335 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.629238 21335 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker4 192.168.126.112] MSG:
node/worker4 labeled
[worker3 192.168.126.111] MSG:
node/worker3 labeled
[worker6 192.168.126.114] MSG:
node/worker6 labeled
[worker2 192.168.126.110] MSG:
node/worker2 labeled
[worker5 192.168.126.113] MSG:
node/worker5 labeled
[worker7 192.168.126.115] MSG:
W0929 14:17:27.215983 7532 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.686161 7532 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker1 192.168.126.109] MSG:
W0929 14:17:27.151066 1065 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W0929 14:17:27.815795 1065 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.17” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker7 192.168.126.115] MSG:
node/worker7 labeled
[worker1 192.168.126.109] MSG:
node/worker1 labeled
INFO[14:17:50 CST] Deploying network plugin …
[master1 192.168.126.131] MSG:
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
[master1 192.168.126.131] MSG:
storageclass.storage.k8s.io/local created
serviceaccount/openebs-maya-operator created
clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created
clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created
deployment.apps/openebs-localpv-provisioner created
INFO[14:17:51 CST] Deploying KubeSphere …
v3.0.0
[master1 192.168.126.131] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system created
[master1 192.168.126.131] MSG:
secret/kube-etcd-client-certs created
[master1 192.168.126.131] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer created
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io created
clusterrole.rbac.authorization.k8s.io/ks-installer created
clusterrolebinding.rbac.authorization.k8s.io/ks-installer created
deployment.apps/ks-installer created
clusterconfiguration.installer.kubesphere.io/ks-installer created

dongoewang

dockerps-a的显示信息

dongoewang

glusterfs-sc.yaml的配置信息
apiVersion: v1
kind: Secret
metadata:
name: heketi-secret
namespace: kube-system
type: kubernetes.io/glusterfs
data:

key: “MTIzNDU2” #请替换为您自己的密钥。Base64 编码。

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.beta.kubernetes.io/is-default-class: “true”
storageclass.kubesphere.io/supported-access-modes: ‘[“ReadWriteOnce”,“ReadOnlyMany”,“ReadWriteMany”]’
name: glusterfs
parameters:
clusterid: “a9ccba59b4db71c5b0a6db520274a668” #请替换为您自己的 GlusterFS 集群 ID。
gidMax: “50000”
gidMin: “40000”
restauthenabled: “true”
resturl: “http://192.168.126.143:8080” #Gluster REST 服务/Heketi 服务 URL 可按需供应 gluster 存储卷。请替换为您自己的 URL。
restuser: admin
secretName: “123456”
secretNamespace: kube-system
volumetype: “replicate:3” #请替换为您自己的存储卷类型。
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

24sama

在 kubekey 的日志中我没有看到失败的信息，看起来 k8s 已经安装成功了，但是 ks 没有安装成功。使用以下命令看看 ks-installer 的日志呢？

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

dongoewang

24sama
运行该命令，提示如下

24sama

我看你粘贴的日杂信息和你最开始的配置信息感觉有点出入，比如日志里面没有master2和master3的消息，所以不太确定是不是配置文件内容有问题。

可以把config-sample.yaml的内容截图发一张吗，直接负责粘贴过来的内容看不出来格式有没有问题

dongoewang

24sama

上边是所有的截图信息，最后一张图片的172行之后，就没有配置内容了

dongoewang

24sama
谢谢回复。出现这个问题是前天出现的，昨天我试过一个master节点，2个工作节点，出现过这个问题，也尝试过试过kubesphere的其他版本的信息3.10或者3.0.0，以及kubernetes版本我也切换过1.20.6,1.17.9这些版本也试过，可能再这些尝试的时候，贴的日志信息有问题。但是所有的尝试，都报的错误信息是 startup timeout（途中我还尝试，把glusterfs的配置去掉，尝试了很多，但报错的信息都是startup timeout）

24sama

dongoewang
我在你之前粘贴的日志中没有看到startup timeout的错误信息，可能你粘贴的日志内容有所缺失，这样可能我不太好帮你排查问题。

现在你可以尝试先删除干净集群环境：

./kk delete cluster -f config-sample.yaml

然后再重新装一下集群：

./kk create cluster -f config-samply.yaml

观察一下日志内容，关键的日志信息放到这个帖子里面再看看是啥问题。

dongoewang

24sama
好的，我之前想直接贴信息的，但是太大，就去掉了一些信息。我按照你说的命令，执行了删除，重新安装了，我把删除的截图的信息，以及安装的截图信息都贴到群里边。文字内容信息的话，在下边（可以直接搜索用）

dongoewang

24sama
删除以及安装的所有信息，可以参照上边的截图，下边的文字信息，删除掉了一些，删除掉的地方做了备注。

[root@master1 k8ssoft]# ./kk delete cluster -f config-sample.yaml
Are you sure to delete this cluster? [yes/no]: yes
INFO[10:55:46 CST] Resetting kubernetes cluster …
[worker3 192.168.126.111] MSG:
sudo -E /bin/sh -c "iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && ip link del kube-ipvs0 && ip link del
删除的信息被去掉了
INFO[10:55:48 CST] Successful.
[root@master1 k8ssoft]# ./kk create cluster -f config-sample.yaml
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| worker7 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| worker6 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| worker5 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| master2 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| worker1 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 10:57:34 |
| worker3 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:34 |
| master3 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| worker4 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 10:57:33 |
| worker2 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 10:57:34 |
| master1 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 10:57:34 |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+

Continue this installation? [yes/no]: yes
INFO[10:57:59 CST] Downloading Installation Files
INFO[10:57:59 CST] Downloading kubeadm …
INFO[10:58:00 CST] Downloading kubelet …
INFO[10:58:00 CST] Downloading kubectl …
INFO[10:58:00 CST] Downloading helm …
INFO[10:58:00 CST] Downloading kubecni …
INFO[10:58:00 CST] Configuring operating system …
[worker1 192.168.126.109] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker6 192.168.126.114] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker2 192.168.126.110] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker5 192.168.126.113] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker3 192.168.126.111] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker4 192.168.126.112] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master3 192.168.126.133] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker7 192.168.126.115] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master2 192.168.126.132] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master1 192.168.126.131] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
INFO[10:58:02 CST] Installing docker …
INFO[10:58:02 CST] Start to download images on all nodes
[worker2] Downloading image: kubesphere/pause:3.2
部分download信息删除
[master3] Downloading image: calico/pod2daemon-flexvol:v3.16.3
INFO[10:58:50 CST] Generating etcd certs
INFO[10:58:52 CST] Synchronizing etcd certs
INFO[10:58:52 CST] Creating etcd service
[master3 192.168.126.133] MSG:
etcd will be installed
[master2 192.168.126.132] MSG:
etcd will be installed
[master1 192.168.126.131] MSG:
etcd will be installed
INFO[10:58:53 CST] Starting etcd cluster
[master1 192.168.126.131] MSG:
Configuration file will be created
[master2 192.168.126.132] MSG:
Configuration file will be created
[master3 192.168.126.133] MSG:
Configuration file will be created
INFO[10:58:53 CST] Refreshing etcd configuration
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
INFO[10:58:59 CST] Backup etcd data regularly
INFO[10:59:05 CST] Get cluster status
[master1 192.168.126.131] MSG:
Cluster will be created.
INFO[10:59:05 CST] Installing kube binaries
Push /home/k8ssoft/kubekey/v1.20.4/amd64/kubeadm to 192.168.126.111:/tmp/kubekey/kubeadm Done
部分push信息删除
Push /home/k8ssoft/kubekey/v1.20.4/amd64/helm to 192.168.126.114:/tmp/kubekey/helm Done
Push /home/k8ssoft/kubekey/v1.20.4/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.114:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
INFO[10:59:10 CST] Initializing kubernetes cluster
[master1 192.168.126.131] MSG:
W0930 10:59:10.431138 8999 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.131 127.0.0.1 192.168.126.146 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115 192.168.126.132 192.168.126.133]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 60.516756 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config-1.20” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see –upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 92vn2n.pgd6ra7bl8y1kh1x
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join lb.kubesphere.local:6443 –token 92vn2n.pgd6ra7bl8y1kh1x \
–discovery-token-ca-cert-hash sha256:ee906f59243114fbb15d023c061badb3a7edbc957bd3bd76ba74be8d39fbee24 \
–control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join lb.kubesphere.local:6443 –token 92vn2n.pgd6ra7bl8y1kh1x \
–discovery-token-ca-cert-hash sha256:ee906f59243114fbb15d023c061badb3a7edbc957bd3bd76ba74be8d39fbee24
[master1 192.168.126.131] MSG:
service “kube-dns” deleted
[master1 192.168.126.131] MSG:
service/coredns created
[master1 192.168.126.131] MSG:
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
[master1 192.168.126.131] MSG:
configmap/nodelocaldns created
[master1 192.168.126.131] MSG:
I0930 11:00:39.136948 11263 version.go:254] remote version is much newer: v1.22.2; falling back to: stable-1.20
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[upload-certs] Using certificate key:
89062d88cf14519ab034e7b2d39a2cbf4ac4b0b9a923944a8530612d8304d03f
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
kubeadm join lb.kubesphere.local:6443 –token 2h0xm8.a4essk84o4w9j0ww –discovery-token-ca-cert-hash sha256:ee906f59243114fbb15d023c061badb3a7edbc957bd3bd76ba74be8d39fbee24
[master1 192.168.126.131] MSG:
master1 v1.20.4 [map[address:192.168.126.131 type:InternalIP] map[address:master1 type:Hostname]]
INFO[11:00:40 CST] Joining nodes to cluster
[worker2 192.168.126.110] MSG:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:40.974044 9903 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker4 192.168.126.112] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.030478 4267 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker3 192.168.126.111] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.072611 16763 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker6 192.168.126.114] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.076304 21496 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker2 192.168.126.110] MSG:
node/worker2 labeled
[worker4 192.168.126.112] MSG:
node/worker4 labeled
[worker6 192.168.126.114] MSG:
node/worker6 labeled
[worker3 192.168.126.111] MSG:
node/worker3 labeled
[worker1 192.168.126.109] MSG:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:40.937872 8241 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker5 192.168.126.113] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.010655 13941 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker7 192.168.126.115] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.094060 22520 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker1 192.168.126.109] MSG:
node/worker1 labeled
[worker5 192.168.126.113] MSG:
node/worker5 labeled
[worker7 192.168.126.115] MSG:
node/worker7 labeled
[master3 192.168.126.133] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:40.985975 1399 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[download-certs] Downloading the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.133 127.0.0.1 192.168.126.146 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115 192.168.126.131 192.168.126.132]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] Valid certificates and keys now exist in “/etc/kubernetes/pki”
[certs] Using the existing “sa” key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
[control-plane-join] using external etcd - no local stacked instance added
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[mark-control-plane] Marking the node master3 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

Certificate signing request was sent to apiserver and approval was received.
The Kubelet was informed of the new secure connection details.
Control plane (master) label and taint were applied to the new node.
The Kubernetes control plane instances scaled up.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.
[master2 192.168.126.132] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 11:00:41.140682 31132 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[download-certs] Downloading the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “front-proxy-client” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.132 127.0.0.1 192.168.126.146 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115 192.168.126.131 192.168.126.133]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Valid certificates and keys now exist in “/etc/kubernetes/pki”
[certs] Using the existing “sa” key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
[control-plane-join] using external etcd - no local stacked instance added
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[mark-control-plane] Marking the node master2 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

Certificate signing request was sent to apiserver and approval was received.
The Kubelet was informed of the new secure connection details.
Control plane (master) label and taint were applied to the new node.
The Kubernetes control plane instances scaled up.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.
INFO[11:00:50 CST] Deploying network plugin …
[master1 192.168.126.131] MSG:
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
INFO[11:00:52 CST] Installing addon [1-1]: glusterfs
secret/heketi-secret serverside-applied
storageclass.storage.k8s.io/glusterfs serverside-applied
INFO[11:00:52 CST] Deploying KubeSphere …
v3.1.1
[master1 192.168.126.131] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system created
[master1 192.168.126.131] MSG:
secret/kube-etcd-client-certs created
[master1 192.168.126.131] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
WARN[11:31:15 CST] Task failed …
WARN[11:31:15 CST] error: KubeSphere startup timeout.
Error: Failed to deploy kubesphere: KubeSphere startup timeout.
Usage:
kk create cluster [flags]

Flags:
–download-cmd string The user defined command to download the necessary binary files. The first param ‘%s’ is output path, the second param ‘%s’, is the URL (default “curl -L -o %s %s”)
-f, –filename string Path to a configuration file
-h, –help help for cluster
–skip-pull-images Skip pre pull images
–with-kubernetes string Specify a supported version of kubernetes (default “v1.19.8”)
–with-kubesphere Deploy a specific version of kubesphere (default v3.1.0)
–with-local-storage Deploy a local PV provisioner
-y, –yes Skip pre-check of the installation

Global Flags:
–debug Print detailed information (default true)
–in-cluster Running inside the cluster

Failed to deploy kubesphere: KubeSphere startup timeout.
You have new mail in /var/spool/mail/root
[root@master1 k8ssoft]#

24sama

dongoewang
这里可以看到是 ks-installer 安装 kubesphere 失败了，k8s 集群已经起来了。
使用以下命令检查集群中节点的状态：

kubectl get node

在其中一个master节点使用命令检查一下有哪些pod没有running

kubectl get po -A

使用命令查看 ks-installer 的日志

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

24sama

dongoewang
我认为你可以尝试修改一下配置文件，把master节点放在所有节点列表的前面试试，示例如下：

apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master1, address: 192.168.126.131, internalAddress: 192.168.126.131, user: root, password: P@ssw0rd}
  - {name: master2, address: 192.168.126.132, internalAddress: 192.168.126.132, user: root, password: P@ssw0rd}
  - {name: master3, address: 192.168.126.133, internalAddress: 192.168.126.133, user: root, password: P@ssw0rd}
  - {name: worker1, address: 192.168.126.109, internalAddress: 192.168.126.109, user: root, password: P@ssw0rd}
  - {name: worker2, address: 192.168.126.110, internalAddress: 192.168.126.110, user: root, password: P@ssw0rd}
  - {name: worker3, address: 192.168.126.111, internalAddress: 192.168.126.111, user: root, password: P@ssw0rd}
  - {name: worker4, address: 192.168.126.112, internalAddress: 192.168.126.112, user: root, password: P@ssw0rd}
  - {name: worker5, address: 192.168.126.113, internalAddress: 192.168.126.113, user: root, password: P@ssw0rd}
  - {name: worker6, address: 192.168.126.114, internalAddress: 192.168.126.114, user: root, password: P@ssw0rd}
  - {name: worker7, address: 192.168.126.115, internalAddress: 192.168.126.115, user: root, password: P@ssw0rd}

因为master节点加入集群时，会在集群控制平面添加证书、建立新的连接等，可能增加集群达到可用状态的时间造成安装超时。
所以我认为可以尝试更改一下 kk 配置文件的顺序，让master节点先加入到集群。
修改配置文件后，还是按照之前的操作删除集群，再创建集群。

dongoewang

24sama
执行命令如下（感觉没有装好，正常装好了之后，命令应该是可以运行的）：

24sama

dongoewang

ls /usr/local/bin/

看下kubectl命令有没有，或者ssh到其他master机器执行一下命令
kubectl 是装到/usr/local/bin/下的，可能这个目录不在你机器的$PATH中，你可以试着加一下。

dongoewang

24sama
下边的信息，是按照上边comment中执行的结果。最后的执行结果，安装失败，错误信息是：startout timeout。（后边我再把执行结果的截图发一下）

[root@master1 k8ssoft]# ./kk delete cluster -f config-sample.yaml
Are you sure to delete this cluster? [yes/no]: yes
INFO[13:37:58 CST] Resetting kubernetes cluster …
[master2 192.168.126.132] MSG:
sudo -E /bin/sh -c “iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && ip link del kube-ipvs0 && ip link del nodelocaldns”
之间的信息去掉一部分
[master1 192.168.126.131] MSG:
sudo -E /bin/sh -c “iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && ip link del kube-ipvs0 && ip link del nodelocaldns”
INFO[13:38:00 CST] Successful.
[root@master1 k8ssoft]# ./kk create cluster -f config-sample.yaml
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| master3 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| worker1 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 13:38:06 |
| worker4 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| worker3 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| master1 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 13:38:06 |
| worker7 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| worker2 | y | y | y | y | y | y | y | 19.03.9 | y | | y | CST 13:38:06 |
| master2 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| worker5 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
| worker6 | y | y | y | y | y | y | y | 20.10.8 | y | | y | CST 13:38:06 |
+———+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+

Continue this installation? [yes/no]: yes
INFO[13:38:08 CST] Downloading Installation Files
INFO[13:38:08 CST] Downloading kubeadm …
INFO[13:38:08 CST] Downloading kubelet …
INFO[13:38:09 CST] Downloading kubectl …
INFO[13:38:09 CST] Downloading helm …
INFO[13:38:09 CST] Downloading kubecni …
INFO[13:38:09 CST] Configuring operating system …
[worker6 192.168.126.114] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker3 192.168.126.111] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker2 192.168.126.110] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker7 192.168.126.115] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master1 192.168.126.131] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker5 192.168.126.113] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker4 192.168.126.112] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[worker1 192.168.126.109] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master2 192.168.126.132] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[master3 192.168.126.133] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
INFO[13:38:11 CST] Installing docker …
INFO[13:38:11 CST] Start to download images on all nodes
[worker7] Downloading image: kubesphere/pause:3.2
download信息去掉一部分
[master2] Downloading image: calico/pod2daemon-flexvol:v3.16.3
INFO[13:40:29 CST] Generating etcd certs
INFO[13:40:32 CST] Synchronizing etcd certs
INFO[13:40:32 CST] Creating etcd service
[master3 192.168.126.133] MSG:
etcd will be installed
[master2 192.168.126.132] MSG:
etcd will be installed
[master1 192.168.126.131] MSG:
etcd will be installed
INFO[13:40:33 CST] Starting etcd cluster
[master1 192.168.126.131] MSG:
Configuration file will be created
[master2 192.168.126.132] MSG:
Configuration file will be created
[master3 192.168.126.133] MSG:
Configuration file will be created
INFO[13:40:33 CST] Refreshing etcd configuration
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
INFO[13:40:39 CST] Backup etcd data regularly
INFO[13:40:45 CST] Get cluster status
[master1 192.168.126.131] MSG:
Cluster will be created.
INFO[13:40:45 CST] Installing kube binaries
Push /home/k8ssoft/kubekey/v1.20.4/amd64/kubeadm to 192.168.126.115:/tmp/kubekey/kubeadm Done
push的信息去掉一部分
Push /home/k8ssoft/kubekey/v1.20.4/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.126.114:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
INFO[13:40:50 CST] Initializing kubernetes cluster
[master1 192.168.126.131] MSG:
W0930 13:40:50.468312 3607 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.131 127.0.0.1 192.168.126.146 192.168.126.132 192.168.126.133 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 61.423479 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config-1.20” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see –upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: emq1j5.0dza5ko1os5puz1o
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join lb.kubesphere.local:6443 –token emq1j5.0dza5ko1os5puz1o \
–discovery-token-ca-cert-hash sha256:cda678e4402a753e7491bd209e209605e10ddd1d801f0cf32dd1f6c4196df95f \
–control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join lb.kubesphere.local:6443 –token emq1j5.0dza5ko1os5puz1o \
–discovery-token-ca-cert-hash sha256:cda678e4402a753e7491bd209e209605e10ddd1d801f0cf32dd1f6c4196df95f
[master1 192.168.126.131] MSG:
service “kube-dns” deleted
[master1 192.168.126.131] MSG:
service/coredns created
[master1 192.168.126.131] MSG:
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
[master1 192.168.126.131] MSG:
configmap/nodelocaldns created
[master1 192.168.126.131] MSG:
I0930 13:42:21.020662 5871 version.go:254] remote version is much newer: v1.22.2; falling back to: stable-1.20
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[upload-certs] Using certificate key:
f17c29922833fe98994bb75ccb5d7a7afd92ac9febefdee44ef0640852434332
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
secret/kubeadm-certs patched
[master1 192.168.126.131] MSG:
kubeadm join lb.kubesphere.local:6443 –token 8sanxe.9cfe3rw266lo0s26 –discovery-token-ca-cert-hash sha256:cda678e4402a753e7491bd209e209605e10ddd1d801f0cf32dd1f6c4196df95f
[master1 192.168.126.131] MSG:
master1 v1.20.4 [map[address:192.168.126.131 type:InternalIP] map[address:master1 type:Hostname]]
INFO[13:42:22 CST] Joining nodes to cluster
[worker1 192.168.126.109] MSG:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.820430 574 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker2 192.168.126.110] MSG:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.850103 14274 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker4 192.168.126.112] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.915165 21194 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker6 192.168.126.114] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.950204 7700 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker3 192.168.126.111] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.951744 1902 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker5 192.168.126.113] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:23.003540 17005 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker1 192.168.126.109] MSG:
node/worker1 labeled
[worker2 192.168.126.110] MSG:
node/worker2 labeled
[worker6 192.168.126.114] MSG:
node/worker6 labeled
[worker4 192.168.126.112] MSG:
node/worker4 labeled
[worker7 192.168.126.115] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.768567 8161 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[worker3 192.168.126.111] MSG:
node/worker3 labeled
[worker5 192.168.126.113] MSG:
node/worker5 labeled
[worker7 192.168.126.115] MSG:
node/worker7 labeled
[master3 192.168.126.133] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.851843 11921 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[download-certs] Downloading the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “front-proxy-client” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.133 127.0.0.1 192.168.126.146 192.168.126.131 192.168.126.132 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Valid certificates and keys now exist in “/etc/kubernetes/pki”
[certs] Using the existing “sa” key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
[control-plane-join] using external etcd - no local stacked instance added
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[mark-control-plane] Marking the node master3 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

Certificate signing request was sent to apiserver and approval was received.
The Kubelet was informed of the new secure connection details.
Control plane (master) label and taint were applied to the new node.
The Kubernetes control plane instances scaled up.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.
[master2 192.168.126.132] MSG:
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.8. Latest validated version: 19.03
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W0930 13:42:22.762043 9855 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[download-certs] Downloading the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.gyhl.k8s.com lb.kubesphere.local localhost master1 master1.gyhl.k8s.com master2 master2.gyhl.k8s.com master3 master3.gyhl.k8s.com worker1 worker1.gyhl.k8s.com worker2 worker2.gyhl.k8s.com worker3 worker3.gyhl.k8s.com worker4 worker4.gyhl.k8s.com worker5 worker5.gyhl.k8s.com worker6 worker6.gyhl.k8s.com worker7 worker7.gyhl.k8s.com] and IPs [10.233.0.1 192.168.126.132 127.0.0.1 192.168.126.146 192.168.126.131 192.168.126.133 192.168.126.109 192.168.126.110 192.168.126.111 192.168.126.112 192.168.126.113 192.168.126.114 192.168.126.115]
[certs] Generating “front-proxy-client” certificate and key
[certs] Valid certificates and keys now exist in “/etc/kubernetes/pki”
[certs] Using the existing “sa” key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
[control-plane-join] using external etcd - no local stacked instance added
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[mark-control-plane] Marking the node master2 as control-plane by adding the labels “node-role.kubernetes.io/master='‘” and “node-role.kubernetes.io/control-plane=’' (deprecated)”
[mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

Certificate signing request was sent to apiserver and approval was received.
The Kubelet was informed of the new secure connection details.
Control plane (master) label and taint were applied to the new node.
The Kubernetes control plane instances scaled up.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.
INFO[13:42:32 CST] Deploying network plugin …
[master1 192.168.126.131] MSG:
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
INFO[13:42:34 CST] Installing addon [1-1]: glusterfs
secret/heketi-secret serverside-applied
storageclass.storage.k8s.io/glusterfs serverside-applied
INFO[13:42:34 CST] Deploying KubeSphere …
v3.1.1
[master1 192.168.126.131] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system created
[master1 192.168.126.131] MSG:
secret/kube-etcd-client-certs created
[master1 192.168.126.131] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
WARN[14:12:56 CST] Task failed …
WARN[14:12:56 CST] error: KubeSphere startup timeout.
Error: Failed to deploy kubesphere: KubeSphere startup timeout.
Usage:
kk create cluster [flags]

Global Flags:
–debug Print detailed information (default true)
–in-cluster Running inside the cluster

Failed to deploy kubesphere: KubeSphere startup timeout.
You have new mail in /var/spool/mail/root
[root@master1 k8ssoft]# cat config-sample.yaml
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:

{name: master1, address: 192.168.126.131, internalAddress: 192.168.126.131, user: root, password: P@ssw0rd}
{name: master2, address: 192.168.126.132, internalAddress: 192.168.126.132, user: root, password: P@ssw0rd}
{name: master3, address: 192.168.126.133, internalAddress: 192.168.126.133, user: root, password: P@ssw0rd}
{name: worker1, address: 192.168.126.109, internalAddress: 192.168.126.109, user: root, password: P@ssw0rd}
{name: worker2, address: 192.168.126.110, internalAddress: 192.168.126.110, user: root, password: P@ssw0rd}
{name: worker3, address: 192.168.126.111, internalAddress: 192.168.126.111, user: root, password: P@ssw0rd}
{name: worker4, address: 192.168.126.112, internalAddress: 192.168.126.112, user: root, password: P@ssw0rd}
{name: worker5, address: 192.168.126.113, internalAddress: 192.168.126.113, user: root, password: P@ssw0rd}
{name: worker6, address: 192.168.126.114, internalAddress: 192.168.126.114, user: root, password: P@ssw0rd}
{name: worker7, address: 192.168.126.115, internalAddress: 192.168.126.115, user: root, password: P@ssw0rd}
roleGroups:
etcd:
- master1
- master2
- master3
  master:
- master1
- master2
- master3
  worker:
- worker1
- worker2
- worker3
- worker4
- worker5
- worker6
- worker7
  controlPlaneEndpoint:
  domain: lb.kubesphere.local
  address: 192.168.126.146
  port: 6443
  kubernetes:
  version: v1.20.4
  imageRepo: kubesphere
  clusterName: gyhl.k8s.com
  network:
  plugin: calico
  kubePodsCIDR: 10.233.64.0/18
  kubeServiceCIDR: 10.233.0.0/18
  registry:
  registryMirrors: []
  insecureRegistries: []
  addons:
name: glusterfs
namespace: kube-system
sources:
yaml:
path:
- /root/glusterfs-sc.yaml

—

apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.1.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
zone: ""
local_registry: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
redis:
enabled: false
redisVolumSize: 2Gi
openldap:
enabled: false
openldapVolumeSize: 2Gi
minioVolumeSize: 20Gi
monitoring:
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
es:
elasticsearchMasterVolumeSize: 4Gi
elasticsearchDataVolumeSize: 20Gi
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchUrl: ""
externalElasticsearchPort: ""
console:
enableMultiLogin: true
port: 30880
alerting:
enabled: false

thanosruler:

replicas: 1

resources: {}

auditing:
enabled: false
devops:
enabled: false
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: false
ruler:
enabled: true
replicas: 2
logging:
enabled: false
logsidecar:
enabled: true
replicas: 2
metrics_server:
enabled: false
monitoring:
storageClass: ""
prometheusMemoryRequest: 400Mi
prometheusVolumeSize: 20Gi
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: false
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
cloudhubPort: “10000”
cloudhubQuicPort: “10001”
cloudhubHttpsPort: “10002”
cloudstreamPort: “10003”
tunnelPort: “10004”
cloudHub:
advertiseAddress:
- ""
nodeLimit: “100”
service:
cloudhubNodePort: “30000”
cloudhubQuicNodePort: “30001”
cloudhubHttpsNodePort: “30002”
cloudstreamNodePort: “30003”
tunnelNodePort: “30004”
edgeWatcher:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []

[root@master1 k8ssoft]#

dongoewang

24sama
kubectl在/usr/local/bin/中，具体如下

kubectl get node的执行结果，如下

kubectl get po -A的执行结果，如下

kubectl logs -n kubesphere-system [root@master1 bin]# ./kubectl 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:47+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:48+08:00 INFO 2021-09-30T13:42:51+08:00 INFO 2021-09-30T13:42:51+08:00 INFO [WARNING]: No inventory [WARNING]: provided hosts the implicit localhost $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath=‘{.items[0].metadata.name}’) -f的执行结果，如下
logs -n kubesphere-system $(./kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath=‘{.items[0].metadata.name}’) -f
: shell-operator latest
: HTTP SERVER Listening on 0.0.0.0:9115
: Use temporary dir: /tmp/shell-operator
: Initialize hooks manager …
: Search and load hooks …
: Load hook config from ‘/hooks/kubesphere/installRunner.py’
: Load hook config from ‘/hooks/kubesphere/schedule.sh’
: Initializing schedule manager …
: KUBE Init Kubernetes client
: KUBE-INIT Kubernetes client is configured successfully
: MAIN: run main loop
: MAIN: add onStartup tasks
: Running schedule manager …
: QUEUE add all HookRun@OnStartup
: MSTOR Create new metric shell_operator_live_ticks
: MSTOR Create new metric shell_operator_tasks_queue_length
: GVR for kind ‘ClusterConfiguration’ is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations
: EVENT Kube event ‘8f20df9d-468e-4213-8f33-b5c52c5520b9’
: QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
: TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
: Running hook ‘kubesphere/installRunner.py’ binding ‘KUBE_EVENTS’ …
was parsed, only implicit localhost is available
list is empty, only localhost is available. Note that
does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Downloading items] ********************************************
skipping: [localhost]

TASK [download : Synchronizing container] **************************************
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [preinstall : KubeSphere | Checking Kubernetes version] *******************
changed: [localhost]

TASK [preinstall : KubeSphere | Initing Kubernetes version] ********************
ok: [localhost]

TASK [preinstall : KubeSphere | Stopping if Kubernetes version is nonsupport] ***
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [preinstall : KubeSphere | Checking StorageClass] *************************
changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if StorageClass was not found] ********
skipping: [localhost]

TASK [preinstall : KubeSphere | Checking default StorageClass] *****************
changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if default StorageClass was not found] ***
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [preinstall : KubeSphere | Checking KubeSphere component] *****************
changed: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost] => (item=ks-openldap)
skipping: [localhost] => (item=ks-redis)
skipping: [localhost] => (item=ks-minio)
skipping: [localhost] => (item=ks-openpitrix)
skipping: [localhost] => (item=elasticsearch-logging)
skipping: [localhost] => (item=elasticsearch-logging-curator)
skipping: [localhost] => (item=istio)
skipping: [localhost] => (item=istio-init)
skipping: [localhost] => (item=jaeger-operator)
skipping: [localhost] => (item=ks-jenkins)
skipping: [localhost] => (item=ks-sonarqube)
skipping: [localhost] => (item=logging-fluentbit-operator)
skipping: [localhost] => (item=uc)
skipping: [localhost] => (item=metrics-server)

PLAY RECAP *********************************************************************
localhost : ok=8 changed=4 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Downloading items] ********************************************
skipping: [localhost]

TASK [download : Synchronizing container] **************************************
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [metrics-server : Metrics-Server | Getting metrics-server installation files] ***
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Creating manifests] ********************
skipping: [localhost] => (item={‘file’: ‘metrics-server.yaml’})

TASK [metrics-server : Metrics-Server | Checking Metrics-Server] ***************
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Uninstalling old metrics-server] *******
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Installing new metrics-server] *********
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Waitting for metrics.k8s.io ready] *****
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Importing metrics-server status] *******
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Downloading items] ********************************************
skipping: [localhost]

TASK [download : Synchronizing container] **************************************
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [common : KubeSphere | Checking kube-node-lease namespace] ****************
changed: [localhost]

TASK [common : KubeSphere | Getting system namespaces] *************************
ok: [localhost]

TASK [common : set_fact] *******************************************************
ok: [localhost]

TASK [common : debug] **********************************************************
ok: [localhost] => {
“msg”: [
“kubesphere-system”,
“kubesphere-controls-system”,
“kubesphere-monitoring-system”,
“kubesphere-monitoring-federated”,
“kube-node-lease”
]
}

TASK [common : KubeSphere | Creating KubeSphere namespace] *********************
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling system-workspace] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Creating ImagePullSecrets] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling namespace for network policy] *************
changed: [localhost]

TASK [common : KubeSphere | Getting Kubernetes master num] *********************
changed: [localhost]

TASK [common : KubeSphere | Setting master num] ********************************
ok: [localhost]

TASK [common : KubeSphere | Getting common component installation files] *******
changed: [localhost] => (item=common)
changed: [localhost] => (item=ks-crds)

TASK [common : KubeSphere | Creating KubeSphere crds] **************************
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/app.k8s.io_applications.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmapplications.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmapplicationversions.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmcategories.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmreleases.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/application.kubesphere.io_helmrepos.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/cluster.kubesphere.io_clusters.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_devopsprojects.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_pipelines.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibinaries.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibuilders.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2ibuildertemplates.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/devops.kubesphere.io_s2iruns.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_globalrolebindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_globalroles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_groupbindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_groups.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_loginrecords.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_rolebases.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_users.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_workspacerolebindings.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/iam.kubesphere.io_workspaceroles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ipamblocks.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ipamhandles.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_ippools.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/network.kubesphere.io_namespacenetworkpolicies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/quota.kubesphere.io_resourcequotas.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/servicemesh.kubesphere.io_servicepolicies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/servicemesh.kubesphere.io_strategies.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/storage.kubesphere.io_provisionercapabilities.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/storage.kubesphere.io_storageclasscapabilities.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/tenant.kubesphere.io_workspaces.yaml)
changed: [localhost] => (item=/kubesphere/kubesphere/ks-crds/tenant.kubesphere.io_workspacetemplates.yaml)

TASK [common : KubeSphere | Creating Storage ProvisionerCapability] ************
changed: [localhost]

TASK [common : KubeSphere | Checking Kubernetes version] ***********************
changed: [localhost]

TASK [common : KubeSphere | Getting common component installation files] *******
changed: [localhost] => (item=snapshot-controller)

TASK [common : KubeSphere | Creating snapshot controller values] ***************
changed: [localhost] => (item={‘name’: ‘custom-values-snapshot-controller’, ‘file’: ‘custom-values-snapshot-controller.yaml’})

TASK [common : KubeSphere | Removing old snapshot crd] *************************
changed: [localhost]

TASK [common : KubeSphere | Deploying snapshot controller] *********************
changed: [localhost]

TASK [common : KubeSphere | Checking openpitrix common component] **************
changed: [localhost]

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item={‘op’: ‘openpitrix-db’, ‘ks’: ‘mysql-pvc’})
skipping: [localhost] => (item={‘op’: ‘openpitrix-etcd’, ‘ks’: ‘etcd-pvc’})

TASK [common : Getting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : KubeSphere | Checking mysql PersistentVolumeClaim] **************
changed: [localhost]

TASK [common : KubeSphere | Setting mysql db pv size] **************************
skipping: [localhost]

TASK [common : KubeSphere | Checking redis PersistentVolumeClaim] **************
changed: [localhost]

TASK [common : KubeSphere | Setting redis db pv size] **************************
skipping: [localhost]

TASK [common : KubeSphere | Checking minio PersistentVolumeClaim] **************
changed: [localhost]

TASK [common : KubeSphere | Setting minio pv size] *****************************
skipping: [localhost]

TASK [common : KubeSphere | Checking openldap PersistentVolumeClaim] ***********
changed: [localhost]

TASK [common : KubeSphere | Setting openldap pv size] **************************
skipping: [localhost]

TASK [common : KubeSphere | Checking etcd db PersistentVolumeClaim] ************
changed: [localhost]

TASK [common : KubeSphere | Setting etcd pv size] ******************************
skipping: [localhost]

TASK [common : KubeSphere | Checking redis ha PersistentVolumeClaim] ***********
changed: [localhost]

TASK [common : KubeSphere | Setting redis ha pv size] **************************
skipping: [localhost]

TASK [common : KubeSphere | Checking es-master PersistentVolumeClaim] **********
changed: [localhost]

TASK [common : KubeSphere | Setting es master pv size] *************************
skipping: [localhost]

TASK [common : KubeSphere | Checking es data PersistentVolumeClaim] ************
changed: [localhost]

TASK [common : KubeSphere | Setting es data pv size] ***************************
skipping: [localhost]

TASK [common : KubeSphere | Creating common component manifests] ***************
changed: [localhost] => (item={‘path’: ‘redis’, ‘file’: ‘redis.yaml’})

TASK [common : KubeSphere | Deploying etcd and mysql] **************************
skipping: [localhost] => (item=etcd.yaml)
skipping: [localhost] => (item=mysql.yaml)

TASK [common : KubeSphere | Getting minio installation files] ******************
skipping: [localhost] => (item=minio-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
skipping: [localhost] => (item={‘name’: ‘custom-values-minio’, ‘file’: ‘custom-values-minio.yaml’})

TASK [common : KubeSphere | Checking minio] ************************************
skipping: [localhost]

TASK [common : KubeSphere | Deploying minio] ***********************************
skipping: [localhost]

TASK [common : debug] **********************************************************
skipping: [localhost]

TASK [common : fail] ***********************************************************
skipping: [localhost]

TASK [common : KubeSphere | Importing minio status] ****************************
skipping: [localhost]

TASK [common : KubeSphere | Checking ha-redis] *********************************
skipping: [localhost]

TASK [common : KubeSphere | Getting redis installation files] ******************
skipping: [localhost] => (item=redis-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
skipping: [localhost] => (item={‘name’: ‘custom-values-redis’, ‘file’: ‘custom-values-redis.yaml’})

TASK [common : KubeSphere | Checking old redis status] *************************
skipping: [localhost]

TASK [common : KubeSphere | Deleting and backup old redis svc] *****************
skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] ***********************************
skipping: [localhost]

TASK [common : KubeSphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Creating redis migration script] *******************
skipping: [localhost] => (item={‘path’: ‘/etc/kubesphere’, ‘file’: ‘redisMigrate.py’})

TASK [common : KubeSphere | Checking redis-ha status] **************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : KubeSphere | Disabling old redis] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : KubeSphere | Importing redis status] ****************************
skipping: [localhost]

TASK [common : KubeSphere | Getting openldap installation files] ***************
skipping: [localhost] => (item=openldap-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
skipping: [localhost] => (item={‘name’: ‘custom-values-openldap’, ‘file’: ‘custom-values-openldap.yaml’})

TASK [common : KubeSphere | Checking old openldap status] **********************
skipping: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Deleting and backup old openldap svc] **************
skipping: [localhost]

TASK [common : KubeSphere | Checking openldap] *********************************
skipping: [localhost]

TASK [common : KubeSphere | Deploying openldap] ********************************
skipping: [localhost]

TASK [common : KubeSphere | Loading old openldap data] *************************
skipping: [localhost]

TASK [common : KubeSphere | Checking openldap-ha status] ***********************
skipping: [localhost]

TASK [common : KubeSphere | Getting openldap-ha pod list] **********************
skipping: [localhost]

TASK [common : KubeSphere | Getting old openldap data] *************************
skipping: [localhost]

TASK [common : KubeSphere | Migrating openldap data] ***************************
skipping: [localhost]

TASK [common : KubeSphere | Disabling old openldap] ****************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting openldap] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]

TASK [common : KubeSphere | Importing openldap status] *************************
skipping: [localhost]

TASK [common : KubeSphere | Checking ha-redis] *********************************
changed: [localhost]

TASK [common : KubeSphere | Getting redis installation files] ******************
changed: [localhost] => (item=redis-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
changed: [localhost] => (item={‘name’: ‘custom-values-redis’, ‘file’: ‘custom-values-redis.yaml’})

TASK [common : KubeSphere | Checking old redis status] *************************
changed: [localhost]

TASK [common : KubeSphere | Deleting and backup old redis svc] *****************
skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] ***********************************
changed: [localhost]

TASK [common : KubeSphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Creating redis migration script] *******************
skipping: [localhost] => (item={‘path’: ‘/etc/kubesphere’, ‘file’: ‘redisMigrate.py’})

TASK [common : KubeSphere | Checking redis-ha status] **************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : KubeSphere | Disabling old redis] *******************************
skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : KubeSphere | Importing redis status] ****************************
changed: [localhost]