创建部署问题时，请参考下面模板：
操作系统信息，例如：物理机，Ubuntu18.04
Kubernetes版本信息，例如：v21.5。多节点。
KubeSphere版本信息，例如：v3.2.0。在线安装。全套安装。
问题是什么，报错日志是什么，最好有截图。
root@172-31-255-3:/DATA/disk1/gitlab.com/wangshan# ./kk create cluster -f sample.yaml
+————–+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+————–+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
| 172-31-255-6 | y | y | y | y | y | y | y | 20.10.6 | | | | CST 20:28:25 |
| 172-31-255-3 | y | y | y | y | y | y | y | 20.10.6 | | | | CST 20:28:25 |
+————–+——+——+———+———-+——-+——-+———–+———+————+————-+——————+————–+
This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Continue this installation? [yes/no]: yes
INFO[20:28:29 CST] Downloading Installation Files
INFO[20:28:29 CST] Downloading kubeadm …
INFO[20:28:29 CST] Downloading kubelet …
INFO[20:28:30 CST] Downloading kubectl …
INFO[20:28:30 CST] Downloading helm …
INFO[20:28:30 CST] Downloading kubecni …
INFO[20:28:30 CST] Downloading etcd …
INFO[20:28:30 CST] Downloading docker …
INFO[20:28:31 CST] Downloading crictl …
INFO[20:28:31 CST] Configuring operating system …
[172-31-255-6 172.31.255.6] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
[172-31-255-3 172.31.255.3] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
INFO[20:28:33 CST] Get cluster status
INFO[20:28:34 CST] Installing Container Runtime …
INFO[20:28:35 CST] Start to download images on all nodes
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.4.1
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.4.1
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.21.5
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.21.5
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.21.5
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.21.5
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.21.5
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
[172-31-255-6] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0
[172-31-255-3] Downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0
INFO[20:28:39 CST] Getting etcd status
[172-31-255-3 172.31.255.3] MSG:
Configuration file will be created
INFO[20:28:40 CST] Generating etcd certs
INFO[20:28:41 CST] Synchronizing etcd certs
INFO[20:28:41 CST] Creating etcd service
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/etcd-v3.4.13-linux-amd64.tar.gz to 172.31.255.3:/tmp/kubekey/etcd-v3.4.13-linux-amd64.tar.gz Done
INFO[20:28:43 CST] Starting etcd cluster
INFO[20:28:43 CST] Refreshing etcd configuration
INFO[20:28:48 CST] Backup etcd data regularly
INFO[20:28:55 CST] Installing kube binaries
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubeadm to 172.31.255.6:/tmp/kubekey/kubeadm Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubelet to 172.31.255.6:/tmp/kubekey/kubelet Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubeadm to 172.31.255.3:/tmp/kubekey/kubeadm Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubectl to 172.31.255.6:/tmp/kubekey/kubectl Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/helm to 172.31.255.6:/tmp/kubekey/helm Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubelet to 172.31.255.3:/tmp/kubekey/kubelet Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/kubectl to 172.31.255.3:/tmp/kubekey/kubectl Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/helm to 172.31.255.3:/tmp/kubekey/helm Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/cni-plugins-linux-amd64-v0.9.1.tgz to 172.31.255.6:/tmp/kubekey/cni-plugins-linux-amd64-v0.9.1.tgz Done
Push /DATA/disk1/gitlab.com/wangshan/kubekey/v1.21.5/amd64/cni-plugins-linux-amd64-v0.9.1.tgz to 172.31.255.3:/tmp/kubekey/cni-plugins-linux-amd64-v0.9.1.tgz Done
INFO[20:28:57 CST] Initializing kubernetes cluster
[172-31-255-3 172.31.255.3] MSG:
W1103 20:28:58.962995 52973 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[init] Using Kubernetes version: v1.21.5
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [172-31-255-3 172-31-255-3.cluster.local 172-31-255-6 172-31-255-6.cluster.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local lb.kubesphere.local localhost] and IPs [10.233.0.1 172.31.255.3 127.0.0.1 172.31.255.6]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.503400 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config-1.21” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see –upload-certs
[mark-control-plane] Marking the node 172-31-255-3 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node 172-31-255-3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 52cw09.qpa9oca553h1d957
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join lb.kubesphere.local:6443 –token 52cw09.qpa9oca553h1d957 \
–discovery-token-ca-cert-hash sha256:ebbbae60d26702c0ac8db4f88ccf8015a19a2e0887753faf7226606b6bdf20b5 \
–control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join lb.kubesphere.local:6443 –token 52cw09.qpa9oca553h1d957 \
–discovery-token-ca-cert-hash sha256:ebbbae60d26702c0ac8db4f88ccf8015a19a2e0887753faf7226606b6bdf20b5
[172-31-255-3 172.31.255.3] MSG:
node/172-31-255-3 untainted
[172-31-255-3 172.31.255.3] MSG:
node/172-31-255-3 labeled
[172-31-255-3 172.31.255.3] MSG:
service “kube-dns” deleted
[172-31-255-3 172.31.255.3] MSG:
service/coredns created
Warning: resource clusterroles/system:coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create –save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/system:coredns configured
[172-31-255-3 172.31.255.3] MSG:
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
[172-31-255-3 172.31.255.3] MSG:
configmap/nodelocaldns created
INFO[20:29:40 CST] Get cluster status
INFO[20:29:42 CST] Joining nodes to cluster
[172-31-255-6 172.31.255.6] MSG:
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
W1103 20:29:44.457861 16486 utils.go:69] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
This node has joined the cluster:
- Certificate signing request was sent to apiserver and a response was received.
- The Kubelet was informed of the new secure connection details.
Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[172-31-255-6 172.31.255.6] MSG:
node/172-31-255-6 labeled
INFO[20:29:51 CST] Deploying network plugin …
[172-31-255-3 172.31.255.3] MSG:
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created
[172-31-255-3 172.31.255.3] MSG:
storageclass.storage.k8s.io/local created
serviceaccount/openebs-maya-operator created
clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created
clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created
deployment.apps/openebs-localpv-provisioner created
INFO[20:29:54 CST] Deploying KubeSphere …
v3.2.0
[172-31-255-3 172.31.255.3] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system created
[172-31-255-3 172.31.255.3] MSG:
secret/kube-etcd-client-certs created
[172-31-255-3 172.31.255.3] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
WARN[21:01:13 CST] Task failed …
WARN[21:01:13 CST] error: KubeSphere startup timeout.
Error: Failed to deploy kubesphere: KubeSphere startup timeout.
Usage:
kk create cluster [flags]
Flags:
–container-manager string Container runtime: docker, crio, containerd and isula. (default “docker”)
–download-cmd string The user defined command to download the necessary binary files. The first param ‘%s’ is output path, the second param ‘%s’, is the URL (default “curl -L -o %s %s”)
-f, –filename string Path to a configuration file
-h, –help help for cluster
–skip-pull-images Skip pre pull images
–with-kubernetes string Specify a supported version of kubernetes (default “v1.21.5”)
–with-kubesphere Deploy a specific version of kubesphere (default v3.2.0)
–with-local-storage Deploy a local PV provisioner
-y, –yes Skip pre-check of the installation
Global Flags:
–debug Print detailed information (default true)
–in-cluster Running inside the cluster
Failed to deploy kubesphere: KubeSphere startup timeout.