rayzhou2017 安装之后继续部署失败
[master-21 10.10.10.21] MSG:
v1.18.6

WARN[13:29:27 CST] Task failed …

WARN[13:29:27 CST] error: Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/sh -c “/usr/local/bin/kubeadm init phase upload-certs –upload-certs”
I0905 13:28:06.466595 116078 version.go:252] remote version is much newer: v1.19.0; falling back to: stable-1.18
W0905 13:28:07.173851 116078 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
error execution phase upload-certs: error uploading certs: error creating token: timed out waiting for the condition
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1
Error: Failed to get cluster status: Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/sh -c “/usr/local/bin/kubeadm init phase upload-certs –upload-certs”
I0905 13:28:06.466595 116078 version.go:252] remote version is much newer: v1.19.0; falling back to: stable-1.18
W0905 13:28:07.173851 116078 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
error execution phase upload-certs: error uploading certs: error creating token: timed out waiting for the condition
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1
Usage:
kk create cluster [flags]

Flags:
-f, –filename string Path to a configuration file
-h, –help help for cluster
–skip-pull-images Skip pre pull images
–with-kubernetes string Specify a supported version of kubernetes
–with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)
-y, –yes Skip pre-check of the installation

Global Flags:
–debug Print detailed information (default true)

Failed to get cluster status: Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/sh -c “/usr/local/bin/kubeadm init phase upload-certs –upload-certs”
I0905 13:28:06.466595 116078 version.go:252] remote version is much newer: v1.19.0; falling back to: stable-1.18
W0905 13:28:07.173851 116078 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
error execution phase upload-certs: error uploading certs: error creating token: timed out waiting for the condition
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1

    Feynman
    我认为先决条件都已经设置好了啊。都是干净的Centos，防火墙也全部关闭了。该提前安装的也都安装了。

      先./kk delete cluster -f <config yaml>清理一下环境再安装试试

        rayzhou2017 好的，我重试一下，vsphere 部署文档有歧义，准备虚拟机应该是8台，VIP不应计算为一台虚拟机。

          rayzhou2017
          [master-21 10.10.10.21] MSG:
          [reset] Reading configuration from the cluster…
          [reset] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
          W0906 12:58:47.629895 40112 reset.go:99] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://lb.kubesphere.local:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s: dial tcp 10.10.10.20:6443: connect: connection refused
          [preflight] Running pre-flight checks
          W0906 12:58:47.630260 40112 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
          [reset] No etcd config found. Assuming external etcd
          [reset] Please, manually reset etcd to prevent further issues
          [reset] Stopping the kubelet service
          [reset] Unmounting mounted directories in “/var/lib/kubelet”
          [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
          [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
          [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

          The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

          The reset process does not reset or clean up iptables rules or IPVS tables.
          If you wish to reset iptables, you must do so manually by using the “iptables” command.

          If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
          to reset your system’s IPVS tables.

          The reset process does not clean your kubeconfig files and you must remove them manually.
          Please, check the contents of the $HOME/.kube/config file.

            rayzhou2017 我换成v1.17.9 安装仍然有问题。
            配置文件
            `

            apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
  name: config-sample
spec:
  hosts:
  - {name: master-21, address: 10.10.10.21, internalAddress: 10.10.10.21, password: an@123@#!}
  - {name: master-22, address: 10.10.10.22, internalAddress: 10.10.10.22, password: an@123@#!}
  - {name: master-23, address: 10.10.10.23, internalAddress: 10.10.10.23, password: an@123@#!}
  - {name: node-24, address: 10.10.10.24, internalAddress: 10.10.10.24, password: an@123@#!}
  - {name: node-25, address: 10.10.10.25, internalAddress: 10.10.10.25, password: an@123@#!}
  - {name: node-26, address: 10.10.10.26, internalAddress: 10.10.10.26, password: an@123@#!}
  - {name: node-27, address: 10.10.10.27, internalAddress: 10.10.10.27, password: an@123@#!}
  - {name: node-28, address: 10.10.10.28, internalAddress: 10.10.10.28, password: an@123@#!}
  roleGroups:
    etcd:
    - master-21
    - master-22
    - master-23
    master:
    - master-21
    - master-22
    - master-23
    worker:
    - node-24
    - node-25
    - node-26
    - node-27
    - node-28
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    # vip
    address: "10.10.10.20"
    port: "6443"
  kubernetes:
    version: v1.17.9
    imageRepo: kubesphere
    clusterName: cluster.local
    masqueradeAll: false  # masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode. [Default: false]
    maxPods: 110  # maxPods is the number of pods that can run on this Kubelet. [Default: 110]
    nodeCidrMaskSize: 24  # internal network node size allocation. This is the size allocated to each node on your network. [Default: 24]
    proxyMode: ipvs  # mode specifies which proxy mode to use. [Default: ipvs]
  network:
    plugin: calico
    calico:
      ipipMode: Always  # IPIP Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, vxlanMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Always]
      vxlanMode: Never  # VXLAN Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, ipipMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Never]
      vethMTU: 1440  # The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. [Default: 1440]
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
  registry:
    registryMirrors: []
    insecureRegistries: []
  addons: [] # add your persistent storage and LoadBalancer plugin configuration here if you have, see https://kubesphere.io/docs/installing-on-linux/introduction/storage-configuration

              zackzhang 就是上面的错误，感觉是lb通信问题。但是我PING了是通的。
              [master-21 10.10.10.21] MSG:
              [reset] Reading configuration from the cluster…
              [reset] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
              W0906 12:58:47.629895 40112 reset.go:99] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://lb.kubesphere.local:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s: dial tcp 10.10.10.20:6443: connect: connection refused
              [preflight] Running pre-flight checks
              W0906 12:58:47.630260 40112 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
              [reset] No etcd config found. Assuming external etcd
              [reset] Please, manually reset etcd to prevent further issues
              [reset] Stopping the kubelet service
              [reset] Unmounting mounted directories in “/var/lib/kubelet”
              [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
              [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
              [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

              The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

              The reset process does not reset or clean up iptables rules or IPVS tables.
              If you wish to reset iptables, you must do so manually by using the “iptables” command.

              If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
              to reset your system’s IPVS tables.

              The reset process does not clean your kubeconfig files and you must remove them manually.
              Please, check the contents of the $HOME/.kube/config file.

                  leejor

                  Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
error execution phase upload-certs: error uploading certs: error creating token: timed out waiting for the condition

                  连接到kube-apiserver超时，能ping通vip并不代表vip的端口能通，要保证vip设置的kub-apiserver端口能通。

                      Cauchy

                      lb是自己搭的？还是用的云lb？lb上是不是挂了防火墙或者安全组

                          leejor
                          那有可能是lb搞得有问题，curl -k https://{masterip}:6443 curl -k https://{vip}:{lbport} 对比下返回结果，不一样的话，就是lb搞得有问题了。