2.1.1 升级到 3.0.0 失败

Ddami · 2021年6月30日

升级日志
`root@master:~# ./kk upgrade –debug –with-kubernetes v1.17.9 –with-kubesphere v3.0.0 -f config-sample.yaml
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+
| node1 | y | y | y | y | y | y | y | y | y | y | y | CST 10:20:57 |
| master | y | y | y | y | y | y | y | y | y | y | y | CST 10:20:58 |
| node2 | y | y | y | y | y | y | y | y | y | y | y | CST 10:21:00 |
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+

Cluster nodes status:
NAME STATUS ROLES AGE VERSION
master Ready master,worker 361d v1.16.7
node1 Ready worker 361d v1.16.7
node2 Ready worker 361d v1.16.7

Components Status:
controller-manager: ok
scheduler: ok
etcd-0: {“health”: “true”}

Upgrade Confirmation:
kubernetes version: v1.16.7 to v1.17.9
kubesphere version: v2.1.1 to v3.0.0

Continue upgrading cluster? [yes/no]: yes
INFO[10:21:02 CST] Get current version
INFO[10:21:07 CST] Configurating operating system …
[master 172.16.0.98] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 6144
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_interval = 60
net.ipv4.neigh.default.gc_stale_time = 120
kernel.perf_event_paranoid = -1
net.ipv4.tcp_slow_start_after_idle = 0
net.core.rmem_max = 16777216
fs.inotify.max_user_watches = 524288
kernel.softlockup_all_cpu_backtrace = 1
kernel.softlockup_panic = 0
kernel.watchdog_thresh = 30
fs.file-max = 2097152
fs.inotify.max_user_instances = 8192
fs.inotify.max_queued_events = 16384
vm.max_map_count = 262144
sysctl: 无法获取/proc/sys/fs/may_detach_mounts 的文件状态(stat): 没有那个文件或目录
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.yama.ptrace_scope = 0
vm.swappiness = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
sysctl: /etc/sysctl.conf(99): invalid syntax, continuing…
[node2 172.16.0.96] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 6144
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_interval = 60
net.ipv4.neigh.default.gc_stale_time = 120
kernel.perf_event_paranoid = -1
net.ipv4.tcp_slow_start_after_idle = 0
net.core.rmem_max = 16777216
fs.inotify.max_user_watches = 524288
kernel.softlockup_all_cpu_backtrace = 1
kernel.softlockup_panic = 0
kernel.watchdog_thresh = 30
fs.file-max = 2097152
fs.inotify.max_user_instances = 8192
fs.inotify.max_queued_events = 16384
vm.max_map_count = 262144
sysctl: 无法获取/proc/sys/fs/may_detach_mounts 的文件状态(stat): 没有那个文件或目录
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.yama.ptrace_scope = 0
vm.swappiness = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
sysctl: /etc/sysctl.conf(99): invalid syntax, continuing…
[node1 172.16.0.97] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 6144
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_interval = 60
net.ipv4.neigh.default.gc_stale_time = 120
kernel.perf_event_paranoid = -1
net.ipv4.tcp_slow_start_after_idle = 0
net.core.rmem_max = 16777216
fs.inotify.max_user_watches = 524288
kernel.softlockup_all_cpu_backtrace = 1
kernel.softlockup_panic = 0
kernel.watchdog_thresh = 30
fs.file-max = 2097152
fs.inotify.max_user_instances = 8192
fs.inotify.max_queued_events = 16384
vm.max_map_count = 262144
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.yama.ptrace_scope = 0
vm.swappiness = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
sysctl: /etc/sysctl.conf(96): invalid syntax, continuing…
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
sysctl: /etc/sysctl.conf(99): invalid syntax, continuing…
INFO[10:21:40 CST] Upgrading kube cluster
INFO[10:21:40 CST] Start Upgrade: v1.16.7 -> v1.17.9
INFO[10:21:40 CST] Downloading Installation Files
INFO[10:21:40 CST] Downloading kubeadm …
INFO[10:21:41 CST] Downloading kubelet …
INFO[10:21:42 CST] Downloading kubectl …
INFO[10:21:44 CST] Downloading helm …
INFO[10:21:44 CST] Downloading kubecni …
[node2] Downloading image: kubesphere/pause:3.1
[master] Downloading image: kubesphere/pause:3.1
[node1] Downloading image: kubesphere/etcd:v3.3.12
[master] Downloading image: kubesphere/kube-apiserver:v1.17.9
[master] Downloading image: kubesphere/kube-controller-manager:v1.17.9
[node2] Downloading image: kubesphere/kube-proxy:v1.17.9
[master] Downloading image: kubesphere/kube-scheduler:v1.17.9
[node2] Downloading image: coredns/coredns:1.6.9
[node1] Downloading image: kubesphere/pause:3.1
[master] Downloading image: kubesphere/kube-proxy:v1.17.9
[node2] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[node1] Downloading image: kubesphere/kube-proxy:v1.17.9
[master] Downloading image: coredns/coredns:1.6.9
[node2] Downloading image: calico/kube-controllers:v3.15.1
[node1] Downloading image: coredns/coredns:1.6.9
[master] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[node2] Downloading image: calico/cni:v3.15.1
[node1] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[master] Downloading image: calico/kube-controllers:v3.15.1
[node2] Downloading image: calico/node:v3.15.1
[node1] Downloading image: calico/kube-controllers:v3.15.1
[master] Downloading image: calico/cni:v3.15.1
[node2] Downloading image: calico/pod2daemon-flexvol:v3.15.1
[node1] Downloading image: calico/cni:v3.15.1
[master] Downloading image: calico/node:v3.15.1
[node1] Downloading image: calico/node:v3.15.1
[master] Downloading image: calico/pod2daemon-flexvol:v3.15.1
[node1] Downloading image: calico/pod2daemon-flexvol:v3.15.1
INFO[10:26:03 CST] Upgrading master [172.16.0.98] node=172.16.0.98
Push /root/kubekey/v1.17.9/amd64/kubeadm to 172.16.0.98:/tmp/kubekey/kubeadm Done
Push /root/kubekey/v1.17.9/amd64/kubelet to 172.16.0.98:/tmp/kubekey/kubelet Done
Push /root/kubekey/v1.17.9/amd64/kubectl to 172.16.0.98:/tmp/kubekey/kubectl Done
Push /root/kubekey/v1.17.9/amd64/helm to 172.16.0.98:/tmp/kubekey/helm Done
Push /root/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.0.98:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
WARN[10:32:09 CST] Task failed …
WARN[10:32:09 CST] error: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c “timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=all –allow-experimental-upgrades –allow-release-candidate-upgrades –etcd-upgrade=false –certificate-renewal=true –force”
W0630 10:32:09.872066 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.872180 29214 validation.go:28] Cannot validate kubelet config - no validator is available
W0630 10:32:09.872188 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
[upgrade/config] Making sure the configuration is correct:
W0630 10:32:09.873953 29214 common.go:94] WARNING: Usage of the –config flag for reconfiguring the cluster during upgrade is not recommended!
W0630 10:32:09.876997 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.877062 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0630 10:32:09.877069 29214 validation.go:28] Cannot validate kubelet config - no validator is available
[preflight] Running pre-flight checks.
[WARNING CoreDNSUnsupportedPlugins]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[WARNING CoreDNSMigration]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade] Making sure the cluster is healthy:
[WARNING APIServerHealth]: the API Server is unhealthy; /healthz didn’t return “ok”
[WARNING ControlPlaneNodesReady]: couldn’t list control-planes in cluster: Get https://172.16.0.98:6443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster%3D: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade/version] You have chosen to change the cluster version to “v1.17.9”
[upgrade/version] FATAL: the –version argument is invalid due to these fatal errors:

    - Unable to fetch cluster version: Couldn't fetch cluster version from the API Server: Get https://172.16.0.98:6443/version?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused

Please fix the misalignments highlighted above and try upgrading again
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1
Error: Failed to upgrade kube cluster: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c “timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=all –allow-experimental-upgrades –allow-release-candidate-upgrades –etcd-upgrade=false –certificate-renewal=true –force”
W0630 10:32:09.872066 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.872180 29214 validation.go:28] Cannot validate kubelet config - no validator is available
W0630 10:32:09.872188 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
[upgrade/config] Making sure the configuration is correct:
W0630 10:32:09.873953 29214 common.go:94] WARNING: Usage of the –config flag for reconfiguring the cluster during upgrade is not recommended!
W0630 10:32:09.876997 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.877062 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0630 10:32:09.877069 29214 validation.go:28] Cannot validate kubelet config - no validator is available
[preflight] Running pre-flight checks.
[WARNING CoreDNSUnsupportedPlugins]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[WARNING CoreDNSMigration]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade] Making sure the cluster is healthy:
[WARNING APIServerHealth]: the API Server is unhealthy; /healthz didn’t return “ok”
[WARNING ControlPlaneNodesReady]: couldn’t list control-planes in cluster: Get https://172.16.0.98:6443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster%3D: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade/version] You have chosen to change the cluster version to “v1.17.9”
[upgrade/version] FATAL: the –version argument is invalid due to these fatal errors:

    - Unable to fetch cluster version: Couldn't fetch cluster version from the API Server: Get https://172.16.0.98:6443/version?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused

Please fix the misalignments highlighted above and try upgrading again
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1
Usage:
kk upgrade [flags]

Flags:
-f, –filename string Path to a configuration file
-h, –help help for upgrade
–skip-pull-images Skip pre pull images
–with-kubernetes string Specify a supported version of kubernetes
–with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)

Global Flags:
–debug Print detailed information (default true)

Failed to upgrade kube cluster: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c “timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=all –allow-experimental-upgrades –allow-release-candidate-upgrades –etcd-upgrade=false –certificate-renewal=true –force”
W0630 10:32:09.872066 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.872180 29214 validation.go:28] Cannot validate kubelet config - no validator is available
W0630 10:32:09.872188 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
[upgrade/config] Making sure the configuration is correct:
W0630 10:32:09.873953 29214 common.go:94] WARNING: Usage of the –config flag for reconfiguring the cluster during upgrade is not recommended!
W0630 10:32:09.876997 29214 defaults.go:186] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0630 10:32:09.877062 29214 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0630 10:32:09.877069 29214 validation.go:28] Cannot validate kubelet config - no validator is available
[preflight] Running pre-flight checks.
[WARNING CoreDNSUnsupportedPlugins]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[WARNING CoreDNSMigration]: Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade] Making sure the cluster is healthy:
[WARNING APIServerHealth]: the API Server is unhealthy; /healthz didn’t return “ok”
[WARNING ControlPlaneNodesReady]: couldn’t list control-planes in cluster: Get https://172.16.0.98:6443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster%3D: dial tcp 172.16.0.98:6443: connect: connection refused
[upgrade/version] You have chosen to change the cluster version to “v1.17.9”
[upgrade/version] FATAL: the –version argument is invalid due to these fatal errors:

    - Unable to fetch cluster version: Couldn't fetch cluster version from the API Server: Get https://172.16.0.98:6443/version?timeout=10s: dial tcp 172.16.0.98:6443: connect: connection refused

Please fix the misalignments highlighted above and try upgrading again
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1`

执行 curl -k https://172.16.0.98:6443/version?timeout=10s 响应正常，
执行 curl -k Get https://172.16.0.98:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s 也正常响应

看日志，可能是 coreDNS服务的问题，请问如何处理呢

Ddami · 2021年6月30日

config-sample.yaml 文件内容