创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
操作系统信息
例如:虚拟机/物理机,Kylin Linux Advanced Server V10 (Halberd),8C/32G
Kubernetes版本信息
Kubernetes version: v1.23.15
容器运行时
docker Version: 24.0.7
KubeSphere版本信息
离线安装 3.4.1 KubeSphere版本
问题是什么
[root@node1 kubekey]# ./kk create cluster -f config-sample.yaml -a ksp-v3.4.1-artifact.tar.gz –skip-push-images
| | / / | | | | / /
| |/ / _ _| |__ ___| |/ / ___ _ _
| \| | | | '_ \ / _ \ \ / _ \ | | |
| |\ \ || | |) | __/ |\ \ __/ |_| |
\| \/\,|./ \\| \/\|\__, |
__/ |
|___/
12:50:10 CST [GreetingsModule] Greetings
12:50:10 CST message: [master1]
Greetings, KubeKey!
12:50:10 CST message: [master2]
Greetings, KubeKey!
12:50:11 CST message: [master3]
Greetings, KubeKey!
12:50:11 CST success: [master1]
12:50:11 CST success: [master2]
12:50:11 CST success: [master3]
12:50:11 CST [NodePreCheckModule] A pre-check on nodes
12:50:13 CST success: [master1]
12:50:13 CST success: [master3]
12:50:13 CST success: [master2]
12:50:13 CST [ConfirmModule] Display confirmation form
+———+——+——+———+———-+——-+——-+———+———–+——–+——–+————+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time |
+———+——+——+———+———-+——-+——-+———+———–+——–+——–+————+————+————-+——————+————–+
| master1 | y | y | y | y | y | y | y | y | y | 24.0.7 | v1.7.6 | y | | | CST 12:50:13 |
| master2 | y | y | y | y | y | y | y | y | y | 24.0.7 | v1.7.6 | y | | | CST 12:50:13 |
| master3 | y | y | y | y | y | y | y | y | y | 24.0.7 | v1.7.6 | y | | | CST 12:50:13 |
+———+——+——+———+———-+——-+——-+———+———–+——–+——–+————+————+————-+——————+————–+
This is a simple check of your environment.
Before installation, ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Install k8s with specify version: v1.23.15
Continue this installation? [yes/no]: yes
12:50:36 CST success: [LocalHost]
12:50:36 CST [UnArchiveArtifactModule] Check the KubeKey artifact md5 value
12:51:11 CST success: [LocalHost]
12:51:11 CST [UnArchiveArtifactModule] UnArchive the KubeKey artifact
12:51:11 CST skipped: [LocalHost]
12:51:11 CST [UnArchiveArtifactModule] Create the KubeKey artifact Md5 file
12:51:11 CST skipped: [LocalHost]
12:51:11 CST [NodeBinariesModule] Download installation binaries
12:51:11 CST message: [localhost]
downloading arm64 kubeadm v1.23.15 …
12:51:11 CST message: [localhost]
kubeadm exists
12:51:11 CST message: [localhost]
downloading arm64 kubelet v1.23.15 …
12:51:11 CST message: [localhost]
kubelet exists
12:51:11 CST message: [localhost]
downloading arm64 kubectl v1.23.15 …
12:51:11 CST message: [localhost]
kubectl exists
12:51:11 CST message: [localhost]
downloading arm64 helm v3.14.3 …
12:51:11 CST message: [localhost]
helm exists
12:51:11 CST message: [localhost]
downloading arm64 kubecni v1.2.0 …
12:51:11 CST message: [localhost]
kubecni exists
12:51:11 CST message: [localhost]
downloading arm64 crictl v1.29.0 …
12:51:11 CST message: [localhost]
crictl exists
12:51:11 CST message: [localhost]
downloading arm64 etcd v3.5.13 …
12:51:11 CST message: [localhost]
etcd exists
12:51:11 CST message: [localhost]
downloading arm64 docker 24.0.9 …
12:51:11 CST message: [localhost]
docker exists
12:51:11 CST message: [localhost]
downloading arm64 calicoctl v3.27.4 …
12:51:12 CST message: [localhost]
calicoctl exists
12:51:12 CST success: [LocalHost]
12:51:12 CST [ConfigureOSModule] Get OS release
12:51:12 CST success: [master3]
12:51:12 CST success: [master2]
12:51:12 CST success: [master1]
12:51:12 CST [ConfigureOSModule] Prepare to init OS
12:51:15 CST success: [master3]
12:51:15 CST success: [master1]
12:51:15 CST success: [master2]
12:51:15 CST [ConfigureOSModule] Generate init os script
12:51:16 CST success: [master3]
12:51:16 CST success: [master1]
12:51:16 CST success: [master2]
12:51:16 CST [ConfigureOSModule] Exec init os script
12:51:17 CST stdout: [master2]
setenforce: SELinux is disabled
Disabled
kernel.sysrq = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.tcp_syncookies = 1
kernel.dmesg_restrict = 1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
net.core.netdev_max_backlog = 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 1048576
net.ipv4.neigh.default.gc_thresh1 = 512
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.tcp_max_orphans = 65535
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.ipv4.conf.all.arp_accept = 1
net.ipv4.conf.default.arp_accept = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
vm.swappiness = 0
vm.overcommit_memory = 0
fs.inotify.max_user_watches = 524288
fs.pipe-max-size = 4194304
fs.aio-max-nr = 262144
kernel.watchdog_thresh = 5
kernel.hung_task_timeout_secs = 5
12:51:18 CST stdout: [master3]
setenforce: SELinux is disabled
Disabled
kernel.sysrq = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.tcp_syncookies = 1
kernel.dmesg_restrict = 1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 0
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
net.core.netdev_max_backlog = 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 1048576
net.ipv4.neigh.default.gc_thresh1 = 512
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.tcp_max_orphans = 65535
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.ipv4.conf.all.arp_accept = 1
net.ipv4.conf.default.arp_accept = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
vm.overcommit_memory = 0
fs.inotify.max_user_watches = 524288
fs.pipe-max-size = 4194304
fs.aio-max-nr = 262144
kernel.watchdog_thresh = 5
kernel.hung_task_timeout_secs = 5
12:51:18 CST stdout: [master1]
setenforce: SELinux is disabled
Disabled
kernel.sysrq = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.tcp_syncookies = 1
kernel.dmesg_restrict = 1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 0
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
net.core.netdev_max_backlog = 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 1048576
net.ipv4.neigh.default.gc_thresh1 = 512
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.tcp_max_orphans = 65535
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.ipv4.conf.all.arp_accept = 1
net.ipv4.conf.default.arp_accept = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
vm.overcommit_memory = 0
fs.inotify.max_user_watches = 524288
fs.pipe-max-size = 4194304
fs.aio-max-nr = 262144
kernel.watchdog_thresh = 5
kernel.hung_task_timeout_secs = 5
12:51:18 CST success: [master2]
12:51:18 CST success: [master3]
12:51:18 CST success: [master1]
12:51:18 CST [ConfigureOSModule] configure the ntp server for each node
12:51:18 CST skipped: [master3]
12:51:18 CST skipped: [master1]
12:51:18 CST skipped: [master2]
12:51:18 CST [KubernetesStatusModule] Get kubernetes cluster status
12:51:18 CST success: [master1]
12:51:18 CST success: [master2]
12:51:18 CST success: [master3]
12:51:18 CST [InstallContainerModule] Sync docker binaries
12:51:18 CST skipped: [master3]
12:51:18 CST skipped: [master1]
12:51:18 CST skipped: [master2]
12:51:18 CST [InstallContainerModule] Generate containerd service
12:51:19 CST success: [master3]
12:51:19 CST success: [master1]
12:51:19 CST success: [master2]
12:51:19 CST [InstallContainerModule] Generate docker service
12:51:20 CST skipped: [master3]
12:51:20 CST skipped: [master2]
12:51:20 CST skipped: [master1]
12:51:20 CST [InstallContainerModule] Generate docker config
12:51:20 CST skipped: [master3]
12:51:20 CST skipped: [master2]
12:51:20 CST skipped: [master1]
12:51:20 CST [InstallContainerModule] Enable containerd
12:51:21 CST success: [master3]
12:51:21 CST success: [master1]
12:51:21 CST success: [master2]
12:51:21 CST [InstallContainerModule] Enable docker
12:51:21 CST skipped: [master3]
12:51:21 CST skipped: [master1]
12:51:21 CST skipped: [master2]
12:51:21 CST [InstallContainerModule] Add auths to container runtime
12:51:22 CST success: [master3]
12:51:22 CST success: [master1]
12:51:22 CST success: [master2]
12:51:22 CST [InstallContainerModule] Sync docker buildx binaries
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [InstallCriDockerdModule] Sync cri-dockerd binaries
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [InstallCriDockerdModule] Generate cri-dockerd service
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [InstallCriDockerdModule] Enable cri-dockerd
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [InstallCriDockerdModule] Sync crictl binaries
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [InstallCriDockerdModule] Generate crictl config
12:51:22 CST skipped: [master3]
12:51:22 CST skipped: [master1]
12:51:22 CST skipped: [master2]
12:51:22 CST [PullModule] Start to pull images on all nodes
12:51:22 CST message: [master3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:22 CST message: [master2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:22 CST message: [master1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:23 CST message: [master3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:23 CST retry: [master3]
12:51:23 CST message: [master1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:23 CST retry: [master1]
12:51:23 CST message: [master2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:23 CST retry: [master2]
12:51:28 CST message: [master3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:28 CST message: [master1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:28 CST message: [master2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:28 CST message: [master1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:28 CST retry: [master1]
12:51:28 CST message: [master3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:28 CST retry: [master3]
12:51:28 CST message: [master2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:28 CST retry: [master2]
12:51:33 CST message: [master3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:33 CST message: [master1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:33 CST message: [master2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.6
12:51:33 CST message: [master3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:33 CST message: [master1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:33 CST message: [master2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
12:51:33 CST failed: [master3]
12:51:33 CST failed: [master1]
12:51:33 CST failed: [master2]
error: Pipeline[CreateClusterPipeline] execute failed: Module[PullModule] exec failed:
failed: [master3] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
failed: [master1] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
failed: [master2] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c “env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.6 –platform arm64”
Error response from daemon: unknown: artifact kubesphereio/pause:3.6 not found: Process exited with status 1
问题1:
我在manifest.yaml 指定的下载的离线镜像的版本好是没带 -arm64的, 但是push到harbor后的镜像tag会自动添加 -arm64 ,导致创建集群的时候,默认没有带-arm64的tag, 导致无法找到镜像, 这个在多个版本测试都一样,是否我缺少什么配置呢?
问题2:
我在制作离线包的时候下载各个离线二进制包都是按照我的manifest.yaml 里面的版本进行下载,为什么我使用 ./kk create cluster -f config-sample.yaml -a ksp-v3.4.1-artifact.tar.gz –skip-push-images 的时候, kk会默认去下载另外版的安装呢? 比如calicoctl 、etcd、 crictl 都默认去下载另外的版本安装包。导致安装的时候会报错
问题3:
我在制作离线包的时候下载镜像已经正常下载完成所有镜像,但是我使用 kk push images的时候, 发现会出现镜像名称错误的问题,但是我查看harbor实际已经有 相关镜像在里面了。