在线安装失败，Pipeline[CreateClusterPipeline] execute failed

Cauchy

wcai_bienue

kubectl describe pod 一下那连个pending的pod看看啥原因

yuanxinghong

解决了吗我也遇到了但是我的pod状态全部是running

zhu

yuanxinghong

我也遇到了。部署等待了两小时就失败了。

我是kubespherev3.4.1版本，k8sv1.23.10版本。

报错只有题主发出的这样一段话，不知道去哪查日志做排查。

frezes

zhu

检查下kubesphere-system 空间下的 ks-installer 的日志呢

zhu

frezes

我在寻求帮助之前，重新进行了部署，查看日志后，显示内容如下，

root@master:~# kubectl logs -n kubesphere-system ks-installer-ddbcf44f8-lqrqw | tail -n 10
TASK [common : KubeSphere | Getting minio installation files] ******************
changed: [localhost] => (item=minio-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
changed: [localhost] => (item={'name': 'custom-values-minio', 'file': 'custom-values-minio.yaml'})

TASK [common : KubeSphere | Checking minio] ************************************
changed: [localhost]

TASK [common : KubeSphere | Deploying minio] ***********************************

zhu

frezes

我查询了日志，显示内容如下。这不是超时显示的日志，是我重新执行部署后，等待很久，查询的日志。

root@master:~# kubectl logs -n kubesphere-system ks-installer-ddbcf44f8-lqrqw | tail -n 10
TASK [common : KubeSphere | Getting minio installation files] ******************
changed: [localhost] => (item=minio-ha)

TASK [common : KubeSphere | Creating manifests] ********************************
changed: [localhost] => (item={'name': 'custom-values-minio', 'file': 'custom-values-minio.yaml'})

TASK [common : KubeSphere | Checking minio] ************************************
changed: [localhost]

TASK [common : KubeSphere | Deploying minio] ***********************************

zhu

frezes

目前查询到以下报错信息


TASK [common : KubeSphere | Checking minio] ************************************
changed: [localhost]

TASK [common : KubeSphere | Deploying minio] ***********************************
changed: [localhost]

TASK [common : debug] **********************************************************
ok: [localhost] => {
    "msg": [
        "1. check the storage configuration and storage server",
        "2. make sure the DNS address in /etc/resolv.conf is available",
        "3. execute 'kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job' to watch logs",
        "4. execute 'helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job'",
        "5. Restart the installer pod in kubesphere-system namespace"
    ]
}

TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "It is suggested to refer to the above methods for troubleshooting problems ."}

PLAY RECAP *********************************************************************
localhost                  : ok=36   changed=28   unreachable=0    failed=1    skipped=78   rescued=0    ignored=0

frezes

zhu

minio 部署超时了吗？检查下 kubesphere-system 下Pod 状态，是否有 Pending ？检查Pending 原因。

可以按照错误提示进行检查。猜测可能是你安装集群时没有配置 storageclass。你可以使用 kk 创建集群时增加 --with-local-storage 选用本地存储或者对接外置其他存储。

      "1. check the storage configuration and storage server",
        "2. make sure the DNS address in /etc/resolv.conf is available",
        "3. execute 'kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job' to watch logs",
        "4. execute 'helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job'",
        "5. Restart the installer pod in kubesphere-system namespace"

zhuyifei_superlinked

frezes

我发现有三个pod提示拉取镜像异常，我手动拉取是正常的。感觉是获取镜像的自动环节有异常。

root@master:~# kubectl get pod -A
NAMESPACE           NAME                                           READY   STATUS             RESTARTS   AGE
kube-system         calico-kube-controllers-67fbf89557-lnjdc       1/1     Running            0          8m7s
kube-system         calico-node-6twn2                              1/1     Running            0          8m7s
kube-system         calico-node-7qw75                              1/1     Running            0          8m7s
kube-system         calico-node-b9fbz                              1/1     Running            0          8m7s
kube-system         coredns-757cd945b-cspcz                        1/1     Running            0          8m15s
kube-system         coredns-757cd945b-t8vtw                        1/1     Running            0          8m15s
kube-system         kube-apiserver-master                          1/1     Running            0          8m30s
kube-system         kube-controller-manager-master                 1/1     Running            0          8m28s
kube-system         kube-proxy-nrs5r                               1/1     Running            0          8m15s
kube-system         kube-proxy-pdfx6                               1/1     Running            0          8m12s
kube-system         kube-proxy-wvc7w                               1/1     Running            0          8m12s
kube-system         kube-scheduler-master                          1/1     Running            0          8m28s
kube-system         nodelocaldns-4hkxg                             1/1     Running            0          8m12s
kube-system         nodelocaldns-qtl5m                             1/1     Running            0          8m15s
kube-system         nodelocaldns-tkflr                             1/1     Running            0          8m12s
kube-system         openebs-localpv-provisioner-7bbcf865cd-zn5th   1/1     Running            0          8m5s
kube-system         snapshot-controller-0                          0/1     ImagePullBackOff   0          6m50s
kubesphere-system   ks-installer-ddbcf44f8-s49w4                   1/1     Running            0          8m5s
kubesphere-system   minio-7879c5dd65-v6rgz                         0/1     ImagePullBackOff   0          6m30s
kubesphere-system   openldap-0                                     0/1     ImagePullBackOff   0          6m38s

zhuyifei_superlinked

frezes

我手动拉取镜像成功后，删除pod，POD依旧提示拉取镜像失败。

root@master:~# kubectl get pod -A
NAMESPACE           NAME                                           READY   STATUS              RESTARTS   AGE
kube-system         calico-kube-controllers-67fbf89557-lnjdc       1/1     Running             0          12m
kube-system         calico-node-6twn2                              1/1     Running             0          12m
kube-system         calico-node-7qw75                              1/1     Running             0          12m
kube-system         calico-node-b9fbz                              1/1     Running             0          12m
kube-system         coredns-757cd945b-cspcz                        1/1     Running             0          12m
kube-system         coredns-757cd945b-t8vtw                        1/1     Running             0          12m
kube-system         kube-apiserver-master                          1/1     Running             0          12m
kube-system         kube-controller-manager-master                 1/1     Running             0          12m
kube-system         kube-proxy-nrs5r                               1/1     Running             0          12m
kube-system         kube-proxy-pdfx6                               1/1     Running             0          12m
kube-system         kube-proxy-wvc7w                               1/1     Running             0          12m
kube-system         kube-scheduler-master                          1/1     Running             0          12m
kube-system         nodelocaldns-4hkxg                             1/1     Running             0          12m
kube-system         nodelocaldns-qtl5m                             1/1     Running             0          12m
kube-system         nodelocaldns-tkflr                             1/1     Running             0          12m
kube-system         openebs-localpv-provisioner-7bbcf865cd-zn5th   1/1     Running             0          12m
kube-system         snapshot-controller-0                          0/1     ImagePullBackOff    0          46s
kubesphere-system   ks-installer-ddbcf44f8-s49w4                   1/1     Running             0          12m
kubesphere-system   minio-7879c5dd65-m22kr                         0/1     ErrImagePull        0          18s
kubesphere-system   openldap-0                                     0/1     ContainerCreating   0          6s
root@master:~# docker images
REPOSITORY                                                              TAG        IMAGE ID       CREATED         SIZE
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers          v3.26.1    1919f2787fa7   22 months ago   75.1MB
registry.cn-beijing.aliyuncs.com/kubesphereio/cni                       v3.26.1    9dee260ef7f5   22 months ago   210MB
registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol        v3.26.1    092a973bb20e   22 months ago   14.8MB
registry.cn-beijing.aliyuncs.com/kubesphereio/node                      v3.26.1    8065b798a4d6   22 months ago   246MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver            v1.23.10   9ca5fafbe8dc   2 years ago     135MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy                v1.23.10   71b9bf9750e1   2 years ago     112MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler            v1.23.10   d5c0efb802d9   2 years ago     53.5MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager   v1.23.10   91a4a0d5de4e   2 years ago     125MB
registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils               3.3.0      e88cfb3a763b   2 years ago     72.4MB
registry.cn-beijing.aliyuncs.com/kubesphereio/coredns                   1.8.6      a4ca41631cc7   3 years ago     46.8MB
registry.cn-beijing.aliyuncs.com/kubesphereio/pause                     3.6        6270bb605e12   3 years ago     683kB
registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache        1.15.12    5340ba194ec9   5 years ago     107MB
root@master:~#
root@master:~# docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
RELEASE.2019-08-07T01-59-21Z: Pulling from kubesphereio/minio
e7c96db7181b: Pull complete
e088a1581a57: Pull complete
9ca5f6ef1275: Pull complete
b645ed203cec: Pull complete
Digest: sha256:cdc1ad6ee7b16b35a8c26c00e7bed0482f21bcd8ce841813fb91600f7d7f2faf
Status: Downloaded newer image for registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
root@master:~# kubectl describe pod -n kubesphere-system minio-7879c5dd65-m22kr | grep Image:
    Image:         minio/minio:RELEASE.2019-08-07T01-59-21Z
root@master:~# docker images
REPOSITORY                                                              TAG                            IMAGE ID       CREATED         SIZE
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers          v3.26.1                        1919f2787fa7   22 months ago   75.1MB
registry.cn-beijing.aliyuncs.com/kubesphereio/cni                       v3.26.1                        9dee260ef7f5   22 months ago   210MB
registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol        v3.26.1                        092a973bb20e   22 months ago   14.8MB
registry.cn-beijing.aliyuncs.com/kubesphereio/node                      v3.26.1                        8065b798a4d6   22 months ago   246MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver            v1.23.10                       9ca5fafbe8dc   2 years ago     135MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler            v1.23.10                       d5c0efb802d9   2 years ago     53.5MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy                v1.23.10                       71b9bf9750e1   2 years ago     112MB
registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager   v1.23.10                       91a4a0d5de4e   2 years ago     125MB
registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils               3.3.0                          e88cfb3a763b   2 years ago     72.4MB
registry.cn-beijing.aliyuncs.com/kubesphereio/coredns                   1.8.6                          a4ca41631cc7   3 years ago     46.8MB
registry.cn-beijing.aliyuncs.com/kubesphereio/pause                     3.6                            6270bb605e12   3 years ago     683kB
registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache        1.15.12                        5340ba194ec9   5 years ago     107MB
registry.cn-beijing.aliyuncs.com/kubesphereio/minio                     RELEASE.2019-08-07T01-59-21Z   29c267893b04   5 years ago     61.3MB
root@master:~#

部署时选择的中国地区的部署方法，实际拉取镜像强制读取国外，本地完全不读取。

root@master:~# kubectl describe pod -n kubesphere-system minio-7879c5dd65-m22kr | grep image
  Warning  Failed     2m26s (x3 over 3m42s)  kubelet            Failed to pull image "minio/minio:RELEASE.2019-08-07T01-59-21Z": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   Pulling    96s (x4 over 3m57s)    kubelet            Pulling image "minio/minio:RELEASE.2019-08-07T01-59-21Z"
  Warning  Failed     81s                    kubelet            Failed to pull image "minio/minio:RELEASE.2019-08-07T01-59-21Z": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 157.240.17.36:443: i/o timeout
  Normal   BackOff    57s (x7 over 3m41s)    kubelet            Back-off pulling image "minio/minio:RELEASE.2019-08-07T01-59-21Z"

frezes

zhuyifei_superlinked

先前 docker 官方的镜像拉取是不受限制的，后续由于某些因素，导致国内无法拉取 docker.io 的镜像，通俗的讲，也就是被墙了。
我印象中 3.x 版本应该是没有完整的国内镜像的仓库地址，且 3.x 会逐步结束支持, 参见 KubeSphere 产品生命周期管理政策公告，建议体验使用 4.1.x 新版本。

回到这个问题上，本质上是镜像拉取受限，一种方式是配置 docker 代理，解决 docker.io 的拉取限制，另外一种则是准备本地镜像仓库harbor或registry，将相关镜像同步到本地仓库，修改 ks-installer 的配置，改为从本地仓库拉取。

zhu

frezes

感谢答复。