kubekey 离线导入镜像到私有仓库报错

thuanghai

操作系统信息
3节点物理机，Centos7.9，40C/128G

Kubernetes版本信息
v1.23.15-多节点部署。

容器运行时
containerd，版本1.6.4

KubeSphere版本信息
kubeky v3.0.13
kubesphere v3.4.1。
离线安装。

问题是什么
现有条件描述，manifest.yaml文件成功下载并打包好离线包。文件内容如下：

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  #operatingSystems:
  #- arch: amd64
  #  type: linux
  #  id: centos
  #  version: "7"
  #  repository:
  #    iso:
  #      localPath:
  #      url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/centos7-rpms-amd64.iso
  #- arch: amd64
  #  type: linux
  #  id: ubuntu
  #  version: "20.04"
  #  repository:
  #    iso:
  #      localPath:
  #      url: https://github.com/kubesphere/kubekey/releases/download/v3.0.10/ubuntu-20.04-debs-amd64.iso
  kubernetesDistributions:
  - type: kubernetes
    version: v1.26.5
  components:
    helm:
      version: v3.9.0
    cni:
      version: v1.2.0
    etcd:
      version: v3.4.13
    calicoctl:
      version: v3.26.1
   ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
   ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
    containerRuntimes:
    - type: docker
      version: 20.10.8
    - type: containerd
      version: 1.6.4
    crictl:
      version: v1.24.0
    #docker-registry:
    #  version: "2"
    #harbor:
    #  version: v2.5.3
    #docker-compose:
    #  version: v2.2.2
  images:
  - docker.io/calico/cni:v3.23.2
  - docker.io/calico/kube-controllers:v3.23.2
  - docker.io/calico/node:v3.23.2
  - docker.io/calico/pod2daemon-flexvol:v3.23.2
  - docker.io/coredns/coredns:1.8.6
  - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
  - docker.io/kubesphere/kube-apiserver:v1.26.5
  - docker.io/kubesphere/kube-controller-manager:v1.26.5
  - docker.io/kubesphere/kube-proxy:v1.26.5
  - docker.io/kubesphere/kube-scheduler:v1.26.5
  - docker.io/kubesphere/pause:3.6

内网自建有centos源，有私有harbor仓库(v2.9.1)，harbor仓库使用的是HTTP方式，非HTTPS
在执行安装的过程中，在导入镜像到私有仓库时报错，安装命令如下：

./kk create cluster -f config.yaml -a kubesphere.tar.gz --with-kubernetes 1.26.5

前面终端打印信息正常，从下面开始展示报错信息，并退出，报错信息如下：

...
21:49:46 CST success: [LocalHost]
21:49:46 CST [CopyImagesToRegistryModule] Path multi-arch manifest to private registry
21:49:46 CST message: [LocalHost]
get manifest list failed by module cache
21:49:46 CST failed: [LocalHost]
error: Pipeline[CreateClusterPipeline] execute failed: Module[CopyImagesToRegistryModule] exec failed:
failed: [LocalHost]  [PushManifest] exec failed after 1 retries: get manifest list failed by module cache

使用的config.yaml文件为

(以上省略，差别不大，下面是关于registry的配置)
...
  registry:
    auths:
      "registry.xxx.xxx":
        username: admin
        password: 123456
    privateRegistry: "registry.xxx.xxx"
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []

其中registry.xxx.xxx是我们的内部域名，肯定是可以使用的，因为，单独docker login/push/pull都没有问题，之前单机部署docker容器也是从这个仓库拉取的，使用HTTP方式，非HTTPS，后面又尝试了单独推送的命令，还是报相同的错误

./kk artifact images push -a kubesphere.tar.gz -f config.yaml

还是报错，后面尝试了更改config配置，如下几种，都不行

  registry:
    auths:
      "registry.xxx.xxx":
        username: admin
        password: 123456
    privateRegistry: "registry.xxx.xxx"
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: ["registry.xxx.xxx"]

  registry:
    auths:
      "registry.xxx.xxx":
        username: admin
        password: 123456
    plainHTTP: true
    privateRegistry: "registry.xxx.xxx"
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: [registry.xxx.xxx]

因此，请问这个报错该如何处理才能通过，当然保持Harbor的HTTP访问方式的前提下。

thuanghai

thuanghai 我已经解决，我自己来回答吧，参照kubekey源代码doc/manifest-example.yaml中最后的配置举例描述，更正registry的相关写法，如下：

  registry:
    auths:
      "dockerhub.kubekey.local":
        username: "xxx"
        password: "***"
        skipTLSVerify: false # Allow contacting registries over HTTPS with failed TLS verification.
        plainHTTP: false # Allow contacting registries over HTTP.
        certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local" # Use certificates at path (*.crt, *.cert, *.key) to connect to the registry.

注意最后三行的写法，可以有两种，注释里面也写的很明白了，就不废话了。

lxj

thuanghai
你好，你这个是 config-sample.yaml吗？

diudiu

呃，也碰到这个问题了，按着这个操作还是报错，现在报的

17:41:17 CST skipped: [LocalHost]

17:41:17 CST [CopyImagesToRegistryModule] Copy images to a private registry from an artifact OCI Path

17:41:17 CST [FATA] Failed to Parse Registry Auths configuration: [123 34 99 101 114 116 115 80 97 116 104 34 58 34 47 101 116 99 47 100 111 99 107 101 114 47 99 101 114 116 115 46 100 47 114 101 103 105 115 116 114 121 46 100 121 110 97 114 111 115 101 116 101 115

angyuang

diudiu

[upl-image-preview url=

示例：

registry:

type: harbor

auths:
  "registry.xxx.cn":
  **username: admin
  **password: Harbor12312
  **certsPath: "/etc/docker/certs.d/registry.xxx.cn"
privateRegistry: "registry.cubigdata.cn"
namespaceOverride: "kubesphereio"

请将username，password，certsPath，前**理解成两个空格，不要要与 “registry.xxx.cn”: 对齐，因为 "registry.xxx.cn"的属性有username，password，certsPath，不能与"registry.xxx.cn"对齐，这个是yaml格式要求，如果对齐就会导致找不到账户信息，如果空两个就是registry的属性

aidan1314520

优秀