离线安装文档疑问

24sama · 2022年8月3日

只需要先准备一台联网主机用于制作离线包即可。继续往下看文档写了两种方式，可以选择第二种，直接复制文档中的manifest

2.同上
3.使用kk搭建私有镜像仓库域名不可变
4.创建集群的过程包含有push镜像

Zzqh · 2022年8月3日

解压镜像的时候报错，能帮忙看一下吗？copy image oci:/opt/kubesphere/kubekey/images:calico:cni:v3.20.0-amd64 to docker://10.50.40.106:30443/kubesphereio/cni:v3.20.0-amd64 failed: trying to reuse blob sha256:a97babae0e73ac29ec5136a8c71a6feda9c92e423775ea8ef2315f1c4af537a5 at destination: checking whether a blob sha256:a97babae0e73ac29ec5136a8c71a6feda9c92e423775ea8ef2315f1c4af537a5 exists in 10.50.40.106:30443/kubesphereio/cni: errors:

denied: requested access to the resource is denied

error parsing HTTP 401 response body: unexpected end of JSON input: ""

24sama · 2022年8月3日

zqh
这个镜像仓库“10.50.40.106:30443”是harbor吗？是不是在harbor里面没有创建对应image namespace（上文日志中的“ calico”）的project？

https://goharbor.io/docs/1.10/working-with-projects/create-projects/

Zzqh · 2022年8月3日

24sama 是harbor，我们自己搭建的，另外咱们config-sample想要使用http的镜像仓库，文档的缩进写的不对，和auth同级不行，要和username同级

registry:

type: harbor

auths:

  "10.50.40.106:30443":

    username: admin

    password: Harbor12345

    skipTLSVerify: true

    plainHTTP: true

privateRegistry: "10.50.40.106:30443"

namespaceOverride: "kubesphereio"

registryMirrors: []

insecureRegistries: []

addons: []

24sama · 2022年8月3日

zqh
你看的是哪篇文档呢，我看官网文档里没有写使用http的这个字段。github的文档也是对的：https://github.com/kubesphere/kubekey/blob/master/docs/config-example.md

Zzqh · 2022年8月3日

24sama

https://v3-2.docs.kubesphere.io/zh/docs/installing-on-linux/introduction/air-gapped-installation/

24sama · 2022年8月3日

zqh
3.2的文档是基于kk 2.0.0版本的，这个版本的字段缩进是这样的。

Zzqh · 2022年8月3日

24sama 嗯嗯好咧，谢谢

Zzqh · 2022年8月3日

24sama 还有个问题哈，我这个推镜像可以了，镜像是在centos x86 机器上kk打包的制品，现在在ubuntu机器上有部分镜像解析push报错如下，

11:27:57 UTC Source: oci:/opt/kubesphere/kubekey/images:library:perl:latest-amd64

11:27:57 UTC Destination: docker://10.50.40.106:30443/kubesphereio/perl:latest-amd64

Getting image source signatures

Copying blob 0e29546d541c done

Copying blob 9b829c73b52b done

Copying blob cb5b7ae36172 done

Copying blob 6494e4811622 done

Copying blob 6f9f74896dfa done

Copying blob 1a99cd2a1d0b done

Copying blob 83b15adf9bf3 done

Copying config 3bfa1a6ecb done

Writing manifest to image destination

Storing signatures

11:28:05 UTC success: [LocalHost]

11:28:05 UTC [CopyImagesToRegistryModule] Push multi-arch manifest to private registry

11:28:05 UTC Push multi-arch manifest list: 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0

INFO[0246] Retrieving digests of member images

11:28:06 UTC message: [LocalHost]

push image 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0 multi-arch manifest failed: Error pushing manifest list/index to registry: sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049: failed commit on ref “index-10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0@sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049”: unexpected status: 415 Unsupported Media Type

11:28:06 UTC failed: [LocalHost]

error: Pipeline[CreateClusterPipeline] execute failed: Module[CopyImagesToRegistryModule] exec failed:

failed: [LocalHost] [PushManifest] exec failed after 1 retires: push image 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0 multi-arch manifest failed: Error pushing manifest list/index to registry: sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049: failed commit on ref “index-10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0@sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049”: unexpected status: 415 Unsupported Media Type

24sama · 2022年8月3日

zqh
现在ubuntu集群是amd64还是arm64呢？然后可以贴一下kk config文件的内容看看

Zzqh · 2022年8月3日

24sama 老哥，加集群的时候貌似报x509,证书过期了吗？

24sama · 2022年8月3日

zqh
把kk执行目录下的kubekey/pki删了再重新试试

Ww281722735 · 2022年8月3日

24sama

如果选择第二种方式, 这些组件的版本应该怎么选呢？

Ww281722735 · 2022年8月3日

24sama

今天尝试了用registry和harbor做离线仓库, 发现了两个新问题

1.拉镜像时都会出现x509证书错误,在/etc/containerd/config.toml指定ca和dockerhub.kubekey.local的证书后才完成部署，这是不是kk的bug?

2.kk create cluster必须放在要部署的集群的某个节点上执行吗？，放在非集群节点会有镜像推送失败的问题

24sama · 2022年8月4日

w281722735

如果是你自己部署的containerd或者docker，kk不会对其config进行重新配置（有覆盖用户自定义配置的风险），所以既然用户选择自行部署容器运行时，那么也需要自行对其配置。
不是必须的，你推送失败了应该是因为非集群节点没有镜像仓库证书

组件版本目前支持的比较少，不用修改。只需改k8s版本和镜像列表即可。官方文档给出的例子即是最佳实践

---
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: centos
    version: "7"
    repository:
      iso:
        localPath:
        url: https://github.com/kubesphere/kubekey/releases/download/v2.2.1/centos7-rpms-amd64.iso
  - arch: amd64
    type: linux
    id: ubuntu
    version: "20.04"
    repository:
      iso:
        localPath:
        url: https://github.com/kubesphere/kubekey/releases/download/v2.2.1/ubuntu-20.04-debs-amd64.iso
  kubernetesDistributions:
  - type: kubernetes
    version: v1.22.10
  components:
    helm:
      version: v3.6.3
    cni:
      version: v0.9.1
    etcd:
      version: v3.4.13
   ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
   ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
    containerRuntimes:
    - type: docker
      version: 20.10.8
    crictl:
      version: v1.22.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.4.1
    docker-compose:
      version: v2.2.2
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.12.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:2.10.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:2.10.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nfs-subdir-external-provisioner:v4.0.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.21.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubefed:v0.8.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tower:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller:v1.1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.0.25-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/netshoot:v1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cloudcore:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/iptables-manager:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/edgeservice:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-controller:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-tools:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-jenkins:v3.3.0-2.319.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/inbound-agent:4.10-2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2ioperator:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2irun:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2i-binary:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-6-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-4-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-36-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-35-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-34-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-27-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd:v2.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset:v0.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/dex:v2.30.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:6.2.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/grafana:8.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-curator:v5.7.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-oss:6.8.22
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/filebeat:6.7.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-operator:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-exporter:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-ruler:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-operator:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-webhook:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pilot:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-operator:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-agent:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-collector:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-query:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-es-index-cleaner:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali-operator:v1.38.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali:v1.38
  - registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:1.31.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx:1.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wget:1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hello:plain-text
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wordpress:4.8-apache
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hpa-example:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java:openjdk-8-jre-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentd:v1.4.2-2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/perl:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-productpage-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v2:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-details-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-ratings-v1:1.16.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/scope:1.13.0

Dduduxiaolong · 2022年8月4日

安装官网的离线文档安装报错：
could not fetch a Kubernetes version from the internet: unable to get URL “https://dl.k8s.io/release/stable-1.txt”: Get “https://dl.k8s.io/release/stable-1.txt”: dial tcp: lookup dl.k8s.io on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable

W0804 09:38:33.138340 50405 version.go:104] falling back to the local client version: v1.22.10

cannot use “0.0.0.0” as the bind address for the API Server

怎么它还是要找互联网

willqy · 2022年9月8日

现在新的制品离线安装方式问题很大：

1、组件版本混乱，不知道哪些版本和哪些版本适配

2、自动安装的harbor仓库证书在各个节点缺失，默认没有在/etc/docker/certs.d，要么containerd证书路径空白，导致登陆和push都会报x509

3、制作的离线制品版本不是实际安装时需要的，导致依然去公网拉镜像

24sama · 2022年9月9日

willqy

官方文档已经给出了版本的最佳实践，如果需要自行修改部署的组件版本，那么需要用户自行配置实践
不清楚你是怎么配置的，kk安装的harbor会在/etc/ssl/registry/ssl和/etc/docker/certs.d下都放置一份证书
版本配置不对，当然会继续去公网寻找

willqy · 2022年9月9日

24sama

2、https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/air-gapped-installation/ 这个文档有个问题，certsPath 这个参数不能少，不然containerd中ca配置是空的

  registry:
    auths:
      "dockerhub.kubekey.local":
        username: "xxx"
        password: "***"
        skipTLSVerify: false # Allow contacting registries over HTTPS with failed TLS verification.
        plainHTTP: false # Allow contacting registries over HTTP.
        certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local" # Use certificates at path (*.crt, *.cert, *.key) to connect to the registry.

Ccici · 7 2月

willqy
請問如果是用自己部署的 harbor 該怎麼改呢?

离线安装文档疑问

24sama

Zzqh

24sama

Zzqh

24sama

Zzqh

24sama

Zzqh

Zzqh

24sama

Zzqh

24sama

Ww281722735

Ww281722735

24sama

Dduduxiaolong

willqyK零S

24sama

willqyK零S

Ccici