请问离线安装参考 https://kubesphere.com.cn/docs/v3.3/installing-on-linux/introduction/air-gapped-installation/ 这个文档吗？

1.本想用kk离线安装k8s和ks，看这前提条件是需要先有一套ks+k8s，这前提条件有点不合理把？

2.源集群有要求吗，用all-in-one还是多节点部署？

3.私有仓库的域名在哪里自定义，必须是dockerhub.kubekey.local吗？

4.最重要的是这文档没有看到Push镜像到私有仓库的步骤啊

  1. 只需要先准备一台联网主机用于制作离线包即可。继续往下看文档写了两种方式，可以选择第二种，直接复制文档中的manifest

2.同上
3.使用kk搭建私有镜像仓库域名不可变
4.创建集群的过程包含有push镜像

    24sama

    解压镜像的时候报错，能帮忙看一下吗？copy image oci:/opt/kubesphere/kubekey/images:calico:cni:v3.20.0-amd64 to docker://10.50.40.106:30443/kubesphereio/cni:v3.20.0-amd64 failed: trying to reuse blob sha256:a97babae0e73ac29ec5136a8c71a6feda9c92e423775ea8ef2315f1c4af537a5 at destination: checking whether a blob sha256:a97babae0e73ac29ec5136a8c71a6feda9c92e423775ea8ef2315f1c4af537a5 exists in 10.50.40.106:30443/kubesphereio/cni: errors:

    denied: requested access to the resource is denied

    error parsing HTTP 401 response body: unexpected end of JSON input: ""

        24sama 是harbor，我们自己搭建的，另外咱们config-sample想要使用http的镜像仓库，文档的缩进写的不对，和auth同级不行，要和username同级

        registry:

        type: harbor

auths:

  "10.50.40.106:30443":

    username: admin

    password: Harbor12345

    skipTLSVerify: true

    plainHTTP: true

privateRegistry: "10.50.40.106:30443"

namespaceOverride: "kubesphereio"

registryMirrors: []

insecureRegistries: []

        addons: []

            zqh
            3.2的文档是基于kk 2.0.0版本的，这个版本的字段缩进是这样的。

            • zqh 回复了此帖

                24sama 还有个问题哈，我这个推镜像可以了，镜像是在centos x86 机器上kk打包的制品，现在在ubuntu机器上有部分镜像 解析push报错如下，

                11:27:57 UTC Source: oci:/opt/kubesphere/kubekey/images:library:perl:latest-amd64

                11:27:57 UTC Destination: docker://10.50.40.106:30443/kubesphereio/perl:latest-amd64

                Getting image source signatures

                Copying blob 0e29546d541c done

                Copying blob 9b829c73b52b done

                Copying blob cb5b7ae36172 done

                Copying blob 6494e4811622 done

                Copying blob 6f9f74896dfa done

                Copying blob 1a99cd2a1d0b done

                Copying blob 83b15adf9bf3 done

                Copying config 3bfa1a6ecb done

                Writing manifest to image destination

                Storing signatures

                11:28:05 UTC success: [LocalHost]

                11:28:05 UTC [CopyImagesToRegistryModule] Push multi-arch manifest to private registry

                11:28:05 UTC Push multi-arch manifest list: 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0

                INFO[0246] Retrieving digests of member images

                11:28:06 UTC message: [LocalHost]

                push image 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0 multi-arch manifest failed: Error pushing manifest list/index to registry: sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049: failed commit on ref “index-10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0@sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049”: unexpected status: 415 Unsupported Media Type

                11:28:06 UTC failed: [LocalHost]

                error: Pipeline[CreateClusterPipeline] execute failed: Module[CopyImagesToRegistryModule] exec failed:

                failed: [LocalHost] [PushManifest] exec failed after 1 retires: push image 10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0 multi-arch manifest failed: Error pushing manifest list/index to registry: sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049: failed commit on ref “index-10.50.40.106:30443/kubesphereio/kube-rbac-proxy:v0.8.0@sha256:49a3f356782e17ad8a0ae31abe7a6a8ba6d07a5e64fd5ae92f624804ceb92049”: unexpected status: 415 Unsupported Media Type

                    zqh
                    现在ubuntu集群是amd64还是arm64呢？然后可以贴一下kk config文件的内容看看

                    • zqh 回复了此帖

                        24sama 老哥，加集群的时候貌似报x509,证书过期了吗？

                            zqh
                            把kk执行目录下的kubekey/pki删了再重新试试

                              24sama

                              今天尝试了用registry和harbor做离线仓库, 发现了两个新问题

                              1.拉镜像时都会出现x509证书错误,在/etc/containerd/config.toml指定ca和dockerhub.kubekey.local的证书后才完成部署，这是不是kk的bug?

                              2.kk create cluster必须放在要部署的集群的某个节点上执行吗？，放在非集群节点会有镜像推送失败的问题

                                  w281722735

                                  1. 如果是你自己部署的containerd或者docker，kk不会对其config进行重新配置（有覆盖用户自定义配置的风险），所以既然用户选择自行部署容器运行时，那么也需要自行对其配置。
                                  2. 不是必须的，你推送失败了应该是因为非集群节点没有镜像仓库证书
                                  3. 组件版本目前支持的比较少，不用修改。只需改k8s版本和镜像列表即可。官方文档给出的例子即是最佳实践 
                                    ---
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: centos
    version: "7"
    repository:
      iso:
        localPath:
        url: https://github.com/kubesphere/kubekey/releases/download/v2.2.1/centos7-rpms-amd64.iso
  - arch: amd64
    type: linux
    id: ubuntu
    version: "20.04"
    repository:
      iso:
        localPath:
        url: https://github.com/kubesphere/kubekey/releases/download/v2.2.1/ubuntu-20.04-debs-amd64.iso
  kubernetesDistributions:
  - type: kubernetes
    version: v1.22.10
  components:
    helm:
      version: v3.6.3
    cni:
      version: v0.9.1
    etcd:
      version: v3.4.13
   ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
   ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
    containerRuntimes:
    - type: docker
      version: 20.10.8
    crictl:
      version: v1.22.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.4.1
    docker-compose:
      version: v2.2.2
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.12.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:2.10.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:2.10.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nfs-subdir-external-provisioner:v4.0.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.21.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubefed:v0.8.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tower:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller:v1.1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.0.25-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/netshoot:v1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cloudcore:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/iptables-manager:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/edgeservice:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-controller:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-tools:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-jenkins:v3.3.0-2.319.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/inbound-agent:4.10-2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2ioperator:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2irun:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2i-binary:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-6-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-4-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-36-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-35-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-34-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-27-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd:v2.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset:v0.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/dex:v2.30.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:6.2.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/grafana:8.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-curator:v5.7.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-oss:6.8.22
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/filebeat:6.7.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-operator:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-exporter:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-ruler:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-operator:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-webhook:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pilot:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-operator:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-agent:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-collector:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-query:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-es-index-cleaner:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali-operator:v1.38.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali:v1.38
  - registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:1.31.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx:1.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wget:1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hello:plain-text
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wordpress:4.8-apache
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hpa-example:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java:openjdk-8-jre-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentd:v1.4.2-2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/perl:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-productpage-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v2:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-details-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-ratings-v1:1.16.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/scope:1.13.0

                                    安装官网的离线文档安装报错：
                                    could not fetch a Kubernetes version from the internet: unable to get URL “https://dl.k8s.io/release/stable-1.txt”: Get “https://dl.k8s.io/release/stable-1.txt”: dial tcp: lookup dl.k8s.io on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable

                                    W0804 09:38:33.138340 50405 version.go:104] falling back to the local client version: v1.22.10

                                    cannot use “0.0.0.0” as the bind address for the API Server

                                    怎么它还是要找互联网

                                    1 个月 后

                                    现在新的制品离线安装方式问题很大：

                                    1、组件版本混乱，不知道哪些版本和哪些版本适配

                                    2、自动安装的harbor仓库证书在各个节点缺失，默认没有在/etc/docker/certs.d，要么containerd证书路径空白，导致登陆和push都会报x509

                                    3、制作的离线制品版本不是实际安装时需要的，导致依然去公网拉镜像

                                      willqy

                                      1. 官方文档已经给出了版本的最佳实践，如果需要自行修改部署的组件版本，那么需要用户自行配置实践
                                      2. 不清楚你是怎么配置的，kk安装的harbor会在/etc/ssl/registry/ssl和/etc/docker/certs.d下都放置一份证书
                                      3. 版本配置不对，当然会继续去公网寻找

                                          24sama

                                          2、https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/air-gapped-installation/ 这个文档有个问题，certsPath 这个参数不能少，不然containerd中ca配置是空的

                                            registry:
    auths:
      "dockerhub.kubekey.local":
        username: "xxx"
        password: "***"
        skipTLSVerify: false # Allow contacting registries over HTTPS with failed TLS verification.
        plainHTTP: false # Allow contacting registries over HTTP.
        certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local" # Use certificates at path (*.crt, *.cert, *.key) to connect to the registry.
                                          • cici 回复了此帖