作者：丁鑫磊，云原生运维工程师，专注于 KubeSphere 与 K8s 的深度应用，致力于自动化方向的探索与实践。热衷于挖掘 KubeSphere 的运维潜力，借助其简化 K8s 操作，提升运维效率，为企业云原生转型推进贡献力量。

1. 背景

1.1 KubeKey 3.1.9 更新

发生了什么变化 ：支持新增 K8s 版本。
漏洞修补 ：
- 修复了 kubelet 的 cgroup 配置始终使用默认 systemd 的问题。
- 修复了禁用 ufw 会清除 ipvs 规则，丢弃缓存会影响稳定性的问题。

1.2 K8s 1.33.0 更新亮点

无需重启的资源动态调整：原地垂直扩展。
Sidecar 容器功能 GA：提升辅助容器的可靠性。
Indexed Jobs 功能 GA：支持批量任务的精细化管理和控制。
服务账户令牌更安全、更智能。
Kubectl子资源支持：简化资源交互。
服务 CIDR 动态扩展：网络增长更灵活。
用户命名空间（User Namespaces）功能增强，提升多租户隔离能力。
OCI 镜像挂载：工具和配置交付更简单。
有序命名空间删除：资源清理更优雅。

1.3 部署目标

本次部署主要展示 KubeKey，可以自定义 Harbor 默认密码、数据目录、Etcd 数据目录、Docker 数据目录、Containerd 数据目录。

2. 环境软件版本信息

软件	版本
操作系统	openEuler 22.03 (LTS-SP3) amd-64
Docker	24.0.9
Harbor	v2.10.1 -b7b88476
Kubernetes	v1.33.0
KubeSphere	v4.1.3
KubeKey	v3.1.9

3. 服务器信息规划

主机 IP 地址	主机名	角色名称
192.168.118.180	k8s-master1	master
192.168.118.181	k8s-node01	worker
192.168.118.182	k8s-node02	worker
192.168.118.183	k8s-harbor	harbor
192.168.118.169	k8s-file	file（外网打包服务器）

4. 主机初始化配置

4.1 配置静态 IP

vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
IPADDR=192.168.118.180
NETMASK=255.255.255.0
GATEWAY=192.168.118.2
DNS1=192.168.118.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
DEVICE=ens33
ONBOOT=yes

4.2 关闭 Selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# 修改 Selinux 配置文件之后，重启机器，Selinux 配置才能永久生效
setenforce 0
# 临时关闭

4.3 修改主机名

hostnamectl set-hostname master && bash
hostnamectl set-hostname node1 && bash
hostnamectl set-hostname node2 && bash
hostnamectl set-hostname k8s-harbor && bash

4.4 关闭交换分区 swap，提升性能

# 临时关闭
swapoff -a
# 永久关闭
vim /etc/fstab
#/dev/mapper/centos-swap swap      swap    defaults        0 0

4.5 关闭 firewalld 防火墙

systemctl stop firewalld ; systemctl disable firewalld

4.6 安装基础包

yum install curl socat conntrack ebtables ipset ipvsadm -y

4.7 创建数据目录

# 这里存放容器运行时 Harbor Etcd 数据目录，要挂载数据盘
mkdir -p /data

5. 离线包准备

5.1 下载 KubeKey 3.1.9

# 默认下载最新版本
curl -sSL https://get-kk.kubesphere.io | sh -
# 受网络限制，可以采用迅雷等第三方工具下载
https://github.com/kubesphere/kubekey/releases/download/v3.1.9/kubekey-v3.1.9-linux-amd64.tar.gz

5.2 准备`manifest.yaml`文件

执行 kk 命令生成 manifest-sample.yaml 文件：
./kk create manifest --with-kubernetes v1.33.0 --with-registry
vim manifest-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems: []
  kubernetesDistributions:
  - type: kubernetes
    version: v1.33.0
  components:
    helm:
      version: v3.14.3
    cni:
      version: v1.2.0
    etcd:
      version: v3.5.13
    containerRuntimes:
    - type: docker
      version: 24.0.9
    - type: containerd
      version: 1.7.13
    calicoctl:
      version: v3.27.4
    crictl:
      version: v1.29.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.10.1
    docker-compose:
      version: v2.26.1
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.33.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.33.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.33.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.33.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.21.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel-cni-plugin:v1.1.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cilium:v1.15.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/operator-generic:v1.15.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hybridnet:v0.8.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-ovn:v1.10.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/multus-cni:v3.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.9.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-vip:v0.7.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kata-deploy:stable
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-feature-discovery:v0.10.0
    ## ks-core
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-apiserver:v4.1.3
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-console:v4.1.3
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-controller-manager:v4.1.3
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.16
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/redis:7.2.4-alpine
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/haproxy:2.9.6-alpine
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-extensions-museum:v1.1.6
   ## metrics-server
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/metrics-server:v0.7.0
  - swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/addon-resizer:1.8.20
  registry:
    auths: {}

5.3 导出镜像制品

export KKZONE=cn
./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz

5.4 下载 KubeSphere Core Helm Chart

# 安装 Helm：
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# 下载 KubeSphere Core Helm Chart：
# 最新 Chart 版本：helm-chart-1.1.5
VERSION=1.1.5    
helm fetch https://charts.kubesphere.io/main/ks-core-${VERSION}.tgz
# 版本需要网站中查看
https://get-images.kubesphere.io/

5.5 将文件拷贝到内网服务器

将以上下载的离线包和配置文件通过 SCP 或其他工具拷贝到目标内网服务器。

6. 安装 Harbor

6.1 创建 config-sample.yaml 配置文件

./kk create config  --with-kubernetes v1.33.0 -f config-sample.yaml

修改示例配置结果如下：

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: k8s-master01, address: 192.168.118.180, internalAddress: 192.168.118.180, user: root, password: "******"}
  - {name: k8s-node01, address: 192.168.118.181, internalAddress: 192.168.118.181, user: root, password: "******"}
  - {name: k8s-node02, address: 192.168.118.182, internalAddress: 192.168.118.182, user: root, password: "******"}
  - {name: k8s-harbor, address: 192.168.118.183, internalAddress: 192.168.118.183, user: root, password: "******"}
  roleGroups:
    etcd:
    - k8s-master01
    control-plane:
    - k8s-master01
    worker:
    - k8s-node01
    - k8s-node02
    registry:
    - k8s-harbor
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    port: 6443
  system:
    # The ntp servers of chrony.
    ntpServers:
      - ntp.aliyun.com
    timezone: "Asia/Shanghai"
  kubernetes:
    version: v1.32.2
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    multusCNI:
      enabled: false
  storage:
    openebs:
      basePath: /data/openebs
  registry:
    type: "harbor"
    auths:
      "dockerhub.kubekey.local":
        username: admin
        # 修改 Harbor 密码
        password: Harbor0987655
        certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local"
        skipTLSVerify: true
        plainHTTP: false
    privateRegistry: "dockerhub.kubekey.local"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
    ### 修改容器运行时数据目录及 Harbor 数据目录
    containerdDataDir: /data/containerd
    dockerDataDir: /data/docker
    registryDataDir: /data/registry
  addons: []

6.2 安装 Harbor

./kk init registry -f config-sample.yaml -a kubesphere.tar.gz

6.3 创建 Harbor 项目

#!/usr/bin/env bash
url="https://dockerhub.kubekey.local"  # 修改为真实镜像仓库地址
user="admin"    
passwd="******"    ## 修改为真实的密码
harbor_projects=(
        ks
        kubesphere
        kubesphereio
        coredns
        calico
        flannel
        cilium
        hybridnetdev
        kubeovn
        openebs
        library
        plndr
        jenkins
        argoproj
        dexidp
        openpolicyagent
        curlimages
        grafana
        kubeedge
        nginxinc
        prom
        kiwigrid
        minio
        opensearchproject
        istio
        jaegertracing
        timberio
        prometheus-operator
        jimmidyson
        elastic
        thanosio
        brancz
        prometheus
)

for project in "${harbor_projects[@]}"; do
    echo "creating $project"
    curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k  # 注意在 curl 命令末尾加上 -k
done

执行脚本创建项目：

chmod +x create_project_harbor.sh
./create_project_harbor.sh

6.4 推送项目到 Harbor

./kk artifact image push -f config-sample.yaml -a kubesphere.tar.gz

7. 安装 K8s

7.1 安装 K8s 集群

./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-local-storage --skip-push-images
查看更多可用参数，可以执行以下命令：
./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --help
参数说明：
- --skip-push-images ：跳过预推送镜像
- --with-packages    ：安装操作系统依赖（需要 ISO 文件）
- --with-local-storage ：部署本地存储（Local PV Provisioner）

7.2 查看集群状态

kubectl get nodes

8. 安装 KubeSphere

8.1 安装 KubeSphere

helm upgrade --install -n kubesphere-system --create-namespace ks-core ks-core-1.1.5.tgz \
     --set global.imageRegistry=dockerhub.kubekey.local/ks \
     --set extension.imageRegistry=dockerhub.kubekey.local/ks \
     --set ksExtensionRepository.image.tag=v1.1.6\ 
     --debug \
     --wait

--set ksExtensionRepository.image.tag=v1.1.6 ：最新扩展组件仓库版本为 v1.1.6。
ks-core ks-core-1.1.5.tgz ：最新 Helm Chart 版本为 helm-chart-1.1.5。

8.2 安装结果

NOTES:
Thank you for choosing KubeSphere Helm Chart.

Please be patient and wait for several seconds for the KubeSphere deployment to complete.

1. Wait for Deployment Completion

    Confirm that all KubeSphere components are running by executing the following command:

    kubectl get pods -n kubesphere-system
2. Access the KubeSphere Console

    Once the deployment is complete, you can access the KubeSphere console using the following URL:  

    http://192.168.118.185:30880

3. Login to KubeSphere Console

    Use the following credentials to log in:

    Account: admin
    Password: P@88w0rd

8.3 验证数据目录

验证 Docker Harbor 数据目录

[root@k8s-harbor data]# ll
总用量 8
drwx--x---. 12 root root 4096  5 月 22 16:10 docker
drwxr-xr-x.  9 root root 4096  5 月 22 16:15 registry

验证 Etcd Containerd 数据目录

[root@master data]# ll
总用量 12
drwx--x--x. 11 root root 4096  5 月 21 17:29 containerd
drwx------   3 root root 4096  5 月 22 09:57 etcd

结语

至此，基于 KubeKey 3.1.9 与 K8s 1.33.0 的完整安装部署流程已全部完成，结合 KubeSphere 提供的管理能力，企业可实现更高效、更可控的云原生平台建设。后续可根据业务需求进行扩展配置和二次优化。