KubeKey 离线环境部署 KubeSphere v3.0.0

robatter

把镜像push到本地仓库了，还是拉取不到

zackzhang

报错很明显了，你看下你的私有仓库是哪个

robatter

zackzhang 私库用的是 dockerhub.kubekey.local

robatter

robatter 上面的 docker images 查的不是我的私库吗？该如何查询我 kubesphere 对就私库的 images?

zackzhang

把镜像推到这个仓库里面去

robatter

zackzhang 就是我上面用docker image查询的镜像仓库和 dockerhub.kuekey.local 的镜像仓库不一样了。请问如何切换查询不同的镜像仓库及推送不同的仓库呢？谢谢！

zackzhang

robatter 打个tag就行了，如：

docker tag kubesphere/examples-bookinfo-details-v1:1.13.0 zackzhangkai/examples-bookinfo-details-v1:1.13.0

robatter

zackzhang
我这个不是已经有 bookinfo 的镜像了吗？但实际运行却报拉不到镜像

robatter

zackzhang
这里老是报正在拉取镜像，我很想知道这个拉取的命令是啥？和我用的 pull 是不是同一个命令？这个拉取的仓库是啥？可以如何去查询和定位我的问题呢？谢谢

robatter

zackzhang 进一步发现一个奇怪的问题，我是三台机器
vm1(192.168.3.101),vm2(192.168.3.102),vm3(192.168.3.103)
在这三台机器上执行命令ping dockerhub.kubekey.local，得到同样的返回：
64 bytes from dockerhub.kubekey.local (192.168.3.101): icmp_seq=1 ttl=64 time=0.508 ms
但只有在 vm1 上执行docker images 才能查到 bookinfo, 在 vm2/vm3 上查不到. 而部署bookinfo pod 的Node在 vm2 上，所以老是查不到。
现在的问题是用同样的账号登录 dockerhub.kubekey.local，vm1上可以查到 bookinfo 的内容而 vm2/vm3 上却查不到？

zackzhang

你没有理解到我说的，你把上下文再好好看一遍，然后在网上搜下怎么把镜像推送到私有仓库

tianlin

images目录里哪些tar是安装k8s+kubesphere 必须的呢，是否有介绍？

zhoachenIt

离线部署KubeSphere v3.0.0我把镜像都推送到Harbor私有仓库中了，KubeSphere也都部署成功了，就是有一个pod jaeger-es-index-cleaner-1614786900-8vns2错误提示：ImagePullBackOff

我查看这个pod的部署文件显示image指定的镜像地址没有指定我的私有仓库地址，但是我看其他部署成功的pod的image地址指定的是我私有仓库的地址，下图是我jaeger-es-index-cleaner-1614786900-8vns2的部署文件

正常来说 image: 10.144.104.127:8001/kubesphere-install/jaegertracing/jaeger-es-index-cleaner:1.17.1 才是正确的，是不是离线部署的整合包漏改了这个pod的部署文件？我这个问题应该怎么解决？

tianlin

@zackzhang , 请帮忙看看
离线安装后，alertmanager启动失败， describe pod信息如下：
Containers:
alertmanager:
Container ID: docker://2e3c1f721ee5164b2ea8c2fa31924616809965416cd67438d2a8d04e9822e855
Image: dockerhub.kubekey.local/prom/alertmanager:v0.21.0
Image ID: docker-pullable://dockerhub.kubekey.local/prom/alertmanager@sha256:913293083cb14085bfc01018bb30d1dcbbc9ed197ae21ef2ca917b0d29265198

Ports:         9093/TCP, 9094/TCP, 9094/UDP
Host Ports:    0/TCP, 0/TCP, 0/UDP
Args:
  --config.file=/etc/alertmanager/config/alertmanager.yaml
  --storage.path=/alertmanager
  --data.retention=120h
  --cluster.listen-address=[$(POD_IP)]:9094
  --web.listen-address=:9093
  --web.route-prefix=/
  --cluster.peer=alertmanager-main-0.alertmanager-operated.kubesphere-monitoring-system.svc:9094
  --cluster.peer=alertmanager-main-1.alertmanager-operated.kubesphere-monitoring-system.svc:9094
  --cluster.peer=alertmanager-main-2.alertmanager-operated.kubesphere-monitoring-system.svc:9094
State:       Waiting
  Reason:    CrashLoopBackOff
Last State:  Terminated
  Reason:    Error
  Message:   level=info ts=2021-03-04T10:45:24.285Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)"level=info ts=2021-03-04T10:45:24.285Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"
  Exit Code:    2

hhh

怎么离线部署 k8s集群

zhq5101

请问可以在银河麒麟V10环境部署KubeSphere吗？谢谢！

Cloud-Shen

zhq5101 应该是完成兼容认证了

wangtao

我在部署离线部署kubesphere时，碰见一下这个报错，请问有人遇到过吗，怎么解决的，谢谢~

li-sen

centos 7.8 systemctl stop firewalld && systemctl disable firewalld 得关闭防火墙。。。不然etcd
etcd无法建立连接

Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
Waiting for etcd to start
ERRO[23:42:24 CST] Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-lisen-master01.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-lisen-master01-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.19.70.80:2379,https://172.19.70.81:2379,https://172.19.70.82:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.19.70.82:2379: connect: no route to host
; error #1: client: endpoint https://172.19.70.80:2379 exceeded header timeout
; error #2: dial tcp 172.19.70.81:2379: connect: no route to host

error #0: dial tcp 172.19.70.82:2379: connect: no route to host
error #1: client: endpoint https://172.19.70.80:2379 exceeded header timeout
error #2: dial tcp 172.19.70.81:2379: connect: no route to host: Process exited with status 1  node=172.19.70.80
Waiting for etcd to start
ERRO[23:42:27 CST] Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-lisen-master03.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-lisen-master03-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.19.70.80:2379,https://172.19.70.81:2379,https://172.19.70.82:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.19.70.80:2379: connect: no route to host
; error #1: dial tcp 172.19.70.81:2379: connect: no route to host
; error #2: client: endpoint https://172.19.70.82:2379 exceeded header timeout

error #0: dial tcp 172.19.70.80:2379: connect: no route to host
error #1: dial tcp 172.19.70.81:2379: connect: no route to host
error #2: client: endpoint https://172.19.70.82:2379 exceeded header timeout: Process exited with status 1  node=172.19.70.82
Waiting for etcd to start
ERRO[23:42:31 CST] Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c "export ETCDCTL_API=2;export ETCDCTL_CERT_FILE='/etc/ssl/etcd/ssl/admin-lisen-master02.pem';export ETCDCTL_KEY_FILE='/etc/ssl/etcd/ssl/admin-lisen-master02-key.pem';export ETCDCTL_CA_FILE='/etc/ssl/etcd/ssl/ca.pem';/usr/local/bin/etcdctl --endpoints=https://172.19.70.80:2379,https://172.19.70.81:2379,https://172.19.70.82:2379 cluster-health | grep -q 'cluster is healthy'"
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.19.70.80:2379: connect: no route to host
; error #1: client: endpoint https://172.19.70.81:2379 exceeded header timeout
; error #2: dial tcp 172.19.70.82:2379: connect: no route to host

error #0: dial tcp 172.19.70.80:2379: connect: no route to host
error #1: client: endpoint https://172.19.70.81:2379 exceeded header timeout
error #2: dial tcp 172.19.70.82:2379: connect: no route to host: Process exited with status 1  node=172.19.70.81
WARN[23:42:31 CST] Task failed ...
WARN[23:42:31 CST] error: interrupted by error
Error: Failed to refresh etcd configuration: interrupted by error
Usage:
  kk create cluster [flags]

Flags:
  -f, --filename string          Path to a configuration file
  -h, --help                     help for cluster
      --skip-pull-images         Skip pre pull images
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-kubesphere          Deploy a specific version of kubesphere (default v3.0.0)
  -y, --yes                      Skip pre-check of the installation

Global Flags:
      --debug   Print detailed information (default true)

Failed to refresh etcd configuration: interrupted by error

在这提几点建议：
1、每台机器手动去关防火墙也是件麻烦事，不能自动做了？
2、离线包导了很多无关东西，时间太久了 k8s版本都好几个了，是不是可以做下判断。。。没开启的插件相关是不是可以选择不导入？
3、bash-completion 安装也加下不能tab也头痛

Jeff

li-sen

li-sen 在这提几点建议：
1、每台机器手动去关防火墙也是件麻烦事，不能自动做了？

每台机器的防火墙可能不相同，如果使用虚拟机，可能防火墙是在虚拟机之外设置的。这个比较复杂，而且防火墙对于安全比较重要，ks要求的是打开特定的端口，不需要完全关闭防火墙

2、离线包导了很多无关东西，时间太久了 k8s版本都好几个了，是不是可以做下判断。。。没开启的插件相关是不是可以选择不导入？

这个导入是为了后续开启组件方便，如果不需要的话可以手动去除下

3、bash-completion 安装也加下不能tab也头痛

bash-completion我们看下，参照官网设置下，很简单的

jiasha

可以在2.1.1的基础上直接升级至3.0吗？

li-sen

Jeff 那把harbor 等以及应用商店里的镜像也整成离线的吧。。。我们有选择的删除，现在装个harbor 都收工整理镜像，我们这基本都是离线环境。。。。