kubesphere_v3.3最小化安装过程monitoring模块timeout

fragments

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息

虚拟化工具：Oracle VM VirtualBox
系统版本：Centos7.9 minimal
内核版本：5.19.10-1.el7.elrepo.x86_64
2Core 4G内存一主两从

Kubernetes版本信息
{Major:“1”, Minor:“24”, GitVersion:“v1.24.2”, GitCommit:“f66044f4361b9f1f96f0053dd46cb7dce5e990a8”, GitTreeState:“clean”, BuildDate:“2022-06-15T14:22:29Z”, GoVersion:“go1.18.3”, Compiler:“gc”, Platform:“linux/amd64”}

ps. 当然我知道1.24还不支持，但是我想试一下

容器运行时

版本：20.10.18
API版本：1.41
Go版本：1.18.6
实际用的应该是containerd+runc

KubeSphere版本信息

k8s上最小化安装， v3.3

问题是什么
无法安装完成

ks-controller-manager卡ContainerCreating

kubectl describe pod ks-controller-manager-6bc65d7478-8pgsv -n kubesphere-system

ks-apiserver卡ContainerCreating

kubectl describe pod ks-apiserver-5496b564c5-zt5jg -n kubesphere-system

关于installer
kubectl logs -n 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:23+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:24+08:00 INFO 2022-09-27T10:52:27+08:00 INFO 2022-09-27T10:52:27+08:00 INFO Delete old cluster configuration Create cluster configuration 2022-09-27T10:52:28+08:00 INFO 2022-09-27T10:52:28+08:00 INFO 2022-09-27T10:52:30+08:00 INFO 2022-09-27T10:52:30+08:00 INFO [WARNING]: No inventory [WARNING]: provided hosts the implicit localhost kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
: shell-operator latest
: Use temporary dir: /tmp/shell-operator
: Initialize hooks manager …
: Search and load hooks …
: Load hook config from ‘/hooks/kubesphere/installRunner.py’
: HTTP SERVER Listening on 0.0.0.0:9115
: Load hook config from ‘/hooks/kubesphere/schedule.sh’
: Initializing schedule manager …
: KUBE Init Kubernetes client
: KUBE-INIT Kubernetes client is configured successfully
: MAIN: run main loop
: MAIN: add onStartup tasks
: QUEUE add all HookRun@OnStartup
: MSTOR Create new metric shell_operator_live_ticks
: MSTOR Create new metric shell_operator_tasks_queue_length
: Running schedule manager …
: GVR for kind ‘ClusterConfiguration’ is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations
: EVENT Kube event ‘ef9fe461-f69a-4032-90e4-ba885e631850’
: QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
: TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
: Running hook ‘kubesphere/installRunner.py’ binding ‘KUBE_EVENTS’ …
successfully
successfully
: EVENT Kube event ‘ef9fe461-f69a-4032-90e4-ba885e631850’
: QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
: TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
: Running hook ‘kubesphere/installRunner.py’ binding ‘KUBE_EVENTS’ …
was parsed, only implicit localhost is available
list is empty, only localhost is available. Note that
does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [download : Generating images list] ***************************************
skipping: [localhost]

TASK [download : Synchronizing images] *****************************************

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [preinstall : KubeSphere | Stopping if Kubernetes version is nonsupport] ***
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [preinstall : KubeSphere | Checking StorageClass] *************************
changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if StorageClass was not found] ********
skipping: [localhost]

TASK [preinstall : KubeSphere | Checking default StorageClass] *****************
changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if default StorageClass was not found] ***
ok: [localhost] => {
“changed”: false,
“msg”: “All assertions passed”
}

TASK [preinstall : KubeSphere | Checking KubeSphere component] *****************
changed: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
changed: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost] => (item=ks-openldap)
skipping: [localhost] => (item=ks-redis)
skipping: [localhost] => (item=ks-minio)
skipping: [localhost] => (item=ks-openpitrix)
skipping: [localhost] => (item=elasticsearch-logging)
skipping: [localhost] => (item=elasticsearch-logging-curator)
skipping: [localhost] => (item=istio)
skipping: [localhost] => (item=istio-init)
skipping: [localhost] => (item=jaeger-operator)
skipping: [localhost] => (item=ks-jenkins)
skipping: [localhost] => (item=ks-sonarqube)
skipping: [localhost] => (item=logging-fluentbit-operator)
skipping: [localhost] => (item=uc)
skipping: [localhost] => (item=metrics-server)

PLAY RECAP *********************************************************************
localhost : ok=7 changed=4 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [download : Generating images list] ***************************************
skipping: [localhost]

TASK [download : Synchronizing images] *****************************************

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [Metrics-Server | Getting metrics-server installation files] **************
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Creating manifests] ********************
skipping: [localhost] => (item={‘file’: ‘metrics-server.yaml’})

TASK [metrics-server : Metrics-Server | Checking Metrics-Server] ***************
skipping: [localhost]

TASK [Metrics-Server | Uninstalling old metrics-server] ************************
skipping: [localhost]

TASK [Metrics-Server | Installing new metrics-server] **************************
skipping: [localhost]

TASK [metrics-server : Metrics-Server | Waitting for metrics.k8s.io ready] *****
skipping: [localhost]

TASK [Metrics-Server | Importing metrics-server status] ************************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’

PLAY [localhost] ***************************************************************

TASK [download : Generating images list] ***************************************
skipping: [localhost]

TASK [download : Synchronizing images] *****************************************

TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
skipping: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
“msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
}

TASK [common : KubeSphere | Checking kube-node-lease namespace] ****************
changed: [localhost]

TASK [common : KubeSphere | Getting system namespaces] *************************
ok: [localhost]

TASK [common : set_fact] *******************************************************
ok: [localhost]

TASK [common : debug] **********************************************************
ok: [localhost] => {
“msg”: [
“kubesphere-system”,
“kubesphere-controls-system”,
“kubesphere-monitoring-system”,
“kubesphere-monitoring-federated”,
“kube-node-lease”
]
}

TASK [common : KubeSphere | Creating KubeSphere namespace] *********************
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling system-workspace] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kubesphere-monitoring-federated)
changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling namespace for network policy] *************
changed: [localhost]

TASK [common : KubeSphere | Getting Kubernetes master num] *********************
changed: [localhost]

TASK [common : KubeSphere | Setting master num] ********************************
ok: [localhost]

TASK [KubeSphere | Getting common component installation files] ****************
changed: [localhost] => (item=common)

TASK [common : KubeSphere | Checking Kubernetes version] ***********************
changed: [localhost]

TASK [KubeSphere | Getting common component installation files] ****************
changed: [localhost] => (item=snapshot-controller)

TASK [common : KubeSphere | Creating snapshot controller values] ***************
changed: [localhost] => (item={‘name’: ‘custom-values-snapshot-controller’, ‘file’: ‘custom-values-snapshot-controller.yaml’})

TASK [common : KubeSphere | Updating snapshot crd] *****************************
changed: [localhost]

TASK [common : KubeSphere | Deploying snapshot controller] *********************
changed: [localhost]

TASK [KubeSphere | Checking openpitrix common component] ***********************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “/usr/local/bin/kubectl get pod -n openpitrix-system\n”, “delta”: “0:00:02.699491”, “end”: “2022-09-27 10:54:39.722422”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-09-27 10:54:37.022931”, “stderr”: “Unable to connect to the server: dial tcp 10.96.0.1:443: connect: no route to host”, “stderr_lines”: [“Unable to connect to the server: dial tcp 10.96.0.1:443: connect: no route to host”], “stdout”: "", “stdout_lines”: []}

PLAY RECAP *********************************************************************
localhost : ok=16 changed=11 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0

关于所有pod

NAMESPACE NAME READY STATUS RES TARTS AGE

calico-apiserver calico-apiserver-87f8c8cdc-745mm 1/1 Running 7 ( 6m29s ago) 57m

calico-apiserver calico-apiserver-87f8c8cdc-x5pbh 1/1 Running 4 ( 64m ago) 20h

calico-system calico-kube-controllers-5b8957ccd7-vfwk4 1/1 Running 4 ( 6m9s ago) 57m

calico-system calico-node-jr7ll 1/1 Running 2 ( 35m ago) 25h

calico-system calico-node-lzxlh 1/1 Running 1 ( 155m ago) 25h

calico-system calico-node-svtd9 1/1 Running 3 ( 63m ago) 25h

calico-system calico-typha-84898fcf9b-jn78k 1/1 Running 4 ( 35m ago) 25h

calico-system calico-typha-84898fcf9b-n2xgp 1/1 Running 2 ( 153m ago) 25h

calico-system csi-node-driver-8zffd 2/2 Running 2 ( 154m ago) 25h

calico-system csi-node-driver-b52sp 2/2 Running 2 ( 155m ago) 25h

calico-system csi-node-driver-wcfz5 2/2 Running 4 ( 35m ago) 25h

kube-system coredns-7f74c56694-d7×8z 1/1 Running 1 ( 154m ago) 20h

kube-system coredns-7f74c56694-vxm84 1/1 Running 0 57m

kube-system etcd-k8s-master-0 1/1 Running 6 ( 87m ago) 4d19h

kube-system kube-apiserver-k8s-master-0 1/1 Running 6 ( 154m ago) 4d19h

kube-system kube-controller-manager-k8s-master-0 1/1 Running 31 (33m ago) 4d19h

kube-system kube-proxy-p6l2f 1/1 Running 1 ( 155m ago) 21h

kube-system kube-proxy-ttszg 1/1 Running 1 ( 154m ago) 21h

kube-system kube-proxy-wdffj 1/1 Running 2 ( 35m ago) 21h

kube-system kube-scheduler-k8s-master-0 1/1 Running 29 (53m ago) 4d19h

kube-system metrics-server-6df598dddf-mf6xs 1/1 Running 1 ( 6m38s ago) 57m

kube-system snapshot-controller-0 1/1 Running 0 132m

kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-slszw 1/1 Running 0 57m

kubernetes-dashboard kubernetes-dashboard-67bd8fc546-kdlvr 1/1 Running 1 ( 5m45s ago) 57m

kubesphere-controls-system default-http-backend-69478ff5f9-99pkw 1/1 Running 0 57m

kubesphere-system ks-apiserver-5496b564c5-zt5jg 0/1 ContainerCreating 0 61m

kubesphere-system ks-console-99dc7cb7c-rnz8q 1/1 Running 0 61m

kubesphere-system ks-controller-manager-6bc65d7478-8pgsv 0/1 ContainerCreating 0 61m

kubesphere-system ks-installer-b6954b57f-nxl76 1/1 Running 3 ( 56m ago) 57m

tigera-operator tigera-operator-6bb888d6fc-hbbmt 1/1 Running 20 (7m32s ago) 26h

关于网络

10.96.0.0/16是服务网段

default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d19h

fragments

kubectl describe pods default-http-backend-56d9d4fdf7-scsc4 -n kubesphere-controls-system
Events:
Type Reason Age From Message

—

Normal Scheduled 16m default-scheduler Successfully assigned kubesphere- controls-system/default-http-backend-56d9d4fdf7-scsc4 to k8s-worker-1
Warning FailedCreatePodSandBox 14m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod “default-http-backend-56d9d4fdf7-s csc4”: operation timeout: context deadline exceeded
Normal SandboxChanged 9m37s kubelet Pod sandbox changed, it will be k illed and re-created.
Warning FailedCreatePodSandBox 3m52s kubelet Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning FailedSync 56s (x8 over 12m) kubelet error determining status: rpc err or: code = DeadlineExceeded desc = context deadline exceeded

kubectl describe pods ks-controller-manager-7b5f77b47f-256cd -n kubesphere-system
Events:

Type Reason Age From Message

—

Normal Scheduled 15m default-scheduler Successfully assigned kubesphere-system/ks -controller-manager-7b5f77b47f-256cd to k8s-master-0

Warning FailedMount 13m kubelet Unable to attach or mount volumes: unmount ed volumes=[kubesphere-config], unattached volumes=[kube-api-access-nsqzn kubesphere-config webhook-secret host-time]: timed out waiting for the condition

Warning FailedMount 4m30s (x4 over 11m) kubelet Unable to attach or mount volumes: unmount ed volumes=[kubesphere-config], unattached volumes=[kubesphere-config webhook-secret host-time kube-api-ac cess-nsqzn]: timed out waiting for the condition

Warning FailedMount 2m12s kubelet Unable to attach or mount volumes: unmount ed volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-nsqzn kubesphere-config webh ook-secret]: timed out waiting for the condition

Warning FailedMount 72s (x15 over 15m) kubelet MountVolume.SetUp failed for volume “kubes phere-config” : configmap “kubesphere-config” not found

kubectl describe pods ks-apiserver-5db774f4f-99vvl -n kubesphere-system
Events:
Type Reason Age From Message

—

Normal Scheduled 15m default-scheduler Successfully assigned kubesphere-system/ ks-apiserver-5db774f4f-99vvl to k8s-master-0
Warning FailedMount 6m17s (x2 over 8m31s) kubelet Unable to attach or mount volumes: unmou nted volumes=[kubesphere-config], unattached volumes=[kubesphere-config host-time kube-api-access-ld8wz ks -router-config]: timed out waiting for the condition
Warning FailedMount 4m1s (x2 over 10m) kubelet Unable to attach or mount volumes: unmou nted volumes=[kubesphere-config], unattached volumes=[ks-router-config kubesphere-config host-time kube-ap i-access-ld8wz]: timed out waiting for the condition
Warning FailedMount 107s (x2 over 13m) kubelet Unable to attach or mount volumes: unmou nted volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-ld8wz ks-router-config kub esphere-config]: timed out waiting for the condition
Warning FailedMount 46s (x15 over 15m) kubelet MountVolume.SetUp failed for volume “kub esphere-config” : configmap “kubesphere-config” not found

frezes

重启下ks-installer，在安装时意外连不上kube-apiserver，导致安装失败，需要重试。

fragments

frezes
你好，我试了一下，好像是内存刚好卡在2g的话会出现这个异常，我把虚拟机从4g调到5.5g，装kubesphere还有3g多就没报错了
另外ks-installer是不支持回滚或者安装超时吗，超时的时候好像不会re-create，也不会把之前已经拉起来的pod删掉
我最后把k8s换成1.22.10拉起来之后Pod数量变得很奇怪