k8s 1.20.7 + KubeSphere 3.1.0安装失败

hvsop

安装程序卡在这里，redis-ha-haproxy的pod无法创建，而且并没有自动创建redis-ha-server。细节如下：

CentOS 7.6 8C/16G/500G
3Masters + 5 workers

[root@kvm-10-115-6-11 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage (default) fuseim.pri/ifs Delete Immediate false 14h

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system get pods
NAME READY STATUS RESTARTS AGE
ks-apiserver-6c6bcff4df-jxnw9 0/1 CrashLoopBackOff 150 12h
ks-apiserver-6c6bcff4df-q5wpr 0/1 CrashLoopBackOff 150 12h
ks-apiserver-6c6bcff4df-rb9h8 0/1 CrashLoopBackOff 150 12h
ks-console-7877478685-b22x7 1/1 Running 0 12h
ks-console-7877478685-rswml 1/1 Running 0 12h
ks-console-7877478685-wqjbn 1/1 Running 0 12h
ks-controller-manager-6ffb9df94c-5txt8 1/1 Running 0 12h
ks-controller-manager-6ffb9df94c-7fz64 1/1 Running 0 12h
ks-controller-manager-6ffb9df94c-lnkkq 1/1 Running 0 12h
ks-installer-6f4c9fcfcf-gfhvk 1/1 Running 0 12h
redis-ha-haproxy-68ff446d79-289pt 0/1 Init:0/1 87 12h
redis-ha-haproxy-68ff446d79-6s2mz 0/1 Init:Error 88 12h
redis-ha-haproxy-68ff446d79-s8z55 0/1 Init:0/1 88 12h

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system describe pod ks-apiserver-6c6bcff4df-jxnw9
Name: ks-apiserver-6c6bcff4df-jxnw9
Namespace: kubesphere-system
Priority: 0
Node: kvm-10-115-6-11/10.115.6.11
Start Time: Wed, 30 Jun 2021 21:00:13 +0800
Labels: app=ks-apiserver
pod-template-hash=6c6bcff4df
tier=backend
version=v3.1.0
Annotations: cni.projectcalico.org/podIP: 10.244.102.89/32
cni.projectcalico.org/podIPs: 10.244.102.89/32
kubectl.kubernetes.io/restartedAt: 2021-06-30T21:00:13+08:00
Status: Running
IP: 10.244.102.89
IPs:
IP: 10.244.102.89
Controlled By: ReplicaSet/ks-apiserver-6c6bcff4df
Containers:
ks-apiserver:
Container ID: containerd://39919caf690a0c5ba1105327abf2b9854ef2e9b30e9ab6e3819db28e6777da48
Image: kubesphere/ks-apiserver:v3.1.0
Image ID: docker.io/kubesphere/ks-apiserver@sha256:4f15c66506aa03b6c6e3928a6440bb2e3c43d13d6ee81bf8cfe73b02247d1c97
Port: 9090/TCP
Host Port: 0/TCP
Command:
ks-apiserver
--logtostderr=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 01 Jul 2021 09:34:31 +0800
Finished: Thu, 01 Jul 2021 09:34:36 +0800
Ready: False
Restart Count: 150
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 20m
memory: 100Mi
Liveness: http-get http://:9090/kapis/version delay=15s timeout=15s period=10s #success=1 #failure=8
Environment:
Mounts:
/etc/kubesphere/ from kubesphere-config (rw)
/etc/kubesphere/ingress-controller from ks-router-config (rw)
/etc/localtime from host-time (rw)
/var/run/docker.sock from docker-sock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kubesphere-token-8sb7j (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
docker-sock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
ks-router-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ks-router-config
Optional: false
kubesphere-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kubesphere-config
Optional: false
host-time:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
kubesphere-token-8sb7j:
Type: Secret (a volume populated by a Secret)
SecretName: kubesphere-token-8sb7j
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
Type Reason Age From Message
Warning BackOff 3m42s (x3500 over 12h) kubelet Back-off restarting failed container
[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system logs ks-apiserver-6c6bcff4df-jxnw9
W0701 09:39:41.942294 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
W0701 09:39:41.944401 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
W0701 09:39:41.959082 1 metricsserver.go:238] Metrics API not available.
Error: failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout
2021/07/01 09:39:46 failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system describe pod redis-ha-haproxy-68ff446d79-289pt
Name: redis-ha-haproxy-68ff446d79-289pt
Namespace: kubesphere-system
Priority: 0
Node: kvm-10-115-6-11/10.115.6.11
Start Time: Wed, 30 Jun 2021 21:07:17 +0800
Labels: app=redis-ha-haproxy
pod-template-hash=68ff446d79
release=ks-redis
Annotations: checksum/config: 46138f8b2005948188bc12c93b08a9c6460354af2db98eaa2158bcf1717e82de
cni.projectcalico.org/podIP: 10.244.102.91/32
cni.projectcalico.org/podIPs: 10.244.102.91/32
prometheus.io/path: /metrics
prometheus.io/port: 9101
prometheus.io/scrape: true
Status: Pending
IP: 10.244.102.91
IPs:
IP: 10.244.102.91
Controlled By: ReplicaSet/redis-ha-haproxy-68ff446d79
Init Containers:
config-init:
Container ID: containerd://edc1cfc94dc84cc3fa04f533448712e6e8734fd6e7e15d3890b1585ff5657eb3
Image: haproxy:2.0.4
Image ID: docker.io/library/haproxy@sha256:cd4b3d4d27ae5931dc96b9632188590b7a6880469bcf07f478a3280dd0955336
Port:
Host Port:
Command:
sh
Args:
/readonly/haproxy_init.sh
State: Running
Started: Thu, 01 Jul 2021 09:35:16 +0800
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 01 Jul 2021 09:27:30 +0800
Finished: Thu, 01 Jul 2021 09:35:01 +0800
Ready: False
Restart Count: 87
Environment:
Mounts:
/data from data (rw)
/readonly from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from redis-ha-haproxy-token-gtkpn (ro)
Containers:
haproxy:
Container ID:
Image: haproxy:2.0.4
Image ID:
Port: 6379/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8888/healthz delay=5s timeout=1s period=3s #success=1 #failure=3
Environment:
Mounts:
/run/haproxy from shared-socket (rw)
/usr/local/etc/haproxy from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from redis-ha-haproxy-token-gtkpn (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-ha-configmap
Optional: false
shared-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
redis-ha-haproxy-token-gtkpn:
Type: Secret (a volume populated by a Secret)
SecretName: redis-ha-haproxy-token-gtkpn
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
Type Reason Age From Message
Normal Pulled 23m (x86 over 12h) kubelet Container image "haproxy:2.0.4" already present on machine
Warning BackOff 6m12s (x433 over 12h) kubelet Back-off restarting failed container

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system logs redis-ha-haproxy-68ff446d79-289pt
Error from server (BadRequest): container "haproxy" in pod "redis-ha-haproxy-68ff446d79-289pt" is waiting to start: PodInitializing

有哪位大神帮忙看一下是什么地方出问题了吗？谢谢！

hongming

hvsop

kubectl -n kubesphere-system logs redis-ha-haproxy-68ff446d79-289pt -c config-init

hvsop

hongming

我卸载了重新安装了一次，这次看到redis-ha-server-0这个pod了，但是状态还是pending:

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system get pods
NAME READY STATUS RESTARTS AGE
ks-apiserver-785fbdb85b-6bp2j 0/1 CrashLoopBackOff 68 5h28m
ks-apiserver-785fbdb85b-mvftr 0/1 CrashLoopBackOff 67 5h28m
ks-apiserver-785fbdb85b-xbv7s 0/1 CrashLoopBackOff 67 5h29m
ks-console-7877478685-nwzwx 1/1 Running 0 5h32m
ks-console-7877478685-vkpzq 1/1 Running 0 5h32m
ks-console-7877478685-xqqhm 1/1 Running 0 5h32m
ks-controller-manager-6c67dfbd8f-77v9j 1/1 Running 0 5h29m
ks-controller-manager-6c67dfbd8f-qvnjl 1/1 Running 0 5h28m
ks-controller-manager-6c67dfbd8f-r2c2g 1/1 Running 0 5h28m
ks-installer-6f4c9fcfcf-sprgg 1/1 Running 0 5h35m
redis-ha-haproxy-68ff446d79-2db98 0/1 Init:0/1 39 5h33m
redis-ha-haproxy-68ff446d79-vw6k6 0/1 Init:0/1 39 5h33m
redis-ha-haproxy-68ff446d79-wchds 0/1 Init:0/1 39 5h33m
redis-ha-server-0 0/2 Pending 0 5h33m

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system logs redis-ha-haproxy-68ff446d79-2db98 -c config-init
Waiting for service redis-ha-announce-0 to be ready (1) …

[root@kvm-10-115-6-11 ~]# kubectl -n kubesphere-system logs redis-ha-server-0
error: a container name must be specified for pod redis-ha-server-0, choose one of: [redis sentinel] or one of the init containers: [config-init]

请问您有其他线索吗？

snifferqq

Kuberneters 版本信息

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c6a2f08fc4378c5381dd948d9ad9d1080e3e6b33", GitTreeState:"clean", BuildDate:"2021-05-12T12:27:07Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c6a2f08fc4378c5381dd948d9ad9d1080e3e6b33", GitTreeState:"clean", BuildDate:"2021-05-12T12:19:22Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

kubesphere/ks-installer:v3.1.1

遇到相同的问题 redis-ha-haproxy有三个Pod ,其中两个Pod 是正常的。异常如下

kubectl logs    -n kubesphere-system   redis-ha-haproxy-5f597cb895-rbhdb config-init
Waiting for service redis-ha-announce-0 to be ready (1) ...
Waiting for service redis-ha-announce-0 to be ready (2) ...
Waiting for service redis-ha-announce-0 to be ready (3) ...
Waiting for service redis-ha-announce-0 to be ready (4) ...
Waiting for service redis-ha-announce-0 to be ready (5) ...
Waiting for service redis-ha-announce-0 to be ready (6) ...
Waiting for service redis-ha-announce-0 to be ready (7) ...
Waiting for service redis-ha-announce-0 to be ready (8) ...
Waiting for service redis-ha-announce-0 to be ready (9) ...
Waiting for service redis-ha-announce-0 to be ready (10) ...
Could not resolve the announce ip for redis-ha-announce-0

正常的如下

kubectl logs    -n kubesphere-system    redis-ha-haproxy-5f597cb895-4lcnw  config-init
10.98.156.184     redis-ha-announce-0.kubesphere-system.svc.cluster.local  redis-ha-announce-0.kubesphere-system.svc.cluster.local redis-ha-announce-0
10.98.111.167     redis-ha-announce-1.kubesphere-system.svc.cluster.local  redis-ha-announce-1.kubesphere-system.svc.cluster.local redis-ha-announce-1
10.98.149.87      redis-ha-announce-2.kubesphere-system.svc.cluster.local  redis-ha-announce-2.kubesphere-system.svc.cluster.local redis-ha-announce-2

snifferqq

另外
kubesphere-system redis-ha-server-0 0/2 Pending 0 17h
kubectl logs -n kubesphere-system redis-ha-server-0 config-init
响应空，没有日志输出