部署失败，无redis服务

Lleo · 2020年2月17日

NAME                         	REVISION	UPDATED                 	STATUS  	CHART                       	APP VERSION   	NAMESPACE
elasticsearch-logging        	1       	Mon Feb 17 20:49:03 2020	DEPLOYED	elasticsearch-1.22.1        	6.7.0-0217    	kubesphere-logging-system
elasticsearch-logging-curator	1       	Mon Feb 17 20:49:08 2020	DEPLOYED	elasticsearch-curator-1.3.3 	5.5.4-0217    	kubesphere-logging-system
gitlab                       	7       	Fri Sep 20 18:52:28 2019	DEPLOYED	gitlab-ce-0.2.1             	              	gitlab
harbor                       	1       	Thu Aug 15 16:08:14 2019	DEPLOYED	harbor-1.1.1                	1.8.1         	harbor
ingress-nginx                	1       	Wed Aug 14 18:14:10 2019	DEPLOYED	nginx-ingress-0.9.5         	0.10.2        	kube-system
ks-openldap                  	1       	Mon Feb 17 20:47:36 2020	DEPLOYED	openldap-ha-0.1.0           	1.0           	kubesphere-system
ks-redis                     	7       	Sat Feb 15 17:33:32 2020	DEPLOYED	redis-ha-3.9.0              	5.0.5         	kubesphere-system
logging-fluentbit-operator   	1       	Mon Feb 17 20:48:53 2020	DEPLOYED	fluentbit-operator-0.1.0    	0.1.0-0217    	kubesphere-logging-system
metrics-server               	1       	Mon Feb 17 20:48:45 2020	DEPLOYED	metrics-server-2.5.0        	0.3.1-0217    	kube-system
nfs-client-provisioner       	1       	Mon Aug 12 11:31:07 2019	DEPLOYED	nfs-client-provisioner-0.1.0	v3.1.0-k8s1.11	kube-system

Cauchy · 2020年2月17日

有ks-redis 改过副本数或者删过redis吗？看下redis-ha（sts）和redis-ha-haproxy（deploy）的副本数都调到3

Lleo · 2020年2月17日

之前部署完失败后曾经删除过ks-redis，ks-redis这个chart可能是没有删除干净。kubesphere-system命名空间看不到任何redis服务。

Cauchy · 2020年2月17日

要删除的话用helm del --purge ks-redis 然后重启下ks-installer

Lleo · 2020年2月17日

我删除了helm redis chart，重新部署ks-installer。这次有redis服务了，但是redis初始化容器有报错

Could not connect to Redis at redis-ha:26379: Name does not resolve
Initializing config..
/readonly-config/init.sh: line 84: Could not resolve the announce ip for this pod: not found

看起来是没有announce服务@Cauchy

Cauchy · 2020年2月18日

看下pod状态 redis-ha起来了么有redis-ha这个service么？

Lleo · 2020年2月18日

Cauchy

NAME                                         READY   STATUS                  RESTARTS   AGE
pod/ks-account-845d86f776-7xlpd              1/1     Running                 0          11h
pod/ks-account-845d86f776-f5jpw              1/1     Running                 0          11h
pod/ks-account-845d86f776-xp8nw              1/1     Running                 0          11h
pod/ks-apigateway-d6f89b96f-np9xn            0/1     CrashLoopBackOff        139        11h
pod/ks-apigateway-d6f89b96f-svvpv            0/1     CrashLoopBackOff        139        11h
pod/ks-apigateway-d6f89b96f-zmvwp            0/1     CrashLoopBackOff        140        11h
pod/ks-apiserver-5dff5c594d-c22dv            1/1     Running                 0          11h
pod/ks-apiserver-5dff5c594d-gh5qd            1/1     Running                 0          11h
pod/ks-apiserver-5dff5c594d-qrn8m            1/1     Running                 0          11h
pod/ks-console-5c666c644-5cttb               1/1     Running                 0          11h
pod/ks-console-5c666c644-d4s97               1/1     Running                 0          11h
pod/ks-console-5c666c644-k5g7p               1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-bsfgz   1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-n2sbw   1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-ts87s   1/1     Running                 0          11h
pod/ks-installer-59fb465b7-c48fz             1/1     Running                 0          11h
pod/openldap-0                               1/1     Running                 0          11h
pod/openldap-1                               1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-5hnwf        1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-9cvxq        1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-g5vw4        1/1     Running                 0          11h
pod/redis-ha-server-0                        0/2     Init:CrashLoopBackOff   140        11h


NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/ks-account      ClusterIP   10.107.110.199   <none>        80/TCP               11h
service/ks-apigateway   ClusterIP   10.106.88.176    <none>        80/TCP               11h
service/ks-apiserver    ClusterIP   10.97.56.222     <none>        80/TCP               11h
service/ks-console      NodePort    10.105.78.128    <none>        80:30880/TCP         11h
service/openldap        ClusterIP   None             <none>        389/TCP              11h
service/redis           ClusterIP   10.103.120.41    <none>        6379/TCP             11h
service/redis-ha        ClusterIP   None             <none>        6379/TCP,26379/TCP   11h


NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-account              3/3     3            3           11h
deployment.apps/ks-apigateway           0/3     3            0           11h
deployment.apps/ks-apiserver            3/3     3            3           11h
deployment.apps/ks-console              3/3     3            3           11h
deployment.apps/ks-controller-manager   3/3     3            3           11h
deployment.apps/ks-installer            1/1     1            1           11h
deployment.apps/redis-ha-haproxy        3/3     3            3           11h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ks-account-845d86f776              3         3         3       11h
replicaset.apps/ks-apigateway-d6f89b96f            3         3         0       11h
replicaset.apps/ks-apiserver-5dff5c594d            3         3         3       11h
replicaset.apps/ks-console-5c666c644               3         3         3       11h
replicaset.apps/ks-controller-manager-779d9c6bbf   3         3         3       11h
replicaset.apps/ks-installer-59fb465b7             1         1         1       11h
replicaset.apps/redis-ha-haproxy-7d856799d5        3         3         3       11h

NAME                               READY   AGE
statefulset.apps/openldap          2/2     11h
statefulset.apps/redis-ha-server   0/3     11h

Jeff · 2020年2月18日

leo 你之前安装高可用成功过么

Lleo · 2020年2月18日

没有，我的测试集群是单master，这是第一次在多master集群部署@Jeff

Cauchy · 2020年2月18日

leo 发个远程瞧瞧

Lleo · 2020年2月18日

没法远程，这是内网集群

Cauchy · 2020年2月18日

leo kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha" 试试能不能解析到redis-ha

Lleo · 2020年2月18日

Cauchy

$ kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha"
ping: bad address 'redis-ha'
pod "test" deleted
pod kubesphere-system/test terminated (Error)

Cauchy · 2020年2月18日

解析不到的话应该是集群的dns有问题
可以先参考下这个：https://github.com/kubesphere/ks-installer/issues/33

Lleo · 2020年2月18日

Cauchy 我用那个命令测了一下我的单master集群，也是解析失败。但是那个集群成功部署了

Cauchy · 2020年2月18日

单master集群的redis跟这个不一样

Lleo · 2020年2月18日

Cauchy 我在pod内ping每个svc都通，但是ping redis-ha不通，如果dns配置有问题，那应该每个svc都Ping不通吧。请问这个有什么好的调试建议吗

NAME                                         READY   STATUS                  RESTARTS   AGE
pod/ks-account-76bff6d655-2vxcs              1/1     Running                 0          2m18s
pod/ks-account-76bff6d655-58l7h              1/1     Running                 0          115s
pod/ks-account-76bff6d655-5f8j7              1/1     Running                 0          2m24s
pod/ks-apigateway-559b894b57-c9p2v           0/1     CrashLoopBackOff        4          2m25s
pod/ks-apigateway-559b894b57-hq7bx           0/1     Error                   4          99s
pod/ks-apigateway-d6f89b96f-b5nv2            0/1     CrashLoopBackOff        5          4m32s
pod/ks-apigateway-d6f89b96f-qhdbf            0/1     CrashLoopBackOff        5          4m33s
pod/ks-apiserver-78dcbb8f69-j9smh            1/1     Running                 0          2m26s
pod/ks-apiserver-78dcbb8f69-t4sff            1/1     Running                 0          2m15s
pod/ks-apiserver-78dcbb8f69-zzx7q            1/1     Running                 0          2m24s
pod/ks-console-668696c7bb-4jn9f              1/1     Running                 0          118s
pod/ks-console-668696c7bb-l4s9c              1/1     Running                 0          2m23s
pod/ks-console-668696c7bb-ntzd4              1/1     Running                 0          2m19s
pod/ks-controller-manager-86f86bf86f-dcpxq   1/1     Running                 0          2m21s
pod/ks-controller-manager-86f86bf86f-hjwmq   1/1     Running                 0          2m5s
pod/ks-controller-manager-86f86bf86f-qttv8   1/1     Running                 0          2m7s
pod/ks-installer-59fb465b7-2sm5q             1/1     Running                 0          4m31s
pod/openldap-0                               1/1     Running                 0          4m30s
pod/openldap-1                               1/1     Running                 0          3m47s
pod/redis-ha-haproxy-7d856799d5-2rjcw        1/1     Running                 0          4m30s
pod/redis-ha-haproxy-7d856799d5-t6jpn        1/1     Running                 0          4m30s
pod/redis-ha-haproxy-7d856799d5-tkfkq        1/1     Running                 0          4m30s
pod/redis-ha-server-0                        0/2     Init:CrashLoopBackOff   5          4m30s
pod/release-name-redis-ha-server-0           2/2     Running                 0          4m30s
pod/release-name-redis-ha-server-1           2/2     Running                 0          4m16s
pod/release-name-redis-ha-server-2           2/2     Running                 0          4m12s


NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/ks-account                         ClusterIP   10.107.110.199   <none>        80/TCP               19h
service/ks-apigateway                      ClusterIP   10.106.88.176    <none>        80/TCP               19h
service/ks-apiserver                       ClusterIP   10.97.56.222     <none>        80/TCP               19h
service/ks-console                         NodePort    10.105.78.128    <none>        80:30880/TCP         19h
service/openldap                           ClusterIP   None             <none>        389/TCP              19h
service/redis                              ClusterIP   10.103.120.41    <none>        6379/TCP             19h
service/redis-ha                           ClusterIP   None             <none>        6379/TCP,26379/TCP   19h
service/release-name-redis-ha              ClusterIP   None             <none>        6379/TCP,26379/TCP   6m1s
service/release-name-redis-ha-announce-0   ClusterIP   10.106.125.136   <none>        6379/TCP,26379/TCP   6m1s
service/release-name-redis-ha-announce-1   ClusterIP   10.97.163.96     <none>        6379/TCP,26379/TCP   6m1s
service/release-name-redis-ha-announce-2   ClusterIP   10.100.108.77    <none>        6379/TCP,26379/TCP   6m1s


NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-account              3/3     3            3           19h
deployment.apps/ks-apigateway           0/3     2            0           19h
deployment.apps/ks-apiserver            3/3     3            3           19h
deployment.apps/ks-console              3/3     3            3           19h
deployment.apps/ks-controller-manager   3/3     3            3           19h
deployment.apps/ks-installer            1/1     1            1           19h
deployment.apps/redis-ha-haproxy        3/3     3            3           19h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ks-account-76bff6d655              3         3         3       2m24s
replicaset.apps/ks-account-845d86f776              0         0         0       19h
replicaset.apps/ks-apigateway-559b894b57           2         2         0       2m25s
replicaset.apps/ks-apigateway-d6f89b96f            2         2         0       19h
replicaset.apps/ks-apiserver-5dff5c594d            0         0         0       19h
replicaset.apps/ks-apiserver-78dcbb8f69            3         3         3       2m26s
replicaset.apps/ks-console-5c666c644               0         0         0       19h
replicaset.apps/ks-console-668696c7bb              3         3         3       2m23s
replicaset.apps/ks-controller-manager-779d9c6bbf   0         0         0       19h
replicaset.apps/ks-controller-manager-86f86bf86f   3         3         3       2m21s
replicaset.apps/ks-installer-59fb465b7             1         1         1       19h
replicaset.apps/redis-ha-haproxy-7d856799d5        3         3         3       19h

NAME                                            READY   AGE
statefulset.apps/openldap                       2/2     19h
statefulset.apps/redis-ha-server                0/3     19h
statefulset.apps/release-name-redis-ha-server   3/3     6m1s

Lleo · 2020年2月18日

不是dns问题，我在ks-installer的redis-ha目录执行helm template导出了一份yaml，手动apply这个yaml以后有了announce服务。然后修改了一下redis的selector。不过现在没有使用haproxy。暂时可以使用了Cauchy @Jeff

NAME                                         READY   STATUS                  RESTARTS   AGE
pod/ks-account-76595547cb-7pmxm              1/1     Running                 0          43m
pod/ks-account-76595547cb-l942l              1/1     Running                 0          43m
pod/ks-account-76595547cb-xfm6t              1/1     Running                 0          43m
pod/ks-apigateway-7b7544c7bc-276km           1/1     Running                 0          43m
pod/ks-apigateway-7b7544c7bc-mz6d5           1/1     Running                 0          43m
pod/ks-apigateway-7b7544c7bc-svzmw           1/1     Running                 0          43m
pod/ks-apiserver-b794989d7-2wbrg             1/1     Running                 0          43m
pod/ks-apiserver-b794989d7-n2nfj             1/1     Running                 0          43m
pod/ks-apiserver-b794989d7-ttwg2             1/1     Running                 0          43m
pod/ks-console-789476b665-kcdfb              1/1     Running                 0          12m
pod/ks-console-789476b665-lvfb2              1/1     Running                 0          12m
pod/ks-console-789476b665-stgq8              1/1     Running                 0          43m
pod/ks-controller-manager-697595cb8c-g6qmt   1/1     Running                 0          43m
pod/ks-controller-manager-697595cb8c-ndl9c   1/1     Running                 0          43m
pod/ks-controller-manager-697595cb8c-nkk9l   1/1     Running                 0          43m
pod/ks-installer-59fb465b7-dpgkw             1/1     Running                 0          45m
pod/openldap-0                               1/1     Running                 0          45m
pod/openldap-1                               1/1     Running                 0          44m
pod/redis-ha-configmap-test                  0/1     Completed               0          16m
pod/redis-ha-haproxy-7d856799d5-7784j        0/1     Init:CrashLoopBackOff   13         45m
pod/redis-ha-haproxy-7d856799d5-p486f        0/1     Init:CrashLoopBackOff   13         45m
pod/redis-ha-haproxy-7d856799d5-rcchr        0/1     Init:CrashLoopBackOff   13         45m
pod/redis-ha-server-0                        2/2     Running                 0          45m
pod/redis-ha-server-1                        2/2     Running                 0          45m
pod/redis-ha-server-2                        2/2     Running                 0          45m
pod/redis-ha-service-test                    0/1     Completed               0          16m


NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/ks-account            ClusterIP   10.107.110.199   <none>        80/TCP               23h
service/ks-apigateway         ClusterIP   10.106.88.176    <none>        80/TCP               23h
service/ks-apiserver          ClusterIP   10.97.56.222     <none>        80/TCP               23h
service/ks-console            NodePort    10.105.78.128    <none>        80:30880/TCP         23h
service/openldap              ClusterIP   None             <none>        389/TCP              23h
service/redis                 ClusterIP   10.103.120.41    <none>        6379/TCP             23h
service/redis-ha              ClusterIP   None             <none>        6379/TCP,26379/TCP   59m
service/redis-ha-announce-0   ClusterIP   10.100.191.108   <none>        6379/TCP,26379/TCP   59m
service/redis-ha-announce-1   ClusterIP   10.96.207.227    <none>        6379/TCP,26379/TCP   59m
service/redis-ha-announce-2   ClusterIP   10.111.13.25     <none>        6379/TCP,26379/TCP   59m


NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-account              3/3     3            3           23h
deployment.apps/ks-apigateway           3/3     3            3           23h
deployment.apps/ks-apiserver            3/3     3            3           23h
deployment.apps/ks-console              3/3     3            3           23h
deployment.apps/ks-controller-manager   3/3     3            3           23h
deployment.apps/ks-installer            1/1     1            1           23h
deployment.apps/redis-ha-haproxy        0/3     3            0           23h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ks-account-675b6db79c              0         0         0       53m
replicaset.apps/ks-account-76595547cb              3         3         3       43m
replicaset.apps/ks-account-76bff6d655              0         0         0       3h43m
replicaset.apps/ks-account-845d86f776              0         0         0       23h
replicaset.apps/ks-account-c495ff4b4               0         0         0       171m
replicaset.apps/ks-apigateway-559b894b57           0         0         0       3h43m
replicaset.apps/ks-apigateway-7b7544c7bc           3         3         3       43m
replicaset.apps/ks-apigateway-86548c4bcf           0         0         0       171m
replicaset.apps/ks-apigateway-bbd8db9bf            0         0         0       53m
replicaset.apps/ks-apigateway-d6f89b96f            0         0         0       23h
replicaset.apps/ks-apiserver-57c648fb67            0         0         0       171m
replicaset.apps/ks-apiserver-5dff5c594d            0         0         0       23h
replicaset.apps/ks-apiserver-74cb6f54cf            0         0         0       53m
replicaset.apps/ks-apiserver-78dcbb8f69            0         0         0       3h43m
replicaset.apps/ks-apiserver-b794989d7             3         3         3       43m
replicaset.apps/ks-console-5c666c644               0         0         0       23h
replicaset.apps/ks-console-644fc8d789              0         0         0       171m
replicaset.apps/ks-console-668696c7bb              0         0         0       3h43m
replicaset.apps/ks-console-789476b665              3         3         3       43m
replicaset.apps/ks-console-7bbf574fdd              0         0         0       53m
replicaset.apps/ks-controller-manager-697595cb8c   3         3         3       43m
replicaset.apps/ks-controller-manager-6dbbc8c774   0         0         0       171m
replicaset.apps/ks-controller-manager-779d9c6bbf   0         0         0       23h
replicaset.apps/ks-controller-manager-77d45b5db6   0         0         0       53m
replicaset.apps/ks-controller-manager-86f86bf86f   0         0         0       3h43m
replicaset.apps/ks-installer-59fb465b7             1         1         1       23h
replicaset.apps/redis-ha-haproxy-7d856799d5        3         3         0       23h

NAME                               READY   AGE
statefulset.apps/openldap          2/2     23h
statefulset.apps/redis-ha-server   3/3     59m

alexd · 2021年10月11日

也可以看下是不是clusterName修改了，我是修改成其他的redis起不来，改回cluster.local就可以了，时间比较紧也没时间去排查具体问题是啥

部署失败，无redis服务

LleoK零S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

JeffK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

LleoK零S

alexd