v2.1 安装失败

cugbzc

环境：kubesphere v2.1 on kubernate v1.14.8
安装kuberinstall安装完成后，发现redis server 的状态为：Init:Error
通过
1 查看POD描述信息： kubectl -n kubesphere-system describe po redis-ha-server-0
发现该pod 定义了3个containers，第一个容器config-init就报错，信息如下：
State: Waiting
Reason: CrashLoopBackOff

2 进一步查看日志：kubectl -n kubesphere-system logs redis-ha-server-0 -c config-init
输出如下：
Could not connect to Redis at redis-ha:26379: Try again
Initializing config..
/readonly-config/init.sh: line 84: Could not resolve the announce ip for this pod: not found

3 通过上一步日志问题出在redies-ha-haproxy，但是redis-ha的状态是running，貌似没什么问题：kubectl -n kubesphere-system get po -owide | grep redis-ha-haproxy
redis-ha-haproxy-5db9c79b8-7kcpp 1/1 Running 0 2d12h 10.244.2.19 k8s-m03 <none> <none>
redis-ha-haproxy-5db9c79b8-ts9kt 1/1 Running 0 2d12h 10.244.1.17 k8s-m02 <none> <none>
redis-ha-haproxy-5db9c79b8-vtjjf 1/1 Running 0 2d12h 10.244.0.17 k8s-m01 <none> <none>
4 查看redies-ha-haproxy描述新，也没有发现问题，下面我该怎么来解决这个问题呢？

Jeff

cugbzc kubectl -n kubesphere-system get svc 看下对应的 svc 是否都在

dubochao

cugbzc 重新调度redis-ha-server，拉起的pod有话变化吗，我这边重试几次发现是在某些节点拉不起来。

可以用下面的命令重试几次观察下情况：

kubectl -n kubesphere-system scale –replicas=0 sts/redis-ha-server deploy/redis-ha-haproxy deploy/ks-apiserver

kubectl -n kubesphere-system scale –replicas=3 sts/redis-ha-server deploy/redis-ha-haproxy deploy/ks-apiserver

kubectl -n kubesphere-system get po -o wide

cugbzc

感谢回复！
Jeff
SVC不在

hongming

https://github.com/helm/charts/issues/18190
应该和这个issue 相关，可以用helm 重新安装一下新的版本，如果只是尝试的话可以安装一个单副本的redis，保证redis.kubesphere-system.svc:6379 端口可达

cugbzc

hongming 感谢回复！我学习下

cugbzc

hongming 请问我是要升级helm版本?还是要重新安装redis-ha-server?

hongming

cugbzc 可以手动创建一下 service 试试

kind: Service
apiVersion: v1
metadata:
  name: redis-ha-announce-0
  namespace: kubesphere-system
  labels:
    app: redis-ha
    chart: redis-ha-3.9.0
    heritage: Tiller
    release: ks-redis
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
spec:
  ports:
    - name: server
      protocol: TCP
      port: 6379
      targetPort: redis
    - name: sentinel
      protocol: TCP
      port: 26379
      targetPort: sentinel
  selector:
    app: redis-ha
    release: ks-redis
    statefulset.kubernetes.io/pod-name: redis-ha-server-0

---

kind: Service
apiVersion: v1
metadata:
  name: redis-ha-announce-1
  namespace: kubesphere-system
  labels:
    app: redis-ha
    chart: redis-ha-3.9.0
    heritage: Tiller
    release: ks-redis
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
spec:
  ports:
    - name: server
      protocol: TCP
      port: 6379
      targetPort: redis
    - name: sentinel
      protocol: TCP
      port: 26379
      targetPort: sentinel
  selector:
    app: redis-ha
    release: ks-redis
    statefulset.kubernetes.io/pod-name: redis-ha-server-1

---

kind: Service
apiVersion: v1
metadata:
  name: redis-ha-announce-2
  namespace: kubesphere-system
  labels:
    app: redis-ha
    chart: redis-ha-3.9.0
    heritage: Tiller
    release: ks-redis
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true'
spec:
  ports:
    - name: server
      protocol: TCP
      port: 6379
      targetPort: redis
    - name: sentinel
      protocol: TCP
      port: 26379
      targetPort: sentinel
  selector:
    app: redis-ha
    release: ks-redis
    statefulset.kubernetes.io/pod-name: redis-ha-server-2

cugbzc

hongming pod都没起来。

cugbzc

hongming pod没起来
kubectl -n kubesphere-system get po NAME READY STATUS RESTARTS AGE ks-account-df78fbf7c-6ndgj 0/1 Init:0/2 0 23s ks-account-df78fbf7c-b9mm8 0/1 Init:0/2 0 23s ks-account-df78fbf7c-p2c5m 0/1 Init:0/2 0 23s ks-apigateway-6cc8d6cb4d-5fwnn 1/1 Running 2 28s ks-apigateway-6cc8d6cb4d-gzhq9 1/1 Running 2 28s ks-apigateway-6cc8d6cb4d-r46pr 1/1 Running 2 28s ks-apiserver-868f8b44fc-8xtfr 1/1 Running 0 24s ks-apiserver-868f8b44fc-dj6l7 1/1 Running 0 24s ks-apiserver-868f8b44fc-jvrp6 1/1 Running 0 24s ks-console-8579f4b8d7-6jz9m 1/1 Running 0 19s ks-console-8579f4b8d7-8gqpk 1/1 Running 0 19s ks-console-8579f4b8d7-vtv4d 1/1 Running 0 19s ks-controller-manager-7bb7c8484b-h52nq 1/1 Running 0 22s ks-controller-manager-7bb7c8484b-ns249 1/1 Running 0 22s ks-controller-manager-7bb7c8484b-wllpk 1/1 Running 0 22s ks-installer-7f487c47df-xjmc4 1/1 Running 0 2m1s openldap-0 1/1 Running 0 62s openldap-1 0/1 Running 0 17s redis-ha-haproxy-5db9c79b8-qfb6s 1/1 Running 0 67s redis-ha-haproxy-5db9c79b8-rnsj2 1/1 Running 0 67s redis-ha-haproxy-5db9c79b8-tc24j 1/1 Running 0 67s redis-ha-server-0 0/2 Init:CrashLoopBackOff 1 67s

hongming

看看 redis-ha-server-0 的错误日志，这个时候 service 应该是存在的

cugbzc

hongming server-0日志：
Could not connect to Redis at redis-ha:26379: Try again
Initializing config..
/readonly-config/init.sh: line 84: Could not resolve the announce ip for this pod: not found

hongming

是重新装了一遍吗，手动创建service 试试，上面的yaml

cugbzc

hongming
对的，我重新装了一遍还是报同样的错误！
后续我使用 helm 将ks-redis删除：helm delete –purge ks-redis。
然后再安装4.0.3版本redis-ha：helm install –name ks-redis –namespace=kubesphere-system –version=4.0.3 stable/redis-h
reids-ha 还是同样的错误，
因为 redis-server-0启动不起来,所以redis-server-1和redis-server-2根本就没有pod出现
我查看SVC，ks-redis-redis-ha-announce-1 和 ks-redis-redis-ha-announce-2 的endpoint 指向都为空

hongming

sts 是顺序启动的，符合预期。如果ks-redis-redis-ha-announce-0 这个svc 存在，但是redis-ha-server-0这个pod 依旧有Could not resolve the announce ip for this pod: not found 的错误的话，还可能是因为dns 的问题，可以看看这个issue
https://github.com/kubesphere/ks-installer/issues/33

cugbzc

hongming
好像离真相越来越近了！
因为我们的集群是在内网，我刚刚查看了主机的resolv.conf，发现是空：

然后编辑 coredns的configmap 发现没有proxy，是forward
kubectl -n kube-system edit configmap coredns
`apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure

upstream

fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: “2019-11-09T09:17:10Z”
name: coredns
namespace: kube-system
resourceVersion: “173″
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: b5bebbd8-02d1-11ea-ab1a-005056a1cd99`

hongming

可以exec 任一pod 里 nslookup 一下 ks-redis-redis-ha-announce-0.kubesphere-system.svc 看看能不能正常解析

cugbzc

hongming 不能解析。

cugbzc

hongming 我按照您给的链接，删除了upstream和 forword 行重新安装 kubesphere，还是出现相同的错误！

hongming

再不要重装了，检查一下redis-ha-announce-0这个SVC 存不存在，如果存在的话再接着检查一下 dns 是否正常解析

如果是dns 的问题，修改coredns 的配置文件之后需要把 coredns重启，再验证一下。

还可能是helm chart 的问题，现象是没有 redis-ha-announce-0、redis-ha-announce-1、redis-ha-announce-2 这三个SVC，先确认一下是哪里的问题， dns 有问题还是 SVC 创建不出来

cugbzc

hongming 修改coredns配置文件后(删除upstream和forword行)，然后对coredns进行了重启，然后再重装kubesphere。问题还有。
然后我去查看kube-system 的core dns日志，发现了日志中在报错：
2019-11-18T09:39:56.276Z [INFO] CoreDNS-1.3.1
2019-11-18T09:39:56.277Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-11-18T09:39:56.277Z [INFO] plugin/reload: Running configuration MD5 = 599b9eb76b8c147408aed6a0bbe0f669
2019-11-18T09:40:02.278Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:43233->8.8.8.8:53: i/o timeout
2019-11-18T09:40:05.278Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:47932->8.8.4.4:53: i/o timeout
2019-11-18T09:40:07.279Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:58377->8.8.4.4:53: i/o timeout
2019-11-18T09:40:08.279Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:36752->8.8.8.8:53: i/o timeout
2019-11-18T09:40:10.279Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:34035->8.8.8.8:53: i/o timeout
2019-11-18T09:40:13.280Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:60176->8.8.4.4:53: i/o timeout
2019-11-18T09:40:16.280Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:39744->8.8.8.8:53: i/o timeout
2019-11-18T09:40:19.281Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:35159->8.8.8.8:53: i/o timeout
2019-11-18T09:40:22.282Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:38069->8.8.8.8:53: i/o timeout
2019-11-18T09:40:25.282Z [ERROR] plugin/errors: 2 8591686032318411992.8857075832873219450. HINFO: read udp 10.244.3.15:36421->8.8.4.4:53: i/o timeout