etcd监控和pod监控失效

Forget-C

因为更改了master节点，所以e t c d的ip也变了。但是发现ks监控的etcd地址并没有改变。 we b
web页面上etcd监控为空，pod监控也为空。查看ks-system空间中的c m
也没有发现对应的配置项，请问这个应该改哪里呢？

shaowenchen

你是说 master 节点的 IP 变了，导致集群监控异常了吗？

Forget-C

shaowenchen 是的

shaowenchen

建议将 master 改回原来的 Ip 地址，Ip 修改之后，可能会导致各种奇怪问题，监控、证书等都会受到影响。

etcd 的 Ip 在 ks-installer cm 里面有。

Forget-C

shaowenchen 其他都没有问题，证书之类的已经确认过了，没有问题。应为变更的时候是先加到集群里，然后踢出老的master，所以这些都没影响

Forget-C

shaowenchen 你说的c m我已经改过了，还是不行。这个c m
cm里的地址应该是安装时候用的，安装完之后就固化了，改了也没效果

shaowenchen

Forget-C
那就是增加了一个 master，剔除了一个之前的 master，这个操作吧？
现在的问题是，etcd 、pod 的监控为空，对吗？
有没有检查一下各个组件的状态是否正常？

Forget-C

shaowenchen 是的。组件都是正常的。现在e t c d
etcd监控中显示的节点就是错的。

shaowenchen

Forget-C

@huanggze 帮忙看下，etcd 监控问题

huanggze

Forget-C

改了 etcd 的话，你看看这些文件是不是要对应改下？我没试过

https://github.com/kubesphere/ks-installer/tree/master/roles/ks-monitor/files/prometheus/etcd

Forget-C

shaowenchen
huanggze
现在e t c d
etcd的监控已经恢复了，但是pod的资源使用监控还是不行

Forget-C

huanggze 这个是要重新部署ks吗？重新部署肯定不现实呀。 k s监控里的etcd地址是从哪里拿到的呢，这个应该有配置文件或者数据库之类的吧？

Forget-C

执行 kubectl get Endpoints -n kube-system ，查看e t c d的endpont还是原ip地址

kubectl get Endpoints -n kube-system
NAME                               ENDPOINTS                                                                  AGE
etcd                               10.110.156.67:2379,10.110.156.68:2379,10.110.156.69:2379                   199d
kube-controller-manager            10.110.156.82:10252,10.110.156.83:10252,10.110.156.84:10252                218d
kube-controller-manager-headless   10.110.156.82:10252,10.110.156.83:10252,10.110.156.84:10252                156d
kube-dns                           10.244.24.122:53,10.244.25.182:53,10.244.27.199:53 + 9 more...             218d

编辑这个endpoint，修改其中的ip为当前的etcd ip

kubectl edit endpoints -n kube-system etcd

再次查看ks中etcd监控已经正常

huanggze

Forget-C 你说的 pod 资源使用是哪里？截图看看？

Forget-C

huanggze 这里

Forget-C

monitor下的服务都是正常运行的

huanggze

Forget-C 那你看看 kubelet ServiceMonitor 是不是改了什么？prometheus-k8s 日志报什么错。尝试定位、调试下看看

huanggze

Forget-C monitor下的服务都是正常运行的只是说明程序进程没挂，不代表不报错

Forget-C

huanggze 这块实在是不熟悉，没啥头绪。
帮忙看一下吧，日志如下：

[root@k8s-master1 ~]# kubectl  logs prometheus-k8s-0  -n kubesphere-monitoring-system prometheus
level=info ts=2020-04-17T08:02:06.721412971Z caller=main.go:244 msg="Starting Prometheus" version="(version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9b)"
level=info ts=2020-04-17T08:02:06.721556227Z caller=main.go:245 build_context="(go=go1.11.1, user=root@578ab108d0b9, date=20181106-11:40:44)"
level=info ts=2020-04-17T08:02:06.721603346Z caller=main.go:246 host_details="(Linux 5.4.15-1.el7.elrepo.x86_64 #1 SMP Sun Jan 26 09:10:24 EST 2020 x86_64 prometheus-k8s-0 (none))"
level=info ts=2020-04-17T08:02:06.721646226Z caller=main.go:247 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-04-17T08:02:06.721681945Z caller=main.go:248 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-04-17T08:02:06.732673778Z caller=main.go:562 msg="Starting TSDB ..."
level=info ts=2020-04-17T08:02:06.732728988Z caller=web.go:399 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-04-17T08:02:06.734449202Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586476800000 maxt=1586498400000 ulid=01E5HN2PWA01ST6KTWV3415EBH
level=info ts=2020-04-17T08:02:06.734983676Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586498400000 maxt=1586520000000 ulid=01E5J9NSF563VE99RYA64FXDCD
level=info ts=2020-04-17T08:02:06.736170321Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586520000000 maxt=1586541600000 ulid=01E5JY8Y7Z4KEV1VST9Z5KDD5E
level=info ts=2020-04-17T08:02:06.736852635Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586541600000 maxt=1586563200000 ulid=01E5KJW87F7XXRW4TZEFP89E3C
level=info ts=2020-04-17T08:02:06.73745925Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586563200000 maxt=1586584800000 ulid=01E5M7FDDRRWEFVEKS40RYB6SN
level=info ts=2020-04-17T08:02:06.738115358Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586584800000 maxt=1586606400000 ulid=01E5MW2FAC45ATRQ8TKHEBFYT4
level=info ts=2020-04-17T08:02:06.73937143Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586606400000 maxt=1586628000000 ulid=01E5NGNMWXHR42GTZX4MH0XNFN
level=info ts=2020-04-17T08:02:06.740035846Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586628000000 maxt=1586649600000 ulid=01E5P598QCD9BQACDSF0V440C7
level=info ts=2020-04-17T08:02:06.740568414Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586649600000 maxt=1586671200000 ulid=01E5PSW0J72M1JCJB9C3JZ4CG5
level=info ts=2020-04-17T08:02:06.754730883Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586671200000 maxt=1586692800000 ulid=01E5QEF6DY4E3CQ8SJ99NC9TM0
level=info ts=2020-04-17T08:02:06.755602384Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586692800000 maxt=1586714400000 ulid=01E5R32GSZHDRP8MJAFY5XFFGJ
level=info ts=2020-04-17T08:02:06.756207619Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586714400000 maxt=1586736000000 ulid=01E5RVTVN4N2BGRJYJ1NP0A04K
level=info ts=2020-04-17T08:02:06.757937029Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586736000000 maxt=1586757600000 ulid=01E5SC8T9SZMMN23M34P1D85RF
level=info ts=2020-04-17T08:02:06.759420863Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586757600000 maxt=1586779200000 ulid=01E5T0VW7ZPAE6V2JQXDP4QC0T
level=info ts=2020-04-17T08:02:06.759979004Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586779200000 maxt=1586800800000 ulid=01E5TNF1SM8KV41J18KR81S5GK
level=info ts=2020-04-17T08:02:06.773872177Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586800800000 maxt=1586822400000 ulid=01E5VA2BFZGZT21ZW98Y944WD5
level=info ts=2020-04-17T08:02:06.774824737Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586822400000 maxt=1586844000000 ulid=01E5VYNMNVFH7AYAK628FX58HE
level=info ts=2020-04-17T08:02:06.775399254Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586844000000 maxt=1586865600000 ulid=01E5WK8P773CQ8BZ7DCGJ7WYQM
level=info ts=2020-04-17T08:02:06.788924204Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586865600000 maxt=1586887200000 ulid=01E5X7VRW3F1VEFKGPV8ZBTNVN
level=info ts=2020-04-17T08:02:06.789602114Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586887200000 maxt=1586908800000 ulid=01E5XWF17GDD3B9FYQJEQJPB83
level=info ts=2020-04-17T08:02:06.790193338Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586908800000 maxt=1586930400000 ulid=01E5YH2A6FSM3MNTEZ97ANXBAS
level=info ts=2020-04-17T08:02:06.790808037Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586930400000 maxt=1586952000000 ulid=01E5Z5NQZKMEQ4FG21S1CVQ983
level=info ts=2020-04-17T08:02:06.791244139Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586952000000 maxt=1586973600000 ulid=01E5ZT8KV0ZA80C359K8ARQJZS
level=info ts=2020-04-17T08:02:06.805382746Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586973600000 maxt=1586995200000 ulid=01E60EVTT0600NNB6SBYNY0Q6B
level=info ts=2020-04-17T08:02:06.806205989Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1586995200000 maxt=1587016800000 ulid=01E613EWDXX630VYW4KCWXVRJC
level=info ts=2020-04-17T08:02:06.806920335Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587016800000 maxt=1587038400000 ulid=01E61R2HP3CZ64RDJ0A566PSP0
level=info ts=2020-04-17T08:02:06.808273592Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587038400000 maxt=1587060000000 ulid=01E62CN7S0T3KMBZNV417025K7
level=info ts=2020-04-17T08:02:06.809209417Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587081600000 maxt=1587088800000 ulid=01E6318ARNP4ABM4MCT98W60WF
level=info ts=2020-04-17T08:02:06.809710198Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587060000000 maxt=1587081600000 ulid=01E6318GJ37VWX5494KSZS456V
level=info ts=2020-04-17T08:02:06.810543832Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587088800000 maxt=1587096000000 ulid=01E63842005AWP0CM3FX6XB89P
level=info ts=2020-04-17T08:02:06.8122049Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587096000000 maxt=1587103200000 ulid=01E63EZS9NXWSN3R5NEZMFS5Q9
level=warn ts=2020-04-17T08:02:38.286105153Z caller=head.go:407 component=tsdb msg="unknown series references" count=193
level=info ts=2020-04-17T08:02:38.395383657Z caller=main.go:572 msg="TSDB started"
level=info ts=2020-04-17T08:02:38.395459284Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-17T08:02:38.400040967Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-17T08:02:38.401114889Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-17T08:02:38.401684391Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-17T08:02:38.420827864Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-17T08:02:38.420904905Z caller=main.go:531 msg="Server is ready to receive web requests."

Forget-C

huanggze 可以帮忙看一下吗，或者给一个思路。我们将ks用在生产环境，现在已经有影响业务了

Forget-C

huanggze 帮忙看一下呗

Forget-C

huanggze 我没有找到 collector.go 这个源代码，您如果没时间看，可以给我源代码的位置么？我自己去看一下代码