apiserver监控失效

SeesawLiu

升级2.1.1后，apiserver无法监控，请问这种情况如何排错

pog2g

SeesawLiu 看来你不是第一个，我也是一样。整个apiserver监控没有，就连坐标轴都没出来，空白一片什么都没有。github提了回复邮件叫查各种日志，但最后啥子都没查出来(我是单节点)

runzexia

SeesawLiu
pog2g
麻烦贴一下 k8s 的版本

SeesawLiu

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

prometheus targets 正常吗

pog2g

runzexia 我的也是1.17.3，单节点
docker:19.3.7
k8s:v1.17.3

runzexia

CC @huanggze

huanggze

pog2g

看看 prometheus-k8s-system-0 有没有错误日志？

SeesawLiu

huanggze 有这些错

huanggze

SeesawLiu 这不是prometheus-k8s-system-0的日志

qwerkinglin

遇到同样的问题监控 kube-apiserver kube-scheduler kube-controller-manager 都没有数据空白一片
Service Components中kube-scheduler-headless kube-controller-manager-headless 是stoppped状态

huanggze

qwerkinglin
看看 prometheus-k8s-system-0 有没有错误日志？

qwerkinglin

huanggze
prometheus-k8s-system-0的日志都是正常的没有报错下面的报错是在kube-state-metrics日志里发现的
E0327 06:56:53.108161 1 reflector.go:125] k8s.io/kube-state-metrics/internal/store/builder.go:295: Failed to list *v1beta1.CertificateSigningRequest: certificatesigningrequests.certificates.k8s.io is forbidden: User “system:serviceaccount:kubesphere-monitoring-system:kube-state-metrics” cannot list resource “certificatesigningrequests” in API group “certificates.k8s.io” at the cluster scope

emotion

huanggze 同样的问题
k8s version:v1.17.5
docker version:19.03.8

prometheus-k8s-system-0,Pod日志

prometheus-k8s-system-0容器日志：

[root@k8s-master-01 ~]# kubectl logs prometheus-k8s-system-0 -n kubesphere-monitoring-system --all-containers
level=info ts=2020-04-24T16:46:36.983539758Z caller=main.go:244 msg="Starting Prometheus" version="(version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9b)"
level=info ts=2020-04-24T16:46:36.983629736Z caller=main.go:245 build_context="(go=go1.11.1, user=root@578ab108d0b9, date=20181106-11:40:44)"
level=info ts=2020-04-24T16:46:36.983662206Z caller=main.go:246 host_details="(Linux 5.6.6-1.el7.elrepo.x86_64 #1 SMP Mon Apr 20 19:32:35 EDT 2020 x86_64 prometheus-k8s-system-0 (none))"
level=info ts=2020-04-24T16:46:36.983694687Z caller=main.go:247 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-04-24T16:46:36.983725562Z caller=main.go:248 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-04-24T16:46:36.984620079Z caller=main.go:562 msg="Starting TSDB ..."
level=info ts=2020-04-24T16:46:36.984678373Z caller=web.go:399 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-04-24T16:46:36.987066536Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587578400000 maxt=1587600000000 ulid=01E6JFMTASJ0A6SBTQSESKXGZC
level=info ts=2020-04-24T16:46:36.988075983Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587600000000 maxt=1587621600000 ulid=01E6K480CYNA9QY27ZH385X11N
level=info ts=2020-04-24T16:46:36.989156904Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587621600000 maxt=1587643200000 ulid=01E6KRV6B8EBE0R155XHG0H9JA
level=info ts=2020-04-24T16:46:36.989848162Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587643200000 maxt=1587664800000 ulid=01E6MDEVGK50QYEMTEVFPDAWTV
level=info ts=2020-04-24T16:46:36.990588908Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587664800000 maxt=1587686400000 ulid=01E6N21DYXS015MSW8QCYHVMT5
level=info ts=2020-04-24T16:46:36.992034454Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587686400000 maxt=1587708000000 ulid=01E6NPMKY5ATK9FJ2AAR1VYB1D
level=info ts=2020-04-24T16:46:36.992820264Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587729600000 maxt=1587736800000 ulid=01E6PB7R1SKVRT5ZM9KECEMWFC
level=info ts=2020-04-24T16:46:36.993626202Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587708000000 maxt=1587729600000 ulid=01E6PB8EH03DKGRDXFZXB6EMHA
level=warn ts=2020-04-24T16:46:59.829223467Z caller=head.go:451 component=tsdb msg="encountered WAL error, attempting repair" err="read records: corruption in segment 8 at 43051908: unexpected checksum d2dc6813, expected 67c81a41"
level=warn ts=2020-04-24T16:46:59.829356029Z caller=wal.go:284 component=tsdb msg="starting corruption repair" segment=8 offset=43051908
level=warn ts=2020-04-24T16:46:59.829475682Z caller=wal.go:292 component=tsdb msg="deleting all segments behind corruption"
level=warn ts=2020-04-24T16:46:59.829504819Z caller=wal.go:314 component=tsdb msg="rewrite corrupted segment"
level=info ts=2020-04-24T16:47:00.048202724Z caller=main.go:572 msg="TSDB started"
level=info ts=2020-04-24T16:47:00.048284916Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.051850706Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.053557216Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.055007106Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.056481566Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.073393434Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.073438206Z caller=main.go:531 msg="Server is ready to receive web requests."
level=info ts=2020-04-24T16:47:00.073492958Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.076818768Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=error ts=2020-04-24T16:47:00.077047769Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.077140303Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.077202297Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.078202633Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2020-04-24T16:47:00.078646841Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.080142696Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.081837699Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.101207498Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2020-04-24T16:46:37.832507005Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."
level=info ts=2020-04-24T16:46:37.832634141Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.101562919Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
2020/04/24 16:46:38 Watching directory: "/etc/prometheus/rules/prometheus-k8s-system-rulefiles-0"

huanggze

qwerkinglin 应该是证书、rbac 相关问题，检查一下看看

vincentgao

这两个东西是干嘛的
kube-controller-manager-headless
kube-scheduler-headless

huanggze

vincentgao 暴露监控指标的 service

vincentgao

huanggze 那这个对二进制部署在host上的kube-scheduler服务有效么

huanggze

vincentgao

不是 kubeadm 安装的嘛？监控失效一般都是证书没配置正确，如果你的环境无法定位问题的话。可以发 tv（12版本）给我们 kubesphere@yunify.com。在邮件里留一个方便的日期和时间，我们帮你远程看看

vincentgao

我都检查了 service servicemonitor endpoint 以及prometheus的配置，感觉没问题，我选择的最小化安装，现在就是为开启的状态：

huanggze

vincentgao 发 tv（12版本）给我们 kubesphere@yunify.com。在邮件里留一个方便的日期和时间，我们帮你远程看看