apiserver监控失效

huanggze

qwerkinglin 应该是证书、rbac 相关问题，检查一下看看

emotion

huanggze 同样的问题
k8s version:v1.17.5
docker version:19.03.8

prometheus-k8s-system-0,Pod日志

prometheus-k8s-system-0容器日志：

[root@k8s-master-01 ~]# kubectl logs prometheus-k8s-system-0 -n kubesphere-monitoring-system --all-containers
level=info ts=2020-04-24T16:46:36.983539758Z caller=main.go:244 msg="Starting Prometheus" version="(version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9b)"
level=info ts=2020-04-24T16:46:36.983629736Z caller=main.go:245 build_context="(go=go1.11.1, user=root@578ab108d0b9, date=20181106-11:40:44)"
level=info ts=2020-04-24T16:46:36.983662206Z caller=main.go:246 host_details="(Linux 5.6.6-1.el7.elrepo.x86_64 #1 SMP Mon Apr 20 19:32:35 EDT 2020 x86_64 prometheus-k8s-system-0 (none))"
level=info ts=2020-04-24T16:46:36.983694687Z caller=main.go:247 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-04-24T16:46:36.983725562Z caller=main.go:248 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-04-24T16:46:36.984620079Z caller=main.go:562 msg="Starting TSDB ..."
level=info ts=2020-04-24T16:46:36.984678373Z caller=web.go:399 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-04-24T16:46:36.987066536Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587578400000 maxt=1587600000000 ulid=01E6JFMTASJ0A6SBTQSESKXGZC
level=info ts=2020-04-24T16:46:36.988075983Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587600000000 maxt=1587621600000 ulid=01E6K480CYNA9QY27ZH385X11N
level=info ts=2020-04-24T16:46:36.989156904Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587621600000 maxt=1587643200000 ulid=01E6KRV6B8EBE0R155XHG0H9JA
level=info ts=2020-04-24T16:46:36.989848162Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587643200000 maxt=1587664800000 ulid=01E6MDEVGK50QYEMTEVFPDAWTV
level=info ts=2020-04-24T16:46:36.990588908Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587664800000 maxt=1587686400000 ulid=01E6N21DYXS015MSW8QCYHVMT5
level=info ts=2020-04-24T16:46:36.992034454Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587686400000 maxt=1587708000000 ulid=01E6NPMKY5ATK9FJ2AAR1VYB1D
level=info ts=2020-04-24T16:46:36.992820264Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587729600000 maxt=1587736800000 ulid=01E6PB7R1SKVRT5ZM9KECEMWFC
level=info ts=2020-04-24T16:46:36.993626202Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1587708000000 maxt=1587729600000 ulid=01E6PB8EH03DKGRDXFZXB6EMHA
level=warn ts=2020-04-24T16:46:59.829223467Z caller=head.go:451 component=tsdb msg="encountered WAL error, attempting repair" err="read records: corruption in segment 8 at 43051908: unexpected checksum d2dc6813, expected 67c81a41"
level=warn ts=2020-04-24T16:46:59.829356029Z caller=wal.go:284 component=tsdb msg="starting corruption repair" segment=8 offset=43051908
level=warn ts=2020-04-24T16:46:59.829475682Z caller=wal.go:292 component=tsdb msg="deleting all segments behind corruption"
level=warn ts=2020-04-24T16:46:59.829504819Z caller=wal.go:314 component=tsdb msg="rewrite corrupted segment"
level=info ts=2020-04-24T16:47:00.048202724Z caller=main.go:572 msg="TSDB started"
level=info ts=2020-04-24T16:47:00.048284916Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.051850706Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.053557216Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.055007106Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.056481566Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.073393434Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.073438206Z caller=main.go:531 msg="Server is ready to receive web requests."
level=info ts=2020-04-24T16:47:00.073492958Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.076818768Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=error ts=2020-04-24T16:47:00.077047769Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.077140303Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.077202297Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-24T16:47:00.078202633Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2020-04-24T16:47:00.078646841Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.080142696Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.081837699Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-24T16:47:00.101207498Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2020-04-24T16:46:37.832507005Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."
level=info ts=2020-04-24T16:46:37.832634141Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-24T16:47:00.101562919Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
2020/04/24 16:46:38 Watching directory: "/etc/prometheus/rules/prometheus-k8s-system-rulefiles-0"

vincentgao

这两个东西是干嘛的
kube-controller-manager-headless
kube-scheduler-headless

huanggze

vincentgao 暴露监控指标的 service

vincentgao

huanggze 那这个对二进制部署在host上的kube-scheduler服务有效么

huanggze

vincentgao

不是 kubeadm 安装的嘛？监控失效一般都是证书没配置正确，如果你的环境无法定位问题的话。可以发 tv（12版本）给我们 kubesphere@yunify.com。在邮件里留一个方便的日期和时间，我们帮你远程看看

vincentgao

我都检查了 service servicemonitor endpoint 以及prometheus的配置，感觉没问题，我选择的最小化安装，现在就是为开启的状态：

huanggze

vincentgao 发 tv（12版本）给我们 kubesphere@yunify.com。在邮件里留一个方便的日期和时间，我们帮你远程看看

vincentgao

这个是证书的错误，能具体指导下怎么做么

vincentgao

不用了，我都解决了。
apiserver是证书的问题，我重新生成证书更换了；
scheduler需要我手动创建endpoints和service以及servicemonitor关联。

huanggze

vincentgao

Nice，最好能整理个文档发到论坛，我看很多人有 apiserver 证书问题。详细说一下你的 k8s 安装方式、原来证书有什么问题、修复方式是什么

vincentgao

huanggze 已整理出来：https://kubesphere.com.cn/forum/d/1374-kube-apisrever-ks

vincentgao

好的，我整理下发出来，不过说实话，对于新手来说，尤其对于k8s可能不是很懂的人来说，你们的安装方式确实很方便，但是一旦有问题新手很难排，因为你们都是封装好了的

« 上一页