查看容器监控，数据为空，该如何排查问题

brosoul

部署kubesphere 2.1后，集群状态监控数据不为空，但是容器查看监控数据是空的，这个如何排查问题。

huanggze

brosoul 看一下prometheus-k8s 的log

Forest-L

brosoul

2020-01-02 09:30:09.052: level=info ts=2020-01-02T01:30:09.025746865Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577880000000 maxt=1577901600000 ulid=01DXHXZK6JCDGFVVPZ7FJPWHJA sources="[01DXHXZJK1BN969JDDFXKTJ9YN 01DXHXZJKTVGHEQKBTFK3S7QDV 01DXHXZJMRS8R9FKZ6MVGRVKXN]"

2020-01-02 09:30:08.971: level=info ts=2020-01-02T01:30:08.963897216Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577858400000 maxt=1577880000000 ulid=01DXHXZK55M54JE1C029ECNVM2 sources="[01DXHXZJDC7NTZH0S1DPT2X1FA 01DXHXZJF8EGQ5SKD552EZZH35 01DXHXZJH3YB1DER7VHJBJ0RWH]"

2020-01-02 09:30:08.921: level=info ts=2020-01-02T01:30:08.918633648Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577836800000 maxt=1577858400000 ulid=01DXHXZK406HYTPHNEAE4HVM4E sources="[01DXHXZJA7DK4AJDVHSWC7CJ6V 01DXHXZJBA4WCZ8RQH5GMXCFVC 01DXHXZJCB44GJDBG2VP5FBJWB]"

2020-01-02 09:30:08.862: level=info ts=2020-01-02T01:30:08.861444647Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577815200000 maxt=1577836800000 ulid=01DXHXZK23MARNRY3Q8C16355Q sources="[01DXHXZJ3GGQFKBZN6RYYTKHNG 01DXHXZJ7NEARFVYMYW28HK0R8 01DXHXZJ8RE0Z0VTTKVS1JR55B]"

2020-01-02 09:30:08.803: level=info ts=2020-01-02T01:30:08.79238308Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577793600000 maxt=1577815200000 ulid=01DXHXZK070Q9WDRAQ56Z36QD9 sources="[01DXHXZH7EQ750J1WE0J3ZFKT0 01DXHXZHPHMYDCYARVT116GX2R 01DXHXZHXHHJX5DPDPC0FEN5Q1]"

2020-01-02 09:30:08.760: level=info ts=2020-01-02T01:30:08.755648754Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1577772000000 maxt=1577793600000 ulid=01DXHXZJS49N2SXS4MM9GN19CV sources="[01DXDNBGQ4YQKBZM8KW1HWDQF6 01DXHXZGWYGPFEJRVNDZNQM81R 01DXHXZH64MHJE7KZJMXQPRJH5]"

2020-01-02 09:30:08.539: level=info ts=2020-01-02T01:30:08.539413812Z caller=head.go:488 component=tsdb msg="head GC completed" duration=596.188µs

2020-01-02 09:30:08.531: level=info ts=2020-01-02T01:30:08.531347701Z caller=compact.go:398 component=tsdb msg="write block" mint=1577916000000 maxt=1577923200000 ulid=01DXHXZJR4YWMN8TA5C5BFVHAK

2020-01-02 09:30:08.510: level=info ts=2020-01-02T01:30:08.510191577Z caller=head.go:488 component=tsdb msg="head GC completed" duration=778.966µs

2020-01-02 09:30:08.504: level=info ts=2020-01-02T01:30:08.499928512Z caller=compact.go:398 component=tsdb msg="write block" mint=1577908800000 maxt=1577916000000 ulid=01DXHXZJQ30HEYNC4KFAVJ3H4W

2020-01-02 09:30:08.478: level=info ts=2020-01-02T01:30:08.477971657Z caller=head.go:488 component=tsdb msg="head GC completed" duration=647.577µs

2020-01-02 09:30:08.470: level=info ts=2020-01-02T01:30:08.470419554Z caller=compact.go:398 component=tsdb msg="write block" mint=1577901600000 maxt=1577908800000 ulid=01DXHXZJP715TRPCHG3G33AQDX

2020-01-02 09:30:08.449: level=info ts=2020-01-02T01:30:08.44823975Z caller=head.go:488 component=tsdb msg="head GC completed" duration=686.476µs

2020-01-02 09:30:08.437: level=info ts=2020-01-02T01:30:08.437176074Z caller=compact.go:398 component=tsdb msg="write block" mint=1577894400000 maxt=1577901600000 ulid=01DXHXZJMRS8R9FKZ6MVGRVKXN

2020-01-02 09:30:08.403: level=info ts=2020-01-02T01:30:08.403405971Z caller=head.go:488 component=tsdb msg="head GC completed" duration=804.945µs

2020-01-02 09:30:08.395: level=info ts=2020-01-02T01:30:08.394940041Z caller=compact.go:398 component=tsdb msg="write block" mint=1577887200000 maxt=1577894400000 ulid=01DXHXZJKTVGHEQKBTFK3S7QDV

2020-01-02 09:30:08.374: level=info ts=2020-01-02T01:30:08.374317374Z caller=head.go:488 component=tsdb msg="head GC completed" duration=674.185µs

2020-01-02 09:30:08.367: level=info ts=2020-01-02T01:30:08.367069611Z caller=compact.go:398 component=tsdb msg="write block" mint=1577880000000 maxt=1577887200000 ulid=01DXHXZJK1BN969JDDFXKTJ9YN

2020-01-02 09:30:08.346: level=info ts=2020-01-02T01:30:08.346215357Z caller=head.go:488 component=tsdb msg="head GC completed" duration=750.564µs

2020-01-02 09:30:08.334: level=info ts=2020-01-02T01:30:08.327719177Z caller=compact.go:398 component=tsdb msg="write block" mint=1577872800000 maxt=1577880000000 ulid=01DXHXZJH3YB1DER7VHJBJ0RWH

2020-01-02 09:30:08.282: level=info ts=2020-01-02T01:30:08.282733707Z caller=head.go:488 component=tsdb msg="head GC completed" duration=1.031982ms

2020-01-02 09:30:08.269: level=info ts=2020-01-02T01:30:08.269085206Z caller=compact.go:398 component=tsdb msg="write block" mint=1577865600000 maxt=1577872800000 ulid=01DXHXZJF8EGQ5SKD552EZZH35

2020-01-02 09:30:08.224: level=info ts=2020-01-02T01:30:08.224744581Z caller=head.go:488 component=tsdb msg="head GC completed" duration=689.584µs

2020-01-02 09:30:08.213: level=info ts=2020-01-02T01:30:08.213656718Z caller=compact.go:398 component=tsdb msg="write block" mint=1577858400000 maxt=1577865600000 ulid=01DXHXZJDC7NTZH0S1DPT2X1FA

2020-01-02 09:30:08.166: level=info ts=2020-01-02T01:30:08.166074599Z caller=head.go:488 component=tsdb msg="head GC completed" duration=614.049µs

2020-01-02 09:30:08.155: level=info ts=2020-01-02T01:30:08.155315365Z caller=compact.go:398 component=tsdb msg="write block" mint=1577851200000 maxt=1577858400000 ulid=01DXHXZJCB44GJDBG2VP5FBJWB

2020-01-02 09:30:08.134: level=info ts=2020-01-02T01:30:08.134230984Z caller=head.go:488 component=tsdb msg="head GC completed" duration=625.075µs

2020-01-02 09:30:08.120: level=info ts=2020-01-02T01:30:08.119877062Z caller=compact.go:398 component=tsdb msg="write block" mint=1577844000000 maxt=1577851200000 ulid=01DXHXZJBA4WCZ8RQH5GMXCFVC

2020-01-02 09:30:08.097: level=info ts=2020-01-02T01:30:08.097530259Z caller=head.go:488 component=tsdb msg="head GC completed" duration=691.585µs

2020-01-02 09:30:08.086: level=info ts=2020-01-02T01:30:08.086232211Z caller=compact.go:398 component=tsdb msg="write block" mint=1577836800000 maxt=1577844000000 ulid=01DXHXZJA7DK4AJDVHSWC7CJ6V

2020-01-02 09:30:08.064: level=info ts=2020-01-02T01:30:08.064025969Z caller=head.go:488 component=tsdb msg="head GC completed" duration=621.772µs

2020-01-02 09:30:08.056: level=info ts=2020-01-02T01:30:08.056361229Z caller=compact.go:398 component=tsdb msg="write block" mint=1577829600000 maxt=1577836800000 ulid=01DXHXZJ8RE0Z0VTTKVS1JR55B

2020-01-02 09:30:08.018: level=info ts=2020-01-02T01:30:08.017978363Z caller=head.go:488 component=tsdb msg="head GC completed" duration=596.05µs

2020-01-02 09:30:08.009: level=info ts=2020-01-02T01:30:08.006259072Z caller=compact.go:398 component=tsdb msg="write block" mint=1577822400000 maxt=1577829600000 ulid=01DXHXZJ7NEARFVYMYW28HK0R8

2020-01-02 09:30:07.983: level=info ts=2020-01-02T01:30:07.9827726Z caller=head.go:488 component=tsdb msg="head GC completed" duration=1.137246ms

2020-01-02 09:30:07.962: level=info ts=2020-01-02T01:30:07.96195351Z caller=compact.go:398 component=tsdb msg="write block" mint=1577815200000 maxt=1577822400000 ulid=01DXHXZJ3GGQFKBZN6RYYTKHNG

2020-01-02 09:30:07.846: level=info ts=2020-01-02T01:30:07.846559015Z caller=head.go:488 component=tsdb msg="head GC completed" duration=1.24488ms

2020-01-02 09:30:07.837: level=info ts=2020-01-02T01:30:07.826391836Z caller=compact.go:398 component=tsdb msg="write block" mint=1577808000000 maxt=1577815200000 ulid=01DXHXZHXHHJX5DPDPC0FEN5Q1

2020-01-02 09:30:07.620: level=info ts=2020-01-02T01:30:07.619060586Z caller=head.go:488 component=tsdb msg="head GC completed" duration=4.042206ms

2020-01-02 09:30:07.584: level=info ts=2020-01-02T01:30:07.583977764Z caller=compact.go:398 component=tsdb msg="write block" mint=1577800800000 maxt=1577808000000 ulid=01DXHXZHPHMYDCYARVT116GX2R

2020-01-02 09:30:07.433: level=info ts=2020-01-02T01:30:07.432406866Z caller=head.go:488 component=tsdb msg="head GC completed" duration=2.261474ms

2020-01-02 09:30:07.418: level=info ts=2020-01-02T01:30:07.416259036Z caller=compact.go:398 component=tsdb msg="write block" mint=1577793600000 maxt=1577800800000 ulid=01DXHXZH7EQ750J1WE0J3ZFKT0

2020-01-02 09:30:07.309: level=info ts=2020-01-02T01:30:06.955133619Z caller=head.go:488 component=tsdb msg="head GC completed" duration=655.319µs

2020-01-02 09:30:06.946: level=info ts=2020-01-02T01:30:06.939127888Z caller=compact.go:398 component=tsdb msg="write block" mint=1577786400000 maxt=1577793600000 ulid=01DXHXZH64MHJE7KZJMXQPRJH5

2020-01-02 09:30:06.911: level=info ts=2020-01-02T01:30:06.907433237Z caller=head.go:488 component=tsdb msg="head GC completed" duration=39.854722ms

2020-01-02 09:30:06.861: level=info ts=2020-01-02T01:30:06.843852412Z caller=compact.go:398 component=tsdb msg="write block" mint=1577779200000 maxt=1577786400000 ulid=01DXHXZGWYGPFEJRVNDZNQM81R

2020-01-02 09:29:58.129: level=error ts=2020-01-02T01:29:58.127769896Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:58.129: level=error ts=2020-01-02T01:29:58.127614429Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:58.129: level=error ts=2020-01-02T01:29:58.127710155Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:58.127: level=error ts=2020-01-02T01:29:58.123381153Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:58.127: level=error ts=2020-01-02T01:29:58.124488279Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:58.127: level=error ts=2020-01-02T01:29:58.12489887Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.206: level=error ts=2020-01-02T01:29:57.204604611Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.206: level=error ts=2020-01-02T01:29:57.204554243Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.206: level=error ts=2020-01-02T01:29:57.204815095Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.206: level=error ts=2020-01-02T01:29:57.204973438Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.205: level=error ts=2020-01-02T01:29:57.203463191Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/namespaces/kubesphere-monitoring-system/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:57.204: level=error ts=2020-01-02T01:29:57.203368768Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout"

2020-01-02 09:29:28.147: level=info ts=2020-01-02T01:29:28.141407281Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

2020-01-02 09:29:28.121: level=error ts=2020-01-02T01:29:28.108187858Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"

2020-01-02 09:29:28.121: level=error ts=2020-01-02T01:29:28.108124028Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"

2020-01-02 09:29:28.121: level=info ts=2020-01-02T01:29:28.107721586Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

2020-01-02 09:29:28.121: level=info ts=2020-01-02T01:29:28.109387226Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

2020-01-02 09:29:28.107: level=info ts=2020-01-02T01:29:28.103884429Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

2020-01-02 09:29:27.223: level=info ts=2020-01-02T01:29:27.210751658Z caller=main.go:531 msg="Server is ready to receive web requests."

2020-01-02 09:29:27.217: level=info ts=2020-01-02T01:29:27.210705527Z caller=main.go:658 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

2020-01-02 09:29:27.175: level=info ts=2020-01-02T01:29:27.175015181Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

2020-01-02 09:29:27.173: level=info ts=2020-01-02T01:29:27.173446519Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

2020-01-02 09:29:27.162: level=info ts=2020-01-02T01:29:27.150660895Z caller=main.go:632 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

2020-01-02 09:29:27.151: level=info ts=2020-01-02T01:29:27.150582521Z caller=main.go:572 msg="TSDB started"

2020-01-02 09:29:27.140: level=warn ts=2020-01-02T01:29:27.140798412Z caller=head.go:407 component=tsdb msg="unknown series references" count=2974

2020-01-02 09:29:26.537: level=info ts=2020-01-02T01:29:26.53689177Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577750400000 maxt=1577772000000 ulid=01DXDNBHYDVGCEQM1WYMJREGKW

2020-01-02 09:29:26.536: level=info ts=2020-01-02T01:29:26.536358127Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577728800000 maxt=1577750400000 ulid=01DXDNBHVJ66AH9KHJBD19MC60

2020-01-02 09:29:26.535: level=info ts=2020-01-02T01:29:26.535164668Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577685600000 maxt=1577707200000 ulid=01DXDNBHQRD8233CG89D1Q9233

2020-01-02 09:29:26.535: level=info ts=2020-01-02T01:29:26.535756382Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577707200000 maxt=1577728800000 ulid=01DXDNBHRQ6H9VMSVBAG0VS18Z

2020-01-02 09:29:26.534: level=info ts=2020-01-02T01:29:26.533996524Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577642400000 maxt=1577664000000 ulid=01DXDNBHNXS9YWJ61NECNVCDEH

2020-01-02 09:29:26.534: level=info ts=2020-01-02T01:29:26.534587751Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577664000000 maxt=1577685600000 ulid=01DXDNBHPVNHZX1E602EC4P3ZK

2020-01-02 09:29:26.533: level=info ts=2020-01-02T01:29:26.532847332Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577599200000 maxt=1577620800000 ulid=01DXDNBHKQ8D46K2FNJ0JHJK86

2020-01-02 09:29:26.533: level=info ts=2020-01-02T01:29:26.533440153Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577620800000 maxt=1577642400000 ulid=01DXDNBHMSFY08CJ7Q1YD14JS1

2020-01-02 09:29:26.532: level=info ts=2020-01-02T01:29:26.532363672Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577577600000 maxt=1577599200000 ulid=01DXDNBHJM1JE55JP6F1KXSJT2

2020-01-02 09:29:26.532: level=info ts=2020-01-02T01:29:26.531709525Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577556000000 maxt=1577577600000 ulid=01DXDNBHHFDWKYX30F096HASY4

2020-01-02 09:29:26.531: level=info ts=2020-01-02T01:29:26.531027653Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577534400000 maxt=1577556000000 ulid=01DXDNBHGBHZTXQ6KEG88C2XAC

2020-01-02 09:29:26.530: level=info ts=2020-01-02T01:29:26.530146576Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577512800000 maxt=1577534400000 ulid=01DXDNBHF6APMVHWEM0266XHF0

2020-01-02 09:29:26.529: level=info ts=2020-01-02T01:29:26.529229491Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577491200000 maxt=1577512800000 ulid=01DXDNBHDZDZ3WQ4EHP7QSAY9V

2020-01-02 09:29:26.527: level=info ts=2020-01-02T01:29:26.526479213Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577448000000 maxt=1577469600000 ulid=01DXDNBHBFQBDDVNBPWCKMR0GR

2020-01-02 09:29:26.527: level=info ts=2020-01-02T01:29:26.527387734Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577469600000 maxt=1577491200000 ulid=01DXDNBHCSRCGRVCV56M07BF91

2020-01-02 09:29:26.526: level=info ts=2020-01-02T01:29:26.525628454Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577426400000 maxt=1577448000000 ulid=01DXDNBHA2486S5HC1JZCRJGX2

2020-01-02 09:29:26.525: level=info ts=2020-01-02T01:29:26.524231695Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577404800000 maxt=1577426400000 ulid=01DXDNBH8ER8XQF0Q71Z2SJ3JM

2020-01-02 09:29:26.523: level=info ts=2020-01-02T01:29:26.521322377Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577383200000 maxt=1577404800000 ulid=01DXDNBH72APHRR23G5PB4NBYG

2020-01-02 09:29:26.519: level=info ts=2020-01-02T01:29:26.518815504Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577361600000 maxt=1577383200000 ulid=01DXDNBH5PZXWYX6WD7AC1SK51

2020-01-02 09:29:26.518: level=info ts=2020-01-02T01:29:26.514042642Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577340000000 maxt=1577361600000 ulid=01DXDNBH4BN2MRM5MT4NX8XAC1

2020-01-02 09:29:26.489: level=info ts=2020-01-02T01:29:26.488965784Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577318400000 maxt=1577340000000 ulid=01DXDNBH30J9970VM1SA5EQX4Y

2020-01-02 09:29:26.464: level=info ts=2020-01-02T01:29:26.463837066Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577296800000 maxt=1577318400000 ulid=01DXDNBH1MSRDN8S5NBQF6CYZG

2020-01-02 09:29:26.428: level=info ts=2020-01-02T01:29:26.427732781Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577275200000 maxt=1577296800000 ulid=01DXDNBH09NRYH9SY703HVHJ9J

2020-01-02 09:29:26.423: level=info ts=2020-01-02T01:29:26.423038554Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577253600000 maxt=1577275200000 ulid=01DXDNBGYX0MA2W5QWNCTPZFR6

2020-01-02 09:29:26.406: level=info ts=2020-01-02T01:29:26.405166523Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577232000000 maxt=1577253600000 ulid=01DXDNBGXJ0Y0T9J7X3ZBWZ1F6

2020-01-02 09:29:26.403: level=info ts=2020-01-02T01:29:26.402497102Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577210400000 maxt=1577232000000 ulid=01DXDNBGW4E998R6N2ZPDM2C8G

2020-01-02 09:29:26.403: level=info ts=2020-01-02T01:29:26.399533073Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577167200000 maxt=1577188800000 ulid=01DXDNBGRJD5SZDT0DEK3K80WK

2020-01-02 09:29:26.403: level=info ts=2020-01-02T01:29:26.401094459Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577188800000 maxt=1577210400000 ulid=01DXDNBGTS4W4MZBZDA8TAYZPY

2020-01-02 09:29:26.396: level=info ts=2020-01-02T01:29:26.395592579Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1577772000000 maxt=1577779200000 ulid=01DXDNBGQ4YQKBZM8KW1HWDQF6

2020-01-02 09:29:26.360: level=info ts=2020-01-02T01:29:26.360123611Z caller=web.go:399 component=web msg="Start listening for connections" address=0.0.0.0:9090

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.341223377Z caller=main.go:246 host_details="(Linux 4.15.0-72-generic #81~16.04.1-Ubuntu SMP Tue Nov 26 16:34:21 UTC 2019 x86_64 prometheus-k8s-0 (none))"

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.341330032Z caller=main.go:248 vm_limits="(soft=unlimited, hard=unlimited)"

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.341204272Z caller=main.go:245 build_context="(go=go1.11.1, user=root@578ab108d0b9, date=20181106-11:40:44)"

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.340819601Z caller=main.go:244 msg="Starting Prometheus" version="(version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9b)"

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.341797823Z caller=main.go:562 msg="Starting TSDB ..."

2020-01-02 09:29:26.341: level=info ts=2020-01-02T01:29:26.341237091Z caller=main.go:247 fd_limits="(soft=1048576, hard=1048576)"

huanggze

brosoul

没看到什么有用的信息，继续debug，把 Prometheus 的console 打开看看

删除原来healess prometheus serive
kubectl delete svc -n kubesphere-monitoring-system prometheus-k8s

创建 NodePort 类型的 service，并重新apply
vi prometheus-k8s-svc.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: kubesphere-monitoring-system
spec:
  ports:
  - name: web
    nodePort: 32655
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    app: prometheus
    prometheus: k8s
  type: NodePort

kubectl apply -f prometheus-k8s-svc.yml

打开Promethus Console，检查 kubelet Error 的原因

brosoul

huanggze

server returned HTTP status 403 Forbidden
Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=metrics)

参考以下的链接，可以使 ‘State’ 变为 ‘UP’ ，但是为什么把IP 从 ‘127.0.0.1’ 替换成 ‘0.0.0.0′ 这可以了？我之前在其他集群中部署过 kubesphere 并没有遇到这个问题。难道是和这次的 minikube的单节点集群相关？
https://github.com/coreos/prometheus-operator/blob/master/Documentation/troubleshooting.md

brosoul

huanggze 你好，我现在是外接原有的监控 prometheus，然后发生了之前一样的问题（容器查看监控数据是空），查看 Promethus Console 中， kubelet 正常。但 kube-proxy 和 kube-etcd 状态为 down，是否是这两个状态的原因导致

huanggze

brosoul 这样啊。请问你是怎么改的呢？有什么步骤，可以记录在这里吗？

brosoul

huanggze
总结一下：容器监控数据返回为空时，
首先检查 prometheus-k8s 的log信息，如未发现有用的信息，进行下一步
使 prometheus-k8s 可以外部访问，发现 /targets 下的 kubesphere-monitoring-system/kubelet 状态不为 UP。
详细参考该文档
执行如下指令问题得到解决：

KUBEADM_SYSTEMD_CONF=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -e "/cadvisor-port=0/d" -i "$KUBEADM_SYSTEMD_CONF"
if ! grep -q "authentication-token-webhook=true" "$KUBEADM_SYSTEMD_CONF"; then
  sed -e "s/--authorization-mode=Webhook/--authentication-token-webhook=true --authorization-mode=Webhook/" -i "$KUBEADM_SYSTEMD_CONF"
fi
systemctl daemon-reload
systemctl restart kubelet

sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml

huanggze

brosoul 应该和 kube-proxy 和 kube-etcd 没关系，kubelet 正常就应该有容器监控数据。kubesphere 原来的 recording rules 都复制到你的 PrometheusRule CRD 里面来没？我们监控需要一些预先内置的 recording rules 指标。

brosoul

huanggze 已经复制了，找到问题了，之前 kubesphere 2.1.0 的rules 不适合 k8s 1.16 的版本，我使用 2.1.1 中的 rules 指标，解决了这个问题

syfad

huanggze 你好，我也是用外部监控，然后kubespher相关监控数据出不来，现在不确定问题，想问下kubesphere 原来的 recording rules怎么复制到我现在的 PrometheusRule CRD里面来？