KS 系统资源面板数据采集异常，后台监控pod状态正常 ,想咨询下该部分属于哪个组件负责

freemankevin

zackzhang

kubectl get servicemonitor -n kubesphere-monitoring-system kube-scheduler -oyaml # 注意检查下里面的label是否匹配
kubectl -n kube-system get svc --show-labels|grep kube-scheduler

freemankevin

zackzhang

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"k8s-app":"kube-scheduler"},"name":"kube-scheduler","namespace":"kubesphere-monitoring-system"},"spec":{"endpoints":[{"interval":"1m","metricRelabelings":[{"action":"drop","regex":"scheduler_(e2e_scheduling_latency_microseconds|scheduling_algorithm_predicate_evaluation|scheduling_algorithm_priority_evaluation|scheduling_algorithm_preemption_evaluation|scheduling_algorithm_latency_microseconds|binding_latency_microseconds|scheduling_latency_seconds)","sourceLabels":["__name__"]}],"port":"http-metrics"}],"jobLabel":"k8s-app","namespaceSelector":{"matchNames":["kube-system"]},"selector":{"matchLabels":{"k8s-app":"kube-scheduler"}}}}
  creationTimestamp: "2020-11-19T05:20:58Z"
  generation: 1
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: kubesphere-monitoring-system
  resourceVersion: "146473"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/kubesphere-monitoring-system/servicemonitors/kube-scheduler
  uid: 4262d0e6-ed36-4cfb-99bb-da6986455d23
spec:
  endpoints:
  - interval: 1m
    metricRelabelings:
    - action: drop
      regex: scheduler_(e2e_scheduling_latency_microseconds|scheduling_algorithm_predicate_evaluation|scheduling_algorithm_priority_evaluation|scheduling_algorithm_preemption_evaluation|scheduling_algorithm_latency_microseconds|binding_latency_microseconds|scheduling_latency_seconds)
      sourceLabels:
      - __name__
    port: http-metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kube-scheduler

freemankevin

zackzhang

[root@master1 ~]# kubectl -n kube-system get svc --show-labels|grep kube-scheduler
kube-scheduler-svc             ClusterIP   None             <none>        10251/TCP                      20d   k8s-app=kube-scheduler

zackzhang

看下日志

kubectl logs -f prometheus-operator-78c5cdbc8f-zwgg9 -c prometheus-operator -n kubesphere-monitoring-system

如果没有看到相关日志，把上面的这个servicemonitor的yaml备份下后，删掉，然后重新apply下

kubectl get servicemonitor kube-scheduler -n kubesphere-monitoring-system  -oyaml > 1.yaml
kubectl delete servicemonitor kube-scheduler -n kubesphere-monitoring-system
kubectl apply -f 1.yaml

然后关注下日志

freemankevin

zackzhang 日志只能等是吗,跑的定时?

level=info ts=2020-12-09T21:15:36.311511737Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:20:36.311710142Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:25:36.312002602Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:30:36.312171761Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:35:36.312330209Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:40:36.312613426Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:45:36.312762305Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
E1209 21:45:47.196355       1 operator.go:310] Sync "kubesphere-monitoring-system/main" failed: synchronizing governing service failed: retrieving service object failed: etcdserver: request timed out
level=info ts=2020-12-09T21:45:47.201598621Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:50:36.313012559Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T21:55:36.313256129Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:00:36.313435611Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:05:36.313706796Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:10:36.313968602Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:15:36.314250995Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:20:36.314488822Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:25:36.314600589Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:30:36.314853643Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:35:36.314912018Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:40:36.315197445Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:45:36.315192738Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:50:36.31559546Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T22:55:36.315778958Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:00:36.315910596Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:05:36.315961843Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:10:36.316143921Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:15:36.316313386Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:20:36.316529755Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:25:36.31679656Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:30:36.31692314Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:35:36.317043142Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:40:36.317310466Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:45:36.317535147Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:50:36.31775562Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-09T23:55:36.317929636Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:00:36.318090252Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=error ts=2020-12-10T00:04:52.807152069Z caller=operator.go:622 component=prometheusoperator msg="syncing nodes into Endpoints object failed" err="listing nodes failed: etcdserver: request timed out"
level=info ts=2020-12-10T00:05:36.318237526Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:10:36.319023517Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:15:36.319289719Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:20:36.319465643Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:25:36.319705184Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:30:36.320022574Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:35:36.320207833Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:40:36.320556992Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:45:36.320700236Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:50:36.320888407Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T00:55:36.321191863Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T01:00:36.321716372Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T01:05:36.322294354Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main
level=info ts=2020-12-10T01:10:36.322304123Z caller=operator.go:447 component=alertmanageroperator msg="sync alertmanager" key=kubesphere-monitoring-system/main

freemankevin

level=error ts=2020-12-09T18:16:52.107235607Z caller=operator.go:622 component=prometheusoperator msg="syncing nodes into Endpoints object failed" err="listing nodes failed: etcdserver: request timed out"

level=error ts=2020-12-10T00:04:52.807152069Z caller=operator.go:622 component=prometheusoperator msg="syncing nodes into Endpoints object failed" err="listing nodes failed: etcdserver: request timed out"

zackzhang

freemankevin 使用的prometheus operator，会自动把servicemonitor转成prometheus的配置

freemankevin

zackzhang 不太明白嘞,啥意思

benjaminhuo

测试下kube-scheduler 的 service 是否能抓取到数据

kubectl run --rm utils -it --generator=run-pod/v1 --image arunvelsriram/utils bash
curl kube-scheduler-svc.kube-system.svc:10251/metrics

benjaminhuo

另外可以看看prometheus ui 上的 target 里面的 kube-scheduler 是否正常，比如下图

需要把 prometheus-k8s 这个 service 改为 nodeport 后访问

freemankevin

benjaminhuo

freemankevin

benjaminhuo ks面板里没有显示异常的地方,只有没有数据的

freemankevin

benjaminhuo

你的k8s 是通过kk 安装的吗？
截图说明 kube-scheduler 的 svc 找不到后端kube-scheduler 的 pod
正常情况应该是这样的
kubectl -n kube-system get svc kube-scheduler-svc -oyaml 显示会找有 component: kube-scheduler 这样的 label 的 pod

kubectl -n kube-system get pod kube-scheduler-i-dwh1ow3b --show-labels 也显示 kube-scheduler 的 pod 也有相应的 label, 数据才能根据 service 从 pod 抓取过来

freemankevin

benjaminhuo

freemankevin

benjaminhuo

freemankevin

benjaminhuo 上面这些应该是正常的对吧?

freemankevin

benjaminhuo

运行下面命令看看能不能抓取到数据吧

kubectl run --rm utils -it --generator=run-pod/v1 --image arunvelsriram/utils bash
curl kube-scheduler-svc.kube-system.svc:10251/metrics

freemankevin

benjaminhuo