KS 系统资源面板数据采集异常，后台监控pod状态正常 ,想咨询下该部分属于哪个组件负责

freemankevin

benjaminhuo

freemankevin 需要把 utils 这个 pod 删掉再运行，这两个命令的意思是启动一个 utils pod，在这个 pod 里面运行 curl 命令看scheduler 的 metrics 是否可以访问

freemankevin

benjaminhuo

说明你 scheduler pod 的端口 10251 获取不到 metrics
你的 k8s 是 kubekey 安装的吗？

贴一下你这个 pod 的yaml
正常情况pod 是这样的

svc 是这样的

端口能对上才行

freemankevin

benjaminhuo
用kubeadm装的，这里端口好像没有

[root@master1 ~]# kubectl describe -n kube-system  pod kube-scheduler-master1
Name:                 kube-scheduler-master1
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master1/10.0.1.216
Start Time:           Thu, 19 Nov 2020 01:10:16 +0800
Labels:               component=kube-scheduler
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: da215ebee0354225c20c7bdf28b467f8
                      kubernetes.io/config.mirror: da215ebee0354225c20c7bdf28b467f8
                      kubernetes.io/config.seen: 2020-11-28T20:58:38.276350708+08:00
                      kubernetes.io/config.source: file
Status:               Running
IP:                   10.0.1.216
IPs:
  IP:           10.0.1.216
Controlled By:  Node/master1
Containers:
  kube-scheduler:
    Container ID:  docker://aa1e8fddad33dc2e155d45e26c9a3667080a2f93d066609dc7d6a6d082181cfa
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.9
    Image ID:      docker-pullable://kubesphere/kube-scheduler@sha256:e43bbf97bb40bb7c0502a588e45695e032827bd9300969bd6a74468660c2c061
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
      --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
      --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
      --bind-address=127.0.0.1
      --kubeconfig=/etc/kubernetes/scheduler.conf
      --leader-elect=true
      --port=0
    State:          Running
      Started:      Thu, 10 Dec 2020 19:18:13 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 10 Dec 2020 18:31:44 +0800
      Finished:     Thu, 10 Dec 2020 19:18:09 +0800
    Ready:          True
    Restart Count:  458
    Requests:
      cpu:        100m
    Liveness:     http-get https://127.0.0.1:10259/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/kubernetes/scheduler.conf from kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/scheduler.conf
    HostPathType:  FileOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:
  Type     Reason     Age                  From              Message
  ----     ------     ----                 ----              -------
  Warning  Unhealthy  13m (x195 over 11d)  kubelet, master1  Liveness probe failed: Get https://127.0.0.1:10259/healthz: dial tcp 127.0.0.1:10259: connect: connection refused
  Normal   Pulled     13m (x245 over 11d)  kubelet, master1  Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.9" already present on machine
  Normal   Created    13m (x245 over 11d)  kubelet, master1  Created container kube-scheduler
  Normal   Started    13m (x245 over 11d)  kubelet, master1  Started container kube-scheduler
[root@master1 ~]#

freemankevin

benjaminhuo

[root@master1 ~]# kubectl get  -n kube-system  svc
NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
etcd                           ClusterIP   None             <none>        2379/TCP                       19d
heapster                       ClusterIP   10.99.236.148    <none>        80/TCP                         21d
kube-controller-manager-svc    ClusterIP   None             <none>        10252/TCP                      21d
kube-dns                       ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP         21d
kube-scheduler-svc             ClusterIP   None             <none>        10251/TCP                      21d
kubelet                        ClusterIP   None             <none>        10250/TCP,10255/TCP,4194/TCP   21d
metrics-server                 ClusterIP   10.96.241.207    <none>        443/TCP                        21d
monitoring-grafana             ClusterIP   10.105.171.235   <none>        80/TCP                         21d
monitoring-influxdb            ClusterIP   10.96.5.20       <none>        8086/TCP                       21d
monitoring-influxdb-nodeport   NodePort    10.102.77.232    <none>        8086:30001/TCP                 21d
[root@master1 ~]#
[root@master1 ~]#
[root@master1 ~]# kubectl describe   -n kube-system  svc kube-scheduler-svc
Name:              kube-scheduler-svc
Namespace:         kube-system
Labels:            k8s-app=kube-scheduler
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kube-scheduler"},"name":"kube-scheduler-svc","namesp...
Selector:          component=kube-scheduler
Type:              ClusterIP
IP:                None
Port:              http-metrics  10251/TCP
TargetPort:        10251/TCP
Endpoints:         10.0.1.216:10251,10.0.1.222:10251,10.0.1.223:10251
Session Affinity:  None
Events:            <none>
[root@master1 ~]#

benjaminhuo

edit kube-scheduler 的 deployment，把 port设为 10251 就可以了，你这是k8s 什么版本？

freemankevin

benjaminhuo 17.9

freemankevin

benjaminhuo

没有deploy想获取监控数据就直接改 kube-scheduler 的 pod 的 spec, port=0 改成 port=10251
看你有 3 个 kube-scheduler，都得改

freemankevin

benjaminhuo 🤣 hahahhahaahh

freemankevin

benjaminhuo 谢谢老哥提供的思路，问题目前已解决。

那个pod是不能直接edit的，而是改/etc下面的yaml文件的 0 —->10251/10252

[root@master3 ~]# kubectl get pod -n kube-system | grep -E 'kube-scheduler|kube-controller-manager'
kube-controller-manager-master1          1/1     Running   0          14s
kube-controller-manager-master2          1/1     Running   0          24s
kube-controller-manager-master3          1/1     Running   0          36s
kube-scheduler-master1                   1/1     Running   0          5m2s
kube-scheduler-master2                   1/1     Running   0          3m17s
kube-scheduler-master3                   1/1     Running   0          4m54s
[root@master3 ~]# kubectl delete  pod kube-controller-manager-master1 -n  kube-system
pod "kube-controller-manager-master1" deleted
[root@master3 ~]# kubectl delete  pod kube-controller-manager-master2 -n  kube-system
pod "kube-controller-manager-master2" deleted
[root@master3 ~]# kubectl delete  pod kube-controller-manager-master3 -n  kube-system
pod "kube-controller-manager-master3" deleted
[root@master3 ~]# kubectl get ep -n kube-system | grep -E 'kube-scheduler|kube-controller-manager'
kube-controller-manager        <none>                                                            23d
kube-controller-manager-svc    10.0.1.216:10252,10.0.1.222:10252,10.0.1.223:10252                23d
kube-scheduler                 <none>                                                            23d
kube-scheduler-svc             10.0.1.216:10251,10.0.1.222:10251,10.0.1.223:10251                23d
[root@master3 ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
  tadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --port=10252
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.9
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10257
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
[root@master3 ~]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --port=10251
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.9
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
[root@master3 ~]#

freemankevin

benjaminhuo 不知这个算不算bug?

v3.1.1也遇到类似问题，补充下：

# 先获取svc的端口信息
kubectl get  -n kube-system svc  kube-scheduler-svc  -oyaml|grep "port:"

# 再将yaml里的port改为上面查到的（多master,就得上去一个个改了）
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
vim /etc/kubernetes/manifests/kube-scheduler.yaml
# 删了让他重建
kubectl delete  pod kube-controller-manager-master3 -n  kube-system
kubectl delete  pod kube-controller-manager-master2 -n  kube-system
kubectl delete  pod kube-controller-manager-master1 -n  kube-system
# 再去确认下
kubectl get po -A |grep controller
kubectl describe -n kube-system  pod kube-scheduler-master1
kubectl describe -n kube-system  pod kube-scheduler-master2
kubectl describe -n kube-system  pod kube-scheduler-master3

不过结果恰恰相反，改了反而没用，现在不改普罗反而是正常的

freemankevin

普罗面板正常了

freemankevin

ks面板右上角落这个调度数据现在也都有了

« 上一页