prometheus-k8s-0 pending

Small-market

按照all-in-one的下载,这里为什么 prometheus-k8s-0 一直处于pending,有大佬教教怎么解决嘛

yinmin2020

Small-market 看看日志，在看看对应的pv是不是绑定了。

Small-market

yinmin2020

[root@k8s-node1 ~]# kubectl describe pod prometheus-k8s-0 --namespace=kubesphere-monitoring-system
Name:           prometheus-k8s-0
Namespace:      kubesphere-monitoring-system
Priority:       0
Node:           <none>
Labels:         app=prometheus
                controller-revision-hash=prometheus-k8s-85b55f7b4b
                prometheus=k8s
                statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/prometheus-k8s
Containers:
  prometheus:
    Image:      prom/prometheus:v2.26.0
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=7d
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --query.max-concurrency=1000
      --web.route-prefix=/
    Limits:
      cpu:     4
      memory:  16Gi
    Requests:
      cpu:        200m
      memory:     400Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-56nfd (ro)
  prometheus-config-reloader:
    Image:      kubesphere/prometheus-config-reloader:v0.42.1
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    Limits:
      memory:  25Mi
    Requests:
      memory:  25Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-56nfd (ro)
  rules-configmap-reloader:
    Image:      jimmidyson/configmap-reload:v0.3.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --webhook-url=http://localhost:9090/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      memory:  25Mi
    Requests:
      memory:     25Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-56nfd (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  prometheus-k8s-token-56nfd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-token-56nfd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     dedicated=monitoring:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  72m   default-scheduler  0/1 nodes are available: 1 Insufficient cpu.
[root@k8s-node1 ~]# kubectl get pv,pvc
No resources found
[root@k8s-node1 ~]#

Small-market

yinmin2020 这个是没挂载还是什么原因

kevendeng

Small-market
是否是1.20版本的k8s，使用的是NFS作为存储类型？

Small-market

kevendeng 是1.20.4的k8s

Small-market

kevendeng nfs?这个要怎么弄?

kevendeng

Small-market 应该跟NFS无关，从你的POD事件来看，你的节点CPU不足了： Warning FailedScheduling 72m default-scheduler 0/1 nodes are available: 1 Insufficient cpu.

Small-market

kevendeng 节点cpu不足?这个是需要将它配置改大点吗?

kevendeng

Small-market
执行以下命令查看你的节点还剩余多少可分配的cpu

kubectl get node $(kubectl get node -o jsonpath='{.items[0].metadata.name}') -o jsonpath='{.status.allocatable.cpu}'

你的prometheus-k8s pod 要求200m的cpu资源，以及4000的cpu上限
这并不是它将持续实际消耗的资源，而是预先规划好的资源
建议的做法当然是调高节点的CPU配置，不然无法完全确保集群有足够的资源正常运作
如果你只是想临时绕过、测试功能，可以手动将prometheus-k8s所需资源调小

kubectl -n kubesphere-monitoring-system edit sts prometheus-k8s

将resources.limits/requests.cpu的数值调低

Small-market

kevendeng 那个memory也要调小吗,我这只有1600m的空闲

kevendeng

Small-market 你的mem需求并没有达到1600，目前是够用的

Small-market

kevendeng 那这个为什么是pending状态啊,怎么弄,控制台显示无法调度,minio也一样

kevendeng

Small-market 每个组件都有它所需的资源，如cpu/mem，它们分别需求多少，总共需求多少，哪些资源不足，需要你自己去检查并解决，如果不太熟悉具体指标，建议直接将节点资源调高。

Small-market

kevendeng 好的,谢谢了