求助：安装kube-state-metrics后kubesphere监控出现问题

huanggze

18342204169

cluster role binding 也看一下

kubectl get clusterrolebinding kube-state-metrics -oyaml

18342204169

huanggze
bash-5.0# kubectl get clusterrolebinding kube-state-metrics -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“rbac.authorization.k8s.io/v1”,“kind”:“ClusterRoleBinding”,“metadata”:{“annotations”:{},“name”:“kube-state-metrics”},“roleRef”:{“apiGroup”:“rbac.authorization.k8s.io”,“kind”:“ClusterRole”,“name”:“kube-state-metrics”},“subjects”:[{“kind”:“ServiceAccount”,“name”:“kube-state-metrics”,“namespace”:“prom”}]}
creationTimestamp: “2020-02-09T04:45:57Z”
name: kube-state-metrics
resourceVersion: “424419”
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kube-state-metrics
uid: 7648fc0a-24b0-4ff1-a0d3-7d076ecf4a2f
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:

kind: ServiceAccount
name: kube-state-metrics
namespace: prom

huanggze

18342204169

最后一行怎么是这个？应该是 kubesphere-monitoring-system
namespace: prom

18342204169

huanggze
我自己装的kube-state-metrics装在了namespace:prom下，也做了认证。是不是冲突了啊？

huanggze

18342204169

整个问题是这个逻辑：

首先，UI 上监控数据是从 kubesphere-monitoring-system 下的 prometheus 的数据库里拿。prometheus 会周期地采集 kubesphere-monitoring-system 下的 kube-state-metrics。

现在 kube-state-metrics 的 ServiceAcount 和 ClusterRole 都在，但 binding 绑定错了，不是绑的 kubesphere-monitoring-system 。所以出现权限相关的 fail 日志。

解决办法是，把这个 namespace 改成 kubesphere-monitoring-system 。不确定你还有没改了其他的，先改这里试试。（如果你要把 prom 下的 kube-state-metrics 接入 prometheus，还有好些步骤要操作。没必要）

18342204169

huanggze
请问一下应该怎么把namespace改回去呢，我当时是新建了一个ServiceAccount绑定到了namespace:prom的。

huanggze

18342204169

使用以下命令改：
kubectl edit clusterrolebinding kube-state-metrics

18342204169

huanggze

改回去了还是没有数据

huanggze

18342204169

需要重启吧，删了等它重建看看呢？
kubectl delete po -n kubesphere-monitoring-system kube-state-metrics-649b4fbc8d-pvglr

18342204169

huanggze
删除了，也重新部署试过了。还是没有。

huanggze

18342204169 现在kubectl log 报什么错

huanggze

18342204169 还有没有改过其他东西？

18342204169

huanggze
还是证书的问题

18342204169

huanggze
—
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-state-metrics

namespace: prom

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-state-metrics
rules:

apiGroups: [""]
resources: [“nodes”, “pods”, “services”, “resourcequotas”, “replicationcontrollers”, “limitranges”, “persistentvolumeclaims”, “persistentvolumes”, “namespaces”, “endpoints”]
verbs: [“list”, “watch”]
apiGroups: [“extensions”]
resources: [“daemonsets”, “deployments”, “replicasets”]
verbs: [“list”, “watch”]
apiGroups: [“apps”]
resources: [“statefulsets”]
verbs: [“list”, “watch”]
apiGroups: [“batch”]
resources: [“cronjobs”, “jobs”]
verbs: [“list”, “watch”]
apiGroups: [“autoscaling”]
resources: [“horizontalpodautoscalers”]
verbs: [“list”, “watch”]
—
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kube-state-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
kind: ServiceAccount
name: kube-state-metrics
namespace: prom

这是我当时部署的yaml文件，部署完这个组件就出问题了

huanggze

18342204169

还改过别的吗？有没有改过 prometheus CRD?

huanggze

18342204169

再打印一次 cluster role binding 看看，我看看改后的内容

kubectl get clusterrolebinding kube-state-metrics -oyaml

18342204169

huanggze
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“rbac.authorization.k8s.io/v1”,“kind”:“ClusterRoleBinding”,“metadata”:{“annotations”:{},“name”:“kube-state-metrics”},“roleRef”:{“apiGroup”:“rbac.authorization.k8s.io”,“kind”:“ClusterRole”,“name”:“kube-state-metrics”},“subjects”:[{“kind”:“ServiceAccount”,“name”:“kube-state-metrics”,“namespace”:“kubesphere-monitoring-system”}]}
creationTimestamp: “2020-02-09T04:45:57Z”
name: kube-state-metrics
resourceVersion: “447930”
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kube-state-metrics
uid: 7648fc0a-24b0-4ff1-a0d3-7d076ecf4a2f
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:

kind: ServiceAccount
name: kube-state-metrics
namespace: kubesphere-monitoring-system

freemankevin

同问，有个类似的监控问题。监控起不来，日志的起不来

yunkunrao

从截图上看prometheus-k8s pod正在创建当中，等pod变为running状态再看一下。

freemankevin

yunkunrao 放了一整晚了，早上过来还是这样的