kubesphere成功安装，prometheus需要单独安装吗？

Mmingr1313 · 2020年11月20日

服务组件 prometheus-k8s 下没有相应的 prometheus-k8s 容器组，显示为 “已停止”。请大神帮忙！！！

Mmingr1313 · 2020年11月23日

@hongming 帮忙看分析一下问题

Ddef__init__ · 2020年11月23日

相同的问题，kubectl-admin也是。我也提了问题 kubesphere平台提供的kubectl工具的镜像版本有问题，看来不是个例

Mmingr1313 · 2020年11月23日

def__init__ 已经解决。
prometheus需要单独安装，kubectl appyl -f prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: kubesphere-monitoring-system
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: kubesphere-monitoring-system
      port: web
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
#  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
#  probeNamespaceSelector: {}
#  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.22.1
  ` ` `

benjaminhuo · 2020年11月23日

贴一下如下命令的输出 kubectl -n kubesphere-monitoring-system logs prometheus-operator-xxx -c prometheus-operator

还有 kubectl -n kubesphere-monitoring-system get prometheus k8s -oyaml

Ddef__init__ · 2020年11月24日

mingr1313 独立安装肯定是没问题的，但是这个问题的来源是什么？如果每次都要手动安装那ks-installer不就等于没用了

Ddef__init__ · 2020年11月24日

benjaminhuo kubectl -n kubesphere-monitoring-system logs prometheus-operator-xxx -c prometheus-operator 的结果
ts=2020-11-18T18:49:47.916901931Z caller=main.go:188 msg="Starting Prometheus Operator version '0.38.3'." ts=2020-11-18T18:49:47.924335649Z caller=main.go:98 msg="Staring insecure server on :8080" level=info ts=2020-11-18T18:49:48.018093419Z caller=operator.go:308 component=thanosoperator msg="connection established" cluster-version=v1.18.8 level=info ts=2020-11-18T18:49:48.018091487Z caller=operator.go:213 component=alertmanageroperator msg="connection established" cluster-version=v1.18.8 level=info ts=2020-11-18T18:49:48.01810684Z caller=operator.go:464 component=prometheusoperator msg="connection established" cluster-version=v1.18.8 level=info ts=2020-11-18T18:49:50.218098666Z caller=operator.go:643 component=alertmanageroperator msg="CRD updated" crd=Alertmanager level=info ts=2020-11-18T18:49:50.4185597Z caller=operator.go:718 component=thanosoperator msg="CRD updated" crd=ThanosRuler level=info ts=2020-11-18T18:49:50.619791003Z caller=operator.go:1941 component=prometheusoperator msg="CRD updated" crd=Prometheus level=info ts=2020-11-18T18:49:50.650358458Z caller=operator.go:1941 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor level=info ts=2020-11-18T18:49:50.716015878Z caller=operator.go:1941 component=prometheusoperator msg="CRD updated" crd=PodMonitor level=info ts=2020-11-18T18:49:50.727115852Z caller=operator.go:1941 component=prometheusoperator msg="CRD updated" crd=PrometheusRule level=info ts=2020-11-18T18:49:53.229173808Z caller=operator.go:229 component=alertmanageroperator msg="CRD API endpoints ready" level=info ts=2020-11-18T18:49:53.515744424Z caller=operator.go:184 component=alertmanageroperator msg="successfully synced all caches" level=info ts=2020-11-18T18:49:56.433016635Z caller=operator.go:324 component=thanosoperator msg="CRD API endpoints ready" level=info ts=2020-11-18T18:49:57.117117552Z caller=operator.go:269 component=thanosoperator msg="successfully synced all caches" level=info ts=2020-11-18T18:50:03.019428553Z caller=operator.go:480 component=prometheusoperator msg="CRD API endpoints ready" level=info ts=2020-11-18T18:50:03.922739774Z caller=operator.go:410 component=prometheusoperator msg="successfully synced all caches"

kubectl -n kubesphere-monitoring-system get prometheus k8s -oyaml 的结果
Error from server (NotFound): prometheuses.monitoring.coreos.com "k8s" not found

benjaminhuo · 2020年11月24日

def__init__ 那是 prometheus 的 instance 没有安装上，可能是dockerhub pull的时候有问题，需要进 ks-installer pod ,重新安装下，默认 prometheus 是会安装的

kubectl -n kubesphere-system exec -it `kubectl -n kubesphere-system get pod|grep ks-installer|awk '{print $1}'` -- /bin/sh
kubectl apply -f /kubesphere/kubesphere/prometheus/prometheus

benjaminhuo · 2020年11月24日

prometheus 这个错已经改正，3.0 安装包应该没问题，近期安装不上prometheus 的错应该是自己从github 的 repo 下载的 ks-installer master 版本导致的。请用kk 安装 3.0 GA指定版本

Ddef__init__ · 2020年11月24日

benjaminhuo
我是在已有k8s集群上安装kubesphere的，用的官方文档的这个指令
重新运行了一下这两个指令，跟之前一样ks-installer运行后会有task失败，日志如下：
`Task ‘monitoring’ failed:

{
“counter”: 105,
“created”: “2020-11-24T09:49:24.714092”,
“end_line”: 103,
“event”: “runner_on_failed”,
“event_data”: {
“duration”: 40.369034,
“end”: “2020-11-24T09:49:24.713963”,
“event_loop”: null,
“host”: “localhost”,
“ignore_errors”: null,
“play”: “localhost”,
“play_pattern”: “localhost”,
“play_uuid”: “bae20c3f-ec61-5e03-b17b-000000000005”,
“playbook”: “/kubesphere/playbooks/monitoring.yaml”,
“playbook_uuid”: “e2c71a74-0ae9-4fda-94b0-4bfb0eef5083”,
“remote_addr”: “127.0.0.1”,
“res”: {
“changed”: true,
“msg”: “All items completed”,
“results”: [
{
“ansible_item_label”: “prometheus”,
“ansible_no_log”: false,
“ansible_loop_var”: “item”,
“attempts”: 5,
“changed”: true,
“cmd”: “/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/prometheus”,
“delta”: “0:00:00.537379”,
“end”: “2020-11-24 17:49:04.573260”,
“failed”: true,
“failed_when_result”: true,
“invocation”: {
“module_args”: {
“raw_params”: “/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/prometheus”,
“uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“item”: “prometheus”,
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2020-11-24 17:49:04.035881”,
“stderr”: “error: error validating \”/kubesphere/kubesphere/prometheus/prometheus/prometheus-prometheus.yaml\": error validating data: ValidationError(Prometheus.spec.serviceMonitorSelector): unknown field \“app.kubernetes.io/vendor\” in com.coreos.monitoring.v1.Prometheus.spec.serviceMonitorSelector; if you choose to ignore these errors, turn validation off with –validate=false",
“stderr_lines”: [
“error: error validating \”/kubesphere/kubesphere/prometheus/prometheus/prometheus-prometheus.yaml\": error validating data: ValidationError(Prometheus.spec.serviceMonitorSelector): unknown field \“app.kubernetes.io/vendor\” in com.coreos.monitoring.v1.Prometheus.spec.serviceMonitorSelector; if you choose to ignore these errors, turn validation off with –validate=false"
],
“stdout”: “secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged\nservicemonitor.monitoring.coreos.com/prometheus-operator unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/kube-controller-manager-svc unchanged\nservice/kube-scheduler-svc unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged”,
“stdout_lines”: [
“secret/additional-scrape-configs unchanged”,
“clusterrole.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged”,
“clusterrolebinding.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged”,
“servicemonitor.monitoring.coreos.com/prometheus-operator unchanged”,
“rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged”,
“role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged”,
“service/prometheus-k8s unchanged”,
“serviceaccount/prometheus-k8s unchanged”,
“service/kube-controller-manager-svc unchanged”,
“service/kube-scheduler-svc unchanged”,
“servicemonitor.monitoring.coreos.com/prometheus unchanged”,
“servicemonitor.monitoring.coreos.com/kube-apiserver unchanged”,
“servicemonitor.monitoring.coreos.com/coredns unchanged”,
“servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged”,
“servicemonitor.monitoring.coreos.com/kube-scheduler unchanged”,
“servicemonitor.monitoring.coreos.com/kubelet unchanged”
]
},
{
“ansible_item_label”: “prometheus”,
“ansible_no_log”: false,
“ansible_loop_var”: “item”,
“attempts”: 5,
“changed”: true,
“cmd”: “/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/prometheus”,
“delta”: “0:00:00.525305”,
“end”: “2020-11-24 17:49:24.668341”,
“failed”: true,
“failed_when_result”: true,
“invocation”: {
“module_args”: {
“raw_params”: “/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/prometheus”,
“uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“item”: “prometheus”,
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2020-11-24 17:49:24.143036”,
“stderr”: “error: error validating \”/kubesphere/kubesphere/prometheus/prometheus/prometheus-prometheus.yaml\": error validating data: ValidationError(Prometheus.spec.serviceMonitorSelector): unknown field \“app.kubernetes.io/vendor\” in com.coreos.monitoring.v1.Prometheus.spec.serviceMonitorSelector; if you choose to ignore these errors, turn validation off with –validate=false",
“stderr_lines”: [
“error: error validating \”/kubesphere/kubesphere/prometheus/prometheus/prometheus-prometheus.yaml\": error validating data: ValidationError(Prometheus.spec.serviceMonitorSelector): unknown field \“app.kubernetes.io/vendor\” in com.coreos.monitoring.v1.Prometheus.spec.serviceMonitorSelector; if you choose to ignore these errors, turn validation off with –validate=false"
],
“stdout”: “secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged\nservicemonitor.monitoring.coreos.com/prometheus-operator unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/kube-controller-manager-svc unchanged\nservice/kube-scheduler-svc unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged”,
“stdout_lines”: [
“secret/additional-scrape-configs unchanged”,
“clusterrole.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged”,
“clusterrolebinding.rbac.authorization.k8s.io/kubesphere-prometheus-k8s unchanged”,
“servicemonitor.monitoring.coreos.com/prometheus-operator unchanged”,
“rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged”,
“role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged”,
“service/prometheus-k8s unchanged”,
“serviceaccount/prometheus-k8s unchanged”,
“service/kube-controller-manager-svc unchanged”,
“service/kube-scheduler-svc unchanged”,
“servicemonitor.monitoring.coreos.com/prometheus unchanged”,
“servicemonitor.monitoring.coreos.com/kube-apiserver unchanged”,
“servicemonitor.monitoring.coreos.com/coredns unchanged”,
“servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged”,
“servicemonitor.monitoring.coreos.com/kube-scheduler unchanged”,
“servicemonitor.monitoring.coreos.com/kubelet unchanged”
]
}
]
},
“role”: “ks-monitor”,
“start”: “2020-11-24T09:48:44.344929”,
“task”: “ks-monitor | Installing prometheus”,
“task_action”: “shell”,
“task_args”: "",
“task_path”: “/kubesphere/installer/roles/ks-monitor/tasks/prometheus.yaml:2”,
“task_uuid”: “bae20c3f-ec61-5e03-b17b-000000000042”,
“uuid”: “019565f5-38a1-4e58-9a3b-076f86d66021”
},
“parent_uuid”: “bae20c3f-ec61-5e03-b17b-000000000042”,
“pid”: 37555,
“runner_ident”: “monitoring”,
“start_line”: 103,
“stdout”: "",
“uuid”: “019565f5-38a1-4e58-9a3b-076f86d66021”
2020-11-24T17:49:35+08:00 INFO : Grace shutdown with terminated signal`

搜索到论坛里也有这个问题 https://kubesphere.com.cn/forum/d/2692-kubesphere-monitoring，但是没人解答

benjaminhuo · 2020年11月24日

这个问题刚fix, 再试应该不会有问题

benjaminhuo · 2020年11月24日

你是在哪个文档看到安装指令的，麻烦给个url
这个错之有 master branch 的 ks-installer才会有

Mmingr1313 · 2020年12月3日

benjaminhuo
按照这个文档安装的

https://kubesphere.com.cn/docs/installing-on-kubernetes/introduction/overview/

kubesphere成功安装，prometheus需要单独安装吗？

Mmingr1313

mingr1313

Mmingr1313

Ddefinit

Mmingr1313

benjaminhuoK零S

Ddefinit

Ddefinit

benjaminhuoK零S

benjaminhuoK零S

Ddefinit

benjaminhuoK零S

benjaminhuoK零S

Mmingr1313