Kubesphere使用问题

xuanyuanaosheng

kubesphere也能脱离 k8s 集群单独安装么？（就是讲kubesphere单独安装在一台Linux服务器上）
对于已经安装了metrics server的k8s集群，在kubesphere里面默认是能看到所有的监控数据的么？ kubesphere 里面使用的是： kube-state-metrics （kubesphere/kube-state-metrics:v1.9.6），监听 Kubernetes API server 以获取集群中各种 API 对象的状态包括节点，工作负载和 Pod 等，并生成相关监控数据供 Prometheus 抓取。
目前遇到的问题（该问题通过重新安装 kubesphere 竟然莫名其妙的解决了）：

这些监控数据没有，但是监控的namespace一切正常。

新的问题，自建集群无法获取 master 节点的 k8s 的服务状态：

- job_name: kubesphere-monitoring-system/kube-scheduler/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  scrape_interval: 1m
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_k8s_app
    regex: kube-scheduler
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_k8s_app
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
  metric_relabel_configs:
  - source_labels:
    - __name__
    regex: scheduler_(e2e_scheduling_latency_microseconds|scheduling_algorithm_predicate_evaluation|scheduling_algorithm_priority_evaluation|scheduling_algorithm_preemption_evaluation|scheduling_algorithm_latency_microseconds|binding_latency_microseconds|scheduling_latency_seconds)
    action: drop

ETCD 的监控有问题。

- job_name: kubesphere-monitoring-system/etcd/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  scrape_interval: 1m
  scheme: https
  tls_config:
    insecure_skip_verify: false
    ca_file: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt
    cert_file: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.crt
    key_file: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.key
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_k8s_app
    regex: etcd
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_k8s_app
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: metrics

请教个问题，想要在 kubesphere 的监控系统里面增加其他的exporter 配置, 请问怎么增加（不想在额外搭建一套了，想直接使用 kubesphere 本身自带的监控系统）？
问题的答案：

- https://kubesphere.com.cn/forum/d/2598-prometheus
- https://kubesphere.io/zh/docs/project-user-guide/custom-application-monitoring/examples/monitor-sample-web/
- https://kubesphere.io/docs/faq/observability/byop/

yuswift

1.你的意思是ks单独安装然后管控这个服务器之外的集群吗这个目前我们不支持 3.1已经确定了可以减少资源占用独立出来正在做发布时间待定
2.可以打开浏览器开发模式看看相关api返回是否有报错

xuanyuanaosheng

yuswift 谢谢您的回复，期待独立版本的发布。

对于api返回没有数值的情况：

初步判断是kube-state-metrics 没有采集相应的数据

API返回的结果：

yuswift

xuanyuanaosheng 这个可以查查kube-state-metrics的日志哈

xuanyuanaosheng

yuswift

I0312 03:14:18.468890 1 main.go:86] Using default collectors
I0312 03:14:18.468951 1 main.go:98] Using all namespace
I0312 03:14:18.469311 1 main.go:139] metric white-blacklisting: blacklisting the following items: kube_service_(spec_.+|status_.+), kube_secret_(type), kube_deployment_(spec_paused|spec_strategy_rollingupdate_.+), kube_pod_container_status_.*terminated_reason, kube_ingress_(info|path|tls), kube_replicaset_(status_.+|spec_.+|owner), kube_persistentvolume_(info|capacity_.+), kube_replicationcontroller_.+, kube_node_info, kube_poddisruptionbudget_status_.+, kube_namespace_(status_phase), kube_.+_version, kube_cronjob_(info|status_.+|spec_.+), kube_endpoint_(info|address_.+), kube_job_(info|owner|spec_(parallelism|active_deadline_seconds)|status_(active|.+_time)), kube_persistentvolumeclaim_(resource_.+|access_.+), kube_.+_created, kube_(hpa|replicaset|replicationcontroller)_.+_generation
W0312 03:14:18.469344 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0312 03:14:18.470752 1 main.go:184] Testing communication with server
I0312 03:14:18.477759 1 main.go:189] Running with Kubernetes cluster version: v1.20. git version: v1.20.2. git tree state: clean. commit: faecb196815e248d3ecfb03c680a4507229c2a56. platform: linux/amd64
I0312 03:14:18.477785 1 main.go:191] Communication with server successful
I0312 03:14:18.477918 1 main.go:225] Starting metrics server: 0.0.0.0:8081
I0312 03:14:18.478005 1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8082
I0312 03:14:18.478010 1 metrics_handler.go:96] Autosharding disabled
I0312 03:14:18.480840 1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments

yuswift

xuanyuanaosheng 你好你是用的k8s 1.20吗 ks 3.0目前支持到1.18 1.19需要手动执行一些步骤 1.20暂未测试过

xuanyuanaosheng

yuswift 嗯嗯是的，用的 1.2.0. 报错信息已经给您发了。我这边属于新的一批用户，哈哈
如果需要手动修复，需要手动执行啥？我这边测试测试。是不是需要新的镜像版本？

Ls569689934

第一个问题：all-in-one安装

xuanyuanaosheng

Ls569689934 谢谢您的回复，目前kubeshpere还不支持
@yuswift 请问有解决办法么？

xuanyuanaosheng

Kubesphere的监控系统的问题的排查方案：

替换 kubesphere 默认的监控系统（kubesphere目前使用 kube-prometheus 来作为监控系统）： https://kubesphere.io/docs/faq/observability/byop/
增加自定义的监控： https://kubesphere.io/docs/project-user-guide/custom-application-monitoring/introduction/
怎样修改prometheus的配置文件： https://kubesphere.com.cn/forum/d/2598-prometheus
怎样修复api-server的监控： https://kubesphere.com.cn/forum/d/1374-kube-apisrever-kubesphere
怎样修复etcd的监控： https://kubesphere.com.cn/forum/d/1386-kubesphere-etcd

xuanyuanaosheng

使用kubesphere 的应用商店已经创建的应用还能查看当时配置的 value. yml么？

xuanyuanaosheng

怎样增加k8s外部的自定义的监控：

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md
也可以使用下面的方法来新增外部的监控

怎样增加 k8e 内部服务的监控：通过 ServiceMonitor 来增加对应的监控