安装monitoring ks-installer中报错

brosoul · 2019年12月11日

在
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f 中出现

task monitoring status is failed
task logging status is successful
task alerting status is successful
task metrics-server status is successful

但是monitoring 的组件都运行成功

主要错误应该是在

 "Error from server (InternalError): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-rules.yaml\": Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.247.69.235:443: connect: connection refused"

因为之前及群里部署过prom。使用kubesphere部署监控的时候，我把原来的crd删除并且把之前的prom相关的dp scale 到0。这里的post 地址好像是原先的prom的operator prometheus-prometheus-oper-operator.monitoring.svc
完整报错信息见下条

brosoul · 2019年12月11日

Task 'monitoring' failed:
******************************************************************************************************************************************************
{
  "counter": 62, 
  "created": "2019-12-11T03:24:53.367978", 
  "end_line": 60, 
  "event": "runner_on_failed", 
  "event_data": {
    "event_loop": null, 
    "host": "localhost", 
    "ignore_errors": null, 
    "play": "localhost", 
    "play_pattern": "localhost", 
    "play_uuid": "0255ac10-0063-ec0c-88fd-000000000005", 
    "playbook": "/kubesphere/playbooks/monitoring.yaml", 
    "playbook_uuid": "a0a078e3-cf18-40f8-90e7-516a802644cd", 
    "remote_addr": "127.0.0.1", 
    "res": {
      "changed": true, 
      "msg": "All items completed", 
      "results": [
        {
          "_ansible_item_label": "sources", 
          "_ansible_no_log": false, 
          "ansible_loop_var": "item", 
          "attempts": 5, 
          "changed": true, 
          "cmd": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", 
          "delta": "0:00:01.820096", 
          "end": "2019-12-11 03:24:25.696958", 
          "failed": true, 
          "failed_when_result": true, 
          "invocation": {
            "module_args": {
              "_raw_params": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", 
              "_uses_shell": true, 
              "argv": null, 
              "chdir": null, 
              "creates": null, 
              "executable": null, 
              "removes": null, 
              "stdin": null, 
              "stdin_add_newline": true, 
              "strip_empty_ends": true, 
              "warn": true
            }
          }, 
          "item": "sources", 
          "msg": "non-zero return code", 
          "rc": 1, 
          "start": "2019-12-11 03:24:23.876862", 
          "stderr": "Error from server (Invalid): error when applying patch:\n{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"node-exporter\"},{\"name\":\"kube-rbac-proxy\"}],\"containers\":[{\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/host/proc\"},{\"mountPath\":\"/host/sys\"},{\"mountPath\":\"/host/root\"}],\"args\":[\"--web.listen-address=127.0.0.1:9100\",\"--path.procfs=/host/proc\",\"--path.sysfs=/host/sys\",\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\",\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"],\"name\":\"node-exporter\",\"volumeMounts\":[{\"mountPath\":\"/host/proc\",\"readOnly\":false},{\"mountPath\":\"/host/sys\",\"readOnly\":false}]},{\"$setElementOrder/ports\":[{\"containerPort\":9100}],\"args\":[\"--secure-listen-address=$(IP):9100\",\"--upstream=http://127.0.0.1:9100/\"],\"name\":\"kube-rbac-proxy\",\"ports\":[{\"containerPort\":9100,\"hostPort\":9100,\"name\":\"https\"}]}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"node-role.kubernetes.io/master\"},{\"key\":\"dedicated\",\"operator\":\"Exists\"}]}}}}\nto:\nResource: \"apps/v1beta2, Resource=daemonsets\", GroupVersionKind: \"apps/v1beta2, Kind=DaemonSet\"\nName: \"node-exporter\", Namespace: \"kubesphere-monitoring-system\"\nObject: &{map[\"status\":map[\"numberReady\":'\\x06' \"observedGeneration\":'\\x04' \"updatedNumberScheduled\":'\\x06' \"numberAvailable\":'\\x06' \"currentNumberScheduled\":'\\x06' \"numberMisscheduled\":'\\x00' \"desiredNumberScheduled\":'\\x06'] \"kind\":\"DaemonSet\" \"apiVersion\":\"apps/v1beta2\" \"metadata\":map[\"selfLink\":\"/apis/apps/v1beta2/namespaces/kubesphere-monitoring-system/daemonsets/node-exporter\" \"uid\":\"b851174e-1bba-11ea-abb4-fa163ec5aba4\" \"resourceVersion\":\"45042072\" \"name\":\"node-exporter\" \"generation\":'\\x04' \"creationTimestamp\":\"2019-12-11T02:05:35Z\" \"labels\":map[\"app\":\"node-exporter\"] \"annotations\":map[\"deprecated.daemonset.template.generation\":\"4\" \"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1beta2\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"},\\\"name\\\":\\\"node-exporter\\\",\\\"namespace\\\":\\\"kubesphere-monitoring-system\\\"},\\\"spec\\\":{\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--web.listen-address=127.0.0.1:9100\\\",\\\"--path.procfs=/host/proc\\\",\\\"--path.sysfs=/host/sys\\\",\\\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\\\",\\\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\\\"],\\\"image\\\":\\\"kubesphere/node-exporter:ks-v0.16.0\\\",\\\"name\\\":\\\"node-exporter\\\",\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"250m\\\",\\\"memory\\\":\\\"180Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"180Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/host/proc\\\",\\\"name\\\":\\\"proc\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/sys\\\",\\\"name\\\":\\\"sys\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/root\\\",\\\"mountPropagation\\\":\\\"HostToContainer\\\",\\\"name\\\":\\\"root\\\",\\\"readOnly\\\":true}]},{\\\"args\\\":[\\\"--secure-listen-address=$(IP):9100\\\",\\\"--upstream=http://127.0.0.1:9100/\\\"],\\\"env\\\":[{\\\"name\\\":\\\"IP\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"status.podIP\\\"}}}],\\\"image\\\":\\\"kubesphere/kube-rbac-proxy:v0.4.1\\\",\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ports\\\":[{\\\"containerPort\\\":9100,\\\"hostPort\\\":9100,\\\"name\\\":\\\"https\\\"}],\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"20m\\\",\\\"memory\\\":\\\"40Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"20Mi\\\"}}}],\\\"hostNetwork\\\":true,\\\"hostPID\\\":true,\\\"imagePullSecrets\\\":[{\\\"name\\\":\\\"qingcloud\\\"}],\\\"nodeSelector\\\":{\\\"beta.kubernetes.io/os\\\":\\\"linux\\\"},\\\"securityContext\\\":{\\\"runAsNonRoot\\\":true,\\\"runAsUser\\\":65534},\\\"serviceAccountName\\\":\\\"node-exporter\\\",\\\"tolerations\\\":[{\\\"effect\\\":\\\"NoSchedule\\\",\\\"key\\\":\\\"node-role.kubernetes.io/master\\\"},{\\\"key\\\":\\\"dedicated\\\",\\\"operator\\\":\\\"Exists\\\"}],\\\"volumes\\\":[{\\\"hostPath\\\":{\\\"path\\\":\\\"/proc\\\"},\\\"name\\\":\\\"proc\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/sys\\\"},\\\"name\\\":\\\"sys\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/\\\"},\\\"name\\\":\\\"root\\\"}]}}}}\\n\"] \"namespace\":\"kubesphere-monitoring-system\"] \"spec\":map[\"selector\":map[\"matchLabels\":map[\"app\":\"node-exporter\"]] \"template\":map[\"metadata\":map[\"creationTimestamp\":<nil> \"labels\":map[\"app\":\"node-exporter\"]] \"spec\":map[\"terminationGracePeriodSeconds\":'\\x1e' \"serviceAccountName\":\"node-exporter\" \"hostPID\":%!q(bool=true) \"securityContext\":map[\"runAsUser\":'\\ufffe' \"runAsNonRoot\":%!q(bool=true)] \"tolerations\":[map[\"operator\":\"Exists\" \"effect\":\"NoSchedule\"]] \"containers\":[map[\"volumeMounts\":[map[\"name\":\"proc\" \"mountPath\":\"/host/proc\"] map[\"name\":\"sys\" \"mountPath\":\"/host/sys\"] map[\"readOnly\":%!q(bool=true) \"mountPath\":\"/host/root\" \"mountPropagation\":\"HostToContainer\" \"name\":\"root\"]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"imagePullPolicy\":\"IfNotPresent\" \"name\":\"node-exporter\" \"image\":\"kubesphere/node-exporter:ks-v0.16.0\" \"args\":[\"--web.listen-address=127.0.0.1:9101\" \"--path.procfs=/host/proc\" \"--path.sysfs=/host/sys\" \"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\" \"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"] \"resources\":map[\"limits\":map[\"cpu\":\"250m\" \"memory\":\"180Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"180Mi\"]]] map[\"terminationMessagePolicy\":\"File\" \"image\":\"kubesphere/kube-rbac-proxy:v0.4.1\" \"args\":[\"--secure-listen-address=$(IP):9101\" \"--upstream=http://127.0.0.1:9101/\"] \"ports\":[map[\"name\":\"https\" \"hostPort\":'\\u238d' \"containerPort\":'\\u238d' \"protocol\":\"TCP\"]] \"env\":[map[\"name\":\"IP\" \"valueFrom\":map[\"fieldRef\":map[\"apiVersion\":\"v1\" \"fieldPath\":\"status.podIP\"]]]] \"resources\":map[\"limits\":map[\"cpu\":\"20m\" \"memory\":\"40Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"20Mi\"]] \"name\":\"kube-rbac-proxy\" \"terminationMessagePath\":\"/dev/termination-log\" \"imagePullPolicy\":\"IfNotPresent\"]] \"restartPolicy\":\"Always\" \"serviceAccount\":\"node-exporter\" \"hostNetwork\":%!q(bool=true) \"imagePullSecrets\":[map[\"name\":\"qingcloud\"]] \"schedulerName\":\"default-scheduler\" \"volumes\":[map[\"name\":\"proc\" \"hostPath\":map[\"path\":\"/proc\" \"type\":\"\"]] map[\"name\":\"sys\" \"hostPath\":map[\"type\":\"\" \"path\":\"/sys\"]] map[\"name\":\"root\" \"hostPath\":map[\"path\":\"/\" \"type\":\"\"]]] \"dnsPolicy\":\"ClusterFirst\"]] \"updateStrategy\":map[\"type\":\"RollingUpdate\" \"rollingUpdate\":map[\"maxUnavailable\":'\\x01']] \"revisionHistoryLimit\":'\\n']]}\nfor: \"/etc/kubesphere/prometheus/sources/node-exporter-daemonset.yaml\": DaemonSet.apps \"node-exporter\" is invalid: spec.template.spec.containers[1].ports[1].name: Duplicate value: \"https\"\nError from server (InternalError): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-rules.yaml\": Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.247.69.235:443: connect: connection refused", 
          "stderr_lines": [
            "Error from server (Invalid): error when applying patch:", 
            "{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"node-exporter\"},{\"name\":\"kube-rbac-proxy\"}],\"containers\":[{\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/host/proc\"},{\"mountPath\":\"/host/sys\"},{\"mountPath\":\"/host/root\"}],\"args\":[\"--web.listen-address=127.0.0.1:9100\",\"--path.procfs=/host/proc\",\"--path.sysfs=/host/sys\",\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\",\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"],\"name\":\"node-exporter\",\"volumeMounts\":[{\"mountPath\":\"/host/proc\",\"readOnly\":false},{\"mountPath\":\"/host/sys\",\"readOnly\":false}]},{\"$setElementOrder/ports\":[{\"containerPort\":9100}],\"args\":[\"--secure-listen-address=$(IP):9100\",\"--upstream=http://127.0.0.1:9100/\"],\"name\":\"kube-rbac-proxy\",\"ports\":[{\"containerPort\":9100,\"hostPort\":9100,\"name\":\"https\"}]}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"node-role.kubernetes.io/master\"},{\"key\":\"dedicated\",\"operator\":\"Exists\"}]}}}}", 
            "to:", 
            "Resource: \"apps/v1beta2, Resource=daemonsets\", GroupVersionKind: \"apps/v1beta2, Kind=DaemonSet\"", 
            "Name: \"node-exporter\", Namespace: \"kubesphere-monitoring-system\"", 
            "Object: &{map[\"status\":map[\"numberReady\":'\\x06' \"observedGeneration\":'\\x04' \"updatedNumberScheduled\":'\\x06' \"numberAvailable\":'\\x06' \"currentNumberScheduled\":'\\x06' \"numberMisscheduled\":'\\x00' \"desiredNumberScheduled\":'\\x06'] \"kind\":\"DaemonSet\" \"apiVersion\":\"apps/v1beta2\" \"metadata\":map[\"selfLink\":\"/apis/apps/v1beta2/namespaces/kubesphere-monitoring-system/daemonsets/node-exporter\" \"uid\":\"b851174e-1bba-11ea-abb4-fa163ec5aba4\" \"resourceVersion\":\"45042072\" \"name\":\"node-exporter\" \"generation\":'\\x04' \"creationTimestamp\":\"2019-12-11T02:05:35Z\" \"labels\":map[\"app\":\"node-exporter\"] \"annotations\":map[\"deprecated.daemonset.template.generation\":\"4\" \"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1beta2\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"},\\\"name\\\":\\\"node-exporter\\\",\\\"namespace\\\":\\\"kubesphere-monitoring-system\\\"},\\\"spec\\\":{\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--web.listen-address=127.0.0.1:9100\\\",\\\"--path.procfs=/host/proc\\\",\\\"--path.sysfs=/host/sys\\\",\\\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\\\",\\\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\\\"],\\\"image\\\":\\\"kubesphere/node-exporter:ks-v0.16.0\\\",\\\"name\\\":\\\"node-exporter\\\",\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"250m\\\",\\\"memory\\\":\\\"180Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"180Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/host/proc\\\",\\\"name\\\":\\\"proc\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/sys\\\",\\\"name\\\":\\\"sys\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/root\\\",\\\"mountPropagation\\\":\\\"HostToContainer\\\",\\\"name\\\":\\\"root\\\",\\\"readOnly\\\":true}]},{\\\"args\\\":[\\\"--secure-listen-address=$(IP):9100\\\",\\\"--upstream=http://127.0.0.1:9100/\\\"],\\\"env\\\":[{\\\"name\\\":\\\"IP\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"status.podIP\\\"}}}],\\\"image\\\":\\\"kubesphere/kube-rbac-proxy:v0.4.1\\\",\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ports\\\":[{\\\"containerPort\\\":9100,\\\"hostPort\\\":9100,\\\"name\\\":\\\"https\\\"}],\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"20m\\\",\\\"memory\\\":\\\"40Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"20Mi\\\"}}}],\\\"hostNetwork\\\":true,\\\"hostPID\\\":true,\\\"imagePullSecrets\\\":[{\\\"name\\\":\\\"qingcloud\\\"}],\\\"nodeSelector\\\":{\\\"beta.kubernetes.io/os\\\":\\\"linux\\\"},\\\"securityContext\\\":{\\\"runAsNonRoot\\\":true,\\\"runAsUser\\\":65534},\\\"serviceAccountName\\\":\\\"node-exporter\\\",\\\"tolerations\\\":[{\\\"effect\\\":\\\"NoSchedule\\\",\\\"key\\\":\\\"node-role.kubernetes.io/master\\\"},{\\\"key\\\":\\\"dedicated\\\",\\\"operator\\\":\\\"Exists\\\"}],\\\"volumes\\\":[{\\\"hostPath\\\":{\\\"path\\\":\\\"/proc\\\"},\\\"name\\\":\\\"proc\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/sys\\\"},\\\"name\\\":\\\"sys\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/\\\"},\\\"name\\\":\\\"root\\\"}]}}}}\\n\"] \"namespace\":\"kubesphere-monitoring-system\"] \"spec\":map[\"selector\":map[\"matchLabels\":map[\"app\":\"node-exporter\"]] \"template\":map[\"metadata\":map[\"creationTimestamp\":<nil> \"labels\":map[\"app\":\"node-exporter\"]] \"spec\":map[\"terminationGracePeriodSeconds\":'\\x1e' \"serviceAccountName\":\"node-exporter\" \"hostPID\":%!q(bool=true) \"securityContext\":map[\"runAsUser\":'\\ufffe' \"runAsNonRoot\":%!q(bool=true)] \"tolerations\":[map[\"operator\":\"Exists\" \"effect\":\"NoSchedule\"]] \"containers\":[map[\"volumeMounts\":[map[\"name\":\"proc\" \"mountPath\":\"/host/proc\"] map[\"name\":\"sys\" \"mountPath\":\"/host/sys\"] map[\"readOnly\":%!q(bool=true) \"mountPath\":\"/host/root\" \"mountPropagation\":\"HostToContainer\" \"name\":\"root\"]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"imagePullPolicy\":\"IfNotPresent\" \"name\":\"node-exporter\" \"image\":\"kubesphere/node-exporter:ks-v0.16.0\" \"args\":[\"--web.listen-address=127.0.0.1:9101\" \"--path.procfs=/host/proc\" \"--path.sysfs=/host/sys\" \"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\" \"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"] \"resources\":map[\"limits\":map[\"cpu\":\"250m\" \"memory\":\"180Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"180Mi\"]]] map[\"terminationMessagePolicy\":\"File\" \"image\":\"kubesphere/kube-rbac-proxy:v0.4.1\" \"args\":[\"--secure-listen-address=$(IP):9101\" \"--upstream=http://127.0.0.1:9101/\"] \"ports\":[map[\"name\":\"https\" \"hostPort\":'\\u238d' \"containerPort\":'\\u238d' \"protocol\":\"TCP\"]] \"env\":[map[\"name\":\"IP\" \"valueFrom\":map[\"fieldRef\":map[\"apiVersion\":\"v1\" \"fieldPath\":\"status.podIP\"]]]] \"resources\":map[\"limits\":map[\"cpu\":\"20m\" \"memory\":\"40Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"20Mi\"]] \"name\":\"kube-rbac-proxy\" \"terminationMessagePath\":\"/dev/termination-log\" \"imagePullPolicy\":\"IfNotPresent\"]] \"restartPolicy\":\"Always\" \"serviceAccount\":\"node-exporter\" \"hostNetwork\":%!q(bool=true) \"imagePullSecrets\":[map[\"name\":\"qingcloud\"]] \"schedulerName\":\"default-scheduler\" \"volumes\":[map[\"name\":\"proc\" \"hostPath\":map[\"path\":\"/proc\" \"type\":\"\"]] map[\"name\":\"sys\" \"hostPath\":map[\"type\":\"\" \"path\":\"/sys\"]] map[\"name\":\"root\" \"hostPath\":map[\"path\":\"/\" \"type\":\"\"]]] \"dnsPolicy\":\"ClusterFirst\"]] \"updateStrategy\":map[\"type\":\"RollingUpdate\" \"rollingUpdate\":map[\"maxUnavailable\":'\\x01']] \"revisionHistoryLimit\":'\\n']]}", 
            "for: \"/etc/kubesphere/prometheus/sources/node-exporter-daemonset.yaml\": DaemonSet.apps \"node-exporter\" is invalid: spec.template.spec.containers[1].ports[1].name: Duplicate value: \"https\"", 
            "Error from server (InternalError): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-rules.yaml\": Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.247.69.235:443: connect: connection refused"
          ], 
          "stdout": "secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\ndeployment.apps/kube-state-metrics configured\nrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\nservice/kube-state-metrics unchanged\nserviceaccount/kube-state-metrics unchanged\nservicemonitor.monitoring.coreos.com/kube-state-metrics unchanged\nclusterrole.rbac.authorization.k8s.io/node-exporter unchanged\nclusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged\nservice/node-exporter unchanged\nserviceaccount/node-exporter unchanged\nservicemonitor.monitoring.coreos.com/node-exporter unchanged\nclusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nprometheus.monitoring.coreos.com/k8s unchanged\nprometheus.monitoring.coreos.com/k8s-system unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/kube-controller-manager-headless unchanged\nservice/kube-scheduler-headless unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged\nservicemonitor.monitoring.coreos.com/prometheus-system unchanged\nservice/prometheus-k8s-system unchanged", 
          "stdout_lines": [
            "secret/additional-scrape-configs unchanged", 
            "clusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "deployment.apps/kube-state-metrics configured", 
            "role.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "rolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "service/kube-state-metrics unchanged", 
            "serviceaccount/kube-state-metrics unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-state-metrics unchanged", 
            "clusterrole.rbac.authorization.k8s.io/node-exporter unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged", 
            "service/node-exporter unchanged", 
            "serviceaccount/node-exporter unchanged", 
            "servicemonitor.monitoring.coreos.com/node-exporter unchanged", 
            "clusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", 
            "prometheus.monitoring.coreos.com/k8s unchanged", 
            "prometheus.monitoring.coreos.com/k8s-system unchanged", 
            "rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", 
            "role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", 
            "service/prometheus-k8s unchanged", 
            "serviceaccount/prometheus-k8s unchanged", 
            "service/kube-controller-manager-headless unchanged", 
            "service/kube-scheduler-headless unchanged", 
            "servicemonitor.monitoring.coreos.com/prometheus unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-apiserver unchanged", 
            "servicemonitor.monitoring.coreos.com/coredns unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-scheduler unchanged", 
            "servicemonitor.monitoring.coreos.com/kubelet unchanged", 
            "servicemonitor.monitoring.coreos.com/prometheus-system unchanged", 
            "service/prometheus-k8s-system unchanged"
          ]
        }, 
        {
          "_ansible_item_label": "sources", 
          "_ansible_no_log": false, 
          "ansible_loop_var": "item", 
          "attempts": 5, 
          "changed": true, 
          "cmd": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", 
          "delta": "0:00:01.959622", 
          "end": "2019-12-11 03:24:53.335803", 
          "failed": true, 
          "failed_when_result": true, 
          "invocation": {
            "module_args": {
              "_raw_params": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", 
              "_uses_shell": true, 
              "argv": null, 
              "chdir": null, 
              "creates": null, 
              "executable": null, 
              "removes": null, 
              "stdin": null, 
              "stdin_add_newline": true, 
              "strip_empty_ends": true, 
              "warn": true
            }
          }, 
          "item": "sources", 
          "msg": "non-zero return code", 
          "rc": 1, 
          "start": "2019-12-11 03:24:51.376181", 
          "stderr": "Error from server (Invalid): error when applying patch:\n{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"node-exporter\"},{\"name\":\"kube-rbac-proxy\"}],\"containers\":[{\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/host/proc\"},{\"mountPath\":\"/host/sys\"},{\"mountPath\":\"/host/root\"}],\"args\":[\"--web.listen-address=127.0.0.1:9100\",\"--path.procfs=/host/proc\",\"--path.sysfs=/host/sys\",\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\",\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"],\"name\":\"node-exporter\",\"volumeMounts\":[{\"mountPath\":\"/host/proc\",\"readOnly\":false},{\"mountPath\":\"/host/sys\",\"readOnly\":false}]},{\"$setElementOrder/ports\":[{\"containerPort\":9100}],\"args\":[\"--secure-listen-address=$(IP):9100\",\"--upstream=http://127.0.0.1:9100/\"],\"name\":\"kube-rbac-proxy\",\"ports\":[{\"containerPort\":9100,\"hostPort\":9100,\"name\":\"https\"}]}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"node-role.kubernetes.io/master\"},{\"key\":\"dedicated\",\"operator\":\"Exists\"}]}}}}\nto:\nResource: \"apps/v1beta2, Resource=daemonsets\", GroupVersionKind: \"apps/v1beta2, Kind=DaemonSet\"\nName: \"node-exporter\", Namespace: \"kubesphere-monitoring-system\"\nObject: &{map[\"kind\":\"DaemonSet\" \"apiVersion\":\"apps/v1beta2\" \"metadata\":map[\"annotations\":map[\"deprecated.daemonset.template.generation\":\"4\" \"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1beta2\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"},\\\"name\\\":\\\"node-exporter\\\",\\\"namespace\\\":\\\"kubesphere-monitoring-system\\\"},\\\"spec\\\":{\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--web.listen-address=127.0.0.1:9100\\\",\\\"--path.procfs=/host/proc\\\",\\\"--path.sysfs=/host/sys\\\",\\\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\\\",\\\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\\\"],\\\"image\\\":\\\"kubesphere/node-exporter:ks-v0.16.0\\\",\\\"name\\\":\\\"node-exporter\\\",\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"250m\\\",\\\"memory\\\":\\\"180Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"180Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/host/proc\\\",\\\"name\\\":\\\"proc\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/sys\\\",\\\"name\\\":\\\"sys\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/root\\\",\\\"mountPropagation\\\":\\\"HostToContainer\\\",\\\"name\\\":\\\"root\\\",\\\"readOnly\\\":true}]},{\\\"args\\\":[\\\"--secure-listen-address=$(IP):9100\\\",\\\"--upstream=http://127.0.0.1:9100/\\\"],\\\"env\\\":[{\\\"name\\\":\\\"IP\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"status.podIP\\\"}}}],\\\"image\\\":\\\"kubesphere/kube-rbac-proxy:v0.4.1\\\",\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ports\\\":[{\\\"containerPort\\\":9100,\\\"hostPort\\\":9100,\\\"name\\\":\\\"https\\\"}],\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"20m\\\",\\\"memory\\\":\\\"40Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"20Mi\\\"}}}],\\\"hostNetwork\\\":true,\\\"hostPID\\\":true,\\\"imagePullSecrets\\\":[{\\\"name\\\":\\\"qingcloud\\\"}],\\\"nodeSelector\\\":{\\\"beta.kubernetes.io/os\\\":\\\"linux\\\"},\\\"securityContext\\\":{\\\"runAsNonRoot\\\":true,\\\"runAsUser\\\":65534},\\\"serviceAccountName\\\":\\\"node-exporter\\\",\\\"tolerations\\\":[{\\\"effect\\\":\\\"NoSchedule\\\",\\\"key\\\":\\\"node-role.kubernetes.io/master\\\"},{\\\"key\\\":\\\"dedicated\\\",\\\"operator\\\":\\\"Exists\\\"}],\\\"volumes\\\":[{\\\"hostPath\\\":{\\\"path\\\":\\\"/proc\\\"},\\\"name\\\":\\\"proc\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/sys\\\"},\\\"name\\\":\\\"sys\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/\\\"},\\\"name\\\":\\\"root\\\"}]}}}}\\n\"] \"name\":\"node-exporter\" \"namespace\":\"kubesphere-monitoring-system\" \"selfLink\":\"/apis/apps/v1beta2/namespaces/kubesphere-monitoring-system/daemonsets/node-exporter\" \"uid\":\"b851174e-1bba-11ea-abb4-fa163ec5aba4\" \"resourceVersion\":\"45042072\" \"labels\":map[\"app\":\"node-exporter\"] \"generation\":'\\x04' \"creationTimestamp\":\"2019-12-11T02:05:35Z\"] \"spec\":map[\"selector\":map[\"matchLabels\":map[\"app\":\"node-exporter\"]] \"template\":map[\"metadata\":map[\"creationTimestamp\":<nil> \"labels\":map[\"app\":\"node-exporter\"]] \"spec\":map[\"containers\":[map[\"args\":[\"--web.listen-address=127.0.0.1:9101\" \"--path.procfs=/host/proc\" \"--path.sysfs=/host/sys\" \"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\" \"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"] \"resources\":map[\"limits\":map[\"cpu\":\"250m\" \"memory\":\"180Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"180Mi\"]] \"volumeMounts\":[map[\"name\":\"proc\" \"mountPath\":\"/host/proc\"] map[\"name\":\"sys\" \"mountPath\":\"/host/sys\"] map[\"mountPath\":\"/host/root\" \"mountPropagation\":\"HostToContainer\" \"name\":\"root\" \"readOnly\":%!q(bool=true)]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"imagePullPolicy\":\"IfNotPresent\" \"name\":\"node-exporter\" \"image\":\"kubesphere/node-exporter:ks-v0.16.0\"] map[\"name\":\"kube-rbac-proxy\" \"resources\":map[\"limits\":map[\"cpu\":\"20m\" \"memory\":\"40Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"20Mi\"]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"image\":\"kubesphere/kube-rbac-proxy:v0.4.1\" \"args\":[\"--secure-listen-address=$(IP):9101\" \"--upstream=http://127.0.0.1:9101/\"] \"ports\":[map[\"name\":\"https\" \"hostPort\":'\\u238d' \"containerPort\":'\\u238d' \"protocol\":\"TCP\"]] \"env\":[map[\"name\":\"IP\" \"valueFrom\":map[\"fieldRef\":map[\"fieldPath\":\"status.podIP\" \"apiVersion\":\"v1\"]]]] \"imagePullPolicy\":\"IfNotPresent\"]] \"terminationGracePeriodSeconds\":'\\x1e' \"serviceAccountName\":\"node-exporter\" \"serviceAccount\":\"node-exporter\" \"hostNetwork\":%!q(bool=true) \"securityContext\":map[\"runAsUser\":'\\ufffe' \"runAsNonRoot\":%!q(bool=true)] \"volumes\":[map[\"name\":\"proc\" \"hostPath\":map[\"path\":\"/proc\" \"type\":\"\"]] map[\"name\":\"sys\" \"hostPath\":map[\"type\":\"\" \"path\":\"/sys\"]] map[\"name\":\"root\" \"hostPath\":map[\"path\":\"/\" \"type\":\"\"]]] \"dnsPolicy\":\"ClusterFirst\" \"hostPID\":%!q(bool=true) \"imagePullSecrets\":[map[\"name\":\"qingcloud\"]] \"schedulerName\":\"default-scheduler\" \"tolerations\":[map[\"operator\":\"Exists\" \"effect\":\"NoSchedule\"]] \"restartPolicy\":\"Always\"]] \"updateStrategy\":map[\"type\":\"RollingUpdate\" \"rollingUpdate\":map[\"maxUnavailable\":'\\x01']] \"revisionHistoryLimit\":'\\n'] \"status\":map[\"desiredNumberScheduled\":'\\x06' \"numberReady\":'\\x06' \"observedGeneration\":'\\x04' \"updatedNumberScheduled\":'\\x06' \"numberAvailable\":'\\x06' \"currentNumberScheduled\":'\\x06' \"numberMisscheduled\":'\\x00']]}\nfor: \"/etc/kubesphere/prometheus/sources/node-exporter-daemonset.yaml\": DaemonSet.apps \"node-exporter\" is invalid: spec.template.spec.containers[1].ports[1].name: Duplicate value: \"https\"\nError from server (InternalError): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-rules.yaml\": Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.247.69.235:443: connect: connection refused", 
          "stderr_lines": [
            "Error from server (Invalid): error when applying patch:", 
            "{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"node-exporter\"},{\"name\":\"kube-rbac-proxy\"}],\"containers\":[{\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/host/proc\"},{\"mountPath\":\"/host/sys\"},{\"mountPath\":\"/host/root\"}],\"args\":[\"--web.listen-address=127.0.0.1:9100\",\"--path.procfs=/host/proc\",\"--path.sysfs=/host/sys\",\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\",\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"],\"name\":\"node-exporter\",\"volumeMounts\":[{\"mountPath\":\"/host/proc\",\"readOnly\":false},{\"mountPath\":\"/host/sys\",\"readOnly\":false}]},{\"$setElementOrder/ports\":[{\"containerPort\":9100}],\"args\":[\"--secure-listen-address=$(IP):9100\",\"--upstream=http://127.0.0.1:9100/\"],\"name\":\"kube-rbac-proxy\",\"ports\":[{\"containerPort\":9100,\"hostPort\":9100,\"name\":\"https\"}]}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"node-role.kubernetes.io/master\"},{\"key\":\"dedicated\",\"operator\":\"Exists\"}]}}}}", 
            "to:", 
            "Resource: \"apps/v1beta2, Resource=daemonsets\", GroupVersionKind: \"apps/v1beta2, Kind=DaemonSet\"", 
            "Name: \"node-exporter\", Namespace: \"kubesphere-monitoring-system\"", 
            "Object: &{map[\"kind\":\"DaemonSet\" \"apiVersion\":\"apps/v1beta2\" \"metadata\":map[\"annotations\":map[\"deprecated.daemonset.template.generation\":\"4\" \"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apps/v1beta2\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"},\\\"name\\\":\\\"node-exporter\\\",\\\"namespace\\\":\\\"kubesphere-monitoring-system\\\"},\\\"spec\\\":{\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"labels\\\":{\\\"app\\\":\\\"node-exporter\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"--web.listen-address=127.0.0.1:9100\\\",\\\"--path.procfs=/host/proc\\\",\\\"--path.sysfs=/host/sys\\\",\\\"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\\\",\\\"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\\\"],\\\"image\\\":\\\"kubesphere/node-exporter:ks-v0.16.0\\\",\\\"name\\\":\\\"node-exporter\\\",\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"250m\\\",\\\"memory\\\":\\\"180Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"180Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/host/proc\\\",\\\"name\\\":\\\"proc\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/sys\\\",\\\"name\\\":\\\"sys\\\",\\\"readOnly\\\":false},{\\\"mountPath\\\":\\\"/host/root\\\",\\\"mountPropagation\\\":\\\"HostToContainer\\\",\\\"name\\\":\\\"root\\\",\\\"readOnly\\\":true}]},{\\\"args\\\":[\\\"--secure-listen-address=$(IP):9100\\\",\\\"--upstream=http://127.0.0.1:9100/\\\"],\\\"env\\\":[{\\\"name\\\":\\\"IP\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"status.podIP\\\"}}}],\\\"image\\\":\\\"kubesphere/kube-rbac-proxy:v0.4.1\\\",\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ports\\\":[{\\\"containerPort\\\":9100,\\\"hostPort\\\":9100,\\\"name\\\":\\\"https\\\"}],\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"20m\\\",\\\"memory\\\":\\\"40Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"20Mi\\\"}}}],\\\"hostNetwork\\\":true,\\\"hostPID\\\":true,\\\"imagePullSecrets\\\":[{\\\"name\\\":\\\"qingcloud\\\"}],\\\"nodeSelector\\\":{\\\"beta.kubernetes.io/os\\\":\\\"linux\\\"},\\\"securityContext\\\":{\\\"runAsNonRoot\\\":true,\\\"runAsUser\\\":65534},\\\"serviceAccountName\\\":\\\"node-exporter\\\",\\\"tolerations\\\":[{\\\"effect\\\":\\\"NoSchedule\\\",\\\"key\\\":\\\"node-role.kubernetes.io/master\\\"},{\\\"key\\\":\\\"dedicated\\\",\\\"operator\\\":\\\"Exists\\\"}],\\\"volumes\\\":[{\\\"hostPath\\\":{\\\"path\\\":\\\"/proc\\\"},\\\"name\\\":\\\"proc\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/sys\\\"},\\\"name\\\":\\\"sys\\\"},{\\\"hostPath\\\":{\\\"path\\\":\\\"/\\\"},\\\"name\\\":\\\"root\\\"}]}}}}\\n\"] \"name\":\"node-exporter\" \"namespace\":\"kubesphere-monitoring-system\" \"selfLink\":\"/apis/apps/v1beta2/namespaces/kubesphere-monitoring-system/daemonsets/node-exporter\" \"uid\":\"b851174e-1bba-11ea-abb4-fa163ec5aba4\" \"resourceVersion\":\"45042072\" \"labels\":map[\"app\":\"node-exporter\"] \"generation\":'\\x04' \"creationTimestamp\":\"2019-12-11T02:05:35Z\"] \"spec\":map[\"selector\":map[\"matchLabels\":map[\"app\":\"node-exporter\"]] \"template\":map[\"metadata\":map[\"creationTimestamp\":<nil> \"labels\":map[\"app\":\"node-exporter\"]] \"spec\":map[\"containers\":[map[\"args\":[\"--web.listen-address=127.0.0.1:9101\" \"--path.procfs=/host/proc\" \"--path.sysfs=/host/sys\" \"--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)\" \"--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$\"] \"resources\":map[\"limits\":map[\"cpu\":\"250m\" \"memory\":\"180Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"180Mi\"]] \"volumeMounts\":[map[\"name\":\"proc\" \"mountPath\":\"/host/proc\"] map[\"name\":\"sys\" \"mountPath\":\"/host/sys\"] map[\"mountPath\":\"/host/root\" \"mountPropagation\":\"HostToContainer\" \"name\":\"root\" \"readOnly\":%!q(bool=true)]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"imagePullPolicy\":\"IfNotPresent\" \"name\":\"node-exporter\" \"image\":\"kubesphere/node-exporter:ks-v0.16.0\"] map[\"name\":\"kube-rbac-proxy\" \"resources\":map[\"limits\":map[\"cpu\":\"20m\" \"memory\":\"40Mi\"] \"requests\":map[\"cpu\":\"10m\" \"memory\":\"20Mi\"]] \"terminationMessagePath\":\"/dev/termination-log\" \"terminationMessagePolicy\":\"File\" \"image\":\"kubesphere/kube-rbac-proxy:v0.4.1\" \"args\":[\"--secure-listen-address=$(IP):9101\" \"--upstream=http://127.0.0.1:9101/\"] \"ports\":[map[\"name\":\"https\" \"hostPort\":'\\u238d' \"containerPort\":'\\u238d' \"protocol\":\"TCP\"]] \"env\":[map[\"name\":\"IP\" \"valueFrom\":map[\"fieldRef\":map[\"fieldPath\":\"status.podIP\" \"apiVersion\":\"v1\"]]]] \"imagePullPolicy\":\"IfNotPresent\"]] \"terminationGracePeriodSeconds\":'\\x1e' \"serviceAccountName\":\"node-exporter\" \"serviceAccount\":\"node-exporter\" \"hostNetwork\":%!q(bool=true) \"securityContext\":map[\"runAsUser\":'\\ufffe' \"runAsNonRoot\":%!q(bool=true)] \"volumes\":[map[\"name\":\"proc\" \"hostPath\":map[\"path\":\"/proc\" \"type\":\"\"]] map[\"name\":\"sys\" \"hostPath\":map[\"type\":\"\" \"path\":\"/sys\"]] map[\"name\":\"root\" \"hostPath\":map[\"path\":\"/\" \"type\":\"\"]]] \"dnsPolicy\":\"ClusterFirst\" \"hostPID\":%!q(bool=true) \"imagePullSecrets\":[map[\"name\":\"qingcloud\"]] \"schedulerName\":\"default-scheduler\" \"tolerations\":[map[\"operator\":\"Exists\" \"effect\":\"NoSchedule\"]] \"restartPolicy\":\"Always\"]] \"updateStrategy\":map[\"type\":\"RollingUpdate\" \"rollingUpdate\":map[\"maxUnavailable\":'\\x01']] \"revisionHistoryLimit\":'\\n'] \"status\":map[\"desiredNumberScheduled\":'\\x06' \"numberReady\":'\\x06' \"observedGeneration\":'\\x04' \"updatedNumberScheduled\":'\\x06' \"numberAvailable\":'\\x06' \"currentNumberScheduled\":'\\x06' \"numberMisscheduled\":'\\x00']]}", 
            "for: \"/etc/kubesphere/prometheus/sources/node-exporter-daemonset.yaml\": DaemonSet.apps \"node-exporter\" is invalid: spec.template.spec.containers[1].ports[1].name: Duplicate value: \"https\"", 
            "Error from server (InternalError): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-rules.yaml\": Internal error occurred: failed calling webhook \"prometheusrulemutate.monitoring.coreos.com\": Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.247.69.235:443: connect: connection refused"
          ], 
          "stdout": "secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\ndeployment.apps/kube-state-metrics unchanged\nrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\nservice/kube-state-metrics unchanged\nserviceaccount/kube-state-metrics unchanged\nservicemonitor.monitoring.coreos.com/kube-state-metrics unchanged\nclusterrole.rbac.authorization.k8s.io/node-exporter unchanged\nclusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged\nservice/node-exporter unchanged\nserviceaccount/node-exporter unchanged\nservicemonitor.monitoring.coreos.com/node-exporter unchanged\nclusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nprometheus.monitoring.coreos.com/k8s unchanged\nprometheus.monitoring.coreos.com/k8s-system unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/kube-controller-manager-headless unchanged\nservice/kube-scheduler-headless unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged\nservicemonitor.monitoring.coreos.com/prometheus-system unchanged\nservice/prometheus-k8s-system unchanged", 
          "stdout_lines": [
            "secret/additional-scrape-configs unchanged", 
            "clusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "deployment.apps/kube-state-metrics unchanged", 
            "role.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "rolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", 
            "service/kube-state-metrics unchanged", 
            "serviceaccount/kube-state-metrics unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-state-metrics unchanged", 
            "clusterrole.rbac.authorization.k8s.io/node-exporter unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged", 
            "service/node-exporter unchanged", 
            "serviceaccount/node-exporter unchanged", 
            "servicemonitor.monitoring.coreos.com/node-exporter unchanged", 
            "clusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged", 
            "clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", 
            "prometheus.monitoring.coreos.com/k8s unchanged", 
            "prometheus.monitoring.coreos.com/k8s-system unchanged", 
            "rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", 
            "role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", 
            "service/prometheus-k8s unchanged", 
            "serviceaccount/prometheus-k8s unchanged", 
            "service/kube-controller-manager-headless unchanged", 
            "service/kube-scheduler-headless unchanged", 
            "servicemonitor.monitoring.coreos.com/prometheus unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-apiserver unchanged", 
            "servicemonitor.monitoring.coreos.com/coredns unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged", 
            "servicemonitor.monitoring.coreos.com/kube-scheduler unchanged", 
            "servicemonitor.monitoring.coreos.com/kubelet unchanged", 
            "servicemonitor.monitoring.coreos.com/prometheus-system unchanged", 
            "service/prometheus-k8s-system unchanged"
          ]
        }
      ]
    }, 
    "role": "ks-monitor", 
    "task": "ks-monitor | Installing prometheus-operator", 
    "task_action": "shell", 
    "task_args": "", 
    "task_path": "/kubesphere/installer/roles/ks-monitor/tasks/main.yaml:65", 
    "task_uuid": "0255ac10-0063-ec0c-88fd-00000000001d", 
    "uuid": "bc4698fb-e727-49f8-9a70-43b28c035965"
  }, 
  "parent_uuid": "0255ac10-0063-ec0c-88fd-00000000001d", 
  "pid": 891, 
  "runner_ident": "monitoring", 
  "start_line": 60, 
  "stdout": "", 
  "uuid": "bc4698fb-e727-49f8-9a70-43b28c035965"
}

brosoul · 2019年12月11日

组件状态正常，但是很多监控信息显示不全

Feynman · 2019年12月11日

@huanggze

huanggze · 2019年12月11日

brosoul 现在问题是什么状态？我没遇见过类似问题，监控数据还没显示的话，可以发我们 tv 远程看看？

brosoul · 2019年12月12日

huanggze 我尝试删除了kubesphere-monitoring和monitoring（原来部署的监控）这两个ns，然后重新部署监控，但是还是

Post https://prometheus-prometheus-oper-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s

现在是 not found。

我不是很清楚这个 webhook prometheusrulemutate.monitoring.coreos.com 属于k8s的哪个资源类型

tv这些远程协助可能不是很方便，请谅解。

brosoul · 2019年12月12日

huanggze 另外我在ks-installer中的源码中没有找到 podmonitors.monitoring.coreos.com 这个crd，是否有影响？

huanggze · 2019年12月12日

brosoul @Cauchy Need Help !

brosoul · 2019年12月13日

huanggze 补充一下，console界面中的监控请求没有报错，只是返回结果是空的。

Cauchy · 2019年12月13日

是不是跟之前prometheus的svc有冲突

brosoul · 2019年12月13日

Cauchy 我已经把原来部署监控的ns都删除了，但还是存在冲突。但是podmonitors.monitoring.coreos.com 这个CRD 我没有删除。正在尝试是不是这个 CRD 存在的影响。

brosoul · 2019年12月13日

Cauchy 请问kubertest组件中 kube-scheduler-headless 和 kube-controller-manager-headless 已停止，是否对监控有影响，以及为什么状态时已停止，我在集群都没有找到对应pod直这是正常现象吗？

brosoul · 2019年12月13日

Cauchy ks-installer中的报错解决了，确实是之前一个crd podmonitors.monitoring.coreos.com (原先Prom 创建的)没有删的原因。感谢大家的帮助。

brosoul · 2019年12月26日

Cauchy 监控现版本是否支持自定义保留的天数呢？还是后续会支持？
我发了一个问题可以在这里回答一下

LuLiMingAmos · 2020年10月15日

请问监控可以部署在ns:monitoring中吗，集群已有监控，如果部署了自带的kubesphere-monitoring-system,还需要迁移原有的东西，我部署在monitoring中，停用了kubesphere-monitoring-system,但是host集群控制面板就显示不了成员集群的资源数据了，请问有办法解决吗

安装monitoring ks-installer中报错

brosoulK零S

brosoulK零S

brosoulK零S

FeynmanK零SK贰SK壹S

huanggzeK零SK壹S

brosoulK零S

brosoulK零S

huanggzeK零SK壹S

brosoulK零S

CauchyK零SK壹S

brosoulK零S

brosoulK零S

brosoulK零S

brosoulK零S

LuLiMingAmosK零S