Kubesphere v3.4.0使用外部prometheus，启用多集群管理功能失败

rzhao1001

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息
物理机，Centos7.5 48c 192g

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.16", GitCommit:"b28e1f370a4a4c428ddbeababcaf0198f048fcac", GitTreeState:"clean", BuildDate:"2022-11-09T13:37:37Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.16", GitCommit:"b28e1f370a4a4c428ddbeababcaf0198f048fcac", GitTreeState:"clean", BuildDate:"2022-11-09T13:31:51Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.41 (downgraded from 1.43)
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:35:25 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:44:05 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

KubeSphere版本信息
例如：v2.1.1/v3.0.0。离线安装还是在线安装。在已有K8s上安装还是使用kk安装。

v3.4 在线安装，在已有K8s上安装

问题是什么
报错日志是什么，最好有截图。

使用外部prometheus启用多集群功能时，controller-manager无法启动

PLAY RECAP *********************************************************************
localhost                  : ok=30   changed=17   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0   
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
**************************************************
Waiting for all tasks to be completed ...
task openpitrix status is successful  (1/4)
task network status is successful  (2/4)
task monitoring status is successful  (3/4)


task multicluster status is failed  (4/4)
**************************************************
Collecting installation results ...


Task 'multicluster' failed:
******************************************************************************************************************************************************
{
  "counter": 65,
  "created": "2023-10-09T04:15:14.794445",
  "end_line": 67,
  "event": "runner_on_failed",
  "event_data": {
    "duration": 610.730374,
    "end": "2023-10-09T04:15:14.793746",
    "event_loop": null,
    "host": "localhost",
    "ignore_errors": null,
    "play": "localhost",
    "play_pattern": "localhost",
    "play_uuid": "8607507a-a2e3-08f1-1c75-000000000005",
    "playbook": "/kubesphere/playbooks/multicluster.yaml",
    "playbook_uuid": "85ee6988-8e4c-4ba7-bc9c-f4897219ea7f",
    "remote_addr": "127.0.0.1",
    "res": {
      "_ansible_no_log": false,
      "attempts": 10,
      "changed": true,
      "cmd": "/usr/local/bin/helm upgrade --install kubefed /kubesphere/kubesphere/kubefed/kubefed -f /kubesphere/kubesphere/kubefed/custom-values-kubefed.yaml --namespace kube-federation-system --wait --timeout 1800s\n",
      "delta": "0:00:00.218527",
      "end": "2023-10-09 12:15:14.683426",
      "invocation": {
        "module_args": {
          "_raw_params": "/usr/local/bin/helm upgrade --install kubefed /kubesphere/kubesphere/kubefed/kubefed -f /kubesphere/kubesphere/kubefed/custom-values-kubefed.yaml --namespace kube-federation-system --wait --timeout 1800s\n",
          "_uses_shell": true,
          "argv": null,
          "chdir": null,
          "creates": null,
          "executable": null,
          "removes": null,
          "stdin": null,
          "stdin_add_newline": true,
          "strip_empty_ends": true,
          "warn": true
        }
      },
      "msg": "non-zero return code",
      "rc": 1,
      "start": "2023-10-09 12:15:14.464899",
      "stderr": "Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress",
      "stderr_lines": [
        "Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress"
      ],
      "stdout": "",
      "stdout_lines": []
    },
    "resolved_action": "command",
    "role": "ks-multicluster",
    "start": "2023-10-09T04:05:04.063372",
    "task": "Kubefed | Initing kube-federation-system",
    "task_action": "command",
    "task_args": "",
    "task_path": "/kubesphere/installer/roles/ks-multicluster/tasks/main.yml:51",
    "task_uuid": "8607507a-a2e3-08f1-1c75-00000000001f",
    "uuid": "2fa0ffc7-7818-454f-812d-6ef727d6a7db"
  },
  "parent_uuid": "8607507a-a2e3-08f1-1c75-00000000001f",
  "pid": 17201,
  "runner_ident": "multicluster",
  "start_line": 66,
  "stdout": "fatal: [localhost]: FAILED! => {\"attempts\": 10, \"changed\": true, \"cmd\": \"/usr/local/bin/helm upgrade --install kubefed /kubesphere/kubesphere/kubefed/kubefed -f /kubesphere/kubesphere/kubefed/custom-values-kubefed.yaml --namespace kube-federation-system --wait --timeout 1800s\\n\", \"delta\": \"0:00:00.218527\", \"end\": \"2023-10-09 12:15:14.683426\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2023-10-09 12:15:14.464899\", \"stderr\": \"Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\", \"stderr_lines\": [\"Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\"], \"stdout\": \"\", \"stdout_lines\": []}",
  "uuid": "2fa0ffc7-7818-454f-812d-6ef727d6a7db"
}

W1009 12:37:11.672363       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1009 12:37:11.675361       1 server.go:197] setting up manager
I1009 12:37:11.773161       1 listener.go:44] "controller-runtime/metrics: Metrics server is starting to listen" addr=":8080"
W1009 12:37:11.858115       1 controllers.go:162] ks-controller-manager starts without ldap provided, it will not sync user into ldap
I1009 12:37:11.870354       1 reconciler.go:149] "controllers/Helm: Watching resource" group="gateway.kubesphere.io" version="v1alpha1" kind="Nginx"
I1009 12:37:11.870397       1 helm_controller.go:76] configured watch gvk gateway.kubesphere.io/v1alpha1, Kind=Nginx chartPath /var/helm-charts/ingress-nginx maxConcurrentReconciles 64 reconcilePeriod 1m0s
I1009 12:37:11.870509       1 reconciler.go:149] "controllers/Helm: Watching resource" group="gateway.kubesphere.io" version="v1alpha1" kind="Gateway"
I1009 12:37:11.870520       1 helm_controller.go:76] configured watch gvk gateway.kubesphere.io/v1alpha1, Kind=Gateway chartPath /var/helm-charts/gateway maxConcurrentReconciles 64 reconcilePeriod 1m0s
I1009 12:37:11.871852       1 loginrecord_controller.go:94] Setting up event handlers
I1009 12:37:11.872144       1 certificatesigningrequest_controller.go:93] Setting up event handlers
I1009 12:37:11.872406       1 clusterrolebinding_controller.go:98] Setting up event handlers
I1009 12:37:11.872746       1 globalrole_controller.go:99] Setting up event handlers
I1009 12:37:11.873056       1 globalrolebinding_controller.go:107] Setting up event handlers
I1009 12:37:11.873284       1 groupbinding_controller.go:95] Setting up event handlers
I1009 12:37:11.873540       1 group_controller.go:103] Setting up event handlers
I1009 12:37:11.873935       1 notification_controller.go:88] Setting up event handlers
W1009 12:37:11.876500       1 reflector.go:424] /workspace/cmd/controller-manager/app/controllers.go:440: failed to list types.kubefed.io/v1beta1, Kind=FederatedGlobalRoleBinding: the server could not find the requested resource
E1009 12:37:11.876561       1 reflector.go:140] /workspace/cmd/controller-manager/app/controllers.go:440: Failed to watch types.kubefed.io/v1beta1, Kind=FederatedGlobalRoleBinding: failed to list types.kubefed.io/v1beta1, Kind=FederatedGlobalRoleBinding: the server could not find the requested resource
W1009 12:37:11.876591       1 reflector.go:424] /workspace/cmd/controller-manager/app/controllers.go:415: failed to list types.kubefed.io/v1beta1, Kind=FederatedGlobalRole: the server could not find the requested resource
E1009 12:37:11.876643       1 reflector.go:140] /workspace/cmd/controller-manager/app/controllers.go:415: Failed to watch types.kubefed.io/v1beta1, Kind=FederatedGlobalRole: failed to list types.kubefed.io/v1beta1, Kind=FederatedGlobalRole: the server could not find the requested resource
E1009 12:37:12.064878       1 notification_controller.go:116] get /, Kind= informer error, no matches for kind "NotificationManager" in version "notification.kubesphere.io/v2beta2"
F1009 12:37:12.064986       1 controllers.go:523] Unable to create Notification controller: no matches for kind "NotificationManager" in version "notification.kubesphere.io/v2beta2"

inksnw

删除掉安装容器, 重新触发一下安装试下呢
kubectl delete pod ks-installer-xxx -n kubesphere-system
或者手动进入容器中执行上面失败的命令

kubectl exec -it ks-installer-xxx -n kubesphere-system – /bin/bash

/usr/local/bin/helm upgrade –install kubefed /kubesphere/kubesphere/kubefed/kubefed -f /kubesphere/kubesphere/kubefed/custom-values-kubefed.yaml –namespace kube-federation-system

rzhao1001

inksnw

rzhao1001

重新触发安装，也无法启用，执行下面的命令，报错

rzhao1001

手动执行

Error: UPGRADE FAILED: cannot patch "clusterroles.rbac.authorization.k8s.io" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "configmaps" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "deployments.apps" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "ingresses.networking.k8s.io" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "jobs.batch" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "namespaces" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "replicasets.apps" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "secrets" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "serviceaccounts" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "services" with kind FederatedTypeConfig: Internal error occurred: failed calling webhook "federatedtypeconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-federatedtypeconfigs?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca") && cannot patch "kubefed" with kind KubeFedConfig: Internal error occurred: failed calling webhook "kubefedconfigs.core.kubefed.io": Post "https://kubefed-admission-webhook.kube-federation-system.svc:443/validate-kubefedconfig?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefed-admission-webhook-ca")

iawia002

rzhao1001
你这个是没卸载干净，执行下面的命令卸载了才能重装

kubectl -n kube-federation-system delete FederatedTypeConfig --all
helm --namespace kube-federation-system uninstall kubefed

rzhao1001

我使用官方提供的*[kubesphere-delete.sh](https://github.com/kubesphere/ks-installer/blob/release-3.1/scripts/kubesphere-delete.sh)*卸载完后，只要使用外接prometheus，多集群功能就无法使用，提示需要E1009 12:37:12.064878 1 notification_controller.go:116] get /, Kind= informer error, no matches for kind “NotificationManager” in version “notification.kubesphere.io/v2beta2” F1009 12:37:12.064986 1 controllers.go:523] Unable to create Notification controller: no matches for kind “NotificationManager” in version “notification.kubesphere.io/v2beta2”

我看playbook里，非外接的prometheus不会生成NotificationManager这个资源？