v3.2.1 kubeconfig 文件错误，user字段为null

wmwm061061

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8", GitCommit:"fd5d41537aee486160ad9b5356a9d82363273721", GitTreeState:"clean", BuildDate:"2021-02-17T12:41:51Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8", GitCommit:"fd5d41537aee486160ad9b5356a9d82363273721", GitTreeState:"clean", BuildDate:"2021-02-17T12:33:08Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

KubeSphere版本信息
例如：v3.2.1。在线安装。

用户角色：

platform-admin

问题

在 devops 中创建的 kubeconfig 凭证无法执行流水线任务中的 helm 命令，提示以下内容

helm3 upgrade test-project -n dev --install -f values-dev.yaml ./
Error: query: failed to query with labels: secrets is forbidden: User "system:anonymous" cannot list resource "secrets" in API group "" in the namespace "dev"
script returned exit code 1

经过查看凭证里面 kubeconfig 详细内容，发现内容不完整，"users"部分的值为 null

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1ERXlOakV4TVRRME5sb1hEVE14TURFeU5ERXhNVFEwTmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTWxICk*****************************1JyclQzWkpJK0VYZENrODk0MEsxRHZsekNFaU51cGVld1hiS2I1TmgKZnNTNVVGUVZFRlI1enRLc25WNXhVN*******d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEQjlFVWV0M1NuL1F4Y2FqOGY1a251UU04ZSsKekVRTkZrWFVrZ0FQajlSTnlaY2JRQk4wRm9uZHllbllYMGNDVXVPaXFJTkVYSnJId3h1TEE5Y0hPTk1UWC9iKwoyUWY1UTRub1JhQ245VkVVVUZjTUVBeDJJRm4vblVPWm9OR0N1TE9qSTJzWTd4******************************YY1VibFVRSzYzajM3amFDb3pDQzIyZGU0OFBNcDR4Z3ZBVGpSRTlLK3g2QTFiZmthTHRjUlZsOGEKTVJCbSs4NHhhUXdJMUc1RTZ0LzRrWHlhNXU3aUhOSEpVd2dERnNHSFpzdVZSdFlWVE5IbHA1YnA5cWxUVlYzVgpqdU8yWURHV0FYT1BJOWNQMlFOUDRDcnBTVzhpczB1aGQ1MXRzU1R6UmtwL3ZZRUo5eEFtaVM5RDM5az0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://10.233.0.1:443
  name: local
contexts:
- context:
    cluster: local
    namespace: default
    user: test111
  name: test111@local
current-context: test111@local
kind: Config
preferences: {}
users: null

集群工具–>kubeconfig 的内容也是一样的

我昨天把 kubesphere 从 v3.1.0 升级到 v3.2.1，kubenetes 从 v1.18.8 升级到 v1.19.8，升级之前是可以获取到完整的 kubeconfig 文件的。 升级之前的用户的 kubeconfig 文件现在查看也是完整的。上面示例的 test111 用户是今天新创建的。

这个问题不知道是不是升级导致的。

另外现在如果想获取到完整的 kubeconfig 文件还有其他方法吗？

DehaoCheng

可以参考这个

https://kubesphere.io/zh/docs/multicluster-management/enable-multicluster/retrieve-kubeconfig/

wmwm061061

DehaoCheng

https://kubesphere.io/zh/docs/multicluster-management/enable-multicluster/retrieve-kubeconfig/

这个链接获取的只是 kubesphere-admin 用户的 kubeconfig 文件，我记得之前的版本会给每个kubesphere用户都生成一个完整的 kubeconfig 文件的。（如下图）

但是现在新创建用户的 kubeconfig 文件是不完整的，"users"部分的值是null。是因为v3.2.1版本对权限相关的内容有更改吗？

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1ERXlOakV4TVRRME5sb1hEVE14TURFeU5ERXhNVFEwTmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTWxICk*****************************1JyclQzWkpJK0VYZENrODk0MEsxRHZsekNFaU51cGVld1hiS2I1TmgKZnNTNVVGUVZFRlI1enRLc25WNXhVN*******d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEQjlFVWV0M1NuL1F4Y2FqOGY1a251UU04ZSsKekVRTkZrWFVrZ0FQajlSTnlaY2JRQk4wRm9uZHllbllYMGNDVXVPaXFJTkVYSnJId3h1TEE5Y0hPTk1UWC9iKwoyUWY1UTRub1JhQ245VkVVVUZjTUVBeDJJRm4vblVPWm9OR0N1TE9qSTJzWTd4******************************YY1VibFVRSzYzajM3amFDb3pDQzIyZGU0OFBNcDR4Z3ZBVGpSRTlLK3g2QTFiZmthTHRjUlZsOGEKTVJCbSs4NHhhUXdJMUc1RTZ0LzRrWHlhNXU3aUhOSEpVd2dERnNHSFpzdVZSdFlWVE5IbHA1YnA5cWxUVlYzVgpqdU8yWURHV0FYT1BJOWNQMlFOUDRDcnBTVzhpczB1aGQ1MXRzU1R6UmtwL3ZZRUo5eEFtaVM5RDM5az0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://10.233.0.1:443
  name: local
contexts:
- context:
    cluster: local
    namespace: default
    user: test111
  name: test111@local
current-context: test111@local
kind: Config
preferences: {}
users: null

johnniang

我这边没有复现。请看下面的截图：

wmwm061061

johnniang

我怀疑可能是我的集群在升级过程中出了什么问题。

请问这个用户 kubeconfig 文件的生成原理是什么？我排查一下是否是某个系统组件出了问题。

johnniang

使用 admin 账号登录集群，然后在命名空间 kubesphere-controls-system 下查找 ConfigMap：kubeconfig-，检查内容是否正确。

johnniang

如果不正确，尝试删除对应的 ConfigMap，然后使用对应账号登录后触发一下用户更新，如切换系统语言（记得切回来）。再次查看对应的 ConfigMap 是否正确。

wmwm061061

johnniang

按照您说的操作了一遍，host集群的用户 kubeconfig 内容还是不完整。

目前我这个 kubesphere 是多集群模式，除了 host 集群之外，还关联了一个 GKE 的集群。我看了 GKE 集群 test111 用户的 kubeconfig 文件是没有问题的。见下图

host集群版本信息：

kubesphere: v3.2.1

kubernetes: v1.19.8

gke 集群版本信息：

kubesphere: v3.1.1

kubernetes: v1.19.14-gke.1900

johnniang

建议查看一下：Deployment ks-controller-manager 的日志。

wmwm061061

johnniang

看了下 Deployment 的日志，没有发现异常

ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.227497       1 basecontroller.go:127] Successfully Synced key:test111-b4f2q in loginrecord-controller

ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.227669       1 event.go:282] Event(v1.ObjectReference{Kind:"LoginRecord", Namespace:"", Name:"test111-b4f2q", UID:"70131e6d-c7a7-4e78-8620-3ab554a6d475", APIVersion:"iam.kubesphere.io/v1alpha2", ResourceVersion:"156757909", FieldPath:""}): type: 'Normal' reason: 'Synced' LoginRecord synced successfully 

ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.702777       1 certificatesigningrequest_controller.go:185] Successfully csrSynced key:test111-csr-1640246865


ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.703003       1 event.go:282] Event(v1.ObjectReference{Kind:"CertificateSigningRequest", Namespace:"", Name:"test111-csr-1640246865"  ,UID:"1c353de9-3a40-4617-9238-d76d1c06ea97", APIVersion:"certificates.k8s.io/v1", ResourceVersion:"156757922", FieldPath:""}): type: 'Normal' reason: 'Synced' CertificateSigningRequest csrSynced successfully


ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.706963       1 certificatesigningrequest_controller.go:185] Successfully csrSynced key:test111-csr-1640246865

ks-controller-manager-95bd45f84-7s75n I1223 16:07:45.706983       1 event.go:282] Event(v1.ObjectReference{Kind:"CertificateSigningRequest", Namespace:"", Name:"test111-csr-1640246865", UID:"1c353de9-3a40-4617-9238-d76d1c06ea97", APIVersion:"certificates.k8s.io/v1", ResourceVersion:"156757923", FieldPath:""}): type: 'Normal' reason: 'Synced' CertificateSigningRequest csrSynced 
successfully

wmwm061061

johnniang

这个 test111 用户的 clusterrolebinding 也是有自动创建的

$ kubectl get clusterrolebindings.rbac.authorization.k8s.io test111-cluster-admin -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2021-12-23T07:43:45Z"
  managedFields:
  - apiVersion: rbac.authorization.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"1d719060-2ff4-420d-b3f7-695a73c9cc03"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:roleRef:
        f:apiGroup: {}
        f:kind: {}
        f:name: {}
      f:subjects: {}
    manager: controller-manager
    operation: Update
    time: "2021-12-23T07:43:45Z"
  name: test111-cluster-admin
  ownerReferences:
  - apiVersion: iam.kubesphere.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: GlobalRoleBinding
    name: test111-platform-admin
    uid: 1d719060-2ff4-420d-b3f7-695a73c9cc03
  resourceVersion: "156739176"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/test111-cluster-admin
  uid: 42e64b74-874a-432a-b1a6-7775c6af6b32
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: test111

也就是说除了除了 kubeconfig 文件里面的 users 字段为空之外，其他的应该都没问题，我尝试着用 master 节 /etc/kubesphere/pki 目录中的 CA 证书为 test111 用户签发了一个客户端证书

[root@master01 pki]# cd /etc/kubernetes/pki

[root@master01 pki]# (umask 077; openssl genrsa -out test111.key 2048)

[root@master01 pki]# openssl req -new -key test111.key -out test111.csr -subj "/CN=test111/"

[root@master01 pki]# openssl x509 -req -in test111.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test111.crt -days 3650

然后把这个 CA 签发的客户端证书和私钥导入到 test111用户的 kubeconfig 文件中，再请求 K8S 的 api-server 是没有问题的。

我怀疑 kubeconfig 文件中 users 为空的原因可能是 ks-controller-manager 这个组件创建 kubeconfig 文件的时候没有成功把客户端证书和私钥信息导入进去或者干脆在签发客户端证书这一步骤就出问题了，但是具体为什么会出现这种情况我无法排查出来。

hongming

wmwm061061 这个是 host 上 ks-controller-manager 的日志吗，看日志证书已经正常签发了，具体逻辑在这里 https://github.com/kubesphere/kubesphere/blob/master/pkg/models/kubeconfig/kubeconfig.go

kubectl get certificatesigningrequests 看看有没有 pending 的 csr

wmwm061061

hongming

$ kubectl get certificatesigningrequests.certificates.k8s.io
NAME                     AGE   SIGNERNAME                            REQUESTOR                                            CONDITION
admin-csr-1640253456     17h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
devops-csr-1640147653    47h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test111-csr-1640149386   46h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test111-csr-1640245426   19h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test111-csr-1640245482   19h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test111-csr-1640246865   19h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test111-csr-1640247292   19h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test222-csr-1640181746   37h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test222-csr-1640243442   20h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test222-csr-1640244720   20h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved
test222-csr-1640244779   20h   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubesphere-system:kubesphere   Approved

使用 kubectl get certificatesigningrequests 命令可以查看到很多 csr。

hongming

wmwm061061

看下 csr 的 status 字段里，Status.Certificate 部分是不是没有值，虽然被 approve 了，kubectl describe 一下csr看看证书为啥签不下来

wmwm061061

hongming

下面是 descripe 信息

$ kubectl describe certificatesigningrequests.certificates.k8s.io test111-csr-1640149386
Name:         test111-csr-1640149386
Labels:       kubesphere.io/username=test111
Annotations:  kubesphere.io/private-key=-----BEGIN PRIVATE KEY-----
MIIEpAIBAAKCAQEAzfMp9IsruKr1i7ltk6BrQUDiH8yRWyEDSAx7cAweWSbKq3jP
aef6QGZg94dYGDcUiDVHWxax5H5ok03ApMXAG4/W6Vs+wGOWbX8gS6IawYuJxLn+
oLGLjg5TPHbohgzCC1eNR71mByMy0tdllubj0mHNz5ka9fjVvDrWe5tmDJC5D41S
422qdMMFsxYqQC2OcnWKKH+ypLTwviPVeBIoShKHL7M+r/LWMLMt/0T+5j7CFraL
hWKOGHubn5XfmDHZUZuLpEZp6i7NnaFy8udb9dlSMjtYVQrntRATm22e8oThfhDn
G9avrwcYgeOZ7EdRDGD8fVkiUJbHh1Epj63bzQIDAQABAoIBAQC91csxf2sAoI0P
iw9nCXJzxpPojc7aOhblPaQ2RuVedfRQlF5Tl6HD5Nqyr03TnEnPt5SbcNLu9Fn+
lafh+em3PiHyHco82k8ZhRnGvZh+GNoXvP3pL16cxbWRwWVZ0r62Z7BxlTJLl/VM
BYG90/vhu4dTOSRx4DlbdLypWTd6ke823Rzs36dS3Vye/hcETqvtHYmo+TaBgTCW
tkaLTlMEThXYS7kJ6b0JX78LEcA3Tjc9BnUcsunv62F+Wi9vhGoDOmFRz5AylMAt
W/HX3M6NoHDMZmkVBMnIIcnO4Ss1UeP4Rt8HM08lIABaYhLyDtsisHcwgwh2Xzcc
CAgH88OBAoGBAPosUfvghC7l4rjUhhIhQhGMhx2W8rl3X69A88WFj2hgQ9XbjMdL
nVW68Mx1bx+HGvRcM0DplikHRLLlVQzRY3yENnqhDq4GxmEy1pu6isUk2f1yRCIj
+keQyzjHPXRqAl6mZggIclwkxlXGz3w1g6NMhVfW9KNyx3+zgLe9r5+hAoGBANK/
KI8XB9MGD5D/qCtD8C2Y9nw82ckJf+m0LpYiVaqzV9BaFKunXSfK3yzJjjkj6Svx
HRr0P83EOsoFkGwltjjMtw+Yqdph/IFXxLhyO/kx0WVd62YhwApxRW+FbvCjjPf9
rCvM6XaSahB0SfW78ZtgWT4zpkZE6XfPi9ODjXytAoGBANWZidZY+ANi3JWbP22z
X8nEBolJHK0mM1hbSMfZJXyk1MZ/0k9HTGqqVnb7vYlZvbXuEwTVbBRT1GA6Vse+
8AacjDdf1DzaI/9RfNMyDtIMiONQ8MOFnGgGRL2Rv/LpoGRYBr3NGHd8HquWOAmB
V4fSCffZ3L7BfXyXMsuu9YiBAoGAU6wZylX2gU4SuBsJBcR/g1sL4d+3CB6BbqRf
KrSPFqsF3m8lbdGRJs9hNEMB+KXNWejtCVqYSnXycWssIAQJCRweQk72gQIOYrwa
GrEgJpwk6v4TDAWyXXMDgneTyG7m9T7+MJW2y6DuA4JoKuxnzKzAQfX3ukZ5+1un
xF6iujECgYBOQ5BKADAbcqO9+wVkBcsxpXglKfsI5I81n5seTpydfz2xA4v3fmmb
H8ShrnuRo22xchTpIPxMDHe+nx+a6aiLKbUMDghBK+CO8Z6LMDUQAxO6FXJwnhr8
KpO7vkCTMQAVWs9Uy52tQKJ2rusg6k0juOwqsGW1tTXEK1Q3S6FkrQ==
-----END PRIVATE KEY-----

CreationTimestamp:  Wed, 22 Dec 2021 13:03:06 +0800
Requesting User:    system:serviceaccount:kubesphere-system:kubesphere
Signer:             kubernetes.io/kube-apiserver-client
Status:             Approved
Subject:
  Common Name:    test111
  Serial Number:  
Events:
  Type    Reason  Age                    From            Message
  ----    ------  ----                   ----            -------
  Normal  Synced  6m32s (x248 over 20h)  csr-controller  CertificateSigningRequest csrSynced successfully

下面是 kubectl get certificatesigningrequests.certificates.k8s.io -oyaml 信息

$ kubectl get certificatesigningrequests.certificates.k8s.io test111-csr-1640149386 -oyaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  annotations:
    kubesphere.io/private-key: |
      -----BEGIN PRIVATE KEY-----
      MIIEpAIBAAKCAQEAzfMp9IsruKr1i7ltk6BrQUDiH8yRWyEDSAx7cAweWSbKq3jP
      aef6QGZg94dYGDcUiDVHWxax5H5ok03ApMXAG4/W6Vs+wGOWbX8gS6IawYuJxLn+
      oLGLjg5TPHbohgzCC1eNR71mByMy0tdllubj0mHNz5ka9fjVvDrWe5tmDJC5D41S
      422qdMMFsxYqQC2OcnWKKH+ypLTwviPVeBIoShKHL7M+r/LWMLMt/0T+5j7CFraL
      hWKOGHubn5XfmDHZUZuLpEZp6i7NnaFy8udb9dlSMjtYVQrntRATm22e8oThfhDn
      G9avrwcYgeOZ7EdRDGD8fVkiUJbHh1Epj63bzQIDAQABAoIBAQC91csxf2sAoI0P
      iw9nCXJzxpPojc7aOhblPaQ2RuVedfRQlF5Tl6HD5Nqyr03TnEnPt5SbcNLu9Fn+
      lafh+em3PiHyHco82k8ZhRnGvZh+GNoXvP3pL16cxbWRwWVZ0r62Z7BxlTJLl/VM
      BYG90/vhu4dTOSRx4DlbdLypWTd6ke823Rzs36dS3Vye/hcETqvtHYmo+TaBgTCW
      tkaLTlMEThXYS7kJ6b0JX78LEcA3Tjc9BnUcsunv62F+Wi9vhGoDOmFRz5AylMAt
      W/HX3M6NoHDMZmkVBMnIIcnO4Ss1UeP4Rt8HM08lIABaYhLyDtsisHcwgwh2Xzcc
      CAgH88OBAoGBAPosUfvghC7l4rjUhhIhQhGMhx2W8rl3X69A88WFj2hgQ9XbjMdL
      nVW68Mx1bx+HGvRcM0DplikHRLLlVQzRY3yENnqhDq4GxmEy1pu6isUk2f1yRCIj
      +keQyzjHPXRqAl6mZggIclwkxlXGz3w1g6NMhVfW9KNyx3+zgLe9r5+hAoGBANK/
      KI8XB9MGD5D/qCtD8C2Y9nw82ckJf+m0LpYiVaqzV9BaFKunXSfK3yzJjjkj6Svx
      HRr0P83EOsoFkGwltjjMtw+Yqdph/IFXxLhyO/kx0WVd62YhwApxRW+FbvCjjPf9
      rCvM6XaSahB0SfW78ZtgWT4zpkZE6XfPi9ODjXytAoGBANWZidZY+ANi3JWbP22z
      X8nEBolJHK0mM1hbSMfZJXyk1MZ/0k9HTGqqVnb7vYlZvbXuEwTVbBRT1GA6Vse+
      8AacjDdf1DzaI/9RfNMyDtIMiONQ8MOFnGgGRL2Rv/LpoGRYBr3NGHd8HquWOAmB
      V4fSCffZ3L7BfXyXMsuu9YiBAoGAU6wZylX2gU4SuBsJBcR/g1sL4d+3CB6BbqRf
      KrSPFqsF3m8lbdGRJs9hNEMB+KXNWejtCVqYSnXycWssIAQJCRweQk72gQIOYrwa
      GrEgJpwk6v4TDAWyXXMDgneTyG7m9T7+MJW2y6DuA4JoKuxnzKzAQfX3ukZ5+1un
      xF6iujECgYBOQ5BKADAbcqO9+wVkBcsxpXglKfsI5I81n5seTpydfz2xA4v3fmmb
      H8ShrnuRo22xchTpIPxMDHe+nx+a6aiLKbUMDghBK+CO8Z6LMDUQAxO6FXJwnhr8
      KpO7vkCTMQAVWs9Uy52tQKJ2rusg6k0juOwqsGW1tTXEK1Q3S6FkrQ==
      -----END PRIVATE KEY-----
  creationTimestamp: "2021-12-22T05:03:06Z"
  labels:
    kubesphere.io/username: test111
  managedFields:
  - apiVersion: certificates.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubesphere.io/private-key: {}
        f:labels:
          .: {}
          f:kubesphere.io/username: {}
      f:spec:
        f:groups: {}
        f:request: {}
        f:signerName: {}
        f:usages: {}
        f:username: {}
      f:status:
        f:conditions:
          .: {}
          k:{"type":"Approved"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: controller-manager
    operation: Update
    time: "2021-12-22T05:03:06Z"
  name: test111-csr-1640149386
  resourceVersion: "157390924"
  selfLink: /apis/certificates.k8s.io/v1/certificatesigningrequests/test111-csr-1640149386
  uid: 313d1b51-1d2f-412e-a826-9cdee86f473d
spec:
  groups:
  - system:serviceaccounts
  - system:serviceaccounts:kubesphere-system
  - system:authenticated
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1Z6Q0NBVDhDQVFBd0VqRVFNQTRHQTFVRUF4TUhkR1Z6ZERFeE1UQ0NBU0l3RFFZSktvWklodmNOQVFFQgpCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNM3pLZlNMSzdpcTlZdTViWk9nYTBGQTRoL01rVnNoQTBnTWUzQU1IbGttCnlxdDR6Mm5uK2tCbVlQZUhXQmczRklnMVIxc1dzZVIrYUpOTndLVEZ3QnVQMXVsYlBzQmpsbTEvSUV1aUdzR0wKaWNTNS9xQ3hpNDRPVXp4MjZJWU13Z3RYalVlOVpnY2pNdExYWlpibTQ5Smh6YytaR3ZYNDFidzYxbnViWmd5UQp1UStOVXVOdHFuVERCYk1XS2tBdGpuSjFpaWgvc3FTMDhMNGoxWGdTS0VvU2h5K3pQcS95MWpDekxmOUUvdVkrCndoYTJpNFZpamhoN201K1YzNWd4MlZHYmk2UkdhZW91eloyaGN2TG5XL1haVWpJN1dGVUs1N1VRRTV0dG52S0UKNFg0UTV4dldyNjhIR0lIam1leEhVUXhnL0gxWklsQ1d4NGRSS1krdDI4MENBd0VBQWFBQU1BMEdDU3FHU0liMwpEUUVCQ3dVQUE0SUJBUUFBUG03cmMyRWZyWTRUbnVlYzlwR0JpL29PL2s1UjJPTmptZDVkVEdGV1JIeVBZa3duCjRPbFJhV3lPV3AyM21neUhPVjdQQW1jalE2V0xjU3lselZNeWt5dzh0aDUxZy90UW16Nm9nVi83ZTZKMnBMeGQKUGxZQmdtbDVVRFJIdG1nL2Q4TUpxbFVOMlhlYWZLaVcyL1BXSHpOc0U4aWZub2ZzZm90TmV4Vm1NQU1LZDM4NAo5Tnd1Y1lXZUhQcm8yMEN6T2FNdGx5SGxSbFduekhQdFFVM25QVWJhYU1hK2ljV0pKRlg3MWZ5cDVlSFlJRzNiCmRTUS9QSFpwRjdCNkVuVCtEblQ4ekQ5c0dvVVdTTlhtUGVMbXg1SVZkNEZxYTFqVkwxZGM5b3hjbzZSNVd3cUsKRWZYSy8xWnRENHZxQkZ3NDlValJaQnNJU2pPVlBFRjZtcE5KCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  signerName: kubernetes.io/kube-apiserver-client
  uid: f7b9f484-b23a-49e9-971f-ee6036698636
  usages:
  - key encipherment
  - client auth
  - digital signature
  username: system:serviceaccount:kubesphere-system:kubesphere
status:
  conditions:
  - lastTransitionTime: "2021-12-22T05:03:06Z"
    lastUpdateTime: "2021-12-24T05:24:50Z"
    message: This CSR was approved by KubeSphere
    reason: KubeSphereApprove
    status: "True"
    type: Approved

hongming

wmwm061061

跟你这个集群环境有关系，CSR approve 了但是证书没有签发下来。 K8s 什么版本，看看 kube-controller-manager 的启动参数和日志。

https://discuss.kubernetes.io/t/certificate-signing-request-approved-but-certificate-not-issued/14324/5

wmwm061061

hongming

@johnniang

@DehaoCheng

非常感谢三位大佬帮忙解答这个问题，这个问题的根本原因就是 kube-controller-manager 导致的。

我的集群是三个 master 节点的高可用模式，在三天之前进行了升级操作，kubesphere由 v3.1.1 升级至 v3.2.1，kubernetes 由 v1.18.8 升级至 v1.19.8。刚刚查看了 kube-controller-manager 日志发现 master03 节点的 kube-controller-manager 升级之后启动报错了。日志信息如下

Flag --experimental-cluster-signing-duration has been deprecated, use --cluster-signing-duration
Flag --port has been deprecated, see --secure-port instead.
I1221 21:41:55.367887       1 serving.go:331] Generated self-signed cert in-memory
I1221 21:41:55.695667       1 controllermanager.go:175] Version: v1.19.8
I1221 21:41:55.697103       1 secure_serving.go:197] Serving securely on [::]:10257
I1221 21:41:55.698631       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/kube-controller-manager...
I1221 21:41:55.702499       1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt
I1221 21:41:55.702570       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
I1221 21:41:55.702666       1 tlsconfig.go:240] Starting DynamicServingCertificateController
E1221 21:41:58.492775       1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: endpoints "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "endpoints" in API group "" in the namespace "kube-system"

我尝试使用 kubectl delete pods kube-controller-manager-master03 命令重启这个容器，查看日志发现实际并没有重启成功，最后登录 master03 节点，使用 docker restart 命令重启成功，启动日志信息也没有报错。

Flag --experimental-cluster-signing-duration has been deprecated, use --cluster-signing-duration
Flag --port has been deprecated, see --secure-port instead.
I1224 16:12:27.777365       1 serving.go:331] Generated self-signed cert in-memory
I1224 16:12:27.985002       1 controllermanager.go:175] Version: v1.19.8
I1224 16:12:27.985851       1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt
I1224 16:12:27.985908       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
I1224 16:12:27.986231       1 secure_serving.go:197] Serving securely on [::]:10257
I1224 16:12:27.986279       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/kube-controller-manager...
I1224 16:12:27.986303       1 tlsconfig.go:240] Starting DynamicServingCertificateController

之前堆积的 csr 也全部重新签发。使用 kubectl get certificatesigningrequests.certificates.k8s.io 命令已经看不到之前的 csr

我又查看了新创建用户的 kubeconfig 文件，users 字段也有了完整内容，使用这个 kubeconfig 也可以正常访问 K8S 的 kube-apiserver