member集群用户同步异常

hulengjing

host集群修改为外部ldap认证，添加完member集群(也修改了ldap
ldap认证)后，member集群出现用户不停的在进行增加删除和授权。

yuswift

member集群不需要添加ldap配置能把ks-apiserver日志贴一下吗

hulengjing

这个时候通过host
host集群去进行查看member集群的k’8s
k8s数据就会抛500异常

hulengjing

W0106 19:16:02.633526 1 client_config.go:543] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.

I0106 19:16:02.651239 1 apiserver.go:300] Start cache objects

W0106 19:16:02.810081 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1beta1.Ingress ended with: too old resource version: 528439 (528705)

W0106 19:16:03.044607 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha3.Pipeline ended with: too old resource version: 529859 (531727)

W0106 19:16:03.044729 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha1.S2iBuilderTemplate ended with: too old resource version: 529756 (531745)

W0106 19:16:03.045374 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha2.WorkspaceRoleBinding ended with: too old resource version: 531572 (531747)

W0106 19:16:03.047064 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha2.Strategy ended with: too old resource version: 531727 (532829)

W0106 19:16:03.240063 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotclasses not exists in the cluster

W0106 19:16:03.240092 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshots not exists in the cluster

W0106 19:16:03.240100 1 apiserver.go:473] resource snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotcontents not exists in the cluster

I0106 19:16:03.440290 1 apiserver.go:502] Finished caching objects

I0106 19:16:03.440314 1 apiserver.go:232] Start listening on :9090

W0106 19:16:04.048243 1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha1.S2iBuilderTemplate ended with: too old resource version: 529756 (531727)

E0107 09:42:19.124550 1 v2.go:105] websocket: close 1001 (going away)

W0107 09:42:27.126055 1 terminal.go:136] 1Process exited

E0107 09:52:14.522705 1 authenticator.go:102] identity mapping not match

E0107 09:52:14.522762 1 handler.go:275] identity mapping not match

I0107 09:52:14.522793 1 apiserver.go:539] 10.233.116.168 - “POST /oauth/token HTTP/1.1” 401 40 0ms

I0107 09:55:32.596123 1 apiserver.go:539] 10.161.100.27, 10.233.64.18, 172.18.1.110 - “GET /kapis/devops.kubesphere.io/v1alpha2/devops/otc-servicen8tgb/pipelines/hujing-test/branches/master/sonarstatus HTTP/1.1” 404 19 0ms

hulengjing

意思是member集群不开启ldap
ldap然后通过host进行认证嘛？那如果单独访问member集群的console可以正常通过ldap
账号密码登录嘛？

yuswift

hulengjing ldap账号应该在host集群登陆这样统一做权限管理如果member不配置不能通过ldap登陆
你的报错是没有权限你是不是在member集群操作的?

hulengjing

controller manager 出现下列一直进行同步的异常
E0107 10:19:20.712661 1 kubectl.go:116] user.iam.kubesphere.io “hujing” not found

I0107 10:19:20.712687 1 clusterrolebinding_controller.go:186] Successfully synced key:hujing-cluster-admin

I0107 10:19:20.712872 1 event.go:281] Event(v1.ObjectReference{Kind:“ClusterRoleBinding”, Namespace:"", Name:“hujing-cluster-admin”, UID:“382e727e-fda8-4cc2-a866-6a0aa5c8593e”, APIVersion:“rbac.authorization.k8s.io/v1”, ResourceVersion:“965312”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ ClusterRoleBinding synced successfully

I0107 10:19:20.712985 1 globalrolebinding_controller.go:200] Successfully synced key:hujing-platform-admin

I0107 10:19:20.713000 1 event.go:281] Event(v1.ObjectReference{Kind:“GlobalRoleBinding”, Namespace:"", Name:“hujing-platform-admin”, UID:“9cd44386-602c-471f-840d-a88555485f36”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965311”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ GlobalRoleBinding synced successfully

I0107 10:19:20.863371 1 workspacerolebinding_controller.go:205] Successfully synced key:admin-otc-hae-admin

I0107 10:19:20.863400 1 event.go:281] Event(v1.ObjectReference{Kind:“WorkspaceRoleBinding”, Namespace:"", Name:“admin-otc-hae-admin”, UID:“28d12f5c-c697-4a4b-8450-f5116e0c1e6a”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965315”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ WorkspaceRoleBinding synced successfully

I0107 10:19:20.863557 1 workspacerolebinding_controller.go:205] Successfully synced key:admin-otc-hae-admin

I0107 10:19:20.863570 1 event.go:281] Event(v1.ObjectReference{Kind:“WorkspaceRoleBinding”, Namespace:"", Name:“admin-otc-hae-admin”, UID:“28d12f5c-c697-4a4b-8450-f5116e0c1e6a”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965316”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ WorkspaceRoleBinding synced successfully

I0107 10:19:21.341516 1 workspacerolebinding_controller.go:205] Successfully synced key:admin-otc-ops-admin

I0107 10:19:21.341555 1 workspacerolebinding_controller.go:205] Successfully synced key:admin-otc-ops-admin

I0107 10:19:21.341802 1 event.go:281] Event(v1.ObjectReference{Kind:“WorkspaceRoleBinding”, Namespace:"", Name:“admin-otc-ops-admin”, UID:“3058baa5-3cb0-4194-81c4-a71d6bdd1b26”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965321”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ WorkspaceRoleBinding synced successfully

I0107 10:19:21.341824 1 event.go:281] Event(v1.ObjectReference{Kind:“WorkspaceRoleBinding”, Namespace:"", Name:“admin-otc-ops-admin”, UID:“3058baa5-3cb0-4194-81c4-a71d6bdd1b26”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965320”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ WorkspaceRoleBinding synced successfully

I0107 10:19:22.320017 1 workspacerolebinding_controller.go:205] Successfully synced key:admin-huobi-otc-admin

I0107 10:19:22.320122 1 event.go:281] Event(v1.ObjectReference{Kind:“WorkspaceRoleBinding”, Namespace:"", Name:“admin-huobi-otc-admin”, UID:“ddbfa6ef-48f9-4f00-b684-279932dead93”, APIVersion:“iam.kubesphere.io/v1alpha2”, ResourceVersion:“965327”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ WorkspaceRoleBinding synced successfully

I0107 10:19:22.531467 1 certificatesigningrequest_controller.go:182] Successfully csrSynced key:hujing-csr-1609985962

I0107 10:19:22.531530 1 event.go:281] Event(v1.ObjectReference{Kind:“CertificateSigningRequest”, Namespace:"", Name:“hujing-csr-1609985962”, UID:“7b5bc1c1-3cb5-4125-8287-b1354fa28c29”, APIVersion:“certificates.k8s.io/v1beta1”, ResourceVersion:“965328”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ CertificateSigningRequest csrSynced successfully

I0107 10:19:22.534300 1 certificatesigningrequest_controller.go:182] Successfully csrSynced key:hujing-csr-1609985962

I0107 10:19:22.534318 1 event.go:281] Event(v1.ObjectReference{Kind:“CertificateSigningRequest”, Namespace:"", Name:“hujing-csr-1609985962”, UID:“7b5bc1c1-3cb5-4125-8287-b1354fa28c29”, APIVersion:“certificates.k8s.io/v1beta1”, ResourceVersion:“965330”, FieldPath:""}): type: ‘Normal’ reason: ‘Synced’ CertificateSigningRequest csrSynced successfully

I0107 10:19:22.547841 1 user_controller.go:221] Successfully synced key:hujing

hongming

hulengjing

member 托管到 host 之后，user/ workspace这类资源是通过 host (kubefed) 向 member 分发的，member上单独创建的资源没办法同步到host上，也不推荐独立使用 member cluster

hulengjing

yuswift 我是在host集群上登录的ldap，然后member之前是有ldap配置，用户我已经删除了，现在一直是host
的用户同步过来异常，一直在进行用户的增加授权删除

hongming

正确配置 member 的 clusterRole，确认 member cluster kubesphere-config 中 accessTokenMaxAge 为 0
只需要在 host 配置LDAP，通过LDAP登录后会关联创建 kubesphere 账户，可以通过kubectl get user / kubectl edit user 查看和编辑
host 上的user会同步到member cluster
尽量不要直接在member 上直接创建资源，无法反向同步到 host 集群，在熟悉kubesphere的前提下，可以通过 kubectl 对资源进行标记，方便 host 集群进行管理

hulengjing

hongming 最开始的情况是当前的member集群设置过host集群，然后新建了一个host集群将最开始的host
修改为了member并导入到了新的host集群中，老集群中新建过企业空间/用户,用户我这边已经删除了，在新的host集群中又新建了相同的企业空间，发现是数据可以获取到的

hongming

hulengjing

这个没明白什么意思

member集群出现用户不停的在进行增加删除和授权，

member 进群中的user是通过kubefed 进行同步的可以在 host cluster 上 kubectl get federatedusers xxx -o yaml 查看状态

ks-apiserver 的日志，是正常的

E0107 09:42:19.124550 1 v2.go:105] websocket: close 1001 (going away)
W0107 09:42:27.126055 1 terminal.go:136] 1Process exited
E0107 09:52:14.522705 1 authenticator.go:102] identity mapping not match
E0107 09:52:14.522762 1 handler.go:275] identity mapping not match

ks-controller-manager 中这一行异常， reconcile 之后同步成功了

E0107 10:19:20.712661 1 kubectl.go:116] user.iam.kubesphere.io “hujing” not found
I0107 10:19:22.547841 1 user_controller.go:221] Successfully synced key:hujing

hulengjing

kubesphere的最佳实践是什么样子的？host安装所有组件进行devops相关操作，member集群只需要安装最小化进行托管k8s这样嘛？

hulengjing

hongming 我的意思是，我现在host新增了一个用户 hu’jing
member集群同步过来之后一直在重复执行创建用户、授权为admin、删除用户这三个操作。
也就是controller-manager的日志一直在重复刚才的异常日志找不到用户、创建用户、同步数据、删除用户、找不到用户

hongming

hulengjing
在 member 上kubectl get user -o yaml 看看是不是因为之前没有删除成功，看看是有没有这个label kubefed.io/managed: true

controller 中的reconcile 编辑修改都会触发，不一定是一直重复删除创建的过程

hulengjing

hongming
这边看到member上的是false

apiVersion: v1
items:
- apiVersion: iam.kubesphere.io/v1alpha2
  kind: User
  metadata:
    annotations:
      iam.kubesphere.io/password-encrypted: "true"
    creationTimestamp: "2021-01-04T02:31:00Z"
    finalizers:
    - finalizers.kubesphere.io/users
    generation: 23
    labels:
      kubefed.io/managed: "false"
    name: admin
    resourceVersion: "936438"
    selfLink: /apis/iam.kubesphere.io/v1alpha2/users/admin
    uid: 5a5c2cb7-17cf-4c2f-bdf7-fe6510a9dc8e
  spec:
    email: admin@kubesphere.io
    password: $2a$10$ryYJfBXlVFx6TgVQab5JKOKuWOPZJWXA3sZAso2z/krp1ESIO9m1K
  status:
    lastLoginTime: "2021-01-07T01:07:39Z"
    lastTransitionTime: "2021-01-04T02:42:19Z"
    state: Active
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

hongming

目前部分组件和功能模块必须在 member cluster 上启用，比如说日志和 devops。但是 member cluster 上可以只启用一些必须的功能。正常情况下通过 host 进行管理，特殊情况下如 host 集群异常，也可以直接登录到 member 集群上对集群资源进行管理。

社区已经在着手裁剪ks-core ，尽可能的轻量化，减少member cluster的资源占用，将组件模块化按需启用

hulengjing

hongming
我能理解到kubesphere的最佳实践应该是这样的host管控所有集群
host: 统一入口、统一认证、统一权限、所需组件全开(devops、servicemesh、日志、监控。。。)
member: 最小化安装、只需开启日志和监控组件

不知道我理解的对不对

hongming

hulengjing

kubectl get user hujing -o yaml 看看，把 kubefed.io/managed 都改为 true

hulengjing

hongming 这个是我连续几次执行查看hujing用户命令

root@kubesphere-01:~# kubectl get user hujing -o yaml
apiVersion: iam.kubesphere.io/v1alpha2
kind: User
metadata:
  annotations:
    iam.kubesphere.io/password-encrypted: "true"
  creationTimestamp: "2021-01-07T05:37:57Z"
  finalizers:
  - finalizers.kubesphere.io/users
  generation: 2
  labels:
    kubefed.io/managed: "true"
  name: hujing
  resourceVersion: "1057717"
  selfLink: /apis/iam.kubesphere.io/v1alpha2/users/hujing
  uid: 128df234-51e9-4623-b33a-95f84021ac50
spec:
  email: hujing@huobi.com
status:
  lastTransitionTime: "2021-01-07T05:37:57Z"
  state: Active
root@kubesphere-01:~# kubectl get user hujing -o yaml
Error from server (NotFound): users.iam.kubesphere.io "hujing" not found
root@kubesphere-01:~# kubectl get user hujing -o yaml
Error from server (NotFound): users.iam.kubesphere.io "hujing" not found
root@kubesphere-01:~# kubectl get user hujing -o yaml
Error from server (NotFound): users.iam.kubesphere.io "hujing" not found
root@kubesphere-01:~# kubectl get user hujing -o yaml
Error from server (NotFound): users.iam.kubesphere.io "hujing" not found
root@kubesphere-01:~# kubectl get user hujing -o yaml
apiVersion: iam.kubesphere.io/v1alpha2
kind: User
metadata:
  annotations:
    iam.kubesphere.io/password-encrypted: "true"
  creationTimestamp: "2021-01-07T05:38:07Z"
  finalizers:
  - finalizers.kubesphere.io/users
  generation: 2
  labels:
    kubefed.io/managed: "true"
  name: hujing
  resourceVersion: "1057793"
  selfLink: /apis/iam.kubesphere.io/v1alpha2/users/hujing
  uid: 8b8a6aa4-f420-4051-a4b4-e5aca5ebc158
spec:
  email: hujing@huobi.com
status:
  lastTransitionTime: "2021-01-07T05:38:07Z"
  state: Active

会发现中间无法找到hujing用户，然后又出现了

hongming

hulengjing

在 member 和 host 上都执行一下看看，应该是哪里冲突了

kubectl get ns
kubectl get federatedusers