Devops工具中的凭证页面打开报错

freemankevin

想咨询下各位大佬这个怎么排查异常，普罗那边目前没有异常

Jeff

freemankevin 看下 ks-apiserver 的日志

freemankevin

Jeff 您看下

Jeff

freemankevin 你看下你的主机时间都是对的么

freemankevin

Jeff
和windows时间相差3s左右算吗？

Rick

所有创建的凭据都是这种情况吗？检查下Jenkins是否可以正常访问。

freemankevin

Rick 是的，都是类似这种，jenkins那边一切正常

freemankevin

Rick 您看下

shaowenchen

freemankevin

再创建一个凭证，看看ks-controller-manager的日志，会有报错提醒。很有可能是节点通信有问题，看看 ks-controller-manager 和 jenkins 在不在一个节点。

Rick

看看你的 Jenkins 上是否包含 demo-devopsgj8zz 的文件夹

freemankevin

Rick 只有一个init-job,正常应该是两个初始化的文件夹对吧，记得之前是

Rick

就是因为没有对应的文件夹导致的404
这个文件夹，正常是在创建DevOps工程后会自动创建的。可能是在你创建工程的时候Jenkins状态不正常导致的。
临时解决方案：手动创建一个和DevOps工程同名的文件夹到Jenkins上
不过，这块的逻辑上确实有缺陷，感谢你帮忙发现了这个问题。我们会尽快修复。

freemankevin

Rick 好的，我试下，您太客气了，我一直以为是我操作哪里的问题

freemankevin

shaowenchen

这是新建的凭证信息：(帖子不知为啥现在无法上传图片了—糟糕，出错啦！请刷新页面重试。)
kubeconfig-id
kubeconfig
kubeconfig默认值
2020-12-23 10:06:59

shaowenchen 新增凭证，无最新日志输出，然后ks-controller-manager确实存在异常的日志

这里的pod数量是正常的吗？

[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]# kubectl get pod -A -owide|grep master1|grep control
kube-system                    calico-kube-controllers-cb45b5ff7-vjvt4                           1/1     Running     0          3d15h   10.233.97.3    master1   <none>           <none>
kube-system                    kube-controller-manager-master1                                   1/1     Running     73         3d15h   10.0.1.223     master1   <none>           <none>
kubesphere-system              ks-controller-manager-9f4ff655d-qbd2f                             1/1     Running     39         2d10h   10.233.97.14   master1   <none>           <none>
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]# kubectl get pod -A -owide|grep master2|grep control
kube-system                    kube-controller-manager-master2                                   1/1     Running     78         3d15h   10.0.1.224     master2   <none>           <none>
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]#
[root@master2 ~]# kubectl get pod -A -owide|grep master3|grep control
kube-system                    kube-controller-manager-master3                                   1/1     Running     78         3d15h   10.0.1.225     master3   <none>           <none>
kubesphere-system              ks-controller-manager-9f4ff655d-x5tvg                             1/1     Running     5          7h28m   10.233.96.20   master3   <none>           <none>
[root@master2 ~]#

freemankevin

shaowenchen

两个 ks-controller-manager的日志一直停在这里

[root@master2 ~]# kubectl logs -f -n kubesphere-system ks-controller-manager-9f4ff655d-x5tvg
W1223 09:32:59.611634       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1223 09:32:59.631754       1 server.go:175] setting up manager
I1223 09:32:59.660909       1 user_controller.go:134] Setting up event handlers
I1223 09:32:59.661119       1 certificatesigningrequest_controller.go:90] Setting up event handlers
I1223 09:32:59.661248       1 clusterrolebinding_controller.go:94] Setting up event handlers
I1223 09:32:59.661357       1 globalrole_controller.go:95] Setting up event handlers
I1223 09:32:59.661448       1 workspacerole.go:106] Setting up event handlers
I1223 09:32:59.661533       1 globalrolebinding_controller.go:101] Setting up event handlers
I1223 09:32:59.661615       1 workspacerolebinding_controller.go:106] Setting up event handlers
I1223 09:32:59.661704       1 workspacetemplate_controller.go:122] Setting up event handlers
I1223 09:32:59.661994       1 server.go:225] Starting cache resource from apiserver...
I1223 09:32:59.662103       1 server.go:236] Starting the controllers.
I1223 09:32:59.662169       1 leaderelection.go:242] attempting to acquire leader lease  kubesphere-system/ks-controller-manager-leader-election...
E1223 09:51:47.252200       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1.PersistentVolumeClaim: Get https://10.233.0.1:443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=1225417&timeout=9m43s&timeoutSeconds=583&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.252278       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1alpha3.DestinationRule: Get https://10.233.0.1:443/apis/networking.istio.io/v1alpha3/destinationrules?allowWatchBookmarks=true&resourceVersion=1129383&timeout=6m22s&timeoutSeconds=382&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.252278       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1.Pod: Get https://10.233.0.1:443/api/v1/pods?allowWatchBookmarks=true&resourceVersion=1549393&timeout=9m52s&timeoutSeconds=592&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.252501       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1alpha2.GlobalRoleBinding: Get https://10.233.0.1:443/apis/iam.kubesphere.io/v1alpha2/globalrolebindings?allowWatchBookmarks=true&resourceVersion=1129384&timeout=6m34s&timeoutSeconds=394&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.255584       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1alpha1.NamespaceNetworkPolicy: Get https://10.233.0.1:443/apis/network.kubesphere.io/v1alpha1/namespacenetworkpolicies?allowWatchBookmarks=true&resourceVersion=1129384&timeout=8m20s&timeoutSeconds=500&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.255650       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1alpha2.WorkspaceRole: Get https://10.233.0.1:443/apis/iam.kubesphere.io/v1alpha2/workspaceroles?allowWatchBookmarks=true&resourceVersion=1129384&timeout=9m19s&timeoutSeconds=559&watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E1223 09:51:47.257345       1 reflector.go:307] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to watch *v1.Secret: Get https://10.233.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=1548019&timeout=8m42s&timeoutSeconds=522&watch=true: dial tcp 10.233.0.1:443: connect: connection refused

shaowenchen

https://kubesphere.com.cn/forum/d/2408-kubesphere-devops-30 3.15 有给出这种情况

freemankevin

shaowenchen 好的

freemankevin

不知为啥这个论坛的帖子现在连长点的日志都无法贴了，要分两次贴，这里是另外一个的

[root@master2 ~]# kubectl logs -f -n kubesphere-system ks-controller-manager-9f4ff655d-qbd2f
W1223 09:09:33.501889       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1223 09:09:33.521247       1 server.go:175] setting up manager
I1223 09:09:33.548955       1 user_controller.go:134] Setting up event handlers
I1223 09:09:33.549144       1 certificatesigningrequest_controller.go:90] Setting up event handlers
I1223 09:09:33.549275       1 clusterrolebinding_controller.go:94] Setting up event handlers
I1223 09:09:33.549403       1 globalrole_controller.go:95] Setting up event handlers
I1223 09:09:33.550332       1 workspacerole.go:106] Setting up event handlers
I1223 09:09:33.550612       1 globalrolebinding_controller.go:101] Setting up event handlers
I1223 09:09:33.550796       1 workspacerolebinding_controller.go:106] Setting up event handlers
I1223 09:09:33.550963       1 workspacetemplate_controller.go:122] Setting up event handlers
I1223 09:09:33.551332       1 server.go:225] Starting cache resource from apiserver...
I1223 09:09:33.551676       1 server.go:236] Starting the controllers.
I1223 09:09:33.551746       1 leaderelection.go:242] attempting to acquire leader lease  kubesphere-system/ks-controller-manager-leader-election...
E1223 09:09:33.601962       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.601987       1 user_controller.go:146] Failed to enqueue login object, error: user ws-admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602130       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602146       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602154       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602160       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602166       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602171       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602176       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602181       1 user_controller.go:146] Failed to enqueue login object, error: user project-admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602187       1 user_controller.go:146] Failed to enqueue login object, error: user ws-admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602193       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602201       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602207       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602212       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602221       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602229       1 user_controller.go:146] Failed to enqueue login object, error: user user-manager doesn't exist any more, login record will be deleted later
E1223 09:09:33.602234       1 user_controller.go:146] Failed to enqueue login object, error: user ws-admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602240       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602245       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602250       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602256       1 user_controller.go:146] Failed to enqueue login object, error: user project-regular doesn't exist any more, login record will be deleted later
E1223 09:09:33.602261       1 user_controller.go:146] Failed to enqueue login object, error: user ws-manager doesn't exist any more, login record will be deleted later
E1223 09:09:33.602268       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602274       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602278       1 user_controller.go:146] Failed to enqueue login object, error: user admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602284       1 user_controller.go:146] Failed to enqueue login object, error: user project-admin doesn't exist any more, login record will be deleted later
E1223 09:09:33.602292       1 user_controller.go:146] Failed to enqueue login object, error: user project-admin doesn't exist any more, login record will be deleted later
E1223 09:20:51.027258       1 user_controller.go:146] Failed to enqueue login object, error: Operation cannot be fulfilled on users.iam.kubesphere.io "project-admin": the object has been modified; please apply your changes to the latest version and try again
E1223 09:27:11.267452       1 user_controller.go:146] Failed to enqueue login object, error: Operation cannot be fulfilled on users.iam.kubesphere.io "project-admin": the object has been modified; please apply your changes to the latest version and try again
E1223 09:27:28.629430       1 user_controller.go:146] Failed to enqueue login object, error: Operation cannot be fulfilled on users.iam.kubesphere.io "project-regular": the object has been modified; please apply your changes to the latest version and try again
error: unexpected EOF
[root@master2 ~]#

shaowenchen

freemankevin

直接将 ks-controller-manager 和 jenkins 放在一个节点试一下，看看是不是节点通信问题。看着错误日志，可能还有其他问题。

Rick

shaowenchen @freemankevin 两位，我提交了一个提议，可以一起帮忙看看 https://github.com/kubesphere/kubesphere/issues/3225

freemankevin

Rick 好的，感谢

shaowenchen 可能要晚点验这个问题了，资源见底，监控告警和优化的需求来了

freemankevin

shaowenchen

这是过一段时间再看的结果：直接挂了，登录不了

Error
com.thoughtworks.xstream.mapper.CannotResolveClassException: com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
	at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:379)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at org.jenkinsci.jruby.JRubyMapper.realClass(JRubyMapper.java:34)
	at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
	at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48)
	at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:459)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:325)
Caused: jenkins.util.xstream.CriticalXStreamException: com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy : com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
---- Debugging information ----
message             : com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
cause-exception     : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message       : com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
class               : hudson.model.Hudson
required-type       : hudson.model.Hudson
converter-type      : hudson.util.RobustReflectionConverter
path                : /hudson/authorizationStrategy
line number         : 13
version             : not available
-------------------------------
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:354)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
	at hudson.util.XStream2.unmarshal(XStream2.java:161)
	at hudson.util.XStream2.unmarshal(XStream2.java:132)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
	at hudson.XmlFile.unmarshal(XmlFile.java:178)
Caused: java.io.IOException: Unable to read /var/jenkins_home/config.xml
	at hudson.XmlFile.unmarshal(XmlFile.java:181)
	at hudson.XmlFile.unmarshal(XmlFile.java:161)
	at jenkins.model.Jenkins.loadConfig(Jenkins.java:3043)
	at jenkins.model.Jenkins.access$1200(Jenkins.java:310)
	at jenkins.model.Jenkins$13.run(Jenkins.java:3145)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: org.jvnet.hudson.reactor.ReactorException
	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
	at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1129)
	at jenkins.model.Jenkins.<init>(Jenkins.java:936)
	at hudson.model.Hudson.<init>(Hudson.java:85)
	at hudson.model.Hudson.<init>(Hudson.java:81)
	at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
	at hudson.WebAppMain$3.run(WebAppMain.java:250)

freemankevin

shaowenchen

ks面板那边也是报错

[root@master1 ~]# kubectl get pod -A |grep jenkins
kubesphere-devops-system       ks-jenkins-84c4fff98c-bwppv                                       1/1     Running     0          2d15h
kubesphere-devops-system       uc-jenkins-update-center-7b885fb95f-59bsm                         1/1     Running     0          38h
[root@master1 ~]#
[root@master1 ~]#
[root@master1 ~]# kubectl get sc -A |grep jenkins
[root@master1 ~]# kubectl get svc -A |grep jenkins
kubesphere-devops-system       ks-jenkins                                NodePort    10.233.1.189    <none>        80:30180/TCP                                   10d
kubesphere-devops-system       ks-jenkins-agent                          ClusterIP   10.233.19.204   <none>        50000/TCP                                   10d
kubesphere-devops-system       uc-jenkins-update-center                  ClusterIP   10.233.53.144   <none>        80/TCP                                   10d
[root@master1 ~]#