• DevOps

  • devops流水线未运行

walle 看日志确实是权限相关的问题，你上传的 configmap devops-config 和 kubesphere-config 截图不全，如果方便最好上传完整文件。

      walle kubesphere-config 和 devops-config 里的 devops.password 是有问题的，不是 x.y.z 格式的；

      kubesphere-secret 看不出来里面的原始内容，把这个里面的内容贴出来；

          yudong 这个devops默认就是这样 要怎么才能变成x.y.z的格式呢?

              walle 重启下 devops-controller-manager ，应该就好重新生成 x.y.z 格式的 devops.password； 然后在看下 devops-config 里的配置；

                  yudong devops-controller-manager，和ks-controller-manager都要重启对吧

                      Jason 如果 devops-config 里的 devops.password 格式是正确的 x.y.z 的，就不用重启 devops-controller-manager ；

                          8 天 后

                          zhaojun-xj yaml出错是因为 token需要base64编码，把password 编码一下就可以，例如：

                          echo “password” | base64

                            25 天 后

                            同样的问题，流水线不能运行，流水线项目无法创建，s2i构建正常。按照上面三步操作后还是不行，能否使用登录jenkins的账号密码转base64后替换密码

                            W0531 09:24:18.126595 1 jwt.go:53] jwt: token is expired by 16h28m30s
W0531 09:24:18.126670 1 jwt.go:53] jwt: token is expired by 16h28m30s
W0531 09:24:18.126815 1 jwt.go:53] jwt: token is expired by 16h28m30s
E0531 09:24:21.949083 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:24:21.949110 1 token.go:100] dial tcp 10.43.147.75:6379: connect: connection refused
I0531 09:24:21.949155 1 apiserver.go:599] ::ffff:10.42.34.128 - "POST /oauth/token HTTP/1.1" 500 109 1118ms
E0531 09:24:24.014172 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:24:24.014210 1 token.go:100] dial tcp 10.43.147.75:6379: connect: connection refused
I0531 09:24:24.014278 1 apiserver.go:599] ::ffff:10.42.34.128 - "POST /oauth/token HTTP/1.1" 500 109 1143ms
E0531 09:24:26.497278 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:24:26.497315 1 token.go:100] dial tcp 10.43.147.75:6379: connect: connection refused
I0531 09:24:26.497373 1 apiserver.go:599] ::ffff:10.42.34.128 - "POST /oauth/token HTTP/1.1" 500 109 1091ms
W0531 09:26:01.175135 1 jwt.go:53] jwt: token is expired by 16h36m53s
W0531 09:26:01.175167 1 jwt.go:53] jwt: token is expired by 16h36m53s
W0531 09:26:01.175296 1 jwt.go:53] jwt: token is expired by 16h36m53s
E0531 09:26:06.557269 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:26:06.557312 1 token.go:100] dial tcp 10.43.147.75:6379: connect: connection refused
I0531 09:26:06.557384 1 apiserver.go:599] ::ffff:10.42.34.128 - "POST /oauth/token HTTP/1.1" 500 109 1150ms
E0531 09:26:08.965813 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:26:08.965877 1 token.go:100] dial tcp 10.43.147.75:6379: connect: connection refused
I0531 09:26:08.965992 1 apiserver.go:599] ::ffff:10.42.34.128 - "POST /oauth/token HTTP/1.1" 500 109 1142ms
E0531 09:29:00.769422 1 token.go:143] dial tcp 10.43.147.75:6379: connect: connection refused
E0531 09:29:00.769479 1 token.go:100] dial tcp 10.43.1

                              igor 这个日志是 ks-apiserver 的吗 ？是最新的吗 ？

                              • igor 回复了此帖

                                  这个密码是用来干啥的 好像也不是登录密码

                                  kubesphere-secret 配置base64 解码后的token，比对了一下确实是和jenkins-config中的密码已经一致了

                                  2 个月 后

                                  按照楼主说的3个步骤做了，但是还是不能运行流水线。

                                  walle 我按照步骤改了token和password，但是流水线还是不运行。跟你情况一样。

                                      13 天 后

                                      zhangzl419 如果执行了上面3步，还不能运行是流水线，麻烦在重启下 devops-controller 服务试下，如果还不行，看下 devops-controller 日志有没有相关报错信息；

                                          7 天 后

                                          yudong

                                          碰到类似的问题：
                                          1，开始发现流水线 “未运行”，查看devops-jenkins日志：

                                          023-08-21 03:07:25.422+0000 [id=20295] WARNING i.k.j.d.a.KubesphereApiTokenAuthenticator#authenticate: API token matched for user liding2 but the impersonation failed

                                          org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.

                                          at org.acegisecurity.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:126)

                                          at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:1320)

                                          at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:1273)

                                          at org.acegisecurity.userdetails.UserDetailsService.lambda$toSpring$1(UserDetailsService.java:52)

                                          Caused: org.springframework.security.core.userdetails.UsernameNotFoundException

                                          at org.acegisecurity.userdetails.UsernameNotFoundException.toSpring(UsernameNotFoundException.java:51)

                                          at org.acegisecurity.userdetails.UsernameNotFoundException.toSpring(UsernameNotFoundException.java:34)

                                          at org.acegisecurity.userdetails.UserDetailsService.lambda$toSpring$1(UserDetailsService.java:54)

                                          at jenkins.security.ImpersonatingUserDetailsService2.loadUserByUsername(ImpersonatingUserDetailsService2.java:29)

                                          at hudson.model.User.getUserDetailsForImpersonation2(User.java:406)

                                          at hudson.model.User.getUserDetailsForImpersonation(User.java:429)

                                          Caused: org.acegisecurity.userdetails.UsernameNotFoundException: org.springframework.security.core.userdetails.UsernameNotFoundException: org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.; nested exception is org.springframework.security.core.userdetails.UsernameNotFoundException: org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.

                                          2，按照你前面提的三个步骤操作了，流水线又成功运行了，一会儿后又发现流水线"未运行"
                                          3，检查Jenkins的role-strategy/assign-roles


                                          4，查看devops-controller日志
                                          E0821 03:07:25.426300 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“elf” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“elf-gqmc7”} “namespace”=“tools6rwrl” “pipeline”=“elf”

                                          E0821 03:07:25.426407 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“elf-gqmc7” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“elf-gqmc7”,“namespace”:“tools6rwrl”} “reconcileID”=“6d87a5d2-3543-40d7-94d3-ec3d14fd0bef”

                                          <nil>

                                          <nil>

                                          E0821 03:09:32.357775 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

                                          E0821 03:09:32.357826 1 devopscredential_controller.go:167] error syncing ‘cssgzndq/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

                                          E0821 03:09:32.357834 1 devopscredential_controller.go:168] error syncing ‘cssgzndq/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

                                          E0821 03:09:32.428121 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

                                          E0821 03:09:32.428175 1 devopscredential_controller.go:167] error syncing ‘cssgzndq/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

                                          E0821 03:09:32.428183 1 devopscredential_controller.go:168] error syncing ‘cssgzndq/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

                                          E0821 03:09:32.449208 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

                                          E0821 03:09:32.449252 1 devopscredential_controller.go:167] error syncing ‘uosjmdbk/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

                                          E0821 03:09:32.449260 1 devopscredential_controller.go:168] error syncing ‘uosjmdbk/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

                                          E0821 03:09:32.461410 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

                                          E0821 03:09:32.461456 1 devopscredential_controller.go:167] error syncing ‘uosjmdbk/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

                                          E0821 03:09:32.461465 1 devopscredential_controller.go:168] error syncing ‘uosjmdbk/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

                                          <nil>

                                          <nil>

                                          <nil>

                                          <nil>

                                          E0821 03:14:01.416187 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“css” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“css-vc4d8”} “namespace”=“tools6rwrl” “pipeline”=“css”

                                          E0821 03:14:01.416736 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“css-vc4d8” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“css-vc4d8”,“namespace”:“tools6rwrl”} “reconcileID”=“7917f670-219b-4738-b3fa-311dedcd39fe”

                                          E0821 03:14:25.682510 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“elf” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“elf-rhgbf”} “namespace”=“tools6rwrl” “pipeline”=“elf”

                                          E0821 03:14:25.682684 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“elf-rhgbf” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“elf-rhgbf”,“namespace”:“tools6rwrl”} “reconcileID”=“1489e14b-bb48-4ca7-8472-2a6ca2ee2ad6”

                                          5，对比devops-config，kubesphere-config，kubesphere-secret

                                          6，疑惑的是kubesphere-devops-system/secrets/devops-jenkins中定义的jenkins-admin-password无法登录Jenkins，重启了Jenkins之后发现也登陆不了，最终用的是默认密码登录。