• DevOps
  • devops流水线未运行

yudong

碰到类似的问题:
1,开始发现流水线 “未运行”,查看devops-jenkins日志:

023-08-21 03:07:25.422+0000 [id=20295] WARNING i.k.j.d.a.KubesphereApiTokenAuthenticator#authenticate: API token matched for user liding2 but the impersonation failed

org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.

at org.acegisecurity.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:126)

at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:1320)

at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:1273)

at org.acegisecurity.userdetails.UserDetailsService.lambda$toSpring$1(UserDetailsService.java:52)

Caused: org.springframework.security.core.userdetails.UsernameNotFoundException

at org.acegisecurity.userdetails.UsernameNotFoundException.toSpring(UsernameNotFoundException.java:51)

at org.acegisecurity.userdetails.UsernameNotFoundException.toSpring(UsernameNotFoundException.java:34)

at org.acegisecurity.userdetails.UserDetailsService.lambda$toSpring$1(UserDetailsService.java:54)

at jenkins.security.ImpersonatingUserDetailsService2.loadUserByUsername(ImpersonatingUserDetailsService2.java:29)

at hudson.model.User.getUserDetailsForImpersonation2(User.java:406)

at hudson.model.User.getUserDetailsForImpersonation(User.java:429)

Caused: org.acegisecurity.userdetails.UsernameNotFoundException: org.springframework.security.core.userdetails.UsernameNotFoundException: org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.; nested exception is org.springframework.security.core.userdetails.UsernameNotFoundException: org.acegisecurity.userdetails.UsernameNotFoundException: User liding2 not found in directory.

2,按照你前面提的三个步骤操作了,流水线又成功运行了,一会儿后又发现流水线"未运行"
3,检查Jenkins的role-strategy/assign-roles


4,查看devops-controller日志
E0821 03:07:25.426300 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“elf” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“elf-gqmc7”} “namespace”=“tools6rwrl” “pipeline”=“elf”

E0821 03:07:25.426407 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“elf-gqmc7” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“elf-gqmc7”,“namespace”:“tools6rwrl”} “reconcileID”=“6d87a5d2-3543-40d7-94d3-ec3d14fd0bef”

<nil>

<nil>

E0821 03:09:32.357775 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

E0821 03:09:32.357826 1 devopscredential_controller.go:167] error syncing ‘cssgzndq/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

E0821 03:09:32.357834 1 devopscredential_controller.go:168] error syncing ‘cssgzndq/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

E0821 03:09:32.428121 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

E0821 03:09:32.428175 1 devopscredential_controller.go:167] error syncing ‘cssgzndq/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

E0821 03:09:32.428183 1 devopscredential_controller.go:168] error syncing ‘cssgzndq/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

E0821 03:09:32.449208 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

E0821 03:09:32.449252 1 devopscredential_controller.go:167] error syncing ‘uosjmdbk/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

E0821 03:09:32.449260 1 devopscredential_controller.go:168] error syncing ‘uosjmdbk/gitlab-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

E0821 03:09:32.461410 1 devopscredential_controller.go:294] unexpected error type: not found resources, should be *restful.ServiceError

E0821 03:09:32.461456 1 devopscredential_controller.go:167] error syncing ‘uosjmdbk/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuingcould not reconcile devopsProject

E0821 03:09:32.461465 1 devopscredential_controller.go:168] error syncing ‘uosjmdbk/k8s-token’: failed to remove devops credential finalizer due to bad communication with Jenkins, requeuing

<nil>

<nil>

<nil>

<nil>

E0821 03:14:01.416187 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“css” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“css-vc4d8”} “namespace”=“tools6rwrl” “pipeline”=“css”

E0821 03:14:01.416736 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“css-vc4d8” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“css-vc4d8”,“namespace”:“tools6rwrl”} “reconcileID”=“7917f670-219b-4738-b3fa-311dedcd39fe”

E0821 03:14:25.682510 1 pipelinerun_controller.go:205] pipelinerun-controller “msg”=“unable to run pipeline” “error”=“unexpected status code: 500” “Pipeline”=“elf” “PipelineRun”={“Namespace”:“tools6rwrl”,“Name”:“elf-rhgbf”} “namespace”=“tools6rwrl” “pipeline”=“elf”

E0821 03:14:25.682684 1 controller.go:326] “msg”=“Reconciler error” “error”=“unexpected status code: 500” “controller”=“pipelinerun” “controllerGroup”=“devops.kubesphere.io” “controllerKind”=“PipelineRun” “name”=“elf-rhgbf” “namespace”=“tools6rwrl” “pipelineRun”={“name”:“elf-rhgbf”,“namespace”:“tools6rwrl”} “reconcileID”=“1489e14b-bb48-4ca7-8472-2a6ca2ee2ad6”

5,对比devops-config,kubesphere-config,kubesphere-secret

6,疑惑的是kubesphere-devops-system/secrets/devops-jenkins中定义的jenkins-admin-password无法登录Jenkins,重启了Jenkins之后发现也登陆不了,最终用的是默认密码登录。

    BombLing 感谢反馈,请问环境是 Kubesphere 哪个版本 ?然后麻烦对比 Kubesphere-config 和 devops-config 里的 jwtSecret 是不是一样的;

      yudong

      1,对比Kubesphere-config 和 devops-config 里的 jwtSecret 是一样的;

      2,Kubesphere是3.4.0,从3.2.1升级的;
      3,回顾时间线:应该devops项目中删除了流水线,或者删除了devops项目,导致拥有该项目的权限的用户出现“未运行”

      4,刚刚发现有devops项目未能成功删除,原有是删除secrets时卡住,使用patch才能删除;

        BombLing 哦 如果 jwtSecret 和 password(截图中的)是一样的,并且重启过 ks-controller-manager 服务,那就没有问题了。
        对删除 devops 项目或者流水线卡住问题,可以在 k8s 节点终端上手动操作删除;

          yudong

          对比了kubesphere-secret中的secret,kubesphere-config中的jwtSecret,devops-config中的 jwtSecret三者一致,也重启了ks-controller-manager,但是Jenkins的用户显示

          在什么情况下会触发更新Jenkins的role-strategy/assign-roles?

            BombLing 哦,同步用户信息在修改用户密码时会触发,您可以试着修改下用户密码。

              @yudong
              ks: 3.3.2
              k8s: 1.24.9
              现象是在多分支流水线中,经常会出现个别流水线一直未运行的情况,其它流水线都正常,重启相关组件都未解决。通过删除重新创建就可以重新运行。
              还有一个现象是流水线已经执行完成,jenkins中也已经结束了,但是ks上流水线的状态一直是运行中

                littlejiancc 可能是同时运行的流水线太对了,这些流水线在排队等待中;可以看下 kubesphere-devops-worker 空间下是否有多个 pod ?

                  13 天 后
                  8 天 后

                  yudong 您好,我是在jenkins安装插件后,流水线点击构建提示未运行,您说的这3步我不清楚怎么操作

                    runaway

                    kubectl get pod -n kubesphere-devops-system

                    看下所有 pod 都启动了吗,包括 devops-controller 和 jenkins

                    如果是,可以看下 devops-controller 中是否有什么报错。

                      chilianyi 感谢回复,我说明一下我这边的情况,k8s是腾讯云的TKE,在TKE上安装kubesphere,因为需要用到钉钉构建通知,我就找到jenkins的web界面,进去安装了一个dingtalk插件,安装后会重启jenkins,重启之后,所有项目就不能构建了。

                      1. 使用 devops-config 中这个 devops/password 替换 secret kubesphere-secret 中的 token;

                      2. 使用 devops-config 中这个 devops/password,替换 kubesphere-config 中的 devops/password;

                      3. 重启 Deployment ks-controller-manager ;

                      以上三步操作我后来也找到了但是执行后,还是点击构建流水线,流水线未运行

                      chilianyi pod 启动了,包括 devops-controller 和 jenkins

                      devops-controller日志中的提示:

                        chilianyi

                        操作系统信息
                        腾讯云:TencentOS Server 3.1(TK4) 8C/16G 3台

                        (腾讯云容器服务,创建集群时创建的)

                        Kubernetes版本信息
                        版本:1.22.5

                        容器运行时
                        containerd

                        KubeSphere版本信息
                        v3.3.2,通过 ks-installer 执行最小化部署

                          runaway 这种情况,是不是 jenkins 中有什么报错呢,看下 ks-jenkins 这个 pod 的 log 有没有什么报错信息。