• DevOps

  • 流水线加载超慢问题

KubeSphere版本信息
v3.1.1

流水线运行一段时间后打开非常的慢

我们有4个devops工程 大概20多条流水线，最多的构建记录大概200左右

jenkins所在节点 6C16G

jenkins配置 cpu 0.1～♾️ 内存1500～♾️

请帮忙核实 看看如何优化

加载时间截图

E1116 18:55:41.382178 1 pure_request.go:62] Get "http://ks-jenkins.kubesphere-devops-system.svc/blue/rest/organizations/jenkins/pipelines/servicegn2pb/pipelines/health-statistics-service/runs/?limit=10000&start=0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

E1116 18:55:41.382221 1 pipeline.go:215] Get "http://ks-jenkins.kubesphere-devops-system.svc/blue/rest/organizations/jenkins/pipelines/servicegn2pb/pipelines/health-statistics-service/runs/?limit=10000&start=0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

E1116 18:55:41.382242 1 devops.go:432] Get "http://ks-jenkins.kubesphere-devops-system.svc/blue/rest/organizations/jenkins/pipelines/servicegn2pb/pipelines/health-statistics-service/runs/?limit=10000&start=0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

E1116 18:55:41.382256 1 devops.go:900] Get "http://ks-jenkins.kubesphere-devops-system.svc/blue/rest/organizations/jenkins/pipelines/servicegn2pb/pipelines/health-statistics-service/runs/?limit=10000&start=0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I1116 18:55:41.382301 1 apiserver.go:613] 10.240.95.86 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/servicegn2pb/pipelines/health-statistics-service/runs/?limit=10000&start=0 HTTP/1.1" 500 245 30000ms

E1116 19:02:07.343802 1 clusterclient.go:157] Failed to get client config, clientcmd.errConfigurationInvalid{(*errors.errorString)(0xc000446be0)}

W1116 19:07:08.448980 1 jwt_token.go:52] token is expired by 38m43s

W1116 19:07:13.405400 1 jwt_token.go:52] token is expired by 38m48s

W1116 19:07:22.448546 1 jwt_token.go:52] token is expired by 38m57s

W1116 19:07:39.398789 1 jwt_token.go:52] token is expired by 39m14s

E1116 19:12:07.347670 1 clusterclient.go:157] Failed to get client config, clientcmd.errConfigurationInvalid{(*errors.errorString)(0xc000446be0)}

W1116 19:13:12.557143 1 jwt_token.go:52] token is expired by 24m32s

W1116 19:13:17.508819 1 jwt_token.go:52] token is expired by 24m37s

W1116 19:13:26.474976 1 jwt_token.go:52] token is expired by 24m46s

W1116 19:13:43.411308 1 jwt_token.go:52] token is expired by 25m3s

E1116 19:22:07.348547 1 clusterclient.go:157] Failed to get client config, clientcmd.errConfigurationInvalid{(*errors.errorString)(0xc000446be0)}

W1116 19:25:42.406661 1 jwt_token.go:52] token is expired by 1h58m59s

W1116 19:25:42.406663 1 jwt_token.go:52] token is expired by 1h58m59s

E1116 19:26:50.045713 1 utils.go:69] context deadline exceeded (Client.Timeout or context cancellation while reading body)

E1116 19:26:50.045742 1 pipeline.go:223] unexpected end of JSON input

E1116 19:26:50.045749 1 devops.go:432] unexpected end of JSON input

E1116 19:26:50.045755 1 devops.go:900] unexpected end of JSON input

I1116 19:26:50.045801 1 apiserver.go:613] 10.240.95.86 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/front-siter852f/pipelines/cloud-service-manage/runs/?limit=10000&start=0 HTTP/1.1" 500 28 30001ms

E1116 19:27:34.534422 1 utils.go:69] context deadline exceeded (Client.Timeout or context cancellation while reading body)

E1116 19:27:34.534453 1 pipeline.go:223] unexpected end of JSON input

E1116 19:27:34.534460 1 devops.go:432] unexpected end of JSON input

E1116 19:27:34.534464 1 devops.go:900] unexpected end of JSON input

I1116 19:27:34.534495 1 apiserver.go:613] 10.240.95.86 - "GET /kapis/devops.kubesphere.io/v1alpha2/devops/front-siter852f/pipelines/cloud-service-manage/runs/?limit=10000&start=0 HTTP/1.1" 500 28 30000ms

E1116 19:32:07.348168 1 clusterclient.go:157] Failed to get client config, clientcmd.errConfigurationInvalid{(*errors.errorString)(0xc000446be0)}

E1116 19:41:56.996095 1 utils.go:69] context deadline exceeded (Client.Timeout or context cancellation while reading body)

发现默认jvm分配内存才512 修改后观察观察 同时也修改了cpu预留

丢弃掉旧的构建历史可以显著提升性能 但似乎多分支流水这里没办法设置 有没有大佬可以指导一下

  • wade 回复了此帖

    3.2这个排序有问题吧？ 另外升级后jenkins admin的ldap账号密码好像会出问题

    johnniang

      Java 程序普遍比较吃资源（内存、CPU），建议在可观测性那边把 Jenkins 的监控打开，看看 Jenkins 的资源消耗情况，并酌情考虑调整 limit。

        Rick 3.1的时候我把默认的jenkins 的limit及jvm参数调整后 性能好了很多 昨天晚上升级3.2后感觉还有很多其他问题

            johnniang

            流水线有询问窗口的时候 点确认后也不刷新 可能跟其他几个人的反馈类似

                johnniang

                英语不太好 我就贴这里吧

                点proceed无法审核通过

                流水线代码 input(id: ‘deploy-to-prod’, message: ‘deploy to prod?’)

                看controller报错如下

                E1122 03:23:32.133518 1 pipelinerun_controller.go:165] pipelinerun-controller “msg”=“unable to update PipelineRun labels and annotations.” “error”=“rpc error: code = Unavailable desc = transport is closing” “Pipeline”=“easst-call-service” “PipelineRun”={“Namespace”:“servicegn2pb”,“Name”:“easst-call-service-2xzfd”} “namespace”=“servicegn2pb”

                E1122 03:23:32.133574 1 controller.go:246] controller “msg”=“Reconciler error” “error”=“rpc error: code = Unavailable desc = transport is closing” “controller”=“pipelinerun” “name”=“easst-call-service-2xzfd” “namespace”=“servicegn2pb” “reconcilerGroup”=“devops.kubesphere.io” “reconcilerKind”=“PipelineRun”

                3.2.1的devops分支 nightly build版本如何安装 等不了正式发布了

                    hysoft 目前的 nightly 是针对 master 分支打的包，不建议直接使用，里面会包含一些不稳定的新功能。

                    如果想提前解决前端的小问题，可以临时替换一下 ks-console 镜像为：kubespheredev/ks-console:release-3.2。

                    注意：kubespheredev/ks-console:release-3.2 是一个动态的镜像，类似于 latest tag。

                      3 个月 后

                      3.2.1 devops 安装的时候jvm 内存分配 2048M发现 存活，保活探针超时，无限重启，将 jvm 内存设到1000m以下devops 组件正常。不知啥原因造成的。

                      hysoft 这个可以在pipeline 里通过代码设置的保留最新的10个

                      options{

                        buildDiscarder(logRotator(numToKeepStr: '10'))

                      }