• DevOps

  • 4.1.2安装devops 集群 Agent 报错

pky 是同样的报错吗 ？看下 devops-jenkins pod 的状态呢；

  • pky 回复了此帖

      yudong 我看了是启动探针和就绪探针不行，我尝试延长探针等待时间，也不行。后来直接在yaml里删除探针，发现启动jenkins容器是没成功的：容器内部的日志到这里就结束了：

      我怎么都记得jenkins启动日志老长了，不是只有这么一点点啊

          pky 如果只有这点日志，jenkins 应该还没有启动；看下是不是 devops-jenkins 资源配置的太少了。

          • pky 回复了此帖

              yudong 命令执行到解压缩war包的时候很慢，长达几分钟，而探针检测机制直接让其重启，也就是说还没来得及对外提供服务就被探针重启了。我的建议是延长探针时间或者修改检测方式。

                  15 天 后

                  碰到了同样的问题，请问各路大神，如何处理？

                  重启 containerd， 重启 kubelet，删除 kube-proxy再生成，试试，我后来好像是这样解决了的。

                  pky 什么样的CI场景要解压缩war包？

                  • pky 回复了此帖

                      klj890

                      Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=70 -XX:MaxRAMPercentage=70 -Dhudson.slaves.NodeProvisioner.initialDelay=20 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Dhudson.model.LoadStatistics.clock=5000 -Dhudson.model.LoadStatistics.decay=0.2 -Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000 -Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true -Dhudson.plugins.git.GitStatus.NOTIFY_COMMIT_ACCESS_CONTROL=disabled -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=10000 -Djenkins.install.runSetupWizard=false -XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions

                      Running from: /usr/share/jenkins/jenkins.war

                      webroot: EnvVars.masterEnvVars.get(“JENKINS_HOME”)

                      2024-11-30 09:40:04.451+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @1046ms to org.eclipse.jetty.util.log.JavaUtilLog

                      2024-11-30 09:40:04.562+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file

                      2024-11-30 09:49:03.230+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath

                      2024-11-30 09:49:03.305+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.45.v20220203; built: 2022-02-03T09:14:34.105Z; git: 4a0c91c0be53805e3fcffdcdcc9587d5301863db; jvm 11.0.16+8

                      2024-11-30 09:49:03.703+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet

                      2024-11-30 09:49:03.747+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0

                      2024-11-30 09:49:03.748+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults

                      2024-11-30 09:49:03.749+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms

                      2024-11-30 09:49:04.474+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get(“JENKINS_HOME”)

                      2024-11-30 09:49:06.756+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@4acb2510{Jenkins v2.346.3,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}

                      2024-11-30 09:49:06.798+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@260e86a1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}

                      2024-11-30 09:49:06.799+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @543396ms

                      2024-11-30 09:49:06.800+0000 [id=23] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled

                      2024-11-30 09:49:08.801+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization

                      2024-11-30 09:58:36.286+0000 [id=28] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins_home/plugins/job-dsl/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness

                      2024-11-30 10:19:20.847+0000 [id=29] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins_home/plugins/node-iterator-api/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness

                      2024-11-30 10:19:26.289+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins

                      2024-11-30 10:19:36.544+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins

                      2024-11-30 10:19:36.565+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins

                      WARNING: An illegal reflective access operation has occurred

                      WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/jenkins_home/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)

                      WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1

                      WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations

                      WARNING: All illegal access operations will be denied in a future release

                      这是Jenkins启动的某段日志，你可以看到在2024-11-30 09:49:08.801到2024-11-30 09:58:36.286用了几十分钟初始化，这个时间太久了，如果探针时间过段就永远启动不了，所以我干脆删除探针，这也许是我的二手服务器性能拉跨，但是想来其他人的服务器应该也好不到哪里去，毕竟现在的服务器大多数是虚拟机的，我这个是英特尔至强E5 2698 V4的处理器，20核心40线程的。

                          7 天 后

                          我也遇到了同样的问题，请问有解决的了吗？

                          而且我的扩展中心的devops组件一直停在“安装中”的状态，没法继续任何操作了。

                          7 天 后
                          3 个月 后

                          • brian

                            • 发布 #18

                            破案了应该是这个原因
                            devops-jenkins 这个pod中jvm的参数设置如下
                            -XX:InitialRAMPercentage=70 -XX:MaxRAMPercentage=70 -Dhudson.slaves.NodeProvisioner.initialDelay=20 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Dhudson.model.LoadStatistics.clock=5000 -Dhudson.model.LoadStatistics.decay=0.2 -Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000 -Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true -Dhudson.plugins.git.GitStatus.NOTIFY_COMMIT_ACCESS_CONTROL=disabled -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=10000 -Djenkins.install.runSetupWizard=false -XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions

                            就是说启动的时候使用系统上限的70%内存，这个百分比的基数大概率是分配到的节点的最大内存，打比方节点内存32G这时候启动然后就会占用 32*0.7 22.4GB内存，此时pod配置limit是默认的6GB 直接被k8s OOM= =、所以配置12G也会蛋疼
                            至于解决方案就是修改这些配置啦