KubeSphere DevOps 3.0 流水线运维指南

shaowenchen

Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/s4h665_jenkinsfile-in-scm_master/deploy/dev-ol/devops-sample-svc.yaml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:391)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:379)
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.100.0.120/10.100.0.120:48812
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at hudson.FilePath.act(FilePath.java:1160)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
… 1 more
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: Forbidden
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedServiceWithHttpInfo(CoreV1Api.java:26889)
at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedService(CoreV1Api.java:26865)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:388)
… 16 more
**Api call failed with code 403, detailed message: {
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {

},
“status”: “Failure”,
“message”: “services \“ks-sample-dev\” is forbidden: User \“project-regular\” cannot get resource \“services\” in API group \“\” in the namespace \“kubesphere-sample-dev\””,
“reason”: “Forbidden”,
“details”: {
“name”: “ks-sample-dev”,
“kind”: “services”
},
“code”: 403
}
Kubernetes deployment ended with HasError**

shaowenchen
通过创建凭证（kubeconfig）默认生成的无效，将/etc/kubernetes/admin.conf中的内容覆盖默认生成的，部署成功！
如果要使project-regular生效，目前是不是要通过kubectl配置？有相关介绍没？新手感谢！！！

流水线最后一步报错，前面ci push镜像都成功了。集群，两台hapoxy，3台master，3台worker。yaml文件在一台worke上apply是成功的。这个提示是什么意思？
Starting Kubernetes deployment
ERROR: ERROR: java.lang.NullPointerException
hudson.remoting.ProxyException: java.lang.NullPointerException
at io.kubernetes.client.util.KubeConfig.loadKubeConfig(KubeConfig.java:85)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.<init>(KubernetesClientWrapper.java:169)
at com.microsoft.jenkins.kubernetes.KubernetesDeployContext$ClientWrapperFactoryImpl.buildClient(KubernetesDeployContext.java:482)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:143)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.85.135/10.233.85.135:43122
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at hudson.FilePath.act(FilePath.java:1160)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
… 1 more
Kubernetes deployment ended with HasError

leonanor
寄几回复寄几。凭证有问题，yaml文件nodeport端口超限。凭证问题怎么解决呢？楼主没有说具体怎么操作。我来补充一下，用admin登录kubesphere，创建一个类型是kubeconfig的凭证，然后登出，再用流水线用户登进去启动流水线就可以了。

leonanor 其实用project-admin邀请project-regular用户进入devops项目就有权限了

通过自定义podTemplate升级maven和jdk，同时按照文档修改了Jenkins配置文件，也登录Jenkins重新加载了

1. maven构建问题解决了
2. 但是docker打包镜像时提示无docker这个命令
3. docker build -f Dockerfile -t 192.168.3.142:8082/library/finai-demo-devops-api:SNAPSHOT-master-9 .
   /home/jenkins/agent/workspace/ch_finai-demo-devops-cicd_master@tmp/durable-6b21eb3a/script.sh: 1: /home/jenkins/agent/workspace/ch_finai-demo-devops-cicd_master@tmp/durable-6b21eb3a/script.sh: docker: not found
   script returned exit code 127

这种问题一般要怎么排查

打包的dockerfile如下

FROM kubesphere/builder-base:v2.1.0
MAINTAINER xxx

ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8

RUN yum remove -y java-1.8.0-openjdk \
&& mkdir /usr/local/java \
&& mkdir /opt/ant

WORKDIR /home/jenkins

ADD jdk-11.0.8_linux-x64_bin.tar.gz /usr/local/java
ENV JAVA_HOME /usr/local/java/jdk-11.0.8
ENV PATH $JAVA_HOME/bin:$PATH

ADD apache-maven-3.6.1-bin.tar.gz /opt
ENV MAVEN_VERSION=3.6.1
ENV M2_HOME=/opt/apache-maven-3.6.1
ENV maven.home=/opt/apache-maven-3.6.1
ENV M2=/opt/apache-maven-3.6.1/bin
ENV PATH $M2:$PATH

ADD apache-ant-1.10.9-bin.tar.gz /opt/ant
ENV ANT_VERSION=1.10.9
ENV ANT_HOME=/opt/ant
ENV PATH $ANT_HOME/bin:$PATH

CMD mvn -version

bb99gh 你的环境是什么，在节点上执行 docker info 有输出么

刚才跑文档例子，发现无论admin还是project-admin生成Kubeconfig凭证都无法跑通。建议生成kubeconfig的时候还是手动粘贴进去比较好

大佬 轮询构建 有没有办法？
https://kubesphere.com.cn/forum/d/3375-svn-scm/2

jenkins点了升级后就各种完蛋草了。。。 不能单独重置ks-jenkins么

shaowenchen 老哥，想请问一下，ci编译的maven镜像里面能打一个kubectl命令嘛？我们这边想直接在ci的时候执行 kubeclt set image把集群内的镜像修改成最新的，开发的项目里就不用写deployment.yml了

jenkins 插件 kubernetesDeploy 有bug, 创建serviceaccount,role, rolebind， secrets会重复创建, 使用kubectl apply -f 就不会重复生成， 而且secrets里面的uuid都是同一个

shaowenchen 是的，新创建了一个secrets出来， 我每次构建都会创建，今天才发现, secrets下面新建了上千个

zqh 没有的吧，不可能挂载这个文件的， 你没有指定容器 选择maven节点