3 个月 后

【 可以在节点上直接执行 kubectl apply 进行验证。怀疑凭证有问题时，可以直接使用 admin 的凭证，进行确认】

通过admin的凭证可以发布成功，通过web控台创建的kubeconfig凭证 403，这是什么原因？

    bb99gh

    贴一下具体报错信息，这个接口有 bug，可以再生成一个 kubecofig 试试，下个版本修复了这个问题。

        shaowenchen

        Starting Kubernetes deployment
        Loading configuration: /home/jenkins/agent/workspace/s4h665_jenkinsfile-in-scm_master/deploy/dev-ol/devops-sample-svc.yaml
        ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
        hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
        at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
        at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:391)
        at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:379)
        at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
        at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
        at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.100.0.120/10.100.0.120:48812
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
        at hudson.remoting.Channel.call(Channel.java:957)
        at hudson.FilePath.act(FilePath.java:1160)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
        at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
        at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
        at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
        at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
        at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        … 1 more
        Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: Forbidden
        at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
        at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
        at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedServiceWithHttpInfo(CoreV1Api.java:26889)
        at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedService(CoreV1Api.java:26865)
        at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:388)
        … 16 more
        **Api call failed with code 403, detailed message: {
        “kind”: “Status”,
        “apiVersion”: “v1”,
        “metadata”: {

        },
        “status”: “Failure”,
        “message”: “services \“ks-sample-dev\” is forbidden: User \“project-regular\” cannot get resource \“services\” in API group \“\” in the namespace \“kubesphere-sample-dev\””,
        “reason”: “Forbidden”,
        “details”: {
        “name”: “ks-sample-dev”,
        “kind”: “services”
        },
        “code”: 403
        }
        Kubernetes deployment ended with HasError**

            shaowenchen
            通过创建凭证（kubeconfig）默认生成的无效，将/etc/kubernetes/admin.conf中的内容覆盖默认生成的，部署成功！
            如果要使project-regular生效，目前是不是要通过kubectl配置？有相关介绍没？新手感谢！！！

                bb99gh

                创建的凭证得有目标 ns 的权限。这里regular 用户没有 kubesphere-sample-dev 的权限。

                  5 天 后

                  流水线最后一步报错，前面ci push镜像都成功了。集群，两台hapoxy，3台master，3台worker。yaml文件在一台worke上apply是成功的。这个提示是什么意思？
                  Starting Kubernetes deployment
                  ERROR: ERROR: java.lang.NullPointerException
                  hudson.remoting.ProxyException: java.lang.NullPointerException
                  at io.kubernetes.client.util.KubeConfig.loadKubeConfig(KubeConfig.java:85)
                  at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.<init>(KubernetesClientWrapper.java:169)
                  at com.microsoft.jenkins.kubernetes.KubernetesDeployContext$ClientWrapperFactoryImpl.buildClient(KubernetesDeployContext.java:482)
                  at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:143)
                  at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
                  at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
                  at hudson.remoting.UserRequest.perform(UserRequest.java:212)
                  at hudson.remoting.UserRequest.perform(UserRequest.java:54)
                  at hudson.remoting.Request$2.run(Request.java:369)
                  at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
                  at java.lang.Thread.run(Thread.java:748)
                  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.85.135/10.233.85.135:43122
                  at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
                  at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                  at hudson.remoting.Channel.call(Channel.java:957)
                  at hudson.FilePath.act(FilePath.java:1160)
                  at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
                  at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
                  at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
                  at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
                  at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
                  at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
                  at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
                  at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
                  at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
                  at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  … 1 more
                  Kubernetes deployment ended with HasError

                    leonanor
                    寄几回复寄几。凭证有问题，yaml文件nodeport端口超限。凭证问题怎么解决呢？楼主没有说具体怎么操作。我来补充一下，用admin登录kubesphere，创建一个类型是kubeconfig的凭证，然后登出，再用流水线用户登进去启动流水线就可以了。

                        leonanor 其实用project-admin邀请project-regular用户进入devops项目就有权限了

                          通过自定义podTemplate升级maven和jdk，同时按照文档修改了Jenkins配置文件，也登录Jenkins重新加载了

                          1. maven构建问题解决了
                          2. 但是docker打包镜像时提示无docker这个命令
                          3. docker build -f Dockerfile -t 192.168.3.142:8082/library/finai-demo-devops-api:SNAPSHOT-master-9 .
                            /home/jenkins/agent/workspace/ch_finai-demo-devops-cicd_master@tmp/durable-6b21eb3a/script.sh: 1: /home/jenkins/agent/workspace/ch_finai-demo-devops-cicd_master@tmp/durable-6b21eb3a/script.sh: docker: not found
                            script returned exit code 127

                          这种问题一般要怎么排查

                          打包的dockerfile如下

                          FROM kubesphere/builder-base:v2.1.0
                          MAINTAINER xxx

                          ENV LANG=en_US.UTF-8
                          ENV LANGUAGE=en_US:en
                          ENV LC_ALL=en_US.UTF-8

                          RUN yum remove -y java-1.8.0-openjdk \
                          && mkdir /usr/local/java \
                          && mkdir /opt/ant

                          WORKDIR /home/jenkins

                          ADD jdk-11.0.8_linux-x64_bin.tar.gz /usr/local/java
                          ENV JAVA_HOME /usr/local/java/jdk-11.0.8
                          ENV PATH $JAVA_HOME/bin:$PATH

                          ADD apache-maven-3.6.1-bin.tar.gz /opt
                          ENV MAVEN_VERSION=3.6.1
                          ENV M2_HOME=/opt/apache-maven-3.6.1
                          ENV maven.home=/opt/apache-maven-3.6.1
                          ENV M2=/opt/apache-maven-3.6.1/bin
                          ENV PATH $M2:$PATH

                          ADD apache-ant-1.10.9-bin.tar.gz /opt/ant
                          ENV ANT_VERSION=1.10.9
                          ENV ANT_HOME=/opt/ant
                          ENV PATH $ANT_HOME/bin:$PATH

                          CMD mvn -version

                            bb99gh 你的环境是什么，在节点上执行 docker info 有输出么

                              bb99gh

                              1，首先得用 label 选中合适的 agent
                              2，在 agent 中需要挂载 docker.socket
                              3，执行 docker 时，需要在 container 包裹下执行(默认使用 jnlp 容器没有 docker 命令)，该容器需要有 docker 命令。

                              可能是 3 导致的，需要检查一下

                                  shaowenchen 确实是第3点导致，没用container 包裹下执行，感谢！！！

                                  自定义的pod镜像1个多G，有什么好的方法可以瘦身~

                                    刚才跑文档例子，发现无论admin还是project-admin生成Kubeconfig凭证都无法跑通。建议生成kubeconfig的时候还是手动粘贴进去比较好

                                    1 个月 后
                                    13 天 后

                                    jenkins点了升级后就各种完蛋草了。。。 不能单独重置ks-jenkins么

                                    shaowenchen 老哥，想请问一下，ci编译的maven镜像里面能打一个kubectl命令嘛？我们这边想直接在ci的时候执行 kubeclt set image把集群内的镜像修改成最新的，开发的项目里就不用写deployment.yml了

                                        jenkins 插件 kubernetesDeploy 有bug, 创建serviceaccount,role, rolebind， secrets会重复创建, 使用kubectl apply -f 就不会重复生成， 而且secrets里面的uuid都是同一个