3.0 devops 踩坑记录

344089386

一直想用kubesphere的devops，因为可视化界面很友好，操作简单。但事不如我愿，一直遇到各种坑，现在分享下我遇到的坑，我是用的官方的demo，链接：https://v2-1.docs.kubesphere.io/docs/zh-CN/quick-start/devops-online。

第一个坑，多节点部署的情况下，devops流水线明明创建成功了，但是devops列表页面没有显示，再次创建同名字的 devops流水线，提示已存在。如下图：

这个问题原因是 ks-controller-manager 组件和 ks-jenkins 组件不在一个节点上，并且他们有通信的问题，解决思路有3个：
方法1：在部署的时候让这2个组件放到一个节点上（我没试过）。
方法2：解决通信问题（我还不清楚怎么解决）。
方法3：单节点部署（我就玩个demo，所以用的这个方法）
具体解决过程可以看：https://kubesphere.com.cn/forum/d/1931-3-0-devops

2.第二个坑，具体错误：

Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/demo-devopsqh6lr_demo1_master/deploy/dev-ol/devops-sample-svc.yaml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: Forbidden
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:391)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:379)
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.90.94/10.233.90.94:34394
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at hudson.FilePath.act(FilePath.java:1160)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
… 1 more
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: Forbidden
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedServiceWithHttpInfo(CoreV1Api.java:26889)
at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedService(CoreV1Api.java:26865)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.getCurrentResource(V1ResourceManager.java:388)
… 16 more
Api call failed with code 403, detailed message: {
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {
},
“status”: “Failure”,
“message”: "services \“ks-sample-dev\” is forbidden: User \“system:anonymous\” cannot get resource \“services\” in API group \“\” in the namespace \“kubesphere-sample-dev\”",
“reason”: “Forbidden”,
“details”: {
“name”: “ks-sample-dev”,
“kind”: “services”
},
“code”: 403
}
Kubernetes deployment ended with HasError

这个问题是 kubeconfig不对的问题，原因是3.0版本的bug，最后官方大佬给出了临时解决方法。

kubectl -n kubesphere-controls-system delete cm kubeconfig-[用户名]
kubectl -n kubesphere-system rollout restart deployment ks-controller-manager ks-apiserver

例：kubectl -n kubesphere-controls-system delete cm kubeconfig-admin
然后重建demo-kubeconfig，运行流水线，一次搞定。
单独说一下：Received fatal alert: bad_certificate，遇到这个问题其实原因是一个，就是 kubeconfig不对的问题。
解决过程参考：https://kubesphere.com.cn/forum/d/2591-spring-boot-received-fatal-alert-bad-certificate

其他问题请参考官方大佬文档：https://kubesphere.com.cn/forum/d/2408-kubesphere-devops-30

NIO

344089386 ,这里的用户名是创建这个凭证的用户名吗还是，任意一个啊

denticle

344089386 第一个坑我也遇到了，最终解决方法是在安装的时候将网络插件calico替换为Flannel之后就没问题了

Jeff

344089386 感谢反馈，这些问题我们在下个版本会全部解决，目前遇到的一些问题我们也提供了临时的解决办法

mycolo1983

😃 ,支持分享！

johnniang

linzea

johnniang 我使用这个脚本, 出了新问题,

这里提示我用户名密码, 是不是搞错了, 我这里用的kubeconfig, 这个用户名密码是什么呢, 我用 3.1.1 版本会好吗

johnniang

linzea 请提供你的完整的 Jenkinsfile，或者提供一个稳定复现的样例。

linzea

johnniang

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('clone code') {
      agent none
      steps {
        container('maven') {
          git(url: 'https://gitee.com/xxx/xxxx.git', branch: 'gray', credentialsId: 'giee-account', changelog: true, poll: false)
          sh 'ls -al'
        }
      }
    }
    stage('unit test') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean -Dmaven.test.skip=true'
        }
      }
    }
    stage('build & push') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean package -Dmaven.test.skip=true '
          sh 'ls ./$MODULE_NAME/target'
          sh 'docker build -t $MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER:latest -f ./$MODULE_NAME/Dockerfile ./$MODULE_NAME/'
          withCredentials([usernamePassword(credentialsId : "$DOCKER_CREDENTIAL_ID" ,usernameVariable : 'DOCKER_USERNAME' ,passwordVariable : 'DOCKER_PASSWORD' ,)]) {
            sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
            sh 'docker tag $MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER:latest $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME-$BRANCH_NAME:$MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER '
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME-$BRANCH_NAME:$MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER '
            echo '删除本地镜像'
            sh 'docker rmi $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME-$BRANCH_NAME:$MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER'
            sh 'docker rmi $MODULE_NAME-$BRANCH_NAME-$BUILD_NUMBER:latest'
          }
        }
      }
    }
    stage('deploy to gray') {
         steps {
             container ('maven') {
                  withCredentials([ kubeconfigFile( credentialsId: "$KUBECONFIG_CREDENTIAL_ID", variable: 'KUBECONFIG') ]) {
                      sh 'envsubst < devops/gray/deploy/deployment.yaml | kubectl apply -f -'
                  }
             }
         }
    }
    stage('deploy to production') {
      steps {
        input(id: 'deploy-to-production', message: 'deploy to production?', submitter: 'admin,ws-manager')
        kubernetesDeploy(configs: 'devops/prod/deploy-prod.yml', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID")
      }
    }
  }
  environment {
    DOCKER_CREDENTIAL_ID = 'docker-hub'
    KUBECONFIG_CREDENTIAL_ID = 'kubeconfig'
    REGISTRY = 'docker.io'
    DOCKERHUB_NAMESPACE = 'docker_namespace'
    APP_NAME = 'object-cloud'
    MODULE_NAME = 'object-job'
    BRANCH_NAME = 'gray'
  }
  parameters {
    string(name: 'TAG_NAME', defaultValue: 'gray', description: '')
  }
}

这是我完整的jenkinsfile , 我使用的是 aws eks, 在前面有提到, 都正常按照官方demo 和视频操作
https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-eks/#%e5%9c%a8-eks-%e4%b8%8a%e5%ae%89%e8%a3%85-kubesphere
按照这个教程安装
我可以讲我的测试环境提供给你, 请给个联系方式

linzea

johnniang

你好, 大佬, 我的 devops-jenkins 无法启动了, 执行流水线的时候突然卡了下, 然后后台页面访问就找不到页面, 等一下恢复后, 流水线进不去, 查看后发现。devops-jenkins 卡在删除中, 然后变成无法调度 , 这个我要怎么恢复它呢, 我试着删除后手动创建也不行