集群是1master 4node,先是用kubekey,始终卡在节点检测那,后用ks-installer升级,ks-installer有一个这样的报错
`
Task ‘devops’ failed:


{
“counter”: 227,
“created”: “2021-05-14T00:41:20.403203”,
“end_line”: 219,
“event”: “runner_on_failed”,
“event_data”: {
“duration”: 1.002557,
“end”: “2021-05-14T00:41:20.403008”,
“event_loop”: null,
“host”: “localhost”,
“ignore_errors”: null,
“play”: “localhost”,
“play_pattern”: “localhost”,
“play_uuid”: “a6ae04f7-4cda-caab-3bd7-000000000005”,
“playbook”: “/kubesphere/playbooks/devops.yaml”,
“playbook_uuid”: “ddc514fb-4b74-4795-8afa-76d9a09a68e6”,
“remote_addr”: “127.0.0.1”,
“res”: {
ansible_no_log”: false,
“changed”: true,
“cmd”: “/usr/local/bin/helm upgrade –install ks-jenkins /kubesphere/kubesphere/jenkins/jenkins -f /kubesphere/kubesphere/jenkins/custom-values-jenkins.yaml –namespace kubesphere-devops-system\n”,
“delta”: “0:00:00.796131”,
“end”: “2021-05-14 08:41:20.379328”,
“invocation”: {
“module_args”: {
raw_params”: “/usr/local/bin/helm upgrade –install ks-jenkins /kubesphere/kubesphere/jenkins/jenkins -f /kubesphere/kubesphere/jenkins/custom-values-jenkins.yaml –namespace kubesphere-devops-system\n”,
“_uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2021-05-14 08:41:19.583197”,
“stderr”: "coalesce.go:199: warning: destination for InitScripts is a table. Ignoring non-table value <nil>\nError: UPGRADE FAILED: cannot patch \“ks-jenkins\” with kind PersistentVolumeClaim: PersistentVolumeClaim \“ks-jenkins\” is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims",
“stderr_lines”: [
“coalesce.go:199: warning: destination for InitScripts is a table. Ignoring non-table value <nil>”,
"Error: UPGRADE FAILED: cannot patch \“ks-jenkins\” with kind PersistentVolumeClaim: PersistentVolumeClaim \“ks-jenkins\” is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims"
],
“stdout”: "",
“stdout_lines”: []
},
“role”: “ks-devops/jenkins”,
“start”: “2021-05-14T00:41:19.400451”,
“task”: “Jenkins | Deploy Jenkins”,
“task_action”: “command”,
“task_args”: "",
“task_path”: “/kubesphere/installer/roles/ks-devops/jenkins/tasks/main.yaml:66”,
“task_uuid”: “a6ae04f7-4cda-caab-3bd7-000000000049”,
“uuid”: “a2da7d52-ebc7-4022-8857-5cbf32412e12”
},
“parent_uuid”: “a6ae04f7-4cda-caab-3bd7-000000000049”,
“pid”: 7079,
“runner_ident”: “devops”,
“start_line”: 218,
“stdout”: "fatal: [localhost]: FAILED! => {\“changed\”: true, \“cmd\”: \“/usr/local/bin/helm upgrade –install ks-jenkins /kubesphere/kubesphere/jenkins/jenkins -f /kubesphere/kubesphere/jenkins/custom-values-jenkins.yaml –namespace kubesphere-devops-system\n\”, \“delta\”: \“0:00:00.796131\”, \“end\”: \“2021-05-14 08:41:20.379328\”, \“msg\”: \“non-zero return code\”, \“rc\”: 1, \“start\”: \“2021-05-14 08:41:19.583197\”, \“stderr\”: \"coalesce.go:199: warning: destination for InitScripts is a table. Ignoring non-table value <nil>\nError: UPGRADE FAILED: cannot patch \\“ks-jenkins\\” with kind PersistentVolumeClaim: PersistentVolumeClaim \\“ks-jenkins\\” is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims\", \“stderr_lines\”: [\“coalesce.go:199: warning: destination for InitScripts is a table. Ignoring non-table value <nil>\”, \"Error: UPGRADE FAILED: cannot patch \\“ks-jenkins\\” with kind PersistentVolumeClaim: PersistentVolumeClaim \\“ks-jenkins\\” is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims\"], \“stdout\”: \“\”, \“stdout_lines\”: []}",
“uuid”: “a2da7d52-ebc7-4022-8857-5cbf32412e12”
}


`
发现集群不能登录,于是apply了一个admin.yaml,手动生成admin用户,登录集群后发现企业空间都没了,只有一个默认的,用户也没有,集群有100多个项目,现在可以正常访问,正常运行流水线,就是集群上看不到,请看看应该如何解决

     invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

    hetao 升级失败了, 看看你的环境 是不是改过 jenkinsVolumeSize 或者是扩容过?

    kubectl -n kubesphere-devops-system get pvc

    把 jenkinsVolumeSize 改成一致的大小, 再升级一遍

      hongming

      # kubectl -n kubesphere-devops-system get pvc
      NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      ks-jenkins                Bound    pvc-8e692ce0-543a-4cd8-be5f-3f20dd696af2   8Gi        RWO            nfs-client     492d
      ks-sonarqube-postgresql   Bound    pvc-8b9d2592-0a96-4b03-a732-aacfcfab4068   8Gi        RWO            nfs-client     388d

      没有改过大小,再重新apply那两个文件吗?

        hetao

        kubectl -n kubesphere-system get cc ks-installer -o yaml | grep jenkinsVolumeSize

        看看配置的是多少,改成一致再升级

          hongming

          # kubectl -n kubesphere-system get cc ks-installer -o yaml | grep jenkinsVolumeSize
                {"apiVersion":"installer.kubesphere.io/v1alpha1","kind":"ClusterConfiguration","metadata":{"annotations":{},"labels":{"version":"v3.0.0"},"name":"ks-installer","namespace":"kubesphere-system"},"spec":{"alerting":{"enabled":true},"auditing":{"enabled":false},"authentication":{"jwtSecret":""},"common":{"es":{"elasticsearchDataVolumeSize":"20Gi","elasticsearchMasterVolumeSize":"4Gi","elkPrefix":"logstash","logMaxAge":7},"etcdVolumeSize":"20Gi","minioVolumeSize":"20Gi","mysqlVolumeSize":"20Gi","openldapVolumeSize":"2Gi","redisVolumSize":"2Gi"},"console":{"enableMultiLogin":true,"port":30880},"devops":{"enabled":true,"jenkinsJavaOpts_MaxRAM":"2g","jenkinsJavaOpts_Xms":"512m","jenkinsJavaOpts_Xmx":"512m","jenkinsMemoryLim":"2Gi","jenkinsMemoryReq":"1500Mi","jenkinsVolumeSize":"8Gi"},"etcd":{"endpointIps":"localhost","monitoring":false,"port":2379,"tlsEnable":true},"events":{"enabled":true,"ruler":{"enabled":true,"replicas":2}},"logging":{"enabled":false,"logsidecarReplicas":2},"metrics_server":{"enabled":false},"monitoring":{"prometheusMemoryRequest":"400Mi","prometheusVolumeSize":"20Gi"},"multicluster":{"clusterRole":"none"},"networkpolicy":{"enabled":false},"notification":{"enabled":false},"openpitrix":{"enabled":false},"persistence":{"storageClass":""},"servicemesh":{"enabled":false}}}
              jenkinsVolumeSize: 8Gi

            hongming

            devops_enabled: true
            jenkins_memory_lim: 8Gi
            jenkins_memory_req: 4Gi
            jenkins_volume_size: 8Gi
            jenkinsJavaOpts_Xms: 3g
            jenkinsJavaOpts_Xmx: 6g
            jenkinsJavaOpts_MaxRAM: 8g
            sonarqube_enabled: false
            #sonar_server_url: SHOULD_BE_REPLACED
            #sonar_server_token: SHOULD_BE_REPLACED

            这个是2.1.1的common.yaml文件,是这两个大小要配置成一样的吗?

            hetao

            是不是点过给 jenkins 的这个 pvc 扩容呀,用下边命令检查下这个pvc的内容,是不是里边的 status.capacity.storagespec.resources.request.storage 的值不匹配呀?

            kubectl -n kubesphere-devops-system get pvc ks-jenkins -o yaml

              Cauchy

              # kubectl -n kubesphere-devops-system get pvc ks-jenkins -o yaml
              apiVersion: v1
              kind: PersistentVolumeClaim
              metadata:
                annotations:
                  pv.kubernetes.io/bind-completed: "yes"
                  pv.kubernetes.io/bound-by-controller: "yes"
                  volume.beta.kubernetes.io/storage-provisioner: cluster.local/nfs-client-nfs-client-provisioner
                creationTimestamp: "2020-01-07T02:40:49Z"
                finalizers:
                - kubernetes.io/pvc-protection
                labels:
                  app: ks-jenkins
                  chart: jenkins-0.19.0
                  heritage: Tiller
                  release: ks-jenkins
                name: ks-jenkins
                namespace: kubesphere-devops-system
                resourceVersion: "6918"
                selfLink: /api/v1/namespaces/kubesphere-devops-system/persistentvolumeclaims/ks-jenkins
                uid: 8e692ce0-543a-4cd8-be5f-3f20dd696af2
              spec:
                accessModes:
                - ReadWriteOnce
                resources:
                  requests:
                    storage: 8Gi
                storageClassName: nfs-client
                volumeMode: Filesystem
                volumeName: pvc-8e692ce0-543a-4cd8-be5f-3f20dd696af2
              status:
                accessModes:
                - ReadWriteOnce
                capacity:
                  storage: 8Gi
                phase: Bound

              没有扩容过,就前段时间更新过一次k8s证书。

                hetao

                进到ks-installer里检查下/kubesphere/kubesphere/jenkins/custom-values-jenkins.yaml中的Persistence.Size字段跟pvc中的resources.requests对比下是否一致,主要看下大小、单位以及是否有多余的空格。

                如果一致的话,直接在ks-installer中执行

                /usr/local/bin/helm upgrade –install ks-jenkins /kubesphere/kubesphere/jenkins/jenkins -f /kubesphere/kubesphere/jenkins/custom-values-jenkins.yaml –namespace kubesphere-devops-system

                如果还是报那个错误的话,可能需要删除旧的pvc重新创建下了,注意:重新创建pvc时,对应pv中的persistentVolumeReclaimPolicy字段修改为Retain ,挂载pvc的服务副本调为0