• DevOps

  • AWS的ESK启用插件kubesphere Devops异常 namespaces "kubesphere-devops-worker" not found

操作系统信息
AWS EKS

Kubernetes版本信息
1.2.6

KubeSphere版本信息
v3.4.0

安装文档参照
https://kubesphere.io/zh/docs/v3.4/pluggable-components/devops/

错误信息
Error from server (NotFound): namespaces \“kubesphere-devops-worker\” not found

结果验证


图二无法添加，提交成功后看不到

启用DevOps插件安装异常日志：
PLAY RECAP *********************************************************************
localhost : ok=25 changed=19 unreachable=0 failed=0 skipped=22 rescued=0 ignored=0
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
Start installing devops

Waiting for all tasks to be completed …
task openpitrix status is successful (⅕)
task network status is successful (⅖)
task multicluster status is successful (⅗)
task devops status is failed (⅘)
task monitoring status is successful (5/5)

Collecting installation results …

Task ‘devops’ failed:

{
“counter”: 146,
“created”: “2023-09-19T13:14:59.160780”,
“end_line”: 175,
“event”: “runner_on_failed”,
“event_data”: {
“duration”: 1.398957,
“end”: “2023-09-19T13:14:59.160713”,
“event_loop”: null,
“host”: “localhost”,
“ignore_errors”: null,
“play”: “localhost”,
“play_pattern”: “localhost”,
“play_uuid”: “6ad67388-8da4-79c8-ac37-000000000005”,
“playbook”: “/kubesphere/playbooks/devops.yaml”,
“playbook_uuid”: “39def82b-a22f-4ec9-951b-12b901c8610e”,
“remote_addr”: “127.0.0.1”,
“res”: {
“changed”: true,
“msg”: “All items completed”,
“results”: [
{
“ansible_item_label”: “kubesphere-devops-worker”,
“ansible_no_log”: false,
“ansible_loop_var”: “item”,
“changed”: true,
“cmd”: “/usr/local/bin/kubectl label –overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n”,
“delta”: “0:00:00.181447”,
“end”: “2023-09-19 13:14:58.453348”,
“failed”: true,
“invocation”: {
“module_args”: {
“raw_params”: “/usr/local/bin/kubectl label –overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n”,
“uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“item”: “kubesphere-devops-worker”,
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2023-09-19 13:14:58.271901”,
“stderr”: "Error from server (NotFound): namespaces \“kubesphere-devops-worker\” not found",
“stderr_lines”: [
"Error from server (NotFound): namespaces \“kubesphere-devops-worker\” not found"
],
“stdout”: "",
“stdout_lines”: []
},
{
“ansible_item_label”: “argocd”,
“ansible_no_log”: false,
“ansible_loop_var”: “item”,
“changed”: true,
“cmd”: “/usr/local/bin/kubectl label –overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n”,
“delta”: “0:00:00.196361”,
“end”: “2023-09-19 13:14:59.146858”,
“failed”: false,
“invocation”: {
“module_args”: {
“raw_params”: “/usr/local/bin/kubectl label –overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n”,
“uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“item”: “argocd”,
“rc”: 0,
“start”: “2023-09-19 13:14:58.950497”,
“stderr”: "",
“stderr_lines”: [],
“stdout”: “namespace/argocd not labeled”,
“stdout_lines”: [
“namespace/argocd not labeled”
]
}
]
},
“resolved_action”: “shell”,
“role”: “ks-devops”,
“start”: “2023-09-19T13:14:57.761756”,
“task”: “ks-devops | Labelling devops namespaces as system workspace”,
“task_action”: “shell”,
“task_args”: "",
“task_path”: “/kubesphere/installer/roles/ks-devops/tasks/main.yaml:232”,
“task_uuid”: “6ad67388-8da4-79c8-ac37-000000000043”,
“uuid”: “36923566-4f6e-438b-a9f0-5276ec313362”
},
“parent_uuid”: “6ad67388-8da4-79c8-ac37-000000000043”,
“pid”: 3507,
“runner_ident”: “devops”,
“start_line”: 175,
“stdout”: "",
“uuid”: “36923566-4f6e-438b-a9f0-5276ec313362”
}

    Youngo 更改标题为「AWS的ESK启用插件kubesphere Devops异常 namespaces "kubesphere-devops-worker" not found」

    chilianyi

    大佬，按照这个你这个回复。我当前状态和上面发的一样。
    检查=>
    第一步：
    第二步：
    重装=>
    第一步：
    第二步：
    然后等重启完：完全卸载安装失败的DevOps按照文档：https://www.kubesphere.io/zh/docs/v3.4/pluggable-components/uninstall-pluggable-components/#%E5%8D%B8%E8%BD%BD-kubesphere-devops


    再重新重新启用。按照文档：https://www.kubesphere.io/zh/docs/v3.4/pluggable-components/devops/#%E5%9C%A8%E5%AE%89%E8%A3%85%E5%90%8E%E5%90%AF%E7%94%A8-devops
    结果和上面我发的内容一样。

      这个看上去还是 devops 这个 chart 没有安装上。

      # helm ls -a -n kubesphere-devops-system

      # kubectl get secret -n kubesphere-devops-system

      有什么结果吗， 看是不是 helm 安装失败了。

      再就是 # kubectl get pod -n kubesphere-devops-system 有内容吗。

      如果都没有，那只能是还没到这步， 但 argocd 安装上了， 下一步其实就是安装 kubesphere-devops-system 下的 chart，

      # kubectl exec -it deploy/ks-installer -n kubesphere-system bash

      进入到 kubesphere ks-installer 容器内部。

      $ cd results/devops/devops/$ ls -l
      total 48
      -rw——-    1 kubesphe kubesphe      3177 Sep 16 09:34 command
      drwxr-xr-x    2 kubesphe kubesphe      4096 Sep 16 09:34 fact_cache
      drwx——    2 kubesphe kubesphe     20480 Sep 16 09:34 job_events
      -rw——-    1 kubesphe kubesphe         1 Sep 16 09:34 rc
      -rw——-    1 kubesphe kubesphe        10 Sep 16 09:34 status
      -rw-r–r–    1 kubesphe kubesphe         0 Sep 16 09:34 stderr
      -rw——-    1 kubesphe kubesphe      7970 Sep 16 09:34 stdout

      看下 stderr 和 stdout 里面有什么更详细的报错信息吗。

        chilianyi

        上面操作结果是这样。
        查看内容日志：

        _[WARNING]: No inventory was parsed, only implicit localhost is available
        [WARNING]: provided hosts list is empty, only localhost is available. Note that
        the implicit localhost does not match ‘all’

        PLAY [localhost] ***************************************************************

        TASK [download : Generating images list] ***************************************
        skipping: [localhost]

        TASK [download : Synchronizing images] *****************************************

        TASK [kubesphere-defaults : KubeSphere | Setting images’ namespace override] ***
        skipping: [localhost]

        TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
        ok: [localhost] => {
        “msg”: “Check roles/kubesphere-defaults/defaults/main.yml”
        }

        TASK [ks-devops : Jenkins | Check jenkins status] ******************************
        fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “/usr/local/bin/kubectl get deploy -n kubesphere-devops-system ks-jenkins\n”, “delta”: “0:00:00.571465”, “end”: “2023-09-22 16:03:20.982943”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2023-09-22 16:03:20.411478”, “stderr”: "Error from server (NotFound): deployments.apps \“ks-jenkins\” not found", “stderr_lines”: ["Error from server (NotFound): deployments.apps \“ks-jenkins\” not found"], “stdout”: "", “stdout_lines”: []}
        …ignoring

        TASK [ks-devops | Getting Jenkins config] **************************************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_SMTP_HOST)] *******************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_FROM_ADDR)] *******************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_FROM_NAME)] *******************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_USE_SSL)] *********************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_SMTP_PORT)] *******************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins configs (EMAIL_FROM_PASS)] *******************
        skipping: [localhost]

        TASK [ks-devops | Checking Jenkins PersistentVolumeClaim] **********************
        skipping: [localhost]

        TASK [ks-devops | Setting Jenkins pv size] *************************************
        skipping: [localhost]

        TASK [ks-devops | Checking Jenkins update center status] ***********************
        changed: [localhost]

        TASK [ks-devops | Uninstall update center] *************************************
        skipping: [localhost]

        TASK [ks-devops | Getting sonarqube host] **************************************
        changed: [localhost]

        TASK [ks-devops | Getting sonarqube token] *************************************
        changed: [localhost]

        TASK [ks-devops : set_fact] ****************************************************
        skipping: [localhost]

        TASK [ks-devops : set_fact] ****************************************************
        skipping: [localhost]

        TASK [ks-devops | Getting KubeSphere JWT secret] *******************************
        changed: [localhost]

        TASK [ks-devops | Debug KubeSphere JWT secret] *********************************
        ok: [localhost] => {
        “ks_jwt_secret”: {
        “changed”: true,
        “cmd”: “/usr/local/bin/kubectl get secret -n kubesphere-system kubesphere-secret -ojsonpath=\”{.data.secret}\“ | base64 -d\n”,
        “delta”: “0:00:00.491604”,
        “end”: “2023-09-22 16:03:31.884673”,
        “failed”: false,
        “rc”: 0,
        “start”: “2023-09-22 16:03:31.393069”,
        “stderr”: "",
        “stderr_lines”: [],
        “stdout”: “ZNmPX3E27kYjMqFGLA9w1wfCq9o1ffnK”,
        “stdout_lines”: [
        “ZNmPX3E27kYjMqFGLA9w1wfCq9o1ffnK”
        ]
        }
        }

        TASK [ks-devops : Get Argo CD installation] ************************************
        changed: [localhost]

        TASK [ks-devops : Get Argo CD namespace] ***************************************
        changed: [localhost]

        TASK [ks-devops | Getting devops installation charts] **************************
        changed: [localhost] => (item=ks-devops)

        TASK [ks-devops | Checking existing PVC named ks-jenkins] **********************
        changed: [localhost]

        TASK [ks-devops | Debug exisiting PVC named ks-jenkins] ************************
        ok: [localhost] => {
        “ks_jenkins_pvc”: {
        “changed”: true,
        “cmd”: “/usr/local/bin/kubectl -n kubesphere-devops-system get pvc ks-jenkins -o jsonpath=‘{.metadata.name}’ –ignore-not-found\n”,
        “delta”: “0:00:00.385517”,
        “end”: “2023-09-22 16:03:44.883366”,
        “failed”: false,
        “rc”: 0,
        “start”: “2023-09-22 16:03:44.497849”,
        “stderr”: "",
        “stderr_lines”: [],
        “stdout”: "",
        “stdout_lines”: []
        }
        }

        TASK [ks-devops | Getting name of existing PVC named ks-jenkins] ***************
        ok: [localhost]

        TASK [ks-devops | Getting Kubernetes Node info] ********************************
        changed: [localhost]

        TASK [ks-devops | Setting container runtime of agent builder] ******************
        ok: [localhost]

        TASK [ks-devops | Creating manifests] ******************************************
        changed: [localhost] => (item={‘path’: ‘ks-devops’, ‘file’: ‘ks-devops-values.yaml’})
        changed: [localhost] => (item={‘path’: ‘ks-devops’, ‘file’: ‘argo-cd-values.yaml’})

        TASK [ks-devops | Checking if ks-devops has installed] *************************
        changed: [localhost]

        TASK [ks-devops | Deleting service monitor of devops] **************************
        changed: [localhost]

        TASK [ks-devops | Checking ks-devops Helm Release] *****************************
        changed: [localhost]

        TASK [ks-devops | Convert ks-devops to helm mananged] **************************
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘secrets’, ‘resource’: ‘s2i-webhook-server-cert’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘configmap’, ‘resource’: ‘jenkins-casc-config’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘configmap’, ‘resource’: ‘ks-devops-agent’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘clusterrole’, ‘resource’: ‘manager-role’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘clusterrole’, ‘resource’: ‘proxy-role’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘ClusterRoleBinding’, ‘resource’: ‘manager-rolebinding’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘ClusterRoleBinding’, ‘resource’: ‘proxy-rolebinding’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘service’, ‘resource’: ‘s2ioperator-metrics-service’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘service’, ‘resource’: ‘s2i-trigger-service’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘service’, ‘resource’: ‘webhook-server-service’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘service’, ‘resource’: ‘s2ioperator-trigger-service’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘StatefulSet’, ‘resource’: ‘s2ioperator’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘MutatingWebhookConfiguration’, ‘resource’: ‘mutating-webhook-configuration’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘ValidatingWebhookConfiguration’, ‘resource’: ‘validating-webhook-configuration’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘S2iBuilderTemplate’, ‘resource’: ‘binary’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘S2iBuilderTemplate’, ‘resource’: ‘java’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘S2iBuilderTemplate’, ‘resource’: ‘nodejs’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘S2iBuilderTemplate’, ‘resource’: ‘tomcat’, ‘release’: ‘devops’})
        changed: [localhost] => (item={‘ns’: ‘kubesphere-devops-system’, ‘kind’: ‘S2iBuilderTemplate’, ‘resource’: ‘python’, ‘release’: ‘devops’})

        TASK [ks-devops | Stop the existing Jenkins] ***********************************
        fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “/usr/local/bin/kubectl -n kubesphere-devops-system delete service ks-jenkins-agent –ignore-not-found\n/usr/local/bin/kubectl -n kubesphere-devops-system delete service ks-jenkins –ignore-not-found\n/usr/local/bin/kubectl -n kubesphere-devops-system scale deployment ks-jenkins –replicas=0\n”, “delta”: “0:00:00.790432”, “end”: “2023-09-22 16:04:25.283475”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2023-09-22 16:04:24.493043”, “stderr”: "Error from server (NotFound): deployments.apps \“ks-jenkins\” not found", “stderr_lines”: ["Error from server (NotFound): deployments.apps \“ks-jenkins\” not found"], “stdout”: "", “stdout_lines”: []}
        …ignoring

        TASK [ks-devops | Upgrading or installing Argo CD] *****************************
        changed: [localhost]

        TASK [ks-devops | Upgrading or installing ks-devops] ***************************
        changed: [localhost]

        TASK [ks-devops | Labelling devops namespaces as system workspace] *************
        failed: [localhost] (item=kubesphere-devops-worker) => {“ansible_loop_var”: “item”, “changed”: true, “cmd”: “/usr/local/bin/kubectl label –overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n”, “delta”: “0:00:00.201544”, “end”: “2023-09-22 16:04:48.794894”, “item”: “kubesphere-devops-worker”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2023-09-22 16:04:48.593350”, “stderr”: "Error from server (NotFound): namespaces \“kubesphere-devops-worker\” not found", “stderr_lines”: ["Error from server (NotFound): namespaces \“kubesphere-devops-worker\” not found"], “stdout”: "", “stdout_lines”: []}
        changed: [localhost] => (item=argocd)

        PLAY RECAP *********************************************************************
        localhost : ok=23 changed=18 unreachable=0 failed=1 skipped=15 rescued=0 ignored=2
        ks-installer-79545d8896-q8fz2:/kubesphere/results/devops/devops$_

          上面图片这里， 是 重启 ks-installer， 这个是不是没重启，所以没重装， 或者手动 delete 下 installer 的 pod 试试。

          kubectl -n kubesphere-system rollout restart deploy ks-installer

            chilianyi
            先按照文档卸载：https://kubesphere.io/zh/docs/v3.4/pluggable-components/uninstall-pluggable-components/#%E5%8D%B8%E8%BD%BD-kubesphere-devops
            再执行命令;

            pod也重启了


            再按照文档重装：https://kubesphere.io/zh/docs/v3.4/pluggable-components/devops/#%E5%9C%A8%E5%AE%89%E8%A3%85%E5%90%8E%E5%90%AF%E7%94%A8-devops
            然后结果还是和上面问题的内容一样:
            PLAY RECAP *********************************************************************
            localhost : ok=30 changed=18 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
            Start installing monitoring
            Start installing multicluster
            Start installing openpitrix
            Start installing network
            Start installing devops
            Start installing events

            Waiting for all tasks to be completed …
            task network status is successful (⅙)
            task openpitrix status is successful (2/6)
            task multicluster status is successful (3/6)
            task events status is successful (4/6)
            task devops status is failed (⅚)
            task monitoring status is successful (6/6)

            Collecting installation results …

                Task 'devops' failed:
    ******************************************************************************************************************************************************
    {
    "counter": 146,
    "created": "2023-09-23T06:17:15.089587",
    "end_line": 175,
    "event": "runner_on_failed",
    "event_data": {
    "duration": 1.787483,
    "end": "2023-09-23T06:17:15.089482",
    "event_loop": null,
    "host": "localhost",
    "ignore_errors": null,
    "play": "localhost",
    "play_pattern": "localhost",
    "play_uuid": "cadf39de-a05e-3964-1a4d-000000000005",
    "playbook": "/kubesphere/playbooks/devops.yaml",
    "playbook_uuid": "3b527603-0a82-406e-a34b-e61f368034f6",
    "remote_addr": "127.0.0.1",
    "res": {
    "changed": true,
    "msg": "All items completed",
    "results": [
    {
    "_ansible_item_label": "kubesphere-devops-worker",
    "_ansible_no_log": false,
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "/usr/local/bin/kubectl label --overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n",
    "delta": "0:00:00.193820",
    "end": "2023-09-23 06:17:14.284541",
    "failed": true,
    "invocation": {
    "module_args": {
    "_raw_params": "/usr/local/bin/kubectl label --overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n",
    "_uses_shell": true,
    "argv": null,
    "chdir": null,
    "creates": null,
    "executable": null,
    "removes": null,
    "stdin": null,
    "stdin_add_newline": true,
    "strip_empty_ends": true,
    "warn": true
    }
    },
    "item": "kubesphere-devops-worker",
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2023-09-23 06:17:14.090721",
    "stderr": "Error from server (NotFound): namespaces \"kubesphere-devops-worker\" not found",
    "stderr_lines": [
    "Error from server (NotFound): namespaces \"kubesphere-devops-worker\" not found"
    ],
    "stdout": "",
    "stdout_lines": []
    },
    {
    "_ansible_item_label": "argocd",
    "_ansible_no_log": false,
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "/usr/local/bin/kubectl label --overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n",
    "delta": "0:00:00.200808",
    "end": "2023-09-23 06:17:15.001559",
    "failed": false,
    "invocation": {
    "module_args": {
    "_raw_params": "/usr/local/bin/kubectl label --overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n",
    "_uses_shell": true,
    "argv": null,
    "chdir": null,
    "creates": null,
    "executable": null,
    "removes": null,
    "stdin": null,
    "stdin_add_newline": true,
    "strip_empty_ends": true,
    "warn": true
    }
    },
    "item": "argocd",
    "rc": 0,
    "start": "2023-09-23 06:17:14.800751",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "namespace/argocd not labeled",
    "stdout_lines": [
    "namespace/argocd not labeled"
    ]
    }
    ]
    },
    "resolved_action": "shell",
    "role": "ks-devops",
    "start": "2023-09-23T06:17:13.301999",
    "task": "ks-devops | Labelling devops namespaces as system workspace",
    "task_action": "shell",
    "task_args": "",
    "task_path": "/kubesphere/installer/roles/ks-devops/tasks/main.yaml:232",
    "task_uuid": "cadf39de-a05e-3964-1a4d-000000000043",
    "uuid": "b0e4d4ee-d60c-41f6-a76e-31dee73ea001"
    },
    "parent_uuid": "cadf39de-a05e-3964-1a4d-000000000043",
    "pid": 10228,
    "runner_ident": "devops",
    "start_line": 175,
    "stdout": "",
    "uuid": "b0e4d4ee-d60c-41f6-a76e-31dee73ea001"
    }
    ******************************************************************************************************************************************************

            当前状态：

              # kubectl exec -it deploy/ks-installer -n kubesphere-system bash

              进入 ks-installer 中，手动 install 一下，看是不是报了什么错误呢。
              helm upgrade –install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

                chilianyi
                不好意思，刚我使用k8s不久。
                请问是先
                kubectl exec -it deploy/ks-installer -n kubesphere-system bash
                再
                helm upgrade –install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

                  Youngo 哦，命令贴近来有问题了。 是 – install , 两个 -

                  helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml
                      kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds

kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

rm -rf kubesphere/ks-devops/charts/s2i-templates.yaml

helm template kubesphere/ks-devops/charts/ks-devops/charts/s2i/ \
      -f kubesphere/ks-devops/ks-devops-values.yaml \
      -s templates/binary.yaml \
      -s templates/java.yaml \
      -s templates/nodejs.yaml \
      -s templates/python.yaml \
      -s templates/tomcat.yaml > kubesphere/ks-devops/charts/s2i-templates.yaml

rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/binary.yaml
rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/java.yaml
rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/nodejs.yaml
rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/python.yaml
rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/tomcat.yaml

helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

kubectl apply -f kubesphere/ks-devops/charts/s2i-templates.yaml

                      你这个报错，是没有安装 crd ，可能是之前彻底卸载过。

                      标准安装是这些命令。 手动 一行一行执行下，看是不是 执行过程中有什么报错。

                        Youngo
                        这个是 ls kubesphere/ks-devops/charts 这个目录下 有个 ks-devops-xxx.tgz

                        这个 目录解压以后得到的。

                            我也是同样的版本，同样的问题。

                            EKS1.26+kubeshpere3.4，我也按照大神们建议的方式，试了一下，还是一样的错误。

                            chilianyi 这个是 ls kubesphere/ks-devops/charts 这个目录下 有个 ks-devops-xxx.tgz

                            这个 目录解压以后得到的。

                            kubesphere/ks-devops/charts目录下没有s2i这个子目录

                                panpan ks-3.4.0 devops 是这个包：https://github.com/kubesphere/ks-installer/blob/v3.4.0/roles/ks-devops/files/ks-devops/charts/ks-devops-0.2.1.tgz

                                这个 解压后，我看是有 相应目录的。 解压的目录是 kubesphere/ks-devops/charts，

                                https://github.com/kubesphere/ks-installer/blob/62d60e496926e07391b78a19357c6ea94d125b92/roles/ks-devops/tasks/main.yaml#L204

                                这个 s2i 应该是在 kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

                                kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
                                kubectl apply -f kubesphere/ks-devops/charts/s2i/crds

                                这两句上面目录发错了，按照代码，目录应该是

                                kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
                                kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

                                    chilianyi 这个 s2i 应该是在 kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

                                    kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
                                    kubectl apply -f kubesphere/ks-devops/charts/s2i/crds

                                    这两句上面目录发错了，按照代码，目录应该是

                                    kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
                                    kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

                                    Usage: helm upgrade [RELEASE] [CHART] [flags]

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

                                    Release "devops" does not exist. Installing it now.

                                    Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "devops" namespace: "" from "": no matches for kind "CronJob" in version "batch/v1beta1"

                                    ensure CRDs are installed first

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds

                                    customresourcedefinition.apiextensions.k8s.io/clustersteptemplates.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/clustertemplates.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/devopsprojects.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/gitrepositories.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/pipelineruns.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/pipelines.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/templates.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/applications.gitops.kubesphere.io configured

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

                                    customresourcedefinition.apiextensions.k8s.io/s2ibinaries.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/s2ibuilders.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/s2ibuildertemplates.devops.kubesphere.io configured

                                    customresourcedefinition.apiextensions.k8s.io/s2iruns.devops.kubesphere.io configured

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ rm -rf kubesphere/ks-devops/charts/s2i-templates.yaml

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ helm template kubesphere/ks-devops/charts/ks-devops/charts/s2i/ \

                                    > -f kubesphere/ks-devops/ks-devops-values.yaml \

                                    > -s templates/binary.yaml \

                                    > -s templates/java.yaml \

                                    > -s templates/nodejs.yaml \

                                    > -s templates/python.yaml \

                                    > -s templates/tomcat.yaml > kubesphere/ks-devops/charts/s2i-templates.yaml

                                    Error: could not find template templates/binary.yaml in chart

                                    ks-installer-6697794f55-8dmbz:/kubesphere$

                                    ks-installer-6697794f55-8dmbz:/kubesphere$

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ ll kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/

                                    bash: ll: command not found

                                    ks-installer-6697794f55-8dmbz:/kubesphere$ ls kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/

                                    NOTES.txt operator.yaml tests

                                    _helpers.tpl prometheus-servicemonitor.yaml

                                    之前两句执行 OK 了，但是往下执行，还是不行