• DevOps
  • AWS的ESK启用插件kubesphere Devops异常 namespaces "kubesphere-devops-worker" not found

chilianyi
先按照文档卸载:https://kubesphere.io/zh/docs/v3.4/pluggable-components/uninstall-pluggable-components/#%E5%8D%B8%E8%BD%BD-kubesphere-devops
再执行命令;

pod也重启了


再按照文档重装:https://kubesphere.io/zh/docs/v3.4/pluggable-components/devops/#%E5%9C%A8%E5%AE%89%E8%A3%85%E5%90%8E%E5%90%AF%E7%94%A8-devops
然后结果还是和上面问题的内容一样:
PLAY RECAP *********************************************************************
localhost : ok=30 changed=18 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
Start installing devops
Start installing events


Waiting for all tasks to be completed …
task network status is successful (⅙)
task openpitrix status is successful (2/6)
task multicluster status is successful (3/6)
task events status is successful (4/6)
task devops status is failed (⅚)
task monitoring status is successful (6/6)


Collecting installation results …

    Task 'devops' failed:
    ******************************************************************************************************************************************************
    {
    "counter": 146,
    "created": "2023-09-23T06:17:15.089587",
    "end_line": 175,
    "event": "runner_on_failed",
    "event_data": {
    "duration": 1.787483,
    "end": "2023-09-23T06:17:15.089482",
    "event_loop": null,
    "host": "localhost",
    "ignore_errors": null,
    "play": "localhost",
    "play_pattern": "localhost",
    "play_uuid": "cadf39de-a05e-3964-1a4d-000000000005",
    "playbook": "/kubesphere/playbooks/devops.yaml",
    "playbook_uuid": "3b527603-0a82-406e-a34b-e61f368034f6",
    "remote_addr": "127.0.0.1",
    "res": {
    "changed": true,
    "msg": "All items completed",
    "results": [
    {
    "_ansible_item_label": "kubesphere-devops-worker",
    "_ansible_no_log": false,
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "/usr/local/bin/kubectl label --overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n",
    "delta": "0:00:00.193820",
    "end": "2023-09-23 06:17:14.284541",
    "failed": true,
    "invocation": {
    "module_args": {
    "_raw_params": "/usr/local/bin/kubectl label --overwrite namespaces kubesphere-devops-worker kubesphere.io/workspace=system-workspace\n",
    "_uses_shell": true,
    "argv": null,
    "chdir": null,
    "creates": null,
    "executable": null,
    "removes": null,
    "stdin": null,
    "stdin_add_newline": true,
    "strip_empty_ends": true,
    "warn": true
    }
    },
    "item": "kubesphere-devops-worker",
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2023-09-23 06:17:14.090721",
    "stderr": "Error from server (NotFound): namespaces \"kubesphere-devops-worker\" not found",
    "stderr_lines": [
    "Error from server (NotFound): namespaces \"kubesphere-devops-worker\" not found"
    ],
    "stdout": "",
    "stdout_lines": []
    },
    {
    "_ansible_item_label": "argocd",
    "_ansible_no_log": false,
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "/usr/local/bin/kubectl label --overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n",
    "delta": "0:00:00.200808",
    "end": "2023-09-23 06:17:15.001559",
    "failed": false,
    "invocation": {
    "module_args": {
    "_raw_params": "/usr/local/bin/kubectl label --overwrite namespaces argocd kubesphere.io/workspace=system-workspace\n",
    "_uses_shell": true,
    "argv": null,
    "chdir": null,
    "creates": null,
    "executable": null,
    "removes": null,
    "stdin": null,
    "stdin_add_newline": true,
    "strip_empty_ends": true,
    "warn": true
    }
    },
    "item": "argocd",
    "rc": 0,
    "start": "2023-09-23 06:17:14.800751",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "namespace/argocd not labeled",
    "stdout_lines": [
    "namespace/argocd not labeled"
    ]
    }
    ]
    },
    "resolved_action": "shell",
    "role": "ks-devops",
    "start": "2023-09-23T06:17:13.301999",
    "task": "ks-devops | Labelling devops namespaces as system workspace",
    "task_action": "shell",
    "task_args": "",
    "task_path": "/kubesphere/installer/roles/ks-devops/tasks/main.yaml:232",
    "task_uuid": "cadf39de-a05e-3964-1a4d-000000000043",
    "uuid": "b0e4d4ee-d60c-41f6-a76e-31dee73ea001"
    },
    "parent_uuid": "cadf39de-a05e-3964-1a4d-000000000043",
    "pid": 10228,
    "runner_ident": "devops",
    "start_line": 175,
    "stdout": "",
    "uuid": "b0e4d4ee-d60c-41f6-a76e-31dee73ea001"
    }
    ******************************************************************************************************************************************************

当前状态:

# kubectl exec -it deploy/ks-installer -n kubesphere-system bash

进入 ks-installer 中,手动 install 一下,看是不是报了什么错误呢。
helm upgrade –install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

    chilianyi
    不好意思,刚我使用k8s不久。
    请问是先
    kubectl exec -it deploy/ks-installer -n kubesphere-system bash

    helm upgrade –install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

    Youngo 哦,命令贴近来有问题了。 是 – install , 两个 -

    helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml
      kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
      
      kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds
      
      rm -rf kubesphere/ks-devops/charts/s2i-templates.yaml
      
      helm template kubesphere/ks-devops/charts/ks-devops/charts/s2i/ \
            -f kubesphere/ks-devops/ks-devops-values.yaml \
            -s templates/binary.yaml \
            -s templates/java.yaml \
            -s templates/nodejs.yaml \
            -s templates/python.yaml \
            -s templates/tomcat.yaml > kubesphere/ks-devops/charts/s2i-templates.yaml
      
      rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/binary.yaml
      rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/java.yaml
      rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/nodejs.yaml
      rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/python.yaml
      rm -rf kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/tomcat.yaml
      
      helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml
      
      kubectl apply -f kubesphere/ks-devops/charts/s2i-templates.yaml

      你这个报错,是没有安装 crd ,可能是之前彻底卸载过。

      标准安装是这些命令。 手动 一行一行执行下,看是不是 执行过程中有什么报错。

        Youngo
        这个是 ls kubesphere/ks-devops/charts 这个目录下 有个 ks-devops-xxx.tgz

        这个 目录解压以后得到的。

          我也是同样的版本,同样的问题。

          EKS1.26+kubeshpere3.4,我也按照大神们建议的方式,试了一下,还是一样的错误。

          chilianyi 这个是 ls kubesphere/ks-devops/charts 这个目录下 有个 ks-devops-xxx.tgz

          这个 目录解压以后得到的。

          kubesphere/ks-devops/charts目录下没有s2i这个子目录

            panpan ks-3.4.0 devops 是这个包:https://github.com/kubesphere/ks-installer/blob/v3.4.0/roles/ks-devops/files/ks-devops/charts/ks-devops-0.2.1.tgz

            这个 解压后,我看是有 相应目录的。 解压的目录是 kubesphere/ks-devops/charts,

            https://github.com/kubesphere/ks-installer/blob/62d60e496926e07391b78a19357c6ea94d125b92/roles/ks-devops/tasks/main.yaml#L204

            这个 s2i 应该是在 kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

            kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
            kubectl apply -f kubesphere/ks-devops/charts/s2i/crds

            这两句上面目录发错了,按照代码,目录应该是

            kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
            kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

              chilianyi 这个 s2i 应该是在 kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

              kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
              kubectl apply -f kubesphere/ks-devops/charts/s2i/crds

              这两句上面目录发错了,按照代码,目录应该是

              kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds
              kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

              Usage: helm upgrade [RELEASE] [CHART] [flags]

              ks-installer-6697794f55-8dmbz:/kubesphere$ helm upgrade --install devops kubesphere/ks-devops/charts/ks-devops*.tgz -n kubesphere-devops-system -f kubesphere/ks-devops/ks-devops-values.yaml

              Release "devops" does not exist. Installing it now.

              Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "devops" namespace: "" from "": no matches for kind "CronJob" in version "batch/v1beta1"

              ensure CRDs are installed first

              ks-installer-6697794f55-8dmbz:/kubesphere$ kubectl apply -f kubesphere/ks-devops/charts/ks-devops/crds

              customresourcedefinition.apiextensions.k8s.io/clustersteptemplates.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/clustertemplates.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/devopsprojects.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/gitrepositories.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/pipelineruns.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/pipelines.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/templates.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/applications.gitops.kubesphere.io configured

              ks-installer-6697794f55-8dmbz:/kubesphere$ kubectl apply -f kubesphere/ks-devops/charts/ks-devops/charts/s2i/crds

              customresourcedefinition.apiextensions.k8s.io/s2ibinaries.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/s2ibuilders.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/s2ibuildertemplates.devops.kubesphere.io configured

              customresourcedefinition.apiextensions.k8s.io/s2iruns.devops.kubesphere.io configured

              ks-installer-6697794f55-8dmbz:/kubesphere$ rm -rf kubesphere/ks-devops/charts/s2i-templates.yaml

              ks-installer-6697794f55-8dmbz:/kubesphere$ helm template kubesphere/ks-devops/charts/ks-devops/charts/s2i/ \

              > -f kubesphere/ks-devops/ks-devops-values.yaml \

              > -s templates/binary.yaml \

              > -s templates/java.yaml \

              > -s templates/nodejs.yaml \

              > -s templates/python.yaml \

              > -s templates/tomcat.yaml > kubesphere/ks-devops/charts/s2i-templates.yaml

              Error: could not find template templates/binary.yaml in chart

              ks-installer-6697794f55-8dmbz:/kubesphere$

              ks-installer-6697794f55-8dmbz:/kubesphere$

              ks-installer-6697794f55-8dmbz:/kubesphere$ ll kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/

              bash: ll: command not found

              ks-installer-6697794f55-8dmbz:/kubesphere$ ls kubesphere/ks-devops/charts/ks-devops/charts/s2i/templates/

              NOTES.txt operator.yaml tests

              _helpers.tpl prometheus-servicemonitor.yaml

              之前两句执行 OK 了,但是往下执行,还是不行

                panpan 你好,看起来是 cronjob 的问题。

                麻烦执行下这两个命令看下输出。

                # kubectl api-versions | grep batch/v1
                batch/v1
                batch/v1beta1
                
                # kubectl version

                  Youngo 后面这个问题,楼主你解决没有呢?我也是 EKS1.26+kubeshpere 3.4.0 实在没有法,我就准备降级了。

                  Youngo 后面这个问题,楼主你解决没有呢?我也是 EKS1.26+kubeshpere 3.4.0 实在没有法,我就准备降级了。

                  chilianyi # kubectl api-versions | grep batch/v1
                  batch/v1
                  batch/v1beta1

                  kubectl version

                  ks-installer-758d459599-592×9:/kubesphere$ kubectl api-versions | grep batch/v1

                  batch/v1

                  ks-installer-758d459599-592×9:/kubesphere$ kubectl version

                  Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.7”, GitCommit:“42c05a547468804b2053ecf60a3bd15560362fc2”, GitTreeState:“clean”, BuildDate:“2022-05-24T12:30:55Z”, GoVersion:“go1.17.10”, Compiler:“gc”, Platform:“linux/amd64”}

                  Server Version: version.Info{Major:“1”, Minor:“26+”, GitVersion:“v1.26.7-eks-2d98532”, GitCommit:“c9c3fc1fdf677f38b8d26568c10b921a1235fa43”, GitTreeState:“clean”, BuildDate:“2023-07-28T16:51:21Z”, GoVersion:“go1.20.6”, Compiler:“gc”, Platform:“linux/amd64”}

                  WARNING: version difference between client (1.23) and server (1.26) exceeds the supported minor version skew of +/-1

                  请大神抽空再帮看看,做最后的努力 了

                    panpan

                    kubesphere/ks-devops#998
                    你好,是个 bug,错误的识别了 k8s 的版本,导致安装时,cronjob 用了 batch/v1beta1, 但 k8s 1.26 里已经只有 batch/v1 没有 batch/v1beta1 了, 导致安装失败。

                    目前已经做了修复,在 helm chart 中,kubesphere-sigs/ks-devops-helm-chart#122/files
                    晚些 会提供下 新的 helm chart,到时 更新下 ks-installer 中对应的 helm 包,就可以正常安装了。

                      panpan 你好,经过我们测试发现是创建 cronjob 的 apiversion 有问题,在 chart 包里的比较当前 k8s 版本没有生效;针对 ks 3.4.0 麻烦先参考下面步骤,先临时手动修复下,后面会在新版本 3.4.1 修复此问题:

                      1. 编辑 ks-installer deployment,修改 securityContext.runAsUser 为 0,以 root 用户运行 ks-installer:
                        kubectl -n kubesphere-system edit deployments.apps ks-installer
                      2. 下载更新后的 ke-devops helm 包 ks-devops-0.2.1.tgz
                      3. 等待 ks-installer 重启完成之后,替换 helm 包:
                        kubectl -n kubesphere-system cp ks-devops-0.2.1.tgz $(kubectl get pod -n kubesphere-system -l app=ks-installer -o jsonpath='{.items[0].metadata.name}'):/kubesphere/installer/roles/ks-devops/files/ks-devops/charts/
                      4. 重新开启 ks-devops 组件
                        kubectl -n kubesphere-system edit cc ks-installer

                        如果 status.devops 存在,删除 status.devops,即删除截图中的三行;

                        修改 spec.devops 配置里任意一项,如图中的 jenkinsCpuReq ,触发重新安装 DevOps 即可;

                        chilianyi

                        感谢大神,按照建议的方式,已安装devops成功,我明天再具体使用一下。

                        拜谢