从3.2.1升级到3.3.0之后，etcd一直报错-【已解决】

lutixia

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息
虚拟机，Centos7.6，4C/8G

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

KubeSphere版本信息
v3.3.0。在线安装。使用kk安装。

问题是什么
etcd集群没有问题，是健康的，但是日志总是报错：

操作界面出现的问题：

devops中，点击创建流水线，没有模板出现，报404；

点击devops的持续部署，会一直转圈，没有反应。

kevendeng

etcd的日志仅是指出这些查询语句执行时间较长，属于性能上可能存在的问题，不属于报错。

devops的问题，建议先看看kubesphere-devops-system命名空间下的组件是否都正常运行了，以及日志是否正常

lutixia

kevendeng

看着没有异常，都正常运行。

devops问题1截图如下：

devops问题2截图如下：

kevendeng

这个得继续看看浏览器network里具体的请求了，应该有报错和卡住的

lutixia

kevendeng

感谢回复，再次访问，通过f12查看如下：

kevendeng

在ks中，devops相关请求是在ks-apiserver通过proxy直接代理转发到devops-apiserver的，gitops相关的请求也包含在其中。

然而network中该请求报错404 not found，说明该路由在apiserver中没有注册。基于以上的前提，可以确定，或是没在ks-apiserver中注册，或是没在devops-apiserver中注册，或两者皆无。

3.3.0版本中的ks-apiserver与devops-apiserver中都包含了相关路由，再注意到你get pod输出中，devops-apiserver的age为26d，所以我猜测是容器镜像版本的问题。

可确定下devops-apiserver与devops-controller（age同为26d）的镜像版本，若我的猜测正确，镜像版本应该仍是老版本，没有升级。

ks-apiserver的版本也可确认下。

如果已按照文档成功完成了升级操作，这里或许是一个版本升级的bug。

lutixia

kevendeng

感谢大佬详细的分析讲解，确实是如大佬所说的，下面是查看的各pod的版本：

kubesphere/ks-apiserver:v3.3.0

kubesphere/devops-controller:v3.2.1

kubesphere/devops-apiserver:v3.2.1

可以直接编辑devops-controller，devops-apiserver的yaml文件的镜像版本进行升级么？还是要怎么升级？

Rick

帮忙提供如下信息以帮助排查问题：

升级步骤
ks-installer 的版本以及日志

lutixia

Rick

升级是参考官方文档中kubekey（kk版本是2.2.1）的多节点升级步骤实现的，我早期k8s版本是1.21.5，ks是3.2.1，在这次升级中，想直接升级k8s和ks的版本，升级命令为：kk upgrade --with-kubernetes v1.23.4 --with-kubesphere v3.3.0 -f sample.yaml

升级过程中ks顺利升级为3.3.0，k8s顺利从1.21.5升级为1.22.10，在从1.22.10升级到1.23.4时报错终止，好像是镜像拉取失败终止，日志实在太多，找不到了。

再次感谢大佬提供帮助！

kevendeng

可以用kubectl -n kubesphere-system logs -l app=ks-installer查看下ks-installer的执行日志

lutixia

kevendeng

好的，查看到日志内容如下：

kevendeng

10.233.0.1这个IP是K8s apiserver的service vip，从这段日志中可以看到，6-27 16:28 时，ks-installer pod中的进程尝试访问这个IP，却没有连接成功（connection refused）

首先，我们并不能确定这就是导致devops版本没有升级的原因，为了进一步确定，需要查看完整的ks-installer日志：1. ks-installer执行成功了哪些步骤？ 2. 这个错误发生后，是否没有自愈，导致升级步骤再也没有进行下一步？

ks-installer通过service访问K8s apiserver，需要经过宿主机、宿主机上的端口转发路由到apiserver所在节点

其中宿主机的端口转发是由该节点上的kube-proxy守护进程负责的

从你可以成功执行kubectl命令来看，K8s的apiserver是正常运行着的，那么访问失败可能是kube-proxy导致的忘了问题，我个人的建议是可以查看对应节点上kube-proxy pod的日志

以及进一步，通过ipvsadm等工具，检查ipvs规则是否正常设置

lutixia

kevendeng

ks-install的日志如下，都是大量重复的内容：

"stdout": "s2ibuildertemplate.devops.kubesphere.io/tomcat annotated\ns2ibuildertemplate.devops.kubesphere.io/tomcat annotated\ns2ibuildertemplate.devops.kubesphere.io/tomcat not labeled",
          "stdout_lines": [
            "s2ibuildertemplate.devops.kubesphere.io/tomcat annotated",
            "s2ibuildertemplate.devops.kubesphere.io/tomcat annotated",
            "s2ibuildertemplate.devops.kubesphere.io/tomcat not labeled"
          ]
        },
        {
          "_ansible_item_label": {
            "kind": "S2iBuilderTemplate",
            "ns": "kubesphere-devops-system",
            "release": "devops",
            "resource": "python"
          },
          "_ansible_no_log": false,
          "ansible_loop_var": "item",
          "changed": true,
          "cmd": "RESOURCE=$(/usr/local/bin/kubectl -n kubesphere-devops-system get S2iBuilderTemplate python 2> /dev/null | head -n 1); if [[ \"$RESOURCE\" != '' ]]; then /usr/local/bin/kubectl -n kubesphere-devops-system annotate --overwrite S2iBuilderTemplate python meta.helm.sh/release-name=devops; /usr/local/bin/kubectl -n kubesphere-devops-system annotate --overwrite S2iBuilderTemplate python meta.helm.sh/release-namespace=kubesphere-devops-system; /usr/local/bin/kubectl -n kubesphere-devops-system label --overwrite S2iBuilderTemplate python app.kubernetes.io/managed-by=Helm; fi\n",
          "delta": "0:00:26.726486",
          "end": "2022-06-27 15:48:00.121406",
          "failed": false,
          "failed_when_result": false,
          "invocation": {
            "module_args": {
              "_raw_params": "RESOURCE=$(/usr/local/bin/kubectl -n kubesphere-devops-system get S2iBuilderTemplate python 2> /dev/null | head -n 1); if [[ \"$RESOURCE\" != '' ]]; then /usr/local/bin/kubectl -n kubesphere-devops-system annotate --overwrite S2iBuilderTemplate python meta.helm.sh/release-name=devops; /usr/local/bin/kubectl -n kubesphere-devops-system annotate --overwrite S2iBuilderTemplate python meta.helm.sh/release-namespace=kubesphere-devops-system; /usr/local/bin/kubectl -n kubesphere-devops-system label --overwrite S2iBuilderTemplate python app.kubernetes.io/managed-by=Helm; fi\n",
              "_uses_shell": true,
              "argv": null,
              "chdir": null,
              "creates": null,
              "executable": "/bin/bash",
              "removes": null,
              "stdin": null,
              "stdin_add_newline": true,
              "strip_empty_ends": true,
              "warn": true
            }
          },
          "item": {
            "kind": "S2iBuilderTemplate",
            "ns": "kubesphere-devops-system",
            "release": "devops",
            "resource": "python"
          },
          "rc": 0,
          "start": "2022-06-27 15:47:33.394920",
          "stderr": "",
          "stderr_lines": [],
          "stdout": "",
          "stdout_lines": []
        }
      ]
    },
    "resolved_action": "shell",
    "role": "ks-devops",
    "start": "2022-06-27T07:43:06.179810",
    "task": "ks-devops | Convert ks-devops to helm mananged",
    "task_action": "shell",
    "task_args": "",
    "task_path": "/kubesphere/installer/roles/ks-devops/tasks/main.yaml:132",
    "task_uuid": "0a66d2c4-1d80-86df-5ee8-00000000003f",
    "uuid": "3cfe877e-0478-4bf5-aa7d-06a4b8ce4f34"
  },
  "parent_uuid": "0a66d2c4-1d80-86df-5ee8-00000000003f",
  "pid": 7601,
  "runner_ident": "devops",
  "start_line": 161,
  "stdout": "",
  "uuid": "3cfe877e-0478-4bf5-aa7d-06a4b8ce4f34"
}
******************************************************************************************************************************************************
E0627 16:06:52.837548       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:27.888367       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Unauthorized
E0627 16:07:43.655221       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": unexpected EOF
E0627 16:07:44.657176       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:45.658643       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:46.660626       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:47.662510       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:48.665779       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:49.667626       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:50.719205       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:51.720731       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:52.722111       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:53.723431       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:54.725096       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:55.726306       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:56.727687       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:57.729592       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
E0627 16:07:58.730774       1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.233.0.1:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: connection refused
之后重复到结束。

几个节点的kube-proxy的日志都差不多是下面这样内容，难道是内核版本不支持？

ipvs的规则如下:

Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  169.254.25.10:30002 rr
  -> 10.233.90.112:8080           Masq    1      0          0         
TCP  169.254.25.10:30004 rr
  -> 10.233.90.112:4443           Masq    1      0          0         
TCP  169.254.25.10:30180 rr
  -> 10.233.96.63:8080            Masq    1      0          0         
TCP  169.254.25.10:30880 rr
  -> 10.233.97.37:8000            Masq    1      0          0         
  -> 10.233.98.46:8000            Masq    1      0          0         
TCP  169.254.25.10:31568 rr
  -> 10.233.92.237:9000           Masq    1      0          0         
TCP  169.254.25.10:31919 rr
  -> 10.233.90.113:15021          Masq    1      0          0         
TCP  169.254.25.10:32049 rr
  -> 10.233.90.113:8443           Masq    1      0          0         
TCP  169.254.25.10:32225 rr
  -> 10.233.92.132:80             Masq    1      0          0         
TCP  169.254.25.10:32339 rr
  -> 10.233.90.113:8080           Masq    1      0          0         
TCP  169.254.25.10:32542 rr
  -> 10.233.92.237:9001           Masq    1      0          0         
TCP  172.17.0.1:30002 rr
  -> 10.233.90.112:8080           Masq    1      0          0         
TCP  172.17.0.1:30004 rr
  -> 10.233.90.112:4443           Masq    1      0          0         
TCP  172.17.0.1:30180 rr
  -> 10.233.96.63:8080            Masq    1      0          0         
TCP  172.17.0.1:30880 rr
  -> 10.233.97.37:8000            Masq    1      0          0         
  -> 10.233.98.46:8000            Masq    1      0          0         
TCP  172.17.0.1:31568 rr
  -> 10.233.92.237:9000           Masq    1      0          0         
TCP  172.17.0.1:31919 rr
  -> 10.233.90.113:15021          Masq    1      0          0         
TCP  172.17.0.1:32049 rr
  -> 10.233.90.113:8443           Masq    1      0          0         
TCP  172.17.0.1:32225 rr
  -> 10.233.92.132:80             Masq    1      0          0         
TCP  172.17.0.1:32339 rr
  -> 10.233.90.113:8080           Masq    1      0          0         
TCP  172.17.0.1:32542 rr
  -> 10.233.92.237:9001           Masq    1      0          0         
TCP  192.168.6.183:30002 rr
  -> 10.233.90.112:8080           Masq    1      0          0         
TCP  192.168.6.183:30004 rr
  -> 10.233.90.112:4443           Masq    1      0          0         
TCP  192.168.6.183:30180 rr
  -> 10.233.96.63:8080            Masq    1      0          0         
TCP  192.168.6.183:30880 rr
  -> 10.233.97.37:8000            Masq    1      0          0         
  -> 10.233.98.46:8000            Masq    1      0          0         
TCP  192.168.6.183:31568 rr
  -> 10.233.92.237:9000           Masq    1      0          0         
TCP  192.168.6.183:31919 rr
  -> 10.233.90.113:15021          Masq    1      0          0         
TCP  192.168.6.183:32049 rr
  -> 10.233.90.113:8443           Masq    1      0          0         
TCP  192.168.6.183:32225 rr
  -> 10.233.92.132:80             Masq    1      0          0         
TCP  192.168.6.183:32339 rr
  -> 10.233.90.113:8080           Masq    1      0          0         
TCP  192.168.6.183:32542 rr
  -> 10.233.92.237:9001           Masq    1      0          0         
TCP  10.233.0.1:443 rr
  -> 192.168.6.149:6443           Masq    1      1          0         
  -> 192.168.6.183:6443           Masq    1      5          0         
TCP  10.233.0.3:53 rr
  -> 10.233.97.41:53              Masq    1      0          0         
  -> 10.233.97.42:53              Masq    1      0          0         
TCP  10.233.0.3:9153 rr
  -> 10.233.97.41:9153            Masq    1      0          0         
  -> 10.233.97.42:9153            Masq    1      0          0         
TCP  10.233.2.8:8080 rr
  -> 10.233.90.32:8080            Masq    1      0          0         
TCP  10.233.5.103:443 rr
  -> 10.233.96.55:15017           Masq    1      0          0         
TCP  10.233.5.103:15010 rr
  -> 10.233.96.55:15010           Masq    1      0          0         
TCP  10.233.5.103:15012 rr
  -> 10.233.96.55:15012           Masq    1      0          0         
TCP  10.233.5.103:15014 rr
  -> 10.233.96.55:15014           Masq    1      0          0         
TCP  10.233.5.147:9000 rr
  -> 10.233.90.135:9000           Masq    1      0          0         
TCP  10.233.5.252:443 rr
  -> 10.233.90.128:9443           Masq    1      0          0         
TCP  10.233.6.34:443 rr
  -> 10.233.92.152:8443           Masq    1      0          0         
  -> 10.233.92.153:8443           Masq    1      0          0         
TCP  10.233.6.63:6379 rr
  -> 10.233.90.116:6379           Masq    1      3          5         
TCP  10.233.6.63:26379 rr
  -> 10.233.90.116:26379          Masq    1      3          15        
TCP  10.233.6.216:19093 rr
  -> 10.233.90.136:19093          Masq    1      0          0         
  -> 10.233.96.56:19093           Masq    1      0          0         
TCP  10.233.7.145:80 rr
  -> 10.233.92.4:9090             Masq    1      0          0         
  -> 10.233.98.45:9090            Masq    1      0          0         
TCP  10.233.7.163:9090 rr
  -> 10.233.90.125:9090           Masq    1      0          0         
TCP  10.233.9.115:80 rr
  -> 10.233.96.63:8080            Masq    1      0          0         
TCP  10.233.11.251:8443 rr
  -> 10.233.90.128:8443           Masq    1      0          0         
TCP  10.233.15.173:7899 rr
  -> 10.233.90.118:7899           Masq    1      0          0         
TCP  10.233.17.172:443 rr
  -> 10.233.92.10:8443            Masq    1      0          0         
  -> 10.233.98.44:8443            Masq    1      0          0         
TCP  10.233.22.61:80 rr
  -> 10.233.90.112:8080           Masq    1      0          0         
TCP  10.233.22.61:4443 rr
  -> 10.233.90.112:4443           Masq    1      0          0         
TCP  10.233.25.123:4443 rr
  -> 10.233.90.108:4443           Masq    1      0          0         
TCP  10.233.28.114:6379 rr
  -> 10.233.98.30:6379            Masq    1      2          5         
TCP  10.233.28.114:26379 rr
  -> 10.233.98.30:26379           Masq    1      2          15        
TCP  10.233.29.37:9200 rr
  -> 10.233.90.114:9200           Masq    1      1          0         
  -> 10.233.92.148:9200           Masq    1      2          0         
  -> 10.233.96.54:9200            Masq    1      1          0         
TCP  10.233.29.42:80 rr
  -> 10.233.90.120:9999           Masq    1      0          0         
TCP  10.233.30.118:80 rr
  -> 10.233.90.113:8080           Masq    1      0          0         
TCP  10.233.30.118:443 rr
  -> 10.233.90.113:8443           Masq    1      0          0         
TCP  10.233.30.118:15021 rr
  -> 10.233.90.113:15021          Masq    1      0          0         
TCP  10.233.32.100:5432 rr
  -> 10.233.90.109:5432           Masq    1      0          0         
TCP  10.233.36.31:8080 rr
  -> 10.233.90.105:8080           Masq    1      0          0         
TCP  10.233.36.104:6379 rr
  -> 10.233.96.57:6379            Masq    1      0          0         
  -> 10.233.97.24:6379            Masq    1      0          0         
  -> 10.233.98.28:6379            Masq    1      0          0         
TCP  10.233.37.5:80 rr
  -> 10.233.90.121:8080           Masq    1      0          0         
TCP  10.233.38.98:443 rr
  -> 10.233.90.106:443            Masq    1      0          0         
TCP  10.233.38.201:80 rr
  -> 10.233.92.132:80             Masq    1      0          0         
TCP  10.233.43.55:8383 rr
TCP  10.233.43.55:8686 rr
TCP  10.233.44.222:16685 rr
  -> 10.233.92.88:16685           Masq    1      0          0         
TCP  10.233.44.222:16686 rr
  -> 10.233.92.88:16686           Masq    1      0          0         
TCP  10.233.46.220:5432 rr
  -> 10.233.92.238:5432           Masq    1      0          0         
TCP  10.233.47.5:8080 rr persistent 10800
  -> 10.233.90.252:8080           Masq    1      0          0         
  -> 10.233.92.11:8080            Masq    1      0          0         
  -> 10.233.96.72:8080            Masq    1      0          0         
TCP  10.233.47.5:9093 rr persistent 10800
  -> 10.233.90.252:9093           Masq    1      0          0         
  -> 10.233.92.11:9093            Masq    1      0          0         
  -> 10.233.96.72:9093            Masq    1      0          0         
TCP  10.233.47.94:80 rr
  -> 10.233.90.127:8080           Masq    1      0          0         
TCP  10.233.47.209:80 rr
  -> 10.233.96.59:8080            Masq    1      0          0         
TCP  10.233.48.169:9411 rr
  -> 10.233.92.169:9411           Masq    1      0          0         
TCP  10.233.48.169:14250 rr
  -> 10.233.92.169:14250          Masq    1      0          0         
TCP  10.233.48.169:14267 rr
  -> 10.233.92.169:14267          Masq    1      0          0         
TCP  10.233.48.169:14268 rr
  -> 10.233.92.169:14268          Masq    1      0          0         
TCP  10.233.49.142:50000 rr
  -> 10.233.96.63:50000           Masq    1      0          0         
TCP  10.233.52.125:9090 rr
  -> 10.233.90.130:9090           Masq    1      0          0         
TCP  10.233.52.125:20001 rr
  -> 10.233.90.130:20001          Masq    1      0          0         
TCP  10.233.52.213:6379 rr
  -> 10.233.97.23:6379            Masq    1      2          22        
TCP  10.233.52.213:26379 rr
  -> 10.233.97.23:26379           Masq    1      1          15        
TCP  10.233.52.236:8081 rr
  -> 10.233.90.106:8081           Masq    1      0          0         
TCP  10.233.55.134:80 rr
  -> 10.233.92.237:9000           Masq    1      0          0         
TCP  10.233.55.134:9001 rr
  -> 10.233.92.237:9001           Masq    1      0          0         
TCP  10.233.58.139:6379 rr
  -> 10.233.90.110:6379           Masq    1      0          0         
TCP  10.233.58.191:80 rr
  -> 10.233.97.37:8000            Masq    1      0          0         
  -> 10.233.98.46:8000            Masq    1      0          0         
TCP  10.233.59.244:8080 rr persistent 10800
  -> 10.233.90.11:8080            Masq    1      0          0         
  -> 10.233.96.71:8080            Masq    1      0          0         
TCP  10.233.59.244:9090 rr persistent 10800
  -> 10.233.90.11:9090            Masq    1      0          0         
  -> 10.233.96.71:9090            Masq    1      0          0         
TCP  10.233.59.249:80 rr
  -> 10.233.90.126:8080           Masq    1      0          0         
TCP  10.233.63.40:5000 rr
  -> 10.233.90.117:5000           Masq    1      0          0         
TCP  10.233.63.40:8080 rr
  -> 10.233.90.117:8080           Masq    1      0          0         
TCP  10.233.63.123:8080 rr
  -> 10.233.90.106:8080           Masq    1      0          0         
TCP  10.233.97.0:30002 rr
  -> 10.233.90.112:8080           Masq    1      0          0         
TCP  10.233.97.0:30004 rr
  -> 10.233.90.112:4443           Masq    1      0          0         
TCP  10.233.97.0:30180 rr
  -> 10.233.96.63:8080            Masq    1      0          0         
TCP  10.233.97.0:30880 rr
  -> 10.233.97.37:8000            Masq    1      0          0         
  -> 10.233.98.46:8000            Masq    1      0          0         
TCP  10.233.97.0:31568 rr
  -> 10.233.92.237:9000           Masq    1      0          0         
TCP  10.233.97.0:31919 rr
  -> 10.233.90.113:15021          Masq    1      0          0         
TCP  10.233.97.0:32049 rr
  -> 10.233.90.113:8443           Masq    1      0          0         
TCP  10.233.97.0:32225 rr
  -> 10.233.92.132:80             Masq    1      0          0         
TCP  10.233.97.0:32339 rr
  -> 10.233.90.113:8080           Masq    1      0          0         
TCP  10.233.97.0:32542 rr
  -> 10.233.92.237:9001           Masq    1      0          0         
UDP  10.233.0.3:53 rr
  -> 10.233.97.41:53              Masq    1      0          0         
  -> 10.233.97.42:53              Masq    1      0          0

kevendeng

这个错误是由于内核版本不支持ipvs的端口复用导致，仅是一个性能优化项，对功能本身不影响。

从设置的ipvs规则来看，kube-proxy已经正常地设置了kube-apiserver的端口转发，问题可能出现在网络栈的其他层面，如节点防火墙等。

建议在ks-installer所在节点上，手动执行curl命令+kube-apiserver service地址及局域网真实地址：curl -k https://10.233.0.1:443、curl -k https://192.168.6.149/183:6443，看能否正常连通。

更进一步，你可以继续在该节点上，进入ks-installer pod所在的 network namespace：

// 获取ks-installer pod 对应container的id
kubectl -n kubesphere-system get pod -l app=ks-installer -o custom-columns=:'{..containerID}'

// 获取该container 对应的pid
docker inspect <容器id（前缀）> -f '{{json .State.Pid}}'

// 进入该进程所属network namespace
nsenter -n -t <容器pid>

// 确认已进入net ns，输出应该只有lo及容器网口
ip a

// 测试
curl xxx

lutixia

kevendeng

果然大佬，还真是网络问题，不管在pod还是宿主机curl，都是被禁止状态，可以ping通，检查了所有节点的firewalld都是关闭状态。

}[root@node3 ~]#curl -k https://10.233.0.1:443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}[root@node3 ~]# curl -k https://192.168.6.149:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}[root@node3 ~]#
[root@node3 ~]#ping 192.168.6.1493
PING 192.168.6.149 (192.168.6.149) 56(84) bytes of data.
64 bytes from 192.168.6.149: icmp_seq=1 ttl=64 time=0.374 ms
64 bytes from 192.168.6.149: icmp_seq=2 ttl=64 time=0.468 ms
^C
--- 192.168.6.149 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.374/0.421/0.468/0.047 ms
[root@node3 ~]# ping 10.233.0.1
PING 10.233.0.1 (10.233.0.1) 56(84) bytes of data.
64 bytes from 10.233.0.1: icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from 10.233.0.1: icmp_seq=2 ttl=64 time=0.033 ms

kevendeng

lutixia

首先，curl service ip已经成功了，只是返回的内容提示你没有权限，这跟connectin refused是不同的
所以，在pod里，curl是能成功的吗？

其次，你使用局域网ip访问时，端口错误了，应该是6443而非443

建议修改后重新试试

firewalld是通过写入iptables规则来实现防火墙规则的，即使firewalld没有启动，iptables也会包含多种过滤规则。除此之外，你的机器所在的环境本身（如某些云厂商的vpc），也可能有防火墙或安全组等规则，阻止某些源ip访问。

lutixia

kevendeng

在pod中curl和上面一样的错误；

你可能看的是我之前删除的图片，后来的图片中修改了访问端口6443；

我的不是云主机，在升级前用3.2.1是没有出现这个问题，所以和安全组策略应该没有关系；

有什么方式可以快速的恢复环境么？如果直接修改devops中的镜像版本有没有可能直接完成升级？

kevendeng

按理来说，pod内能访问到kube-apiserver，ks-installer不该有这个报错。

你可以看看ks-installer最新的日志，是否还是连接不上。

由于可能有其他初始化操作（CRD安装等）直接修改devops的镜像，很可能不会成功，可以试试删除ks-installer的pod，触发一次重新安装。

lutixia

kevendeng

彻底被你们的技术折服了，在web界面重新创建ks-installer后的一切都好了，devops的相关pod镜像版本也顺利升级到3.3.0，前面出现的devops等问题都解决了，在此感谢大佬不辞辛苦的帮忙解决问题。

peteradmin

kevendeng
感谢大佬，我的这个问题也是重建ks-install就好了~~~

kevendeng

lutixia

不客气，问题解决了就好