• 安装部署

  • 升级3.4.0后，web无法登陆提示request to http://ks-apiserver/oauth/token failed

操作系统信息
虚拟机，Centos7.9，8C/16G

Kubernetes版本信息
v1.26.5 多节点。三主三从

容器运行时
使用containerd，版本containerd://1.6.4

KubeSphere版本信息
v3.3.2/v3.4.0。在线安装。全套升级。

问题是什么
v3.3.2升级v3.4.0，控制台服务登陆

ks-apiserver的报错日志

E0920 14:42:52.745803       1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
W0920 14:42:58.765923       1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
E0920 14:42:58.765954       1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
W0920 14:43:04.953994       1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
E0920 14:43:04.954020       1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
W0920 14:43:21.596185       1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
E0920 14:43:21.596213       1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
W0920 14:43:24.536053       1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused
E0920 14:43:24.536079       1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post "https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s": dial tcp 10.233.1.231:443: connect: connection refused

    shadowwalker

    执行kubectl get pod -A -o wide 看一下10.233.1.231和10.233.28.109对应的pod的状态, 看一下出什么错了

        shadowwalker

        kubectl get pod -n kubesphere-system 看一下有没有异常的pod, 使用describe /logs查看一下pod的信息

          16 天 后

          我的也是这种情况，但报错信息是证书发放机构问题：

          W1008 11:44:27.198439 1 client_config.go:618] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.

          W1008 11:44:27.203440 1 client_config.go:618] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.

          I1008 11:44:27.270383 1 interface.go:50] start helm repo informer

          I1008 11:44:28.017930 1 apiserver.go:428] Start cache objects

          W1008 11:44:33.354246 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

          E1008 11:44:33.354325 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

          各位大佬有啥解决方法？ 谢谢

          9 天 后

          你好 我也遇到这个问题，请问您解决了吗 能描述一下处理方案吗？

          可以用下面的脚本修复一下

          caBundle=$(kubectl get validatingWebhookConfiguration notification-manager-validating-webhook -o jsonpath=‘{.webhooks[0].clientConfig.caBundle}’)

          cat > /tmp/patch.yaml <<EOF

          spec:

          conversion:

           webhook: 

      clientConfig: 

        caBundle: ${caBundle}

        service: 

           namespace: kubesphere-monitoring-system

          EOF

          kubectl patch crd configs.notification.kubesphere.io –type=merge –patch-file /tmp/patch.yaml

          kubectl patch crd receivers.notification.kubesphere.io –type=merge –patch-file /tmp/patch.yaml

          2 个月 后
          1. caBundle=$(kubectl get validatingWebhookConfiguration notification-manager-validating-webhook -o jsonpath=‘{.webhooks[0].clientConfig.caBundle}’)
          2. kubectl edit crd configs.notification.kubesphere.io, 将$caBundle的值填入到spec.conversion.webhook.clientConfig.caBundle