创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
发帖前请点击 发表主题 右边的 预览(👀) 按钮,确保帖子格式正确。

操作系统信息
例如:虚拟机,Centos7.9,master: 4C/8G,worker:16C/32G

Kubernetes版本信息
v1.23.17。3主7从。

容器运行时
docker ,V20.10.8

KubeSphere版本信息

v3.4.0。在线安装。全套安装,从V3.3.2升级。

问题是什么
无法登录:

ks-apiserver 三个pod一直反复重启:

### kubectl describe pod ks-apiserver-56546b89bd-dj4bq -n kubesphere-system:

……

Events:

Type Reason Age From Message

Warning BackOff 5m56s (x711 over 3h59m) kubelet Back-off restarting failed container

Warning Unhealthy 55s (x522 over 4h10m) kubelet Liveness probe failed: Get “http://10.233.73.226:9090/healthz”: dial tcp 10.233.73.226:9090: connect: connection refused

### kubectl logs ks-apiserver-56546b89bd-dj4bq -n kubesphere-system:

W1008 15:31:57.124901 1 client_config.go:618] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.

W1008 15:31:57.130008 1 client_config.go:618] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.

I1008 15:31:57.193448 1 interface.go:50] start helm repo informer

I1008 15:31:58.143257 1 apiserver.go:428] Start cache objects

W1008 15:32:02.847106 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:02.847220 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:03.948698 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:03.948874 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:05.848043 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:05.848206 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:06.147139 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:06.147225 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:09.546978 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:09.547090 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:12.248685 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:12.248809 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:16.150053 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:16.150559 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:18.149169 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:18.149267 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:26.053302 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:26.053397 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:27.647285 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:27.647379 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:47.246709 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:47.246847 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:32:51.554669 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:32:51.554798 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Receiver: failed to list *v2beta1.Receiver: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Receiver failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

W1008 15:33:21.448759 1 reflector.go:424] pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

E1008 15:33:21.448892 1 reflector.go:140] pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.Config: failed to list *v2beta1.Config: conversion webhook for notification.kubesphere.io/v2beta2, Kind=Config failed: Post “https://notification-manager-webhook.kubesphere-monitoring-system.svc:443/convert?timeout=30s”: x509: certificate signed by unknown authority

感谢大佬们帮帮忙,谢谢!

    qczrzl
    感谢! 但我执行这个脚本没有成功,是否版本问题?
    `[root@master1 ~]# ./path.sh
    The CustomResourceDefinition “configs.notification.kubesphere.io” is invalid:

    • spec.group: Required value
    • spec.scope: Required value
    • spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion(nil): must have exactly one version marked as storage version
    • spec.names.plural: Required value
    • spec.names.singular: Required value
    • spec.names.kind: Required value
    • spec.names.listKind: Required value
    • spec.scope: Invalid value: "": field is immutable
    • spec.names.kind: Invalid value: "": field is immutable
    • spec.group: Invalid value: "": field is immutable
    • spec.names.plural: Invalid value: "": field is immutable
    • status.storedVersions[0]: Invalid value: “v2beta2”: must appear in spec.versions
      The CustomResourceDefinition “receivers.notification.kubesphere.io” is invalid:
    • spec.group: Required value
    • spec.scope: Required value
    • spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion(nil): must have exactly one version marked as storage version
    • spec.names.plural: Required value
    • spec.names.singular: Required value
    • spec.names.kind: Required value
    • spec.names.listKind: Required value
    • spec.scope: Invalid value: "": field is immutable
    • spec.names.kind: Invalid value: "": field is immutable
    • spec.group: Invalid value: "": field is immutable
    • spec.names.plural: Invalid value: "": field is immutable
    • status.storedVersions[0]: Invalid value: “v2beta2”: must appear in spec.versions`

    path.sh脚本:
    `#!/bin/bash

    caBundle=$(kubectl get validatingWebhookConfiguration notification-manager-validating-webhook -o jsonpath=‘{.webhooks[0].clientConfig.caBundle}’)

    cat > /tmp/patch.yaml <<EOF
    spec:
    conversion:
    webhook:
    clientConfig:
    caBundle: ${caBundle}
    service:
    namespace: kubesphere-monitoring-system
    EOF

    kubectl patch crd configs.notification.kubesphere.io –type=merge –patch-file /tmp/patch.yaml
    kubectl patch crd receivers.notification.kubesphere.io –type=merge –patch-file /tmp/patch.yaml`

    notification-manager 信息如下:
    kubectl describe notification-manager -n kubesphere-monitoring-system

    `Name: global-email-receiver
    Namespace:
    Labels: type=global
    Annotations: kubesphere.io/creator: admin
    reloadtimestamp: 2023-09-21 17:19:13.759526576 +0800 CST m=+504204.023130488
    API Version: notification.kubesphere.io/v2beta2
    Kind: Receiver
    Metadata:
    Creation Timestamp: 2022-11-09T08:19:43Z
    Generation: 1
    Managed Fields:
    API Version: notification.kubesphere.io/v2beta1
    Fields Type: FieldsV1
    fieldsV1:
    f:metadata:
    f:annotations:
    .:
    f:kubesphere.io/creator:
    f:labels:
    .:
    f:type:
    f:spec:
    .:
    f:email:
    .:
    f:enabled:
    f:to:
    Manager: ks-apiserver
    Operation: Update
    Time: 2022-11-09T08:19:43Z
    API Version: notification.kubesphere.io/v2beta2
    Fields Type: FieldsV1
    fieldsV1:
    f:metadata:
    f:annotations:
    f:reloadtimestamp:
    Manager: notification-manager
    Operation: Update
    Time: 2022-11-09T18:22:06Z
    Resource Version: 194184799
    UID: 14022a00-c4d7-43a9-9d56-539b3a13d772
    Spec:
    Email:
    Enabled: false
    To:
    xxx@xxx.com
    Status:
    Events: <none>

    Name: notification-manager
    Namespace:
    Labels: app=notification-manager
    app.kubernetes.io/managed-by=Helm
    Annotations: meta.helm.sh/release-name: notification-manager
    meta.helm.sh/release-namespace: kubesphere-monitoring-system
    API Version: notification.kubesphere.io/v2beta2
    Kind: NotificationManager
    Metadata:
    Creation Timestamp: 2022-08-27T13:04:03Z
    Generation: 2
    Managed Fields:
    API Version: notification.kubesphere.io/v2beta2
    Fields Type: FieldsV1
    fieldsV1:
    f:metadata:
    f:annotations:
    .:
    f:meta.helm.sh/release-name:
    f:meta.helm.sh/release-namespace:
    f:labels:
    .:
    f:app:
    f:app.kubernetes.io/managed-by:
    f:spec:
    .:
    f:affinity:
    f:defaultConfigSelector:
    .:
    f:matchLabels:
    .:
    f:type:
    f:defaultSecretNamespace:
    f:groupLabels:
    f:image:
    f:imagePullPolicy:
    f:nodeSelector:
    f:portName:
    f:receivers:
    .:
    f:globalReceiverSelector:
    .:
    f:matchLabels:
    .:
    f:type:
    f:options:
    .:
    f:email:
    .:
    f:deliveryType:
    f:notificationTimeout:
    f:slack:
    .:
    f:notificationTimeout:
    f:wechat:
    .:
    f:notificationTimeout:
    f:tenantKey:
    f:tenantReceiverSelector:
    .:
    f:matchLabels:
    .:
    f:type:
    f:replicas:
    f:resources:
    .:
    f:limits:
    .:
    f:cpu:
    f:memory:
    f:requests:
    .:
    f:cpu:
    f:memory:
    f:serviceAccountName:
    f:sidecars:
    .:
    f:tenant:
    .:
    f:image:
    f:name:
    f:type:
    f:template:
    .:
    f:language:
    f:languagePack:
    f:reloadCycle:
    f:text:
    .:
    f:name:
    f:namespace:
    f:tolerations:
    f:volumeMounts:
    f:volumes:
    Manager: helm
    Operation: Update
    Time: 2023-09-21T09:19:13Z
    Resource Version: 194184791
    UID: 49f079ea-9d2d-4e85-a7cf-6f43cbaaac1c
    Spec:
    Affinity:
    Batch Max Size: 100
    Batch Max Wait: 1m
    Default Config Selector:
    Match Labels:
    Type: default
    Default Secret Namespace: kubesphere-monitoring-federated
    Group Labels:
    alertname
    namespace
    Image: registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v2.3.0
    Image Pull Policy: IfNotPresent
    Node Selector:
    Port Name: webhook
    Receivers:
    Global Receiver Selector:
    Match Labels:
    Type: global
    Options:
    Email:
    Delivery Type: bulk
    Notification Timeout: 5
    Slack:
    Notification Timeout: 5
    Wechat:
    Notification Timeout: 5
    Tenant Key: user
    Tenant Receiver Selector:
    Match Labels:
    Type: tenant
    Replicas: 2
    Resources:
    Limits:
    Cpu: 500m
    Memory: 500Mi
    Requests:
    Cpu: 5m
    Memory: 20Mi
    Route Policy: All
    Service Account Name: notification-manager-sa
    Sidecars:
    Tenant:
    Image: registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
    Name: tenant
    Type: kubesphere
    Template:
    Language: English
    Language Pack:
    Name: zh-cn
    Namespace: kubesphere-monitoring-system
    Reload Cycle: 1m
    Text:
    Name: notification-manager-template
    Namespace: kubesphere-monitoring-system
    Tolerations:
    Volume Mounts:
    Volumes:
    Events: <none>

    Name: default-email-config
    Namespace:
    Labels: type=default
    Annotations: reloadtimestamp: 2023-09-21 17:18:45.899257459 +0800 CST m=+504176.162861364
    API Version: notification.kubesphere.io/v2beta2
    Kind: Config
    Metadata:
    Creation Timestamp: 2022-11-09T08:19:41Z
    Generation: 2
    Managed Fields:
    API Version: notification.kubesphere.io/v2beta1
    Fields Type: FieldsV1
    fieldsV1:
    f:metadata:
    f:labels:
    .:
    f:type:
    f:spec:
    .:
    f:email:
    .:
    f:authPassword:
    .:
    f:key:
    f:name:
    f:authUsername:
    f:from:
    f:requireTLS:
    f:smartHost:
    .:
    f:host:
    f:port:
    Manager: ks-apiserver
    Operation: Update
    Time: 2022-11-09T08:20:36Z
    API Version: notification.kubesphere.io/v2beta2
    Fields Type: FieldsV1
    fieldsV1:
    f:metadata:
    f:annotations:
    .:
    f:reloadtimestamp:
    Manager: notification-manager
    Operation: Update
    Time: 2023-03-24T05:26:21Z
    Resource Version: 194184602
    UID: df25a41a-432a-4b13-87da-0abb2da839cf
    Spec:
    Email:
    Auth Password:
    Value From:
    Secret Key Ref:
    Key: authPassword
    Name: global-email-config-secret
    Auth Username: xxx@xxx.com
    From: xxx@xxx.com
    Require TLS: false
    Smart Host:
    Host: smtp.exmail.qq.com
    Port: 25
    Status:
    Events: <none>
    `

    7 个月 后
    3 个月 后