修改 CRD 中的 ClusterConfiguration 会重置 Prometheus 等监控组件

LeoDemon

操作系统信息
物理机，CentOS Linux release 7.5.1804 (Core)， 8C/16G 3 台 master

Kubernetes版本信息
1.20.5

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

[root@master1 ~]# docker version

Client: Docker Engine - Community

Version: 20.10.9

API version: 1.41

Go version: go1.16.8

Git commit: c2ea9bc

Built: Mon Oct 4 16:08:14 2021

OS/Arch: linux/amd64

Context: default

Experimental: true

Server: Docker Engine - Community

Engine:

Version: 20.10.9

API version: 1.41 (minimum version 1.12)

Go version: go1.16.8

Git commit: 79ea9d3

Built: Mon Oct 4 16:06:37 2021

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: 1.4.11

GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8

runc:

Version: 1.0.2

GitCommit: v1.0.2-0-g52b36a2

docker-init:

Version: 0.19.0

GitCommit: de40ad0

[root@master1 ~]#

[root@master1 ~]# crictl version

Version: 0.1.0

RuntimeName: docker

RuntimeVersion: 20.10.9

RuntimeApiVersion: 1.41.0

[root@master1 ~]#

KubeSphere版本信息
v3.2.0 ；在已有K8s上安装

问题是什么
1）在 CRD 中将 prometheus, alertmanager 等配置的副本数修改为0，后台会将对应的副本数变成 0，没有问题。

2）在这之后，修改 CRD 中的 ClusterConfiguration ，将 devops、etcd、logging、servicemesh 中任意一项的 enabled 从 true 改成 false（或 false改成true）

3）后台运行之后，会重新将第 1）步的修改重置，即 CRD 中 prometheus, alertmanager 等配置的副本数又变回了默认值（比如1），

如何保证调整 CRD 中的 ClusterConfiguration 后，其他的 CRD 不会受影响？

我看官方文档中 **[KubeSphere 3.2.0 卸载可插拔组件 ](https://kubesphere.io/zh/docs/pluggable-components/uninstall-pluggable-components/)**的开头有一个“准备工作”：

将 CRD 配置文件 ClusterConfiguration 中的 ks-installer 中的 enabled 字段的值从 true 改为 false。

可我始终未找到这个 enabled 字段，请大佬们指点。

LeoDemon

简单看了一下 ks-installer pod 的代码和日志，我理解每次修改 CRD 中的 ClusterConfiguration 配置文件，就会将enabled=true 的组件与第一次安装时的初始值进行比较，如果有变化，就会重新安装对应组件。

这个逻辑不对吧？

我理解应该是这次改动了哪个配置，就重新安装哪个组件，我没有动 Prometheus 相关配置，它就不应该变化呀。

这一次改动的变化，应该与改动前的配置进行比对，为何要与第一次安装时的初始值进行比对，这样设计的考虑是什么？

Cauchy

LeoDemon

任务执行前会检查 ClusterConfiguration 中 status 里的组件状态，如果组件状态为enabled，则会逃过相关组件的部署络流程

我看 prometheus 的安装逻辑里也是会去判断 status.monitoring 的状态

LeoDemon

Cauchy

谢谢您的回复。

现在 CRD 中 ClusterConfiguration 的配置文件如下：

apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"installer.kubesphere.io/v1alpha1","kind":"ClusterConfiguration","metadata":{"annotations":{},"labels":{"version":"v3.2.0"},"name":"ks-installer","namespace":"kubesphere-system"},"spec":{"alerting":{"enabled":false},"auditing":{"enabled":false},"authentication":{"jwtSecret":""},"common":{"es":{"basicAuth":{"enabled":false,"password":"","username":""},"elasticsearchDataVolumeSize":"20Gi","elasticsearchMasterVolumeSize":"4Gi","elkPrefix":"logstash","externalElasticsearchPort":"","externalElasticsearchUrl":"","logMaxAge":7},"minioVolumeSize":"20Gi","monitoring":{"endpoint":"http://prometheus-operated.kubesphere-monitoring-system.svc:9090"},"openldap":{"enabled":false},"openldapVolumeSize":"2Gi","redis":{"enabled":false},"redisVolumSize":"2Gi"},"console":{"enableMultiLogin":true,"port":30880},"devops":{"enabled":false,"jenkinsJavaOpts_MaxRAM":"2g","jenkinsJavaOpts_Xms":"512m","jenkinsJavaOpts_Xmx":"512m","jenkinsMemoryLim":"2Gi","jenkinsMemoryReq":"1500Mi","jenkinsVolumeSize":"8Gi"},"etcd":{"endpointIps":"localhost","monitoring":false,"port":2379,"tlsEnable":true},"events":{"enabled":false,"ruler":{"enabled":true,"replicas":2}},"kubeedge":{"cloudCore":{"cloudHub":{"advertiseAddress":[""],"nodeLimit":"100"},"cloudhubHttpsPort":"10002","cloudhubPort":"10000","cloudhubQuicPort":"10001","cloudstreamPort":"10003","nodeSelector":{"node-role.kubernetes.io/worker":""},"service":{"cloudhubHttpsNodePort":"30002","cloudhubNodePort":"30000","cloudhubQuicNodePort":"30001","cloudstreamNodePort":"30003","tunnelNodePort":"30004"},"tolerations":[],"tunnelPort":"10004"},"edgeWatcher":{"edgeWatcherAgent":{"nodeSelector":{"node-role.kubernetes.io/worker":""},"tolerations":[]},"nodeSelector":{"node-role.kubernetes.io/worker":""},"tolerations":[]},"enabled":false},"local_registry":"","logging":{"enabled":false,"logsidecar":{"enabled":true,"replicas":2}},"metrics_server":{"enabled":false},"monitoring":{"prometheusMemoryRequest":"400Mi","prometheusVolumeSize":"20Gi","storageClass":""},"multicluster":{"clusterRole":"none"},"network":{"ippool":{"type":"none"},"networkpolicy":{"enabled":false},"topology":{"type":"none"}},"openpitrix":{"store":{"enabled":false}},"persistence":{"storageClass":""},"servicemesh":{"enabled":false}}}
  labels:
    version: v3.2.0
  name: ks-installer
  namespace: kubesphere-system
spec:
  alerting:
    enabled: true
  auditing:
    enabled: true
  authentication:
    jwtSecret: ''
  common:
    es:
      basicAuth:
        enabled: false
        password: ''
        username: ''
      elasticsearchDataVolumeSize: 20Gi
      elasticsearchMasterVolumeSize: 4Gi
      elkPrefix: logstash
      externalElasticsearchPort: ''
      externalElasticsearchUrl: ''
      logMaxAge: 7
    minioVolumeSize: 20Gi
    monitoring:
      endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090'
    openldap:
      enabled: false
    openldapVolumeSize: 2Gi
    redis:
      enabled: false
    redisVolumSize: 2Gi
  console:
    enableMultiLogin: true
    port: 30880
  devops:
    enabled: true
    jenkinsJavaOpts_MaxRAM: 2g
    jenkinsJavaOpts_Xms: 512m
    jenkinsJavaOpts_Xmx: 512m
    jenkinsMemoryLim: 2Gi
    jenkinsMemoryReq: 1500Mi
    jenkinsVolumeSize: 8Gi
  etcd:
    endpointIps: '172.16.0.171,172.16.0.220,172.16.0.240'
    monitoring: true
    port: 2379
    tlsEnable: true
  events:
    enabled: false
    ruler:
      enabled: true
      replicas: 2
  kubeedge:
    cloudCore:
      cloudHub:
        advertiseAddress:
          - ''
        nodeLimit: '100'
      cloudhubHttpsPort: '10002'
      cloudhubPort: '10000'
      cloudhubQuicPort: '10001'
      cloudstreamPort: '10003'
      nodeSelector:
        node-role.kubernetes.io/worker: ''
      service:
        cloudhubHttpsNodePort: '30002'
        cloudhubNodePort: '30000'
        cloudhubQuicNodePort: '30001'
        cloudstreamNodePort: '30003'
        tunnelNodePort: '30004'
      tolerations: []
      tunnelPort: '10004'
    edgeWatcher:
      edgeWatcherAgent:
        nodeSelector:
          node-role.kubernetes.io/worker: ''
        tolerations: []
      nodeSelector:
        node-role.kubernetes.io/worker: ''
      tolerations: []
    enabled: false
  local_registry: ''
  logging:
    enabled: true
    logsidecar:
      enabled: true
      replicas: 2
  metrics_server:
    enabled: false
  monitoring:
    prometheusMemoryRequest: 400Mi
    prometheusVolumeSize: 20Gi
    storageClass: ''
  multicluster:
    clusterRole: none
  network:
    ippool:
      type: none
    networkpolicy:
      enabled: false
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  persistence:
    storageClass: ''
  servicemesh:
    enabled: true

我该如何配置这个变量呢，在文件最后加上下面这段是没有用的。

  status:
    monitoring:
      status: enabled

我看 ks-installer 的源码中有下面这段：

def getComponentLists():
    readyToEnabledList = [
        'monitoring',
        'multicluster',
        'openpitrix',
        'network']
    readyToDisableList = []
    global configFile

    if os.path.exists(configFile):
        with open(configFile, 'r') as f:
            configs = json.load(f)
        f.close()
    else:
        print("The configuration file does not exist !  {}".format(configFile))
        exit()

    for component, parameters in configs.items():
        if (not isinstance(parameters, str)) or (
                not isinstance(parameters, int)):
            try:
                for j, value in parameters.items():
                    if (j == 'enabled') and (value):
                        readyToEnabledList.append(component)
                        break
                    elif (j == 'enabled') and (value == False):
                        readyToDisableList.append(component)
                        break
            except BaseException:
                pass
    try:
        readyToEnabledList.remove("metrics_server")
    except BaseException:
        pass
   ...
   ...

这代码的意思是 monitoring 组件默认就被 enabled 吧。

我通过 kubectl describe ClusterConfiguration ks-installer -n kubesphere-system 拿到最后一段内容如下：

...
...
Status:
  Alerting:
    Enabled Time:  2021-12-13T17:25:42CST
    Status:        enabled
  Auditing:
    Enabled Time:  2021-12-13T17:23:36CST
    Status:        enabled
  Cluster Id:      0e488f87-51f2-4513-82f7-5d776f6e8999-1639387570
  Core:
    Enabled Time:  2021-12-13T17:27:39CST
    Status:        enabled
    Version:       v3.2.0
  Devops:
    Enabled Time:  2021-12-13T17:24:27CST
    Status:        enabled
  Es:
    Enabled Time:  2021-12-13T17:21:25CST
    Status:        enabled
  Fluentbit:
    Enabled Time:  2021-12-13T17:21:34CST
    Status:        enabled
  Logging:
    Enabled Time:  2021-12-13T17:23:59CST
    Status:        enabled
  Minio:
    Enabled Time:  2021-12-13T17:21:20CST
    Status:        enabled
  Monitoring:
    Enabled Time:  2021-12-13T17:29:29CST
    Status:        enabled
  Openldap:
    Enabled Time:  2021-12-13T17:21:11CST
    Status:        enabled
  Redis:
    Enabled Time:  2021-12-13T17:21:05CST
    Status:        enabled
  Servicemesh:
    Enabled Time:  2021-12-13T17:24:24CST
    Status:        enabled
Events:            <none>

还劳烦指导下，如何配置，才能不会被默认模板给重置掉，谢谢。