kubesphere: 3.1.0
kubernetes: 1.20.6
elasticsearch: 7.5.1

ClusterConfiguration CRD 为(只输出了有用的部分):

spec:
  alerting:
    enabled: true
  auditing:
    enabled: true
  authentication:
    jwtSecret: NbxwnhExCLSfQnRlvcuudDgwj3G14cKb
  common:
    es:
      basicAuth:
        enabled: true
        password: '=sm%123}Drhg}Oa'
        username: kslog
      elasticsearchDataVolumeSize: 20Gi
      elasticsearchMasterVolumeSize: 4Gi
      elkPrefix: dev
      externalElasticsearchPort: '80'
      externalElasticsearchUrl: es.platform.baidu.cn
      logMaxAge: 7
    minioVolumeSize: 20Gi
    monitoring:
      endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090'
    openldap:
      enabled: false
    openldapVolumeSize: 2Gi
    redis:
      enabled: false
    redisVolumSize: 2Gi

解析 fluent-bit-config

kubectl get secrets -n kubesphere-logging-system fluent-bit-config -oyaml
[Output]
    Name    es
    Match    kube_events
    Host    es.platform.baidu.cn
    Port    80
    HTTP_User    kslog
    HTTP_Passwd    =sm%123}Drhg}Oa
    Logstash_Format    true
    Logstash_Prefix    ks-dev-events
[Output]
    Name    es
    Match    kube_auditing
    Host    es.platform.baidu.cn
    Port    80
    HTTP_User    kslog
    HTTP_Passwd    =sm%123}Drhg}Oa
    Logstash_Format    true
    Logstash_Prefix    ks-dev-auditing
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    es.platform.baidu.cn
    Port    80
    HTTP_User    kslog
    HTTP_Passwd    =sm%123}Drhg}Oa
    Logstash_Format    true
    Logstash_Prefix    ks-dev-log
    Time_Key    @timestamp
apiVersion: logging.kubesphere.io/v1alpha2
kind: Output
metadata:
  name: es
  namespace: kubesphere-logging-system
  resourceVersion: '199132225'
  uid: 1af6ef9e-4fdc-47a7-97a7-6cc1172a8bac
spec:
  es:
    host: es.platform.baidu.cn
    httpPassword:
      valueFrom:
        secretKeyRef:
          key: password
          name: elasticsearch-credentials
    httpUser:
      valueFrom:
        secretKeyRef:
          key: username
          name: elasticsearch-credentials
    logstashFormat: true
    logstashPrefix: ks-dev-log
    port: 80
    timeKey: '@timestamp'
  matchRegex: '(?:kube|service)\.(.*)'
  • hongming 非常感谢您的回复,经过我的测试使用特殊字符没有影响,最终还是权限不足造成的,下面是我配置的最小权限。


kslog 用户具有 ks-** 索引的所有权。

查询抛出 403

这在2天前一切正常,过了2两天后出现的。控制台是哪个 pod 去查询日志的,我需要去排查下是否因为某个 secret 未更新导致,在这两天中我启用过其他组件。

当使用日志查询时:ks-apiserver-5d449cf77c-l5zf6 抛出

E0804 15:28:58.947877       1 handler.go:380] [403 Forbidden] type: security_exception, reason: action [cluster:monitor/main] is unauthorized for user [kslog]

E0804 15:28:58.947903       1 utils.go:76] /root/go/src/kubesphere.io/kubesphere/pkg/kapis/tenant/v1alpha2/handler.go:381 [403 Forbidden] type: security_exception, reason: action [cluster:monitor/main] is unauthorized for user [kslog]

I0804 15:28:58.947960       1 apiserver.go:615] 10.42.4.199 - "GET /kapis/tenant.kubesphere.io/v1alpha2/logs?operation=statistics&start_time=1628006400&end_time=1628062138 HTTP/1.1" 500 113 28ms

这个是es报的错,kslog权限不足,你检查下你的es集群吧

    wanjunlei 我对 kslog 授予了 ks-* 的索引的所有权,日志写入正常,使用 kslog 用户登录 kibana 可以查看日志。 就是在控制台查看报 403。

    hyt05 看着是 es 的密码没被读取到,看看 ks-installer 有没有异常日志,kubectl -n kubesphere-system get cm kubesphere-config 检查一下 es 的配置有没有生效,如果配置是对就重启一下ks-apiserver 重新加载配置文件

      hongming ks-installer 没有异常日志,es的配置是生效的,重启 ks-apiserver 也是一样,请问会不会和密码中有特殊字符有影响,但是第一次配置完各项都没问题。

      kubectl -n kubesphere-system get cm kubesphere-config -oyaml
      
      network:
            ippoolType: none
            weaveScopeHost: weave-scope-app.weave
          monitoring:
            endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
          logging:
            host: http://es.platform.baidu.cn:80
            basicAuth: True
            username: "kslog"
            password: "=sm%8yl3}Drhg}Oa"
            indexPrefix: ks-dev-log
          events:
            host: http://es.platform.baidu.cn:80
            basicAuth: True
            username: "kslog"
            password: "=sm%8yl3}Drhg}Oa"
            indexPrefix: ks-dev-events
          auditing:
            enable: true
            host: http://es.platform.baidu.cn:80
            basicAuth: True
            username: "kslog"
            password: "=sm%8yl3}Drhg}Oa"
            indexPrefix: ks-dev-auditing

      kubectl get secrets -n kubesphere-logging-system fluent-bit-config -oyaml

      [Output]
          Name    es
          Match    kube_events
          Host    es.platform.baidu.cn
          Port    80
          HTTP_User    kslog
          HTTP_Passwd    =sm%8yl3}Drhg}Oa
          Logstash_Format    true
          Logstash_Prefix    ks-dev-events
      [Output]
          Name    es
          Match    kube_auditing
          Host    es.platform.baidu.cn
          Port    80
          HTTP_User    kslog
          HTTP_Passwd    =sm%8yl3}Drhg}Oa
          Logstash_Format    true
          Logstash_Prefix    ks-dev-auditing
      [Output]
          Name    es
          Match_Regex    (?:kube|service)\.(.*)
          Host    es.platform.baidu.cn
          Port    80
          HTTP_User    kslog
          HTTP_Passwd    =sm%8yl3}Drhg}Oa
          Logstash_Format    true
          Logstash_Prefix    ks-dev-log
          Time_Key    @timestamp

        hyt05 es 的密码改过吗,如果是因为改密码引起的,可以把密码中的特殊字符去掉试试

          hongming 非常感谢您的回复,经过我的测试使用特殊字符没有影响,最终还是权限不足造成的,下面是我配置的最小权限。