• 监控日志

  • 修改opensearch的默认密码后，kubesphere自带的日志查询提示授权失败

操作系统信息
虚拟机，openeuler 24.3

Kubernetes版本信息

[root@ksmaster01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:27:46Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

容器运行时

Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.13
RuntimeApiVersion:  v1

KubeSphere版本信息
3.4.1

问题是什么
修改opensearch的默认密码后，fluentbit往opensearch写入日志时报401未授权，

使用用户admin和密码admin可以登录dashboard，但自定义的密码就不行

相同问题： kubesphere/kubesphere#6004

查看secret中的密码是自定义的密码，但登录还是使用的默认密码admin

[root@ksmaster01 ~]# kubectl get secret opensearch-credentials -n kubesphere-logging-system
NAME                     TYPE                       DATA   AGE
opensearch-credentials   kubernetes.io/basic-auth   2      19d

fluentbit错误日志:

2024-12-25T14:44:36.008861974+08:00 [2024/12/25 06:44:36] [ warn] [engine] failed to flush chunk '12-1735109074.972222575.flb', retry in 7 seconds: task_id=1, input=tail.2 > output=opensearch.0 (out_id=0)

2024-12-25T14:44:36.984787300+08:00 [2024/12/25 06:44:36] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:36.984863915+08:00 Unauthorized

2024-12-25T14:44:36.984875211+08:00 

2024-12-25T14:44:36.984883751+08:00 [2024/12/25 06:44:36] [ warn] [engine] chunk '12-1735109069.870988028.flb' cannot be retried: task_id=7, input=tail.2 > output=opensearch.0

2024-12-25T14:44:37.105842401+08:00 [2024/12/25 06:44:37] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:37.105905032+08:00 Unauthorized

2024-12-25T14:44:37.105911647+08:00 

2024-12-25T14:44:37.105917566+08:00 [2024/12/25 06:44:37] [ warn] [engine] failed to flush chunk '12-1735109076.8946702.flb', retry in 7 seconds: task_id=3, input=tail.2 > output=opensearch.0 (out_id=0)

2024-12-25T14:44:38.093340097+08:00 [2024/12/25 06:44:38] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:38.093373010+08:00 Unauthorized

2024-12-25T14:44:38.093375489+08:00 

2024-12-25T14:44:38.093377415+08:00 [2024/12/25 06:44:38] [ warn] [engine] failed to flush chunk '12-1735109076.985056291.flb', retry in 11 seconds: task_id=6, input=tail.2 > output=opensearch.0 (out_id=0)

2024-12-25T14:44:39.012889867+08:00 [2024/12/25 06:44:39] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:39.012936777+08:00 Unauthorized

2024-12-25T14:44:39.012942934+08:00 

2024-12-25T14:44:39.012948293+08:00 [2024/12/25 06:44:39] [ warn] [engine] failed to flush chunk '12-1735109078.93445265.flb', retry in 9 seconds: task_id=7, input=tail.2 > output=opensearch.0 (out_id=0)

2024-12-25T14:44:39.984426983+08:00 [2024/12/25 06:44:39] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:39.984510554+08:00 Unauthorized

2024-12-25T14:44:39.984520870+08:00 

2024-12-25T14:44:39.984527780+08:00 [2024/12/25 06:44:39] [ warn] [engine] failed to flush chunk '12-1735109079.259224636.flb', retry in 6 seconds: task_id=11, input=tail.2 > output=opensearch.0 (out_id=0)

2024-12-25T14:44:40.284674129+08:00 [2024/12/25 06:44:40] [error] [output:opensearch:opensearch.0] HTTP status=401 URI=/_bulk, response:

2024-12-25T14:44:40.284730228+08:00 Unauthorized

2024-12-25T14:44:40.284738141+08:00 

2024-12-25T14:44:40.284744550+08:00 [2024/12/25 06:44:40] [ warn] [engine] failed to flush chunk '12-1735109079.13082982.flb', retry in 9 seconds: task_id=8, input=tail.2 > output=opensearch.0 (out_id=0)

fluent-bit写入配置要检查output cr资源，看看内容是否符合预期

    Birdhk

    cr中是引用了对应的secret，但没有效果

    [root@ksmaster01 ~]# kubectl -n kubesphere-logging-system get outputs.logging.kubesphere.io
NAME                  AGE
opensearch            19d
opensearch-auditing   19d
opensearch-events     19d
[root@ksmaster01 ~]# 
[root@ksmaster01 ~]# 
[root@ksmaster01 ~]# kubectl -n kubesphere-logging-system get outputs.logging.kubesphere.io opensearch -oyaml
apiVersion: logging.kubesphere.io/v1alpha2
kind: Output
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"logging.kubesphere.io/v1alpha2","kind":"Output","metadata":{"annotations":{},"labels":{"logging.kubesphere.io/component":"logging","logging.kubesphere.io/enabled":"true"},"name":"opensearch","namespace":"kubesphere-logging-system"},"spec":{"matchRegex":"(?:kube|service)\\.(.*)","opensearch":{"host":"opensearch-cluster-data.kubesphere-logging-system.svc","httpPassword":{"valueFrom":{"secretKeyRef":{"key":"password","name":"opensearch-credentials"}}},"httpUser":{"valueFrom":{"secretKeyRef":{"key":"username","name":"opensearch-credentials"}}},"logstashFormat":true,"logstashPrefix":"ks-whizard-logging","port":9200,"suppressTypeName":true,"timeKey":"@timestamp","tls":{"verify":false}}}}
  creationTimestamp: "2024-12-06T05:33:27Z"
  generation: 1
  labels:
    logging.kubesphere.io/component: logging
    logging.kubesphere.io/enabled: "true"
  name: opensearch
  namespace: kubesphere-logging-system
  resourceVersion: "3998"
  uid: 6be93e55-e408-4c31-bda2-9cea021738d9
spec:
  matchRegex: (?:kube|service)\.(.*)
  opensearch:
    host: opensearch-cluster-data.kubesphere-logging-system.svc
    httpPassword:
      valueFrom:
        secretKeyRef:
          key: password
          name: opensearch-credentials
    httpUser:
      valueFrom:
        secretKeyRef:
          key: username
          name: opensearch-credentials
    logstashFormat: true
    logstashPrefix: ks-whizard-logging
    port: 9200
    suppressTypeName: true
    timeKey: '@timestamp'
    tls:
      verify: false

      Birdhk

      是的，

      kubesphere.yaml

      
logging:
  host: https://opensearch-cluster-data.kubesphere-logging-system.svc:9200
  basicAuth: True
  username: "admin"
  password: "hJYW3aERxbVBI/YnzNb3"
  indexPrefix: ks-whizard-logging
      [root@ksmaster01]# kubectl -n kubesphere-logging-system get secret opensearch-credentials  -oyaml
apiVersion: v1
data:
  password: aEpZVzNhRVJ4YlZCSS9ZbnpOYjM=
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: "2024-12-06T05:31:28Z"
  name: opensearch-credentials
  namespace: kubesphere-logging-system
  resourceVersion: "2312"
  uid: 3e23bb03-bc15-4ca7-9d2e-b68fbc7c206f
type: kubernetes.io/basic-auth 
[root@ksmaster01]# echo "aEpZVzNhRVJ4YlZCSS9ZbnpOYjM=" | base64 -d && echo
hJYW3aERxbVBI/YnzNb3

        fuent-bit写入opensearch的配置在output里，控制台页面要查看数据配置在kubesphere的config里，保证这两个地方的配置正确。

          Birdhk

          配置都是对的，根本问题是admin用户的密码还是admin，并没有修改为自定义的密码，应该是opensearch安装初始化时要么没有使用自定义的变量，要么就是定死了

            opensearch的密码你要改只能自己单独改，这些配置都是为了读取和写入数据的

              Birdhk

              有一个疑问opensearch这个密码是在第一次安装kubesphere集群时在config-sample.yaml中定义的，此处的默认密码admin是否可以修改？修改后在安装Opensearch时是否会使用自定义密码作为admin的默认密码

                那也是是为了生成配置给fluent和查询使用的，不包括修改密码