KubeSphere v3.4.1 在腾讯云 TKE 上安装，外部ES对接后，工具箱无法按容器组等维度搜索

tmkddskf

操作系统信息
容器集群是腾讯云TKE，TencentOS Server 3.1

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.1-tke.3", GitCommit:"38d7740632d901ebf9040053ca5ece34d1f809be", GitTreeState:"clean", BuildDate:"2023-11-30T13:06:08Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.26) exceeds the supported minor version skew of +/-1

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.6.9-tke.4
RuntimeApiVersion:  v1

KubeSphere版本信息
例如：3.4.1，在线安装。在已有K8s上安装。

问题是什么
1. 配置好后无法，日志那边一直显示“今日总共收录了0个容器0条日志信息”，检查 fluent 容器，发现日志被跳过了，没有解析

其他说明

我想要对接的是外部ES（腾讯云ES），使用过两个版本6.8.2、7.10.1，均不成功

ks-installer内容

   apiVersion: installer.kubesphere.io/v1alpha1
   kind: ClusterConfiguration
   metadata:
     labels:
       version: v3.4.1
     name: ks-installer
     namespace: kubesphere-system
   spec:
     alerting:
       enabled: true
     auditing:
       enabled: true
     authentication:
       jwtSecret: ''
     common:
       core:
         console:
           enableMultiLogin: true
           port: 30880
           type: NodePort
       es:
         basicAuth:
           enabled: true
           password: xxxxx
           username: elastic
         elkPrefix: logstash
         enabled: true
         externalElasticsearchHost: 10.xxx.xxx.xxx
         externalElasticsearchPort: 9200
         logMaxAge: 8
       gpu:
         kinds:
           - default: true
             resourceName: nvidia.com/gpu
             resourceType: GPU
       minio:
         volumeSize: 100Gi
       monitoring:
         GPUMonitoring:
           enabled: false
         endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090'
       openldap:
         enabled: false
         volumeSize: 2Gi
       opensearch:
         basicAuth:
           enabled: true
           password: admin
           username: admin
         dashboard:
           enabled: false
         enabled: false
         externalOpensearchHost: ''
         externalOpensearchPort: ''
         logMaxAge: 7
         opensearchPrefix: whizard
       redis:
         enableHA: false
         enabled: true
         volumeSize: 2Gi
     devops:
       enabled: true
       jenkinsCpuLim: 1
       jenkinsCpuReq: 0.5
       jenkinsMemoryLim: 4Gi
       jenkinsMemoryReq: 4Gi
       jenkinsVolumeSize: 16Gi
     edgeruntime:
       enabled: false
       kubeedge:
         cloudCore:
           cloudHub:
             advertiseAddress:
               - ''
           service:
             cloudhubHttpsNodePort: '30002'
             cloudhubNodePort: '30000'
             cloudhubQuicNodePort: '30001'
             cloudstreamNodePort: '30003'
             tunnelNodePort: '30004'
         enabled: false
         iptables-manager:
           enabled: true
           mode: external
     etcd:
       endpointIps: localhost
       monitoring: false
       port: 2379
       tlsEnable: true
     events:
       enabled: true
       ruler:
         enabled: true
         replicas: 2
     gatekeeper:
       enabled: false
     local_registry: ''
     logging:
       containerruntime: containerd
       enabled: true
       logsidecar:
         enabled: true
         replicas: 2
     metrics_server:
       enabled: true
     monitoring:
       gpu:
         nvidia_dcgm_exporter:
           enabled: false
       node_exporter:
         port: 9100
       storageClass: ''
     multicluster:
       clusterRole: none
     network:
       ippool:
         type: none
       networkpolicy:
         enabled: false
       topology:
         type: none
     openpitrix:
       store:
         enabled: true
     persistence:
       storageClass: ''
     servicemesh:
       enabled: false
       istio:
         components:
           cni:
             enabled: false
           ingressGateways:
             - enabled: false
               name: istio-ingressgateway
     terminal:
       timeout: 600
   status:
     alerting:
       enabledTime: '2024-01-03T23:32:36CST'
       status: enabled
     auditing:
       enabledTime: '2024-01-03T23:29:08CST'
       status: enabled
     clusterId: 6a2d7938-1f47-4e3a-9729-38452c348532-1704296045
     core:
       enabledTime: '2024-01-17T08:47:01CST'
       status: enabled
       version: v3.4.1
     devops:
       enabledTime: '2024-01-03T23:31:32CST'
       status: enabled
     events:
       enabledTime: '2024-01-03T23:29:30CST'
       ruler:
         enabledTime: '2024-01-03T23:29:31CST'
         status: enabled
       status: enabled
     fluentbit:
       enabledTime: '2024-01-03T23:26:13CST'
       status: enabled
     logging:
       enabledTime: '2024-01-03T23:29:26CST'
       status: enabled
     metricsServer:
       enabledTime: '2024-01-03T23:24:40CST'
       status: enabled
     minio:
       enabledTime: '2024-01-03T23:25:57CST'
       status: enabled
     monitoring:
       enabledTime: '2024-01-17T08:48:45CST'
       status: enabled
     openldap:
       enabledTime: '2024-01-03T23:25:05CST'
       status: enabled
     redis:
       enabledTime: '2024-01-03T23:25:00CST'
       status: enabled
     servicemesh:
       enabledTime: '2024-01-03T23:30:12CST'
       status: enabled

kubectl get Input -n kubesphere-logging-system tail -o yaml

执行结果

   apiVersion: logging.kubesphere.io/v1alpha2
   kind: Input
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"logging.kubesphere.io/v1alpha2","kind":"Input","metadata":{"annotations":{},"labels":{"logging.kubesphere.io/component":"logging","logging.kubesphere.io/enabled":"true"},"name":"tail","namespace":"kubesphere-logging-system"},"spec":{"tail":{"db":"/fluent-bit/tail/pos.db","dbSync":"Normal","excludePath":"/var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log","memBufLimit":"5MB","parser":"cri","path":"/var/log/containers/*.log","refreshIntervalSeconds":10,"skipLongLines":true,"tag":"kube.*"}}}
     creationTimestamp: "2024-01-03T15:28:33Z"
     generation: 1
     labels:
       logging.kubesphere.io/component: logging
       logging.kubesphere.io/enabled: "true"
     name: tail
     namespace: kubesphere-logging-system
     resourceVersion: "193183683"
     uid: 41fddf15-1703-4fed-ad6b-3321aacf008d
   spec:
     tail:
       db: /fluent-bit/tail/pos.db
       dbSync: Normal
       excludePath: /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
       memBufLimit: 5MB
       parser: cri
       path: /var/log/containers/*.log
       refreshIntervalSeconds: 10
       skipLongLines: true
       tag: kube.*

kubectl get Output -n kubesphere-logging-system es -o yaml执行结果

apiVersion: logging.kubesphere.io/v1alpha2
kind: Output
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"logging.kubesphere.io/v1alpha2","kind":"Output","metadata":{"annotations":{},"labels":{"logging.kubesphere.io/component":"logging","logging.kubesphere.io/enabled":"true"},"name":"es","namespace":"kubesphere-logging-system"},"spec":{"es":{"generateID":true,"host":"10.xxx.xxx.xxx","httpPassword":{"valueFrom":{"secretKeyRef":{"key":"password","name":"elasticsearch-credentials"}}},"httpUser":{"valueFrom":{"secretKeyRef":{"key":"username","name":"elasticsearch-credentials"}}},"logstashFormat":true,"logstashPrefix":"ks-logstash-log","port":9200,"timeKey":"@timestamp"},"matchRegex":"(?:kube|service)\\.(.*)"}}
  creationTimestamp: "2024-01-03T15:28:34Z"
  generation: 2
  labels:
    logging.kubesphere.io/component: logging
    logging.kubesphere.io/enabled: "true"
  name: es
  namespace: kubesphere-logging-system
  resourceVersion: "867867040"
  uid: 2ff07cc0-a9dc-4afe-acfb-7655037f980c
spec:
  es:
    generateID: true
    host: 10.xxx.xxx.xxx
    httpPassword:
      valueFrom:
        secretKeyRef:
          key: password
          name: elasticsearch-credentials
    httpUser:
      valueFrom:
        secretKeyRef:
          key: username
          name: elasticsearch-credentials
    logstashFormat: true
    logstashPrefix: ks-logstash-log
    port: 9200
    timeKey: '@timestamp'
  matchRegex: (?:kube|service)\.(.*)

fluent-bit 容器开启debug之后，可以看到我的日志文件被 skip 了，这个 tail 的 parser 是 cri ，应该配置是对的。

[2024/01/17 13:30:04] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/zhtappsso-59c78c9d9d-46lpk_team2_app-874711ddd02cbac82eadbf25de389f7b6a03c9ef2eb7e399c04494a001f37d32.log
[2024/01/17 13:30:04] [debug] [input:tail:tail.2] 0 new files found on path '/var/log/containers/*.log'

fluent-bit-config 如下

[Service]
    Log_Level    trace
    Parsers_File    parsers.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.containerd
    Systemd_Filter    _SYSTEMD_UNIT=containerd.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/kubelet.db
    DB.Sync    Normal
    Tag    service.kubelet
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube.*
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    parser
    Match    kube_auditing
    Key_Name    message
    Parser    json
[Filter]
    Name    modify
    Match    kube_auditing
    Condition    Key_does_not_exist    AuditID    
    Add    ignore    true
[Filter]
    Name    grep
    Match    kube_auditing
    Exclude    ignore true
[Filter]
    Name    parser
    Match    kube_events
    Key_Name    message
    Parser    json
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    10.xxx.xxx.xxx
    Port    9200
    HTTP_User    elastic
    HTTP_Passwd    xxx
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp
    Generate_ID    true
    Trace_Error    true
[Output]
    Name    es
    Match    kube_auditing
    Host    10.xxx.xxx.xxx
    Port    9200
    HTTP_User    elastic
    HTTP_Passwd    xxx
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-auditing
    Generate_ID    true
[Output]
    Name    es
    Match    kube_events
    Host    10.xxx.xxx.xxx
    Port    9200
    HTTP_User    elastic
    HTTP_Passwd    xxx
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-events
    Generate_ID    true

tmkddskf

原因知道了

/var/lib/containerd 这个目录，fluentbit访问不了，需要去添加volumes，下面供参考

apiVersion: logging.kubesphere.io/v1alpha2
kind: FluentBit
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"logging.kubesphere.io/v1alpha2","kind":"FluentBit","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"fluent-bit"},"name":"fluent-bit","namespace":"kubesphere-logging-system"},"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}},"fluentBitConfigName":"fluent-bit-config","image":"kubesphere/fluent-bit:v1.9.4","positionDB":{"emptyDir":{}},"resources":{"limits":{"cpu":"500m","memory":"200Mi"},"requests":{"cpu":"10m","memory":"25Mi"}},"tolerations":[{"operator":"Exists"}]}}
  finalizers:
    - fluentbit.logging.kubesphere.io
  labels:
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit
  namespace: kubesphere-logging-system
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: DoesNotExist
  fluentBitConfigName: fluent-bit-config
  image: 'kubesphere/fluent-bit:v1.9.4'
  positionDB:
    emptyDir: {}
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 10m
      memory: 25Mi
  tolerations:
    - operator: Exists
  volumes:
    - hostPath:
        path: /var/lib/containerd
      name: new-volume
  volumesMounts:
    - mountPath: /var/lib/containerd
      name: new-volume
status: {}

比较logging的indexprefix，需要保持一致
kubesphere-system下的configmap kubesphere-config 中logging的部分和kubesphere-logging-system下的secret fluent-bit-conf 中的 fluent-bit.conf

tmkddskf

但是现在出现了新的问题，日志已经搜集到，但是没法儿按容器来统计

tmkddskf

tmkddskf 继续补充，现在发现可以按容器组搜索，但是需要选中“模糊匹配”，不知道什么原因。这个容器统计依然是0