v3.2.1升级到v3.3.1后容器日志、事件日志、审计日志都收集不了了

Wwanjunlei · 2023年1月16日

kubectl get secret -n kubesphere-logging-system fluent-bit-config -oyaml

把 fb的配置文件发出来看看

Vveuve · 2023年1月16日

apiVersion: v1

data:

containerd.lua: ZnVuY3Rpb24gY29udGFpbmVyZCggdGFnLCB0aW1lc3RhbXAsIHJlY29yZCkKICAgICAgIGlmKHJlY29yZFsibG9ndGFnIl1+PW5pbCkKICAgICAgIHRoZW4KICAgICAgIHRpbWVTdHIgPSBvcy5kYXRlKCIhKnQiLCAgdGltZXN0YW1wWyJzZWMiXSkKICAgICAgICB0ID0gc3RyaW5nLmZvcm1hdCgiJTRkLSUwMmQtJTAyZFQlMDJkOiUwMmQ6JTAyZC4lc1oiLAogICAgICAgIHRpbWVTdHJbInllYXIiXSwgdGltZVN0clsibW9udGgiXSwgdGltZVN0clsiZGF5Il0sCiAgICAgICAgdGltZVN0clsiaG91ciJdLCB0aW1lU3RyWyJtaW4iXSwgdGltZVN0clsic2VjIl0sCiAgICAgICAgdGltZXN0YW1wWyJuc2VjIl0pOwogICAgICAgIHJlY29yZFsidGltZSJdID0gdDsKICAgICAgICByZWNvcmRbImxvZyJdID0gcmVjb3JkWyJtZXNzYWdlIl07CiAgICAgICAgcmVjb3JkWyJtZXNzYWdlIl0gPSAgbmlsOwogICAgICAgIHJldHVybiAxLCB0aW1lc3RhbXAsIHJlY29yZAogICAgICAgIGVsc2UKICAgICAgICByZXR1cm4gMCx0aW1lc3RhbXAscmVjb3JkCiAgICAgICBlbmQKZW5k

fluent-bit.conf: W1NlcnZpY2VdCiAgICBQYXJzZXJzX0ZpbGUgICAgcGFyc2Vycy5jb25mCltJbnB1dF0KICAgIE5hbWUgICAgc3lzdGVtZAogICAgUGF0aCAgICAvdmFyL2xvZy9qb3VybmFsCiAgICBEQiAgICAvZmx1ZW50LWJpdC90YWlsL2RvY2tlci5kYgogICAgREIuU3luYyAgICBOb3JtYWwKICAgIFRhZyAgICBzZXJ2aWNlLmNyaW8KICAgIFN5c3RlbWRfRmlsdGVyICAgIF9TWVNURU1EX1VOSVQ9Y3Jpby5zZXJ2aWNlCltJbnB1dF0KICAgIE5hbWUgICAgc3lzdGVtZAogICAgUGF0aCAgICAvdmFyL2xvZy9qb3VybmFsCiAgICBEQiAgICAvZmx1ZW50LWJpdC90YWlsL2RvY2tlci5kYgogICAgREIuU3luYyAgICBOb3JtYWwKICAgIFRhZyAgICBzZXJ2aWNlLmRvY2tlcgogICAgU3lzdGVtZF9GaWx0ZXIgICAgX1NZU1RFTURfVU5JVD1kb2NrZXIuc2VydmljZQpbSW5wdXRdCiAgICBOYW1lICAgIHN5c3RlbWQKICAgIFBhdGggICAgL3Zhci9sb2cvam91cm5hbAogICAgREIgICAgL2ZsdWVudC1iaXQvdGFpbC9rdWJlbGV0LmRiCiAgICBEQi5TeW5jICAgIE5vcm1hbAogICAgVGFnICAgIHNlcnZpY2Uua3ViZWxldAogICAgU3lzdGVtZF9GaWx0ZXIgICAgX1NZU1RFTURfVU5JVD1rdWJlbGV0LnNlcnZpY2UKW0lucHV0XQogICAgTmFtZSAgICB0YWlsCiAgICBQYXRoICAgIC92YXIvbG9nL2NvbnRhaW5lcnMvKi5sb2cKICAgIEV4Y2x1ZGVfUGF0aCAgICAvdmFyL2xvZy9jb250YWluZXJzLypfa3ViZXNwaGVyZS1sb2dnaW5nLXN5c3RlbV9ldmVudHMtZXhwb3J0ZXIqLmxvZywvdmFyL2xvZy9jb250YWluZXJzL2t1YmUtYXVkaXRpbmctd2ViaG9vaypfa3ViZXNwaGVyZS1sb2dnaW5nLXN5c3RlbV9rdWJlLWF1ZGl0aW5nLXdlYmhvb2sqLmxvZwogICAgUmVmcmVzaF9JbnRlcnZhbCAgICAxMAogICAgU2tpcF9Mb25nX0xpbmVzICAgIHRydWUKICAgIERCICAgIC9mbHVlbnQtYml0L3RhaWwvcG9zLmRiCiAgICBEQi5TeW5jICAgIE5vcm1hbAogICAgTWVtX0J1Zl9MaW1pdCAgICA1TUIKICAgIFBhcnNlciAgICBjcmkKICAgIFRhZyAgICBrdWJlLioKW0lucHV0XQogICAgTmFtZSAgICB0YWlsCiAgICBQYXRoICAgIC92YXIvbG9nL2NvbnRhaW5lcnMva3ViZS1hdWRpdGluZy13ZWJob29rKl9rdWJlc3BoZXJlLWxvZ2dpbmctc3lzdGVtX2t1YmUtYXVkaXRpbmctd2ViaG9vayoubG9nCiAgICBSZWZyZXNoX0ludGVydmFsICAgIDEwCiAgICBTa2lwX0xvbmdfTGluZXMgICAgdHJ1ZQogICAgREIgICAgL2ZsdWVudC1iaXQvdGFpbC9wb3MtYXVkaXRpbmcuZGIKICAgIERCLlN5bmMgICAgTm9ybWFsCiAgICBNZW1fQnVmX0xpbWl0ICAgIDVNQgogICAgUGFyc2VyICAgIGNyaQogICAgVGFnICAgIGt1YmVfYXVkaXRpbmcKW0lucHV0XQogICAgTmFtZSAgICB0YWlsCiAgICBQYXRoICAgIC92YXIvbG9nL2NvbnRhaW5lcnMvKl9rdWJlc3BoZXJlLWxvZ2dpbmctc3lzdGVtX2V2ZW50cy1leHBvcnRlcioubG9nCiAgICBSZWZyZXNoX0ludGVydmFsICAgIDEwCiAgICBTa2lwX0xvbmdfTGluZXMgICAgdHJ1ZQogICAgREIgICAgL2ZsdWVudC1iaXQvdGFpbC9wb3MtZXZlbnRzLmRiCiAgICBEQi5TeW5jICAgIE5vcm1hbAogICAgTWVtX0J1Zl9MaW1pdCAgICA1TUIKICAgIFBhcnNlciAgICBjcmkKICAgIFRhZyAgICBrdWJlX2V2ZW50cwpbRmlsdGVyXQogICAgTmFtZSAgICBsdWEKICAgIE1hdGNoICAgIGt1YmUuKgogICAgc2NyaXB0ICAgIC9mbHVlbnQtYml0L2NvbmZpZy9jb250YWluZXJkLmx1YQogICAgY2FsbCAgICBjb250YWluZXJkCiAgICB0aW1lX2FzX3RhYmxlICAgIHRydWUKW0ZpbHRlcl0KICAgIE5hbWUgICAgcGFyc2VyCiAgICBNYXRjaCAgICBrdWJlX2F1ZGl0aW5nCiAgICBLZXlfTmFtZSAgICBtZXNzYWdlCiAgICBQYXJzZXIgICAganNvbgpbRmlsdGVyXQogICAgTmFtZSAgICBtb2RpZnkKICAgIE1hdGNoICAgIGt1YmVfYXVkaXRpbmcKICAgIENvbmRpdGlvbiAgICBLZXlfZG9lc19ub3RfZXhpc3QgICAgQXVkaXRJRCAgICAKICAgIEFkZCAgICBpZ25vcmUgICAgdHJ1ZQpbRmlsdGVyXQogICAgTmFtZSAgICBncmVwCiAgICBNYXRjaCAgICBrdWJlX2F1ZGl0aW5nCiAgICBFeGNsdWRlICAgIGlnbm9yZSB0cnVlCltGaWx0ZXJdCiAgICBOYW1lICAgIHBhcnNlcgogICAgTWF0Y2ggICAga3ViZV9ldmVudHMKICAgIEtleV9OYW1lICAgIG1lc3NhZ2UKICAgIFBhcnNlciAgICBqc29uCltGaWx0ZXJdCiAgICBOYW1lICAgIGt1YmVybmV0ZXMKICAgIE1hdGNoICAgIGt1YmUuKgogICAgS3ViZV9VUkwgICAgaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjOjQ0MwogICAgS3ViZV9DQV9GaWxlICAgIC92YXIvcnVuL3NlY3JldHMva3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9jYS5jcnQKICAgIEt1YmVfVG9rZW5fRmlsZSAgICAvdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvdG9rZW4KICAgIExhYmVscyAgICBmYWxzZQogICAgQW5ub3RhdGlvbnMgICAgZmFsc2UKW0ZpbHRlcl0KICAgIE5hbWUgICAgbmVzdAogICAgTWF0Y2ggICAga3ViZS4qCiAgICBPcGVyYXRpb24gICAgbGlmdAogICAgTmVzdGVkX3VuZGVyICAgIGt1YmVybmV0ZXMKICAgIEFkZF9wcmVmaXggICAga3ViZXJuZXRlc18KW0ZpbHRlcl0KICAgIE5hbWUgICAgbW9kaWZ5CiAgICBNYXRjaCAgICBrdWJlLioKICAgIFJlbW92ZSAgICBzdHJlYW0KICAgIFJlbW92ZSAgICBrdWJlcm5ldGVzX3BvZF9pZAogICAgUmVtb3ZlICAgIGt1YmVybmV0ZXNfaG9zdAogICAgUmVtb3ZlICAgIGt1YmVybmV0ZXNfY29udGFpbmVyX2hhc2gKW0ZpbHRlcl0KICAgIE5hbWUgICAgbmVzdAogICAgTWF0Y2ggICAga3ViZS4qCiAgICBPcGVyYXRpb24gICAgbmVzdAogICAgV2lsZGNhcmQgICAga3ViZXJuZXRlc18qCiAgICBOZXN0X3VuZGVyICAgIGt1YmVybmV0ZXMKICAgIFJlbW92ZV9wcmVmaXggICAga3ViZXJuZXRlc18KW0ZpbHRlcl0KICAgIE5hbWUgICAgbHVhCiAgICBNYXRjaCAgICBzZXJ2aWNlLioKICAgIHNjcmlwdCAgICAvZmx1ZW50LWJpdC9jb25maWcvc3lzdGVtZC5sdWEKICAgIGNhbGwgICAgYWRkX3RpbWUKICAgIHRpbWVfYXNfdGFibGUgICAgdHJ1ZQpbT3V0cHV0XQogICAgTmFtZSAgICBlcwogICAgTWF0Y2hfUmVnZXggICAgKD86a3ViZXxzZXJ2aWNlKVwuKC4qKQogICAgSG9zdCAgICAxMC4zMy4xMC4xNwogICAgUG9ydCAgICA5MjAwCiAgICBMb2dzdGFzaF9Gb3JtYXQgICAgdHJ1ZQogICAgTG9nc3Rhc2hfUHJlZml4ICAgIGtzLWxvZ3N0YXNoLWxvZwogICAgVGltZV9LZXkgICAgQHRpbWVzdGFtcAogICAgR2VuZXJhdGVfSUQgICAgdHJ1ZQogICAgVHJhY2VfRXJyb3IgICAgdHJ1ZQpbT3V0cHV0XQogICAgTmFtZSAgICBlcwogICAgTWF0Y2ggICAga3ViZV9hdWRpdGluZwogICAgSG9zdCAgICAxMC4zMy4xMC4xNwogICAgUG9ydCAgICA5MjAwCiAgICBMb2dzdGFzaF9Gb3JtYXQgICAgdHJ1ZQogICAgTG9nc3Rhc2hfUHJlZml4ICAgIGtzLWxvZ3N0YXNoLWF1ZGl0aW5nCiAgICBHZW5lcmF0ZV9JRCAgICB0cnVlCltPdXRwdXRdCiAgICBOYW1lICAgIGVzCiAgICBNYXRjaCAgICBrdWJlX2V2ZW50cwogICAgSG9zdCAgICAxMC4zMy4xMC4xNwogICAgUG9ydCAgICA5MjAwCiAgICBMb2dzdGFzaF9Gb3JtYXQgICAgdHJ1ZQogICAgTG9nc3Rhc2hfUHJlZml4ICAgIGtzLWxvZ3N0YXNoLWV2ZW50cwogICAgR2VuZXJhdGVfSUQgICAgdHJ1ZQo=

parsers.conf: ""

systemd.lua: ZnVuY3Rpb24gYWRkX3RpbWUodGFnLCB0aW1lc3RhbXAsIHJlY29yZCkKICBuZXdfcmVjb3JkID0ge30KICB0aW1lU3RyID0gb3MuZGF0ZSgiISp0IiwgdGltZXN0YW1wWyJzZWMiXSkKICB0ID0gc3RyaW5nLmZvcm1hdCgiJTRkLSUwMmQtJTAyZFQlMDJkOiUwMmQ6JTAyZC4lc1oiLAoJCXRpbWVTdHJbInllYXIiXSwgdGltZVN0clsibW9udGgiXSwgdGltZVN0clsiZGF5Il0sCgkJdGltZVN0clsiaG91ciJdLCB0aW1lU3RyWyJtaW4iXSwgdGltZVN0clsic2VjIl0sCgkJdGltZXN0YW1wWyJuc2VjIl0pCiAga3ViZXJuZXRlcyA9IHt9CiAga3ViZXJuZXRlc1sicG9kX25hbWUiXSA9IHJlY29yZFsiX0hPU1ROQU1FIl0KICBrdWJlcm5ldGVzWyJjb250YWluZXJfbmFtZSJdID0gcmVjb3JkWyJTWVNMT0dfSURFTlRJRklFUiJdCiAga3ViZXJuZXRlc1sibmFtZXNwYWNlX25hbWUiXSA9ICJrdWJlLXN5c3RlbSIKICBuZXdfcmVjb3JkWyJ0aW1lIl0gPSB0CiAgbmV3X3JlY29yZFsibG9nIl0gPSByZWNvcmRbIk1FU1NBR0UiXQogIG5ld19yZWNvcmRbImt1YmVybmV0ZXMiXSA9IGt1YmVybmV0ZXMKICByZXR1cm4gMSwgdGltZXN0YW1wLCBuZXdfcmVjb3JkCmVuZA==

kind: Secret

metadata:

creationTimestamp: "2022-05-10T12:11:27Z"

managedFields:

- apiVersion: v1

fieldsType: FieldsV1

fieldsV1:

f:data:

.: {}

f:containerd.lua: {}

f:fluent-bit.conf: {}

f:parsers.conf: {}

f:systemd.lua: {}

f:metadata:

f:ownerReferences:

.: {}

k:{"uid":"da908ca0-a481-4fc4-a561-204488c3db90"}:

.: {}

f:apiVersion: {}

f:blockOwnerDeletion: {}

f:controller: {}

f:kind: {}

f:name: {}

f:uid: {}

f:type: {}

manager: manager

operation: Update

time: "2023-01-09T07:55:45Z"

name: fluent-bit-config

namespace: kubesphere-logging-system

ownerReferences:

- apiVersion: logging.kubesphere.io/v1alpha2

blockOwnerDeletion: true

controller: true

kind: FluentBitConfig

name: fluent-bit-config

uid: da908ca0-a481-4fc4-a561-204488c3db90

resourceVersion: "144407134"

selfLink: /api/v1/namespaces/kubesphere-logging-system/secrets/fluent-bit-config

uid: 5edfd227-e937-40c4-a1ef-cb3788cff2f6

type: Opaque

Vveuve · 2023年1月16日

containerd.lua:

function containerd( tag, timestamp, record)
       if(record["logtag"]~=nil)
       then
       timeStr = os.date("!*t",  timestamp["sec"])
        t = string.format("%4d-%02d-%02dT%02d:%02d:%02d.%sZ",
        timeStr["year"], timeStr["month"], timeStr["day"],
        timeStr["hour"], timeStr["min"], timeStr["sec"],
        timestamp["nsec"]);
        record["time"] = t;
        record["log"] = record["message"];
        record["message"] =  nil;
        return 1, timestamp, record
        else
        return 0,timestamp,record
       end
end

fluent-bit.conf:

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.crio
    Systemd_Filter    _SYSTEMD_UNIT=crio.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.docker
    Systemd_Filter    _SYSTEMD_UNIT=docker.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/kubelet.db
    DB.Sync    Normal
    Tag    service.kubelet
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube.*
[Input]
    Name    tail
    Path    /var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos-auditing.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube_auditing
[Input]
    Name    tail
    Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos-events.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube_events
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    parser
    Match    kube_auditing
    Key_Name    message
    Parser    json
[Filter]
    Name    modify
    Match    kube_auditing
    Condition    Key_does_not_exist    AuditID    
    Add    ignore    true
[Filter]
    Name    grep
    Match    kube_auditing
    Exclude    ignore true
[Filter]
    Name    parser
    Match    kube_events
    Key_Name    message
    Parser    json
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp
    Generate_ID    true
    Trace_Error    true
[Output]
    Name    es
    Match    kube_auditing
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-auditing
    Generate_ID    true
[Output]
    Name    es
    Match    kube_events
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-events
    Generate_ID    true

systemd.lua:

function add_time(tag, timestamp, record)
  new_record = {}
  timeStr = os.date("!*t", timestamp["sec"])
  t = string.format("%4d-%02d-%02dT%02d:%02d:%02d.%sZ",
		timeStr["year"], timeStr["month"], timeStr["day"],
		timeStr["hour"], timeStr["min"], timeStr["sec"],
		timestamp["nsec"])
  kubernetes = {}
  kubernetes["pod_name"] = record["_HOSTNAME"]
  kubernetes["container_name"] = record["SYSLOG_IDENTIFIER"]
  kubernetes["namespace_name"] = "kube-system"
  new_record["time"] = t
  new_record["log"] = record["MESSAGE"]
  new_record["kubernetes"] = kubernetes
  return 1, timestamp, new_record
end

Wwanjunlei · 2023年1月16日

从fb的配置文件看你的runtime是containerd，但是你上面给的是docker 的版本信息，先确认下你的runtime。是不是有节点同时安装了docker和containerd？

Vveuve · 2023年1月16日

wanjunlei 腾讯云的TKE集群，运行时是docker的

Vveuve · 2023年1月16日

有用到tke的超级节点池，不知道这里面是不是两个运行时都有

Wwanjunlei · 2023年1月16日

kubectl get node -ojson | jq ‘.items[0].status.nodeInfo.containerRuntimeVersion’

执行下这个看看，所有节点都执行下

Vveuve · 2023年1月16日

wanjunlei 这个eks的就是三个超级节点池，其它普通的节点都是docker的

[root@rhino-c1-c1 ~]# kubectl get node -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}'
eks://2.0[root@rhino-c1-c1 ~]# 
[root@rhino-c1-c1 ~]# kubectl get node -o jsonpath='{.items[1].status.nodeInfo.containerRuntimeVersion}'
eks://2.0[root@rhino-c1-c1 ~]# 
[root@rhino-c1-c1 ~]# kubectl get node -o jsonpath='{.items[2].status.nodeInfo.containerRuntimeVersion}'
eks://2.0[root@rhino-c1-c1 ~]# 
[root@rhino-c1-c1 ~]# kubectl get node -o jsonpath='{.items[3].status.nodeInfo.containerRuntimeVersion}'
docker://19.3.9[root@rhino-c1-c1 ~]# 
[root@rhino-c1-c1 ~]# kubectl get node -o jsonpath='{.items[4].status.nodeInfo.containerRuntimeVersion}'
docker://19.3.9

Vveuve · 2023年1月16日

wanjunlei 3个超级节点7个普通节点

Wwanjunlei · 2023年1月16日

首先，ks 目前不支持不同节点使用不同的 container runtime，其次这个 eks 的 runtime 没有对接过，不确定能不能支持

Vveuve · 2023年1月16日

wanjunlei 可以怎么改成支持docker运行时的吗，先让docker节点的可以正常

Wwanjunlei · 2023年1月16日

编辑 cc

kubectl edit cc -n kubesphere-system ks-installer

删除 status 中的 auditing、fluentbit、logging、events 部分，然后把 ks-installer 调度到 docker 节点上，等待安装完成

Vveuve · 2023年1月16日

wanjunlei 好的，感谢

Vveuve · 2023年1月16日

wanjunlei 装完还是一样的，只有kube-system的日志

Wwanjunlei · 2023年1月17日

先检查 fb 的配置文件，看看是不是 docker 的。

然后看看 fluentbit-operator 的initcontainer，正确的命令应该是

 echo CONTAINER_ROOT_DIR=$(docker info -f {{'{{.DockerRootDir}}'}}) > /fluentbit-operator/fluent-bit.env

都正确的话把 fluentbit-operator 调度到 docker 节点上，然后删除 fluent-bit daemonset 试试

Vveuve · 2023年1月17日

wanjunlei

fb的配置还是containerd的

fluent-bit.conf:
[Service]
    Parsers_File    parsers.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.crio
    Systemd_Filter    _SYSTEMD_UNIT=crio.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.docker
    Systemd_Filter    _SYSTEMD_UNIT=docker.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/kubelet.db
    DB.Sync    Normal
    Tag    service.kubelet
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube.*
[Input]
    Name    tail
    Path    /var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos-auditing.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube_auditing
[Input]
    Name    tail
    Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos-events.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube_events
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    parser
    Match    kube_auditing
    Key_Name    message
    Parser    json
[Filter]
    Name    modify
    Match    kube_auditing
    Condition    Key_does_not_exist    AuditID    
    Add    ignore    true
[Filter]
    Name    grep
    Match    kube_auditing
    Exclude    ignore true
[Filter]
    Name    parser
    Match    kube_events
    Key_Name    message
    Parser    json
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp
    Generate_ID    true
    Trace_Error    true
[Output]
    Name    es
    Match    kube_auditing
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-auditing
    Generate_ID    true
[Output]
    Name    es
    Match    kube_events
    Host    10.33.10.17
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-events
    Generate_ID    true

fluentbit-operator 的initcontainer

echo CONTAINER_ROOT_DIR=/var/log > /fluentbit-operator/fluent-bit.env

Wwanjunlei · 2023年1月17日

现在只能手动改了

1 改 input

kubectl edit input -n kubesphere-logging-system tail

kubectl edit input -n kubesphere-logging-system tail-auditing

kubectl edit input -n kubesphere-logging-system tail-events

修改 parser 为 docker

删除 crio input

kubectl delete input -n kubesphere-logging-system crio

创建 docker input

apiVersion: logging.kubesphere.io/v1alpha2
kind: Input
metadata:
  name: {{ logging_container_runtime | default('docker') }}
  namespace: kubesphere-logging-system
  labels:
    logging.kubesphere.io/enabled: "true"
    logging.kubesphere.io/component: logging
spec:
  systemd:
    tag: service.docker
    path: /var/log/journal
    db: /fluent-bit/tail/docker.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service

2 删除 containerd filter

kubectl delete filter -n kubesphere-logging-system containerd

3 修改 fluent-operator，需要修改 volumes 和 initContainer，可以照这个修改

kind: Deployment
apiVersion: apps/v1
metadata:
  name: fluentbit-operator
  namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/name: fluentbit-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: operator
      app.kubernetes.io/name: fluentbit-operator
  template:
    metadata:
      labels:
        app.kubernetes.io/component: operator
        app.kubernetes.io/name: fluentbit-operator
    spec:
      volumes:
        - name: env
          emptyDir: {}
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
            type: ''
      initContainers:
        - name: setenv
          image: 'docker:19.03'
          command:
            - /bin/sh
            - '-c'
            - >-
              set -ex; echo CONTAINER_ROOT_DIR=$(docker info -f
              {{.DockerRootDir}}) > /fluentbit-operator/fluent-bit.env
          resources: {}
          volumeMounts:
            - name: env
              mountPath: /fluentbit-operator
            - name: dockersock
              readOnly: true
              mountPath: /var/run/docker.sock
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent

flunet-operator 重启完成后，删除 flunet-bit daemonset

Vveuve · 2023年1月17日

wanjunlei 好的，我试试

Vveuve · 2023年1月17日

wanjunlei 现在可以正常收集容器日志了，事件的日志也能收集，就ks里的事件日志查看请求还有问题，是不是需要该哪里的配置呀

Wwanjunlei · 2023年1月17日

kubectl edit filter -n kubesphere-logging-system filter-events

kubectl edit filter -n kubesphere-logging-system filter-auditing

keyName 改成 log

v3.2.1升级到v3.3.1后容器日志、事件日志、审计日志都收集不了了

WwanjunleiK零S

Vveuve

Vveuve

WwanjunleiK零S

Vveuve

Vveuve

WwanjunleiK零S

Vveuve

Vveuve

WwanjunleiK零S

Vveuve

WwanjunleiK零S

Vveuve

Vveuve

WwanjunleiK零S

Vveuve

WwanjunleiK零S

Vveuve

Vveuve

WwanjunleiK零S