member 集群非 kube-system 命名空间日志查询不到

sigboom

主集群版本 v3.2.1

member集群版本 v3.2.1

runtime：containerd

Fluent Bit v1.8.3 日志如下

[2022/06/10 09:52:51] [ info] [engine] started (pid=61)
[2022/06/10 09:52:51] [ info] [storage] version=1.1.1, initializing...
[2022/06/10 09:52:51] [ info] [storage] in-memory
[2022/06/10 09:52:51] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/06/10 09:52:51] [ info] [cmetrics] version=0.1.6
[2022/06/10 09:52:51] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=1e5... OK
[2022/06/10 09:52:51] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=225... OK
[2022/06/10 09:52:51] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443
[2022/06/10 09:52:51] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2022/06/10 09:52:51] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2022/06/10 09:52:51] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2022/06/10 09:52:51] [ info] [sp] stream processor started
level=error msg="Fluent bit exited" error="signal: killed"
level=info msg=backoff delay=5m0s
level=info msg="backoff timer done" actual=5m0.000094737s expected=5m0s
level=info msg="Fluent bit started"
[1mFluent Bit v1.8.3[0m
* [1m[93mCopyright (C) 2019-2021 The Fluent Bit Authors[0m
* [1m[93mCopyright (C) 2015-2018 Treasure Data[0m
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/06/10 09:57:56] [ info] [engine] started (pid=64)
[2022/06/10 09:57:56] [ info] [storage] version=1.1.1, initializing...
[2022/06/10 09:57:56] [ info] [storage] in-memory
[2022/06/10 09:57:56] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/06/10 09:57:56] [ info] [cmetrics] version=0.1.6
[2022/06/10 09:57:56] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=216... OK
[2022/06/10 09:57:56] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=225... OK
[2022/06/10 09:57:56] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443
[2022/06/10 09:57:56] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2022/06/10 09:57:56] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2022/06/10 09:57:56] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2022/06/10 09:57:56] [ info] [sp] stream processor started
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:07] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 09:58:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2022/06/10 11:12:41] [ warn] [engine] failed to flush chunk '64-1654859557.494881885.flb', retry in 11 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
[2022/06/10 11:52:41] [ warn] [engine] failed to flush chunk '64-1654861957.498122714.flb', retry in 7 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
[2022/06/10 17:12:36] [ warn] [engine] failed to flush chunk '64-1654881151.760230362.flb', retry in 6 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
[2022/06/10 17:32:41] [ warn] [engine] failed to flush chunk '64-1654882357.258750322.flb', retry in 10 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
[2022/06/10 18:12:36] [ warn] [engine] failed to flush chunk '64-1654884751.757711333.flb', retry in 10 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
...

[2022/06/20 05:52:41] [ warn] [engine] failed to flush chunk '64-1655704357.599898439.flb', retry in 11 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)
[2022/06/20 07:52:41] [ warn] [engine] failed to flush chunk '64-1655711558.6775275.flb', retry in 7 seconds: task_id=1, input=systemd.1 > output=es.0 (out_id=0)

DehaoCheng

看下es的资源pvc是否占满，es状态情况，如果没有问题，执行一下代码看看配置是否存在问题

kubectl -n kubesphere-logging-system get secret fluent-bit-config -ojson | jq '.data."fluent-bit.conf"' | awk -F '"' '{printf $2}' | base64 --decode

sigboom

DehaoCheng

存储没问题
下面是命令执行返回

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/docker.db
    DB.Sync    Normal
    Tag    service.containerd
    Systemd_Filter    _SYSTEMD_UNIT=containerd.service
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/kubelet.db
    DB.Sync    Normal
    Tag    service.kubelet
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    cri
    Tag    kube.*
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    elasticsearch-logging-data.kubesphere-logging-system.svc
    Port    9200
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp
    Generate_ID    true

DehaoCheng

那时试一试在input中把Mem_Buf_Limit调大一点呢？

sigboom

DehaoCheng

看起来日志是正常了，但是依然查不到对应的日志

2022-06-20T20:48:50.348459852+08:00 level=info msg="Fluent bit started"

2022-06-20T20:48:50.353420513+08:00 Fluent Bit v1.8.3

2022-06-20T20:48:50.353459757+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:48:50.353463784+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:48:50.353467401+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:48:50.353469806+08:00 * https://fluentbit.io

2022-06-20T20:48:50.353471689+08:00 

2022-06-20T20:48:50.354186912+08:00 [2022/06/20 12:48:50] [ info] [engine] started (pid=14)

2022-06-20T20:48:50.354212540+08:00 [2022/06/20 12:48:50] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:48:50.354219152+08:00 [2022/06/20 12:48:50] [ info] [storage] in-memory

2022-06-20T20:48:50.354224312+08:00 [2022/06/20 12:48:50] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:48:50.354227538+08:00 [2022/06/20 12:48:50] [ info] [cmetrics] version=0.1.6

2022-06-20T20:48:50.395640108+08:00 [2022/06/20 12:48:50] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:48:50.396777012+08:00 [2022/06/20 12:48:50] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:48:50.398362117+08:00 [2022/06/20 12:48:50] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:48:50.410635354+08:00 [2022/06/20 12:48:50] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:48:50.411033561+08:00 [2022/06/20 12:48:50] [ info] [sp] stream processor started

2022-06-20T20:48:56.291891092+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:48:56.291922030+08:00 level=info msg=backoff delay=1s

2022-06-20T20:48:57.292084589+08:00 level=info msg="backoff timer done" actual=1.000140689s expected=1s

2022-06-20T20:48:57.292375605+08:00 level=info msg="Fluent bit started"

2022-06-20T20:48:57.313498513+08:00 Fluent Bit v1.8.3

2022-06-20T20:48:57.313522447+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:48:57.313527657+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:48:57.313531755+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:48:57.313534500+08:00 * https://fluentbit.io

2022-06-20T20:48:57.313537015+08:00 

2022-06-20T20:48:57.314324372+08:00 [2022/06/20 12:48:57] [ info] [engine] started (pid=19)

2022-06-20T20:48:57.314334722+08:00 [2022/06/20 12:48:57] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:48:57.314341545+08:00 [2022/06/20 12:48:57] [ info] [storage] in-memory

2022-06-20T20:48:57.314347776+08:00 [2022/06/20 12:48:57] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:48:57.314352285+08:00 [2022/06/20 12:48:57] [ info] [cmetrics] version=0.1.6

2022-06-20T20:48:57.315614063+08:00 [2022/06/20 12:48:57] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=17d... OK

2022-06-20T20:48:57.326413223+08:00 [2022/06/20 12:48:57] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=25e... OK

2022-06-20T20:48:57.328318278+08:00 [2022/06/20 12:48:57] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:48:57.329447768+08:00 [2022/06/20 12:48:57] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:48:57.330638433+08:00 [2022/06/20 12:48:57] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:48:57.353032676+08:00 [2022/06/20 12:48:57] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:48:57.353271916+08:00 [2022/06/20 12:48:57] [ info] [sp] stream processor started

2022-06-20T20:49:03.208412855+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:49:03.208471405+08:00 level=info msg=backoff delay=2s

2022-06-20T20:49:05.208674144+08:00 level=info msg="backoff timer done" actual=2.000145171s expected=2s

2022-06-20T20:49:05.208847298+08:00 level=info msg="Fluent bit started"

2022-06-20T20:49:05.265653692+08:00 Fluent Bit v1.8.3

2022-06-20T20:49:05.265688237+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:49:05.265699678+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:49:05.265703606+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:49:05.265707212+08:00 * https://fluentbit.io

2022-06-20T20:49:05.265710108+08:00 

2022-06-20T20:49:05.266865096+08:00 [2022/06/20 12:49:05] [ info] [engine] started (pid=22)

2022-06-20T20:49:05.266884933+08:00 [2022/06/20 12:49:05] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:49:05.266889942+08:00 [2022/06/20 12:49:05] [ info] [storage] in-memory

2022-06-20T20:49:05.266893810+08:00 [2022/06/20 12:49:05] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:49:05.266908076+08:00 [2022/06/20 12:49:05] [ info] [cmetrics] version=0.1.6

2022-06-20T20:49:05.268346376+08:00 [2022/06/20 12:49:05] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=198... OK

2022-06-20T20:49:05.277354043+08:00 [2022/06/20 12:49:05] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=2f0... OK

2022-06-20T20:49:05.280351860+08:00 [2022/06/20 12:49:05] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:49:05.281561029+08:00 [2022/06/20 12:49:05] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:49:05.282872711+08:00 [2022/06/20 12:49:05] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:49:05.306142037+08:00 [2022/06/20 12:49:05] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:49:05.306447780+08:00 [2022/06/20 12:49:05] [ info] [sp] stream processor started

2022-06-20T20:49:11.309272498+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:49:11.309314607+08:00 level=info msg=backoff delay=4s

2022-06-20T20:49:15.309383982+08:00 level=info msg="backoff timer done" actual=4.000098218s expected=4s

2022-06-20T20:49:15.309869773+08:00 level=info msg="Fluent bit started"

2022-06-20T20:49:15.364576416+08:00 Fluent Bit v1.8.3

2022-06-20T20:49:15.364594980+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:49:15.364602334+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:49:15.364612072+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:49:15.364616100+08:00 * https://fluentbit.io

2022-06-20T20:49:15.364619326+08:00 

2022-06-20T20:49:15.366057686+08:00 [2022/06/20 12:49:15] [ info] [engine] started (pid=25)

2022-06-20T20:49:15.366086660+08:00 [2022/06/20 12:49:15] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:49:15.366091168+08:00 [2022/06/20 12:49:15] [ info] [storage] in-memory

2022-06-20T20:49:15.366094495+08:00 [2022/06/20 12:49:15] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:49:15.366097650+08:00 [2022/06/20 12:49:15] [ info] [cmetrics] version=0.1.6

2022-06-20T20:49:15.367738581+08:00 [2022/06/20 12:49:15] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=1b3... OK

2022-06-20T20:49:15.382407263+08:00 [2022/06/20 12:49:15] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=382... OK

2022-06-20T20:49:15.385887386+08:00 [2022/06/20 12:49:15] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:49:15.387059665+08:00 [2022/06/20 12:49:15] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:49:15.388360978+08:00 [2022/06/20 12:49:15] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:49:15.397744611+08:00 [2022/06/20 12:49:15] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:49:15.398025959+08:00 [2022/06/20 12:49:15] [ info] [sp] stream processor started

2022-06-20T20:49:21.291863943+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:49:21.291904028+08:00 level=info msg=backoff delay=8s

2022-06-20T20:49:29.291988930+08:00 level=info msg="backoff timer done" actual=8.000095562s expected=8s

2022-06-20T20:49:29.292216787+08:00 level=info msg="Fluent bit started"

2022-06-20T20:49:29.348060463+08:00 Fluent Bit v1.8.3

2022-06-20T20:49:29.348095499+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:49:29.348099937+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:49:29.348103765+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:49:29.348106540+08:00 * https://fluentbit.io

2022-06-20T20:49:29.348109195+08:00 

2022-06-20T20:49:29.348885923+08:00 [2022/06/20 12:49:29] [ info] [engine] started (pid=28)

2022-06-20T20:49:29.348910730+08:00 [2022/06/20 12:49:29] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:49:29.348915749+08:00 [2022/06/20 12:49:29] [ info] [storage] in-memory

2022-06-20T20:49:29.348920128+08:00 [2022/06/20 12:49:29] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:49:29.348933613+08:00 [2022/06/20 12:49:29] [ info] [cmetrics] version=0.1.6

2022-06-20T20:49:29.350287995+08:00 [2022/06/20 12:49:29] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=1ce... OK

2022-06-20T20:49:29.362500016+08:00 [2022/06/20 12:49:29] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=413... OK

2022-06-20T20:49:29.365783379+08:00 [2022/06/20 12:49:29] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:49:29.366942675+08:00 [2022/06/20 12:49:29] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:49:29.368164899+08:00 [2022/06/20 12:49:29] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:49:29.379404816+08:00 [2022/06/20 12:49:29] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:49:29.379657951+08:00 [2022/06/20 12:49:29] [ info] [sp] stream processor started

2022-06-20T20:49:34.710558709+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:49:34.710584017+08:00 level=info msg=backoff delay=16s

2022-06-20T20:49:50.710708789+08:00 level=info msg="backoff timer done" actual=16.000098494s expected=16s

2022-06-20T20:49:50.711129589+08:00 level=info msg="Fluent bit started"

2022-06-20T20:49:50.764272787+08:00 Fluent Bit v1.8.3

2022-06-20T20:49:50.764306320+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:49:50.764311550+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:49:50.764315247+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:49:50.764320567+08:00 * https://fluentbit.io

2022-06-20T20:49:50.764323452+08:00 

2022-06-20T20:49:50.765375698+08:00 [2022/06/20 12:49:50] [ info] [engine] started (pid=31)

2022-06-20T20:49:50.765396196+08:00 [2022/06/20 12:49:50] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:49:50.765401346+08:00 [2022/06/20 12:49:50] [ info] [storage] in-memory

2022-06-20T20:49:50.765404542+08:00 [2022/06/20 12:49:50] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:49:50.765407888+08:00 [2022/06/20 12:49:50] [ info] [cmetrics] version=0.1.6

2022-06-20T20:49:50.766681218+08:00 [2022/06/20 12:49:50] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=1ef... OK

2022-06-20T20:49:50.778745011+08:00 [2022/06/20 12:49:50] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=465... OK

2022-06-20T20:49:50.783332511+08:00 [2022/06/20 12:49:50] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:49:50.784857625+08:00 [2022/06/20 12:49:50] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:49:50.786453940+08:00 [2022/06/20 12:49:50] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:49:50.803324706+08:00 [2022/06/20 12:49:50] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:49:50.803609050+08:00 [2022/06/20 12:49:50] [ info] [sp] stream processor started

2022-06-20T20:49:55.806927200+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:49:55.806958458+08:00 level=info msg=backoff delay=32s

2022-06-20T20:50:27.807086219+08:00 level=info msg="backoff timer done" actual=32.000111451s expected=32s

2022-06-20T20:50:27.807320740+08:00 level=info msg="Fluent bit started"

2022-06-20T20:50:27.859339747+08:00 Fluent Bit v1.8.3

2022-06-20T20:50:27.859369143+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:50:27.859377047+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:50:27.859380844+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:50:27.859382848+08:00 * https://fluentbit.io

2022-06-20T20:50:27.859384602+08:00 

2022-06-20T20:50:27.860374750+08:00 [2022/06/20 12:50:27] [ info] [engine] started (pid=34)

2022-06-20T20:50:27.860401621+08:00 [2022/06/20 12:50:27] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:50:27.860407993+08:00 [2022/06/20 12:50:27] [ info] [storage] in-memory

2022-06-20T20:50:27.860412321+08:00 [2022/06/20 12:50:27] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:50:27.860416930+08:00 [2022/06/20 12:50:27] [ info] [cmetrics] version=0.1.6

2022-06-20T20:50:27.861803883+08:00 [2022/06/20 12:50:27] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=221... OK

2022-06-20T20:50:27.874571106+08:00 [2022/06/20 12:50:27] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=465... OK

2022-06-20T20:50:27.878451920+08:00 [2022/06/20 12:50:27] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:50:27.879549410+08:00 [2022/06/20 12:50:27] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:50:27.880859649+08:00 [2022/06/20 12:50:27] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:50:27.899562815+08:00 [2022/06/20 12:50:27] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:50:27.899784601+08:00 [2022/06/20 12:50:27] [ info] [sp] stream processor started

2022-06-20T20:50:32.992561160+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:50:32.992585846+08:00 level=info msg=backoff delay=1m4s

2022-06-20T20:51:36.992691423+08:00 level=info msg="backoff timer done" actual=1m4.000101239s expected=1m4s

2022-06-20T20:51:36.992964506+08:00 level=info msg="Fluent bit started"

2022-06-20T20:51:37.047874610+08:00 Fluent Bit v1.8.3

2022-06-20T20:51:37.047888155+08:00 * Copyright (C) 2019-2021 The Fluent Bit Authors

2022-06-20T20:51:37.047892563+08:00 * Copyright (C) 2015-2018 Treasure Data

2022-06-20T20:51:37.047895599+08:00 * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-06-20T20:51:37.047898324+08:00 * https://fluentbit.io

2022-06-20T20:51:37.047900118+08:00 

2022-06-20T20:51:37.048865680+08:00 [2022/06/20 12:51:37] [ info] [engine] started (pid=37)

2022-06-20T20:51:37.048895075+08:00 [2022/06/20 12:51:37] [ info] [storage] version=1.1.1, initializing...

2022-06-20T20:51:37.048900175+08:00 [2022/06/20 12:51:37] [ info] [storage] in-memory

2022-06-20T20:51:37.048904613+08:00 [2022/06/20 12:51:37] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-06-20T20:51:37.048907699+08:00 [2022/06/20 12:51:37] [ info] [cmetrics] version=0.1.6

2022-06-20T20:51:37.050238066+08:00 [2022/06/20 12:51:37] [ info] [input:systemd:systemd.0] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=257... OK

2022-06-20T20:51:37.063460805+08:00 [2022/06/20 12:51:37] [ info] [input:systemd:systemd.1] seek_cursor=s=66d13f2233484a49a3b310b705691122;i=465... OK

2022-06-20T20:51:37.067459830+08:00 [2022/06/20 12:51:37] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443

2022-06-20T20:51:37.068605601+08:00 [2022/06/20 12:51:37] [ info] [filter:kubernetes:kubernetes.1] local POD info OK

2022-06-20T20:51:37.069767372+08:00 [2022/06/20 12:51:37] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...

2022-06-20T20:51:37.094267378+08:00 [2022/06/20 12:51:37] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

2022-06-20T20:51:37.094544137+08:00 [2022/06/20 12:51:37] [ info] [sp] stream processor started

2022-06-20T20:51:42.092880415+08:00 level=error msg="Fluent bit exited" error="signal: killed"

2022-06-20T20:51:42.092913537+08:00 level=info msg=backoff delay=2m8s

DehaoCheng

sigboom 检查一下es的情况，es日志是否正常，pvc占了多少。看下fluentbit是否挂载在containerd环境下的日志中，以至于采集不了日志。