FeynmanK零SK贰SK壹S
Fluent Bit 和 Fluentd 有错误日志吗?
@DehaoCheng 也请关注一下这个问题吧
Fluent Bit 和 Fluentd 有错误日志吗?
@DehaoCheng 也请关注一下这个问题吧
3.3之前的版本都是采用fluentbit-operator。现在fluent-operator以及releasev1.0.0。欢迎试用部署。fluent-operator可以部署fluentd以及fluentbit,而fluentbit-operator只能部署fluentbit。然后针对这两个问题进行解答:
1.旧的日志可以看到应该是fluentbit环节出现了问题,需要提供fluentbit的日志信息。
2.fluentbit+fluentd的话要看你配置,你可以使用fluent-operator来部署fluentbit以及fluentd。在ks上手动安装fluent-operator的话记得删除旧的crd等资源。
另外你可以使用workthough来体验配置
https://github.com/kubesphere-sigs/fluent-operator-walkthrough
Feynman 这里log文件上传不了。fluentbit日志没什么报错,fluent.log文件有一部分信息不知道是否报错。
日志内容:
level=info msg=“Fluentd started”
2022-04-19 07:18:57 +0000 [info]: parsing config file is succeeded path=“/fluentd/etc/fluent.conf”
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-aws-elasticsearch-service’ version ‘2.4.1’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-dedot_filter’ version ‘1.0.0’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-detect-exceptions’ version ‘0.0.14’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-elasticsearch’ version ‘5.2.1’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-kafka’ version ‘0.17.5’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-label-router’ version ‘0.2.10’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-multi-format-parser’ version ‘1.0.0’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-oss’ version ‘0.0.2’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-record-modifier’ version ‘2.1.0’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-rewrite-tag-filter’ version ‘2.4.0’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-s3’ version ‘1.6.1’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-sumologic_output’ version ‘1.7.3’
2022-04-19 07:18:57 +0000 [info]: gem ‘fluentd’ version ‘1.14.4’
2022-04-19 07:18:57 +0000 [info]: [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] brokers has been set: [“my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091”, “my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092”, “my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093”]
2022-04-19 07:18:57 +0000 [warn]: [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] Use ‘topic’ field of event record for topic but no fallback. Recommend to set default_topic or set ‘tag’ in buffer chunk keys like <buffer topic,tag>
2022-04-19 07:18:58 +0000 [info]: using configuration file: <ROOT>
<system>
rpc_endpoint "127.0.0.1:24444"
log_level info
workers 1
</system>
<source>
@type forward
bind "0.0.0.0"
port 24224
</source>
<match **>
@id main
@type label_router
<route>
@label "@0943890ac248552151615ab88ecb5e43"
<match>
namespaces agcloud-dev,default,kube-system
</match>
</route>
</match>
<label @0943890ac248552151615ab88ecb5e43>
<filter \*\*>
@id ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusterfilter::cluster-fluentd-filter-k8s-0
@type record_transformer
enable_ruby true
<record>
kubernetes_ns ${record["kubernetes"]["namespace_name"]}
</record>
</filter>
<match \*\*>
@id ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0
@type kafka2
brokers my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091,my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092,my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093
topic_key "kubernetes_ns"
use_event_time true
<format>
@type "json"
</format>
</match>
</label>
<match **>
@type null
@id main-no-output
</match>
<label @FLUENT_LOG>
<match fluent.\*>
@type null
@id main-fluentd-log
</match>
</label>
</ROOT>
2022-04-19 07:18:58 +0000 [info]: starting fluentd-1.14.4 pid=11 ruby=“2.7.5”
2022-04-19 07:18:58 +0000 [info]: spawn command to main: cmdline=[“/usr/bin/ruby”, “-Eascii-8bit:ascii-8bit”, “/usr/bin/fluentd”, “-c”, “/fluentd/etc/fluent.conf”, “-p”, “/fluentd/plugins”, “–under-supervisor”]
2022-04-19 07:18:58 +0000 [info]: adding filter in @0943890ac248552151615ab88ecb5e43 pattern=“**” type=“record_transformer”
2022-04-19 07:18:58 +0000 [info]: adding match in @0943890ac248552151615ab88ecb5e43 pattern=“**” type=“kafka2”
2022-04-19 07:18:58 +0000 [info]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] brokers has been set: [“my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091”, “my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092”, “my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093”]
2022-04-19 07:18:58 +0000 [warn]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] Use ‘topic’ field of event record for topic but no fallback. Recommend to set default_topic or set ‘tag’ in buffer chunk keys like <buffer topic,tag>
2022-04-19 07:18:58 +0000 [info]: adding match in @FLUENT_LOG pattern=“fluent.*” type=“null”
2022-04-19 07:18:58 +0000 [info]: adding match pattern=“**” type=“label_router”
2022-04-19 07:18:58 +0000 [info]: adding match pattern=“**” type=“null”
2022-04-19 07:18:58 +0000 [info]: adding source type=“forward”
2022-04-19 07:18:58 +0000 [info]: #0 starting fluentd worker pid=20 ppid=11 worker=0
2022-04-19 07:18:58 +0000 [info]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] initialized kafka producer: fluentd
2022-04-19 07:18:58 +0000 [info]: #0 listening port port=24224 bind=“0.0.0.0”
2022-04-19 07:18:58 +0000 [info]: #0 fluentd worker is now running worker=0
DehaoCheng 这个workthough我也有看,我是本地有k8s集群和kafka集群,不需要再创建。我是使用fluent-operator部署的,下载fluent-operator.tgz后手动执行安装命令:
helm upgrade –install fluent-operator –create-namespace -n fluent fluent-operator.tgz
fluentbit+fluentd的配置我主要是参考了公众号文章https://mp.weixin.qq.com/s/-FpltcXoB6tCKMXaq6DAuQ里面的。
配置如下:
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
name: fluent-bit
namespace: fluent
labels:
app.kubernetes.io/name: fluent-bit
spec:
image: kubesphere/fluent-bit:v1.8.11
positionDB:
hostPath:
path: /var/lib/fluent-bit/
resources:
requests:
cpu: 10m
memory: 25Mi
limits:
cpu: 500m
memory: 200Mi
fluentBitConfigName: fluent-bit-config
tolerations:
- operator: Exists
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
name: fluent-bit-config
labels:
app.kubernetes.io/name: fluent-bit
spec:
service:
parsersFile: parsers.conf
inputSelector:
matchLabels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
filterSelector:
matchLabels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
outputSelector:
matchLabels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
name: tail
labels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
spec:
tail:
tag: kube.\*
path: /var/log/containers/\*.log
parser: docker
refreshIntervalSeconds: 10
memBufLimit: 5MB
skipLongLines: true
db: /fluent-bit/tail/pos.db
dbSync: Normal
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
name: kubernetes
labels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
spec:
match: kube.*
filters:
kubernetes:
kubeURL: https://kubernetes.default.svc:443
kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
labels: false
annotations: false
nest:
operation: lift
nestedUnder: kubernetes
addPrefix: kubernetes_
modify:
rules:
remove: stream
remove: kubernetes_pod_id
remove: kubernetes_host
remove: kubernetes_container_hash
nest:
operation: nest
wildcard:
kubernetes_*
nestUnder: kubernetes
removePrefix: kubernetes_
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
name: fluentd
labels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/component: logging
spec:
matchRegex: (?:kube|service)\.(.*)
forward:
host: fluentd.fluent.svc
port: 24224
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
name: fluentd
namespace: fluent
labels:
app.kubernetes.io/name: fluentd
spec:
globalInputs:
forward:
bind: 0.0.0.0
port: 24224
replicas: 3
image: kubesphere/fluentd:v1.14.4
fluentdCfgSelector:
matchLabels:
config.fluentd.fluent.io/enabled: “true”
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
name: cluster-fluentd-config-kafka
labels:
config.fluentd.fluent.io/enabled: "true"
spec:
watchedNamespaces:
kube-system
default
agcloud-dev
clusterFilterSelector:
matchLabels:
filter.fluentd.fluent.io/type: “k8s”
filter.fluentd.fluent.io/enabled: “true”
clusterOutputSelector:
matchLabels:
output.fluentd.fluent.io/type: “kafka”
output.fluentd.fluent.io/enabled: “true”
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFilter
metadata:
name: cluster-fluentd-filter-k8s
labels:
filter.fluentd.fluent.io/type: "k8s"
filter.fluentd.fluent.io/enabled: "true"
spec:
filters:
recordTransformer:
enableRuby: true
records:
key: kubernetes_ns
value: ${record[“kubernetes”][“namespace_name”]}
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
name: cluster-fluentd-output-kafka
labels:
output.fluentd.fluent.io/type: "kafka"
output.fluentd.fluent.io/enabled: "true"
spec:
outputs:
kafka:
brokers: my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091,my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092,my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093
useEventTime: true
topicKey: kubernetes_ns
容器启动也没有看到报错信息
DehaoCheng 第一个问题的fluentbig部分日志如下:
关于第二个问题
因为你在fluentbit config中定义的
outputSelector:
matchLabels:
fluentbit.fluent.io/enabled: "true"
fluentbit.fluent.io/mode: "k8s"
的label必须匹配fluentbit中的clusteroutput的label,我这边看到是不匹配的,所以没有输入,自然没有输出。
第一个问题需要看一下你的配置,你看一看你的与fluentbit有关的crd是什么配置?
谢谢。现在遇到一个情况fluent-operator部署过滤器ClusterFilter使用- kubernetes插件,有些时候生效,有些时候不生效(日志记录没有kubernetes的元数据信息),看fluentbit的容器日志也没有什么提示,请问这个情况要怎么排查?
zhangwn 使用fluent-operator会产生一个秘钥,就是一个secret,那里存储着fluentbit的配置信息。你可以使用base64转码,看看是否将相关配置加载进去了吗
可以看到是有配置的,现在比较奇怪的如果tail插件的tag是kube.*,kubernetes插件的match也是kube.*的情况下日志记录有元数据信息,其他就没有。实在搞不懂了!!!
kubernetes插件的tag必须满足kube.var.log.containers.
感谢两位的指导,我发现使用kubernetes插件的kubeTagPrefix属性可以拿到元数据信息,默认匹配是kube.var.log.containers.
我添加kubeTagPrefix: agcloud.var.log.containers.配置就可以匹配到agcloud.*的tag了。
再次请教一下两位大佬一些问题:
1、环境变量配置。我看fluentbit配置的spec有containers配置env属性的,在fluent-operator部署FluentBit的配置文件要怎么写?貌似没有这个属性配置。
2、过滤器ClusterFilter插件modify的条件属性keyDoesNotExist配置问题。这个在yaml文件里面要怎么写配置?它被定义为map结构类型,可是我只有一个key,执行的时候会报got “string”, expected "map"错误。
zhangwn 第一个问题,env是代码上写的,可以看代码https://github.com/fluent/fluent-operator/blob/master/pkg/operator/daemonset.go#L80
你的具体需求是什么?
第二个问题可以参考:https://github.com/fluent/fluent-operator/blob/master/config/crd/bases/fluentd.fluent.io_filters.yaml
crd中怎么写的