KubeSphere日志采集相关问题

zhangwn · 2022年4月20日

Feynman 这里log文件上传不了。fluentbit日志没什么报错，fluent.log文件有一部分信息不知道是否报错。

日志内容：

level=info msg=“Fluentd started”

2022-04-19 07:18:57 +0000 [info]: parsing config file is succeeded path=“/fluentd/etc/fluent.conf”

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-aws-elasticsearch-service’ version ‘2.4.1’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-dedot_filter’ version ‘1.0.0’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-detect-exceptions’ version ‘0.0.14’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-elasticsearch’ version ‘5.2.1’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-kafka’ version ‘0.17.5’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-label-router’ version ‘0.2.10’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-multi-format-parser’ version ‘1.0.0’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-oss’ version ‘0.0.2’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-record-modifier’ version ‘2.1.0’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-rewrite-tag-filter’ version ‘2.4.0’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-s3’ version ‘1.6.1’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluent-plugin-sumologic_output’ version ‘1.7.3’

2022-04-19 07:18:57 +0000 [info]: gem ‘fluentd’ version ‘1.14.4’

2022-04-19 07:18:57 +0000 [info]: [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] brokers has been set: [“my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091”, “my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092”, “my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093”]

2022-04-19 07:18:57 +0000 [warn]: [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] Use ‘topic’ field of event record for topic but no fallback. Recommend to set default_topic or set ‘tag’ in buffer chunk keys like <buffer topic,tag>

2022-04-19 07:18:58 +0000 [info]: using configuration file: <ROOT>

rpc_endpoint "127.0.0.1:24444"

log_level info

workers 1

</system>

@type forward

bind "0.0.0.0"

port 24224

</source>

@id main

@type label_router

<route>

  @label "@0943890ac248552151615ab88ecb5e43"

  <match>

    namespaces agcloud-dev,default,kube-system

  </match>

</route>

</match>

<filter \*\*>

  @id ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusterfilter::cluster-fluentd-filter-k8s-0

  @type record_transformer

  enable_ruby true

  <record>

    kubernetes_ns ${record["kubernetes"]["namespace_name"]}

  </record>

</filter>

<match \*\*>

  @id ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0

  @type kafka2

  brokers my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091,my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092,my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093

  topic_key "kubernetes_ns"

  use_event_time true

  <format>

    @type "json"

  </format>

</match>

</label>

@type null

@id main-no-output

</match>

<match fluent.\*>

  @type null

  @id main-fluentd-log

</match>

</label>

</ROOT>

2022-04-19 07:18:58 +0000 [info]: starting fluentd-1.14.4 pid=11 ruby=“2.7.5”

2022-04-19 07:18:58 +0000 [info]: spawn command to main: cmdline=[“/usr/bin/ruby”, “-Eascii-8bit:ascii-8bit”, “/usr/bin/fluentd”, “-c”, “/fluentd/etc/fluent.conf”, “-p”, “/fluentd/plugins”, “–under-supervisor”]

2022-04-19 07:18:58 +0000 [info]: adding filter in @0943890ac248552151615ab88ecb5e43 pattern=“**” type=“record_transformer”

2022-04-19 07:18:58 +0000 [info]: adding match in @0943890ac248552151615ab88ecb5e43 pattern=“**” type=“kafka2”

2022-04-19 07:18:58 +0000 [info]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] brokers has been set: [“my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091”, “my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092”, “my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093”]

2022-04-19 07:18:58 +0000 [warn]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] Use ‘topic’ field of event record for topic but no fallback. Recommend to set default_topic or set ‘tag’ in buffer chunk keys like <buffer topic,tag>

2022-04-19 07:18:58 +0000 [info]: adding match in @FLUENT_LOG pattern=“fluent.*” type=“null”

2022-04-19 07:18:58 +0000 [info]: adding match pattern=“**” type=“label_router”

2022-04-19 07:18:58 +0000 [info]: adding match pattern=“**” type=“null”

2022-04-19 07:18:58 +0000 [info]: adding source type=“forward”

2022-04-19 07:18:58 +0000 [info]: #0 starting fluentd worker pid=20 ppid=11 worker=0

2022-04-19 07:18:58 +0000 [info]: #0 [ClusterFluentdConfig-cluster-cluster-fluentd-config-kafka::cluster::clusteroutput::cluster-fluentd-output-kafka-0] initialized kafka producer: fluentd

2022-04-19 07:18:58 +0000 [info]: #0 listening port port=24224 bind=“0.0.0.0”

2022-04-19 07:18:58 +0000 [info]: #0 fluentd worker is now running worker=0

zhangwn · 2022年4月20日

DehaoCheng 这个workthough我也有看，我是本地有k8s集群和kafka集群，不需要再创建。我是使用fluent-operator部署的，下载fluent-operator.tgz后手动执行安装命令：

helm upgrade –install fluent-operator –create-namespace -n fluent fluent-operator.tgz

fluentbit+fluentd的配置我主要是参考了公众号文章https://mp.weixin.qq.com/s/-FpltcXoB6tCKMXaq6DAuQ里面的。

配置如下：

apiVersion: fluentbit.fluent.io/v1alpha2

kind: FluentBit

metadata:

namespace: fluent

labels:

app.kubernetes.io/name: fluent-bit

spec:

image: kubesphere/fluent-bit:v1.8.11

positionDB:

hostPath:

  path: /var/lib/fluent-bit/

resources:

requests:

  cpu: 10m

  memory: 25Mi

limits:

  cpu: 500m

  memory: 200Mi

fluentBitConfigName: fluent-bit-config

tolerations:

- operator: Exists

---

apiVersion: fluentbit.fluent.io/v1alpha2

kind: ClusterFluentBitConfig

metadata:

labels:

app.kubernetes.io/name: fluent-bit

spec:

service:

parsersFile: parsers.conf

inputSelector:

matchLabels:

  fluentbit.fluent.io/enabled: "true"

  fluentbit.fluent.io/mode: "k8s"

filterSelector:

matchLabels:

  fluentbit.fluent.io/enabled: "true"

  fluentbit.fluent.io/mode: "k8s"

outputSelector:

matchLabels:

  fluentbit.fluent.io/enabled: "true"

  fluentbit.fluent.io/mode: "k8s"

---

apiVersion: fluentbit.fluent.io/v1alpha2

kind: ClusterInput

metadata:

labels:

fluentbit.fluent.io/enabled: "true"

fluentbit.fluent.io/mode: "k8s"

spec:

tail:

tag: kube.\*

path: /var/log/containers/\*.log

parser: docker

refreshIntervalSeconds: 10

memBufLimit: 5MB

skipLongLines: true

db: /fluent-bit/tail/pos.db

dbSync: Normal

---

apiVersion: fluentbit.fluent.io/v1alpha2

kind: ClusterFilter

metadata:

labels:

fluentbit.fluent.io/enabled: "true"

fluentbit.fluent.io/mode: "k8s"

spec:

match: kube.*

filters:

kubernetes:

kubeURL: https://kubernetes.default.svc:443

kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

labels: false

annotations: false
nest:

operation: lift

nestedUnder: kubernetes

addPrefix: kubernetes_
modify:

rules:
- remove: stream
  - remove: kubernetes_pod_id
  - remove: kubernetes_host
  - remove: kubernetes_container_hash
nest:

operation: nest

wildcard:
- kubernetes_*
  
  nestUnder: kubernetes
  
  removePrefix: kubernetes_

---

apiVersion: fluentbit.fluent.io/v1alpha2

kind: ClusterOutput

metadata:

labels:

fluentbit.fluent.io/enabled: "true"

fluentbit.fluent.io/component: logging

spec:

matchRegex: (?:kube|service)\.(.*)

forward:

host: fluentd.fluent.svc

port: 24224

---

apiVersion: fluentd.fluent.io/v1alpha1

kind: Fluentd

metadata:

namespace: fluent

labels:

app.kubernetes.io/name: fluentd

spec:

globalInputs:

forward:

bind: 0.0.0.0

port: 24224

replicas: 3

image: kubesphere/fluentd:v1.14.4

fluentdCfgSelector:

matchLabels:

config.fluentd.fluent.io/enabled: “true”

---

apiVersion: fluentd.fluent.io/v1alpha1

kind: ClusterFluentdConfig

metadata:

labels:

config.fluentd.fluent.io/enabled: "true"

spec:

watchedNamespaces:

kube-system
default
agcloud-dev

clusterFilterSelector:

matchLabels:

filter.fluentd.fluent.io/type: “k8s”

filter.fluentd.fluent.io/enabled: “true”

clusterOutputSelector:

matchLabels:

output.fluentd.fluent.io/type: “kafka”

output.fluentd.fluent.io/enabled: “true”

---

apiVersion: fluentd.fluent.io/v1alpha1

kind: ClusterFilter

metadata:

labels:

filter.fluentd.fluent.io/type: "k8s"

filter.fluentd.fluent.io/enabled: "true"

spec:

filters:

recordTransformer:

enableRuby: true

records:
- key: kubernetes_ns
  
  value: ${record[“kubernetes”][“namespace_name”]}

---

apiVersion: fluentd.fluent.io/v1alpha1

kind: ClusterOutput

metadata:

labels:

output.fluentd.fluent.io/type: "kafka"

output.fluentd.fluent.io/enabled: "true"

spec:

outputs:

kafka:

brokers: my-cluster-kafka-bootstrap.my-cluster-kafka-0.svc:9091,my-cluster-kafka-bootstrap.my-cluster-kafka-1.svc:9092,my-cluster-kafka-bootstrap.my-cluster-kafka-2.svc:9093

useEventTime: true

topicKey: kubernetes_ns

容器启动也没有看到报错信息

zhangwn · 2022年4月20日

DehaoCheng 第一个问题的fluentbig部分日志如下：

DehaoCheng · 2022年4月24日

关于第二个问题
因为你在fluentbit config中定义的

outputSelector:

  matchLabels:

    fluentbit.fluent.io/enabled: "true"

    fluentbit.fluent.io/mode: "k8s"

的label必须匹配fluentbit中的clusteroutput的label，我这边看到是不匹配的，所以没有输入，自然没有输出。

第一个问题需要看一下你的配置，你看一看你的与fluentbit有关的crd是什么配置？

zhangwn · 2022年4月26日

DehaoCheng

谢谢。现在遇到一个情况fluent-operator部署过滤器ClusterFilter使用- kubernetes插件，有些时候生效，有些时候不生效（日志记录没有kubernetes的元数据信息），看fluentbit的容器日志也没有什么提示，请问这个情况要怎么排查？

DehaoCheng · 2022年4月26日

zhangwn 使用fluent-operator会产生一个秘钥，就是一个secret，那里存储着fluentbit的配置信息。你可以使用base64转码，看看是否将相关配置加载进去了吗

zhangwn · 2022年4月26日

DehaoCheng

可以看到是有配置的，现在比较奇怪的如果tail插件的tag是kube.*，kubernetes插件的match也是kube.*的情况下日志记录有元数据信息，其他就没有。实在搞不懂了！！！

Wwanjunlei · 2022年4月26日

kubernetes插件的tag必须满足kube.var.log.containers.

zhangwn · 2022年4月27日

wanjunlei DehaoCheng

感谢两位的指导，我发现使用kubernetes插件的kubeTagPrefix属性可以拿到元数据信息，默认匹配是kube.var.log.containers.

我添加kubeTagPrefix: agcloud.var.log.containers.配置就可以匹配到agcloud.*的tag了。

zhangwn · 2022年4月27日

DehaoCheng wanjunlei

再次请教一下两位大佬一些问题：

1、环境变量配置。我看fluentbit配置的spec有containers配置env属性的，在fluent-operator部署FluentBit的配置文件要怎么写？貌似没有这个属性配置。

2、过滤器ClusterFilter插件modify的条件属性keyDoesNotExist配置问题。这个在yaml文件里面要怎么写配置？它被定义为map结构类型，可是我只有一个key，执行的时候会报got “string”, expected "map"错误。

DehaoCheng · 2022年4月28日

zhangwn 第一个问题，env是代码上写的，可以看代码https://github.com/fluent/fluent-operator/blob/master/pkg/operator/daemonset.go#L80
你的具体需求是什么？
第二个问题可以参考：https://github.com/fluent/fluent-operator/blob/master/config/crd/bases/fluentd.fluent.io_filters.yaml
crd中怎么写的

KubeSphere日志采集相关问题

zhangwn

zhangwn

zhangwn

DehaoChengK零S

zhangwn

DehaoChengK零S

zhangwn

WwanjunleiK零S

zhangwn

zhangwn

DehaoChengK零S