关于kubesphere日志收集到kafka中的问题

still-L

huanggze 我先按照step2中改下cm试试吧

huanggze

still-L 可以，但应该没什么帮助【悲观脸= =】

still-L

huanggze 版主你好感谢您昨天的回复我试了修改cm并没有什么意义。我想和您确认下您的kafka环境和k8s集群在同一网络内吗？有经过nat 映射外网什么的嘛，您给我一个您正常的fluentbit 容器传输到kafka的日志，我经过昨天的观察，发现k8s环境在接收到fluentbit访问kafka的时候先查询的是kafka的hostname ，会不会和这个有关系？

huanggze

still-L 我的环境是在同一个网段，我的 k8s 节点和 kafka 用的青云的服务。不在同一个网段，我需要配 advertised host。官网有使用 kafka 作为日志接收者的教程 https://kubesphere.com.cn/docs/v2.1/zh-CN/platform-settings/output-kafka/

still-L

3:9092′: Name or service not known (after 56ms in state CONNECT)

2019-11-19 17:38:06: [2019/11/19 09:38:06] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-3:9092/97]: kafka-3:9092/97: Failed to resolve ‘kafka-3:9092′: Name or service not known (after 56ms in state CONNECT)

2019-11-19 17:38:06: [2019/11/19 09:38:06] [error] [out_kafka] fluent-bit#producer-1: [thrd:kafka-3:9092/97]: kafka-3:9092/97: Failed to resolve ‘kafka-3:9092’: Name or service not known (after 56ms in state CONNECT)

still-L

huanggze 嗯我的也做了 advertised host配置了，问下能否添加HostAliases 到fluentbit 容器？

huanggze

still-L 好像 fluent-bit 没有支持这个参数

still-L

huanggze 那能mount node /etc/hosts文件么？

huanggze

still-L 你看看官网的教程（下面说明那部分）能不能帮到你 https://kubesphere.com.cn/docs/v2.1/zh-CN/platform-settings/output-kafka/#%E7%AC%AC%E4%B8%80%E6%AD%A5%EF%BC%9A%E5%88%9B%E5%BB%BA-zookeeper–kafka-%E9%9B%86%E7%BE%A4

still-L

huanggze 感谢支持，我改成功了，我把fluentbit的ds配置改了，添加了本地的hosts文件。可以发送日志到kafka了。猜想数据流是这样的，kubesphere平台创建完成日志接收者之后，第一阶段去联系kafka创建topic 这个过程使用的IP通信，第二阶段往topic传输日志的时候使用的是kafka集群的hostname。嗯，这个思路是猜想的，目前看是这样的。

huanggze

still-L 赞！可以把改后的 ds yaml 分享出来，供以后大家参考吗？

still-L

huanggze ok

still-L

[root@k8s-1 ~]# cat Flent-bit-DaemonSet.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  generation: 1
  labels:
    app: fluent-bit
    release: logging-fluentbit-operator
  name: fluent-bit
  namespace: kubesphere-logging-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: fluent-bit
      chart: fluentbit-operator-0.1.0
      release: logging-fluentbit-operator
  template:
    metadata:
      annotations:
        prometheus.io/path: /api/v1/metrics/prometheus
        prometheus.io/port: "2020"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: fluent-bit
        chart: fluentbit-operator-0.1.0
        release: logging-fluentbit-operator
    spec:
      containers:
      - image: kubesphere/fluent-bit:v1.3.2-reload
        imagePullPolicy: IfNotPresent
        name: fluent-bit
        ports:
        - containerPort: 2020
          name: monitor
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/docker/containers
          name: varlibcontainers
          readOnly: true
        - mountPath: /fluent-bit/etc/fluent-bit.conf
          name: config
          subPath: fluent-bit.conf
        - mountPath: /fluent-bit/app-config/
          name: app-config
        - mountPath: /tail-db
          name: positions
        - mountPath: /var/log/
          name: varlogs
        - mountPath: /etc/hosts  ####新增hosts文件
          name: hosts
          readOnly: true
      - args:
        - -volume-dir=/fluent-bit/app-config/
        - -webhook-url=http://127.0.0.1:24444/api/config.reload
        image: kubesphere/configmap-reload:v0.0.1
        imagePullPolicy: IfNotPresent
        name: config-reloader
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /fluent-bit/app-config/
          name: app-config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: fluentbit
      serviceAccountName: fluentbit
      terminationGracePeriodSeconds: 30
      tolerations:
      - operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: varlibcontainers
      - configMap:
          defaultMode: 420
          name: fluent-bit-config
        name: config
      - configMap:
          defaultMode: 420
          name: fluent-bit-app-config
        name: app-config
      - hostPath:
          path: /var/log
          type: ""
        name: varlogs
      - hostPath:
          path: /etc/hosts  ####新增hosts文件
          type: ""
        name: hosts
      - emptyDir: {}
        name: positions
  templateGeneration: 1
  updateStrategy:
    type: OnDelete
status:
  currentNumberScheduled: 4
  desiredNumberScheduled: 4
  numberAvailable: 4
  numberMisscheduled: 0
  numberReady: 4
  observedGeneration: 1
  updatedNumberScheduled: 4

« 上一页