控制台日志查询功能无法打印日志

huanggze · 2019年11月23日

lan-liang 好的。我在你开的原贴里有回复

rysinal · 2020年2月4日

huanggze huanggze 我这边是做了映射后，按照 https://github.com/kubesphere/kubesphere/issues/680 和 https://github.com/kubesphere/kubesphere/issues/703 操作后仍无法解决，现象是：仅能通过管理员账号可以在日志界面查看到ks的系统级别日志，无法搜到应用容器的日志：

# kubectl logs -n kubesphere-logging-system logging-fluentbit-operator-6798c5bf8-vpsv7

time="2020-02-04T09:51:48Z" level=info msg="Registering plugin: fluentbit_service"
time="2020-02-04T09:51:48Z" level=info msg="Registering plugin: fluentbit_input"
time="2020-02-04T09:51:48Z" level=info msg="Registering plugin: fluentbit_filter"
time="2020-02-04T09:51:48Z" level=info msg="Registering plugin: fluentbit_output"
time="2020-02-04T09:51:48Z" level=info msg="Registering plugin: fluentbit_settings"
time="2020-02-04T09:51:48Z" level=info msg="Gettint current environment: ns: \"kubesphere-logging-system\" pod: \"logging-fluentbit-operator-6798c5bf8-vpsv7\""
time="2020-02-04T09:51:49Z" level=info msg="Go Version: go1.10.8"
time="2020-02-04T09:51:49Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-02-04T09:51:49Z" level=info msg="operator-sdk Version: 0.0.5+git"
time="2020-02-04T09:51:49Z" level=info msg="Operator namespace: kubesphere-logging-system"
time="2020-02-04T09:51:49Z" level=info msg="Watching logging.kubesphere.io/v1alpha1, FluentBit, kubesphere-logging-system, 0"
time="2020-02-04T09:51:49Z" level=info msg="Initializing configuration"
time="2020-02-04T09:51:49Z" level=info msg="New config file change \"Initial\": CREATE"
time="2020-02-04T09:51:49Z" level=info msg="Trying to init fluent-bit"
time="2020-02-04T09:51:49Z" level=info msg="FluentBit DaemonSet does not exists!"
time="2020-02-04T09:51:49Z" level=error msg="daemonsets.extensions \"fluent-bit\" not found"
time="2020-02-04T09:51:49Z" level=info msg="Deploying fluent-bit"
time="2020-02-04T09:51:49Z" level=error msg="<nil>"
time="2020-02-04T09:51:49Z" level=error msg="serviceaccounts \"fluentbit\" already exists"
time="2020-02-04T09:51:49Z" level=error msg="<nil>"
time="2020-02-04T09:51:49Z" level=error msg="clusterroles.rbac.authorization.k8s.io \"LoggingRole\" already exists"
time="2020-02-04T09:51:49Z" level=error msg="<nil>"
time="2020-02-04T09:51:49Z" level=info msg="New CRD arrived &v1alpha1.FluentBit{TypeMeta:v1.TypeMeta{Kind:\"FluentBit\", APIVersion:\"logging.kubesphere.io/v1alpha1\"}, ObjectMeta:v1.ObjectMeta{Name:\"fluent-bit\", GenerateName:\"\", Namespace:\"kubesphere-logging-system\", SelfLink:\"/apis/logging.kubesphere.io/v1alpha1/namespaces/kubesphere-logging-system/fluentbits/fluent-bit\", UID:\"990ffcf0-112e-4a2b-9b6b-c65385091db7\", ResourceVersion:\"8024\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63714067288, loc:(*time.Location)(0x19245e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"logging.kubesphere.io/v1alpha1\\\",\\\"kind\\\":\\\"FluentBit\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"fluent-bit\\\",\\\"namespace\\\":\\\"kubesphere-logging-system\\\"},\\\"spec\\\":{\\\"filter\\\":[{\\\"name\\\":\\\"fluentbit-filter\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"kubernetes\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Kube_URL\\\",\\\"value\\\":\\\"https://kubernetes.default.svc:443\\\"},{\\\"name\\\":\\\"Kube_CA_File\\\",\\\"value\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt\\\"},{\\\"name\\\":\\\"Kube_Token_File\\\",\\\"value\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount/token\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"nest\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Operation\\\",\\\"value\\\":\\\"lift\\\"},{\\\"name\\\":\\\"Nested_under\\\",\\\"value\\\":\\\"kubernetes\\\"},{\\\"name\\\":\\\"Prefix_with\\\",\\\"value\\\":\\\"kubernetes_\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter-remove-stream\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"modify\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Remove\\\",\\\"value\\\":\\\"stream\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter-remove-labels\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"modify\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Remove\\\",\\\"value\\\":\\\"kubernetes_labels\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter-remove-annotations\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"modify\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Remove\\\",\\\"value\\\":\\\"kubernetes_annotations\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter-remove-podid\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"modify\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Remove\\\",\\\"value\\\":\\\"kubernetes_pod_id\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"},{\\\"name\\\":\\\"fluentbit-filter-nest\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"nest\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Operation\\\",\\\"value\\\":\\\"nest\\\"},{\\\"name\\\":\\\"Wildcard\\\",\\\"value\\\":\\\"kubernetes_*\\\"},{\\\"name\\\":\\\"Nested_under\\\",\\\"value\\\":\\\"kubernetes\\\"},{\\\"name\\\":\\\"Remove_prefix\\\",\\\"value\\\":\\\"kubernetes_\\\"}],\\\"type\\\":\\\"fluentbit_filter\\\"}],\\\"input\\\":[{\\\"name\\\":\\\"fluentbit-input\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"tail\\\"},{\\\"name\\\":\\\"Path\\\",\\\"value\\\":\\\"/var/log/containers/*.log\\\"},{\\\"name\\\":\\\"Parser\\\",\\\"value\\\":\\\"docker\\\"},{\\\"name\\\":\\\"Tag\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Refresh_Interval\\\",\\\"value\\\":\\\"5\\\"},{\\\"name\\\":\\\"Mem_Buf_Limit\\\",\\\"value\\\":\\\"5MB\\\"},{\\\"name\\\":\\\"Skip_Long_Lines\\\",\\\"value\\\":\\\"On\\\"},{\\\"name\\\":\\\"DB\\\",\\\"value\\\":\\\"/tail-db/tail-containers-state.db\\\"},{\\\"name\\\":\\\"DB.Sync\\\",\\\"value\\\":\\\"Normal\\\"}],\\\"type\\\":\\\"fluentbit_input\\\"}],\\\"output\\\":[{\\\"name\\\":\\\"fluentbit-output\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Name\\\",\\\"value\\\":\\\"es\\\"},{\\\"name\\\":\\\"Match\\\",\\\"value\\\":\\\"kube.*\\\"},{\\\"name\\\":\\\"Host\\\",\\\"value\\\":\\\"elasticsearch-logging-data.kubesphere-logging-system.svc\\\"},{\\\"name\\\":\\\"Port\\\",\\\"value\\\":\\\"9200\\\"},{\\\"name\\\":\\\"Logstash_Format\\\",\\\"value\\\":\\\"On\\\"},{\\\"name\\\":\\\"Replace_Dots\\\",\\\"value\\\":\\\"on\\\"},{\\\"name\\\":\\\"Retry_Limit\\\",\\\"value\\\":\\\"False\\\"},{\\\"name\\\":\\\"Type\\\",\\\"value\\\":\\\"flb_type\\\"},{\\\"name\\\":\\\"Time_Key\\\",\\\"value\\\":\\\"@timestamp\\\"},{\\\"name\\\":\\\"Logstash_Prefix\\\",\\\"value\\\":\\\"ks-logstash-log\\\"}],\\\"type\\\":\\\"fluentbit_output\\\"}],\\\"service\\\":[{\\\"name\\\":\\\"fluentbit-service\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Flush\\\",\\\"value\\\":\\\"1\\\"},{\\\"name\\\":\\\"Daemon\\\",\\\"value\\\":\\\"Off\\\"},{\\\"name\\\":\\\"Log_Level\\\",\\\"value\\\":\\\"info\\\"},{\\\"name\\\":\\\"Parsers_File\\\",\\\"value\\\":\\\"parsers.conf\\\"}],\\\"type\\\":\\\"fluentbit_service\\\"}],\\\"settings\\\":[{\\\"name\\\":\\\"fluentbit-settings\\\",\\\"parameters\\\":[{\\\"name\\\":\\\"Enable\\\",\\\"value\\\":\\\"true\\\"}],\\\"type\\\":\\\"fluentbit_settings\\\"}]}}\\n\"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\"}, Spec:v1alpha1.FluentBitSpec{Service:[]v1alpha1.Plugin{v1alpha1.Plugin{Type:\"fluentbit_service\", Name:\"fluentbit-service\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Flush\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"1\"}, v1alpha1.Parameter{Name:\"Daemon\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"Off\"}, v1alpha1.Parameter{Name:\"Log_Level\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"info\"}, v1alpha1.Parameter{Name:\"Parsers_File\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"parsers.conf\"}}}}, Input:[]v1alpha1.Plugin{v1alpha1.Plugin{Type:\"fluentbit_input\", Name:\"fluentbit-input\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"tail\"}, v1alpha1.Parameter{Name:\"Path\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"/var/log/containers/*.log\"}, v1alpha1.Parameter{Name:\"Parser\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"docker\"}, v1alpha1.Parameter{Name:\"Tag\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Refresh_Interval\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"5\"}, v1alpha1.Parameter{Name:\"Mem_Buf_Limit\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"5MB\"}, v1alpha1.Parameter{Name:\"Skip_Long_Lines\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"On\"}, v1alpha1.Parameter{Name:\"DB\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"/tail-db/tail-containers-state.db\"}, v1alpha1.Parameter{Name:\"DB.Sync\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"Normal\"}}}}, Filter:[]v1alpha1.Plugin{v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Kube_URL\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"https://kubernetes.default.svc:443\"}, v1alpha1.Parameter{Name:\"Kube_CA_File\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt\"}, v1alpha1.Parameter{Name:\"Kube_Token_File\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"/var/run/secrets/kubernetes.io/serviceaccount/token\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"nest\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Operation\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"lift\"}, v1alpha1.Parameter{Name:\"Nested_under\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes\"}, v1alpha1.Parameter{Name:\"Prefix_with\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter-remove-stream\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"modify\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Remove\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"stream\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter-remove-labels\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"modify\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Remove\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_labels\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter-remove-annotations\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"modify\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Remove\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_annotations\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter-remove-podid\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"modify\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Remove\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_pod_id\"}}}, v1alpha1.Plugin{Type:\"fluentbit_filter\", Name:\"fluentbit-filter-nest\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"nest\"},v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Operation\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"nest\"}, v1alpha1.Parameter{Name:\"Wildcard\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_*\"}, v1alpha1.Parameter{Name:\"Nested_under\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes\"}, v1alpha1.Parameter{Name:\"Remove_prefix\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kubernetes_\"}}}}, Output:[]v1alpha1.Plugin{v1alpha1.Plugin{Type:\"fluentbit_output\", Name:\"fluentbit-output\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Name\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"es\"}, v1alpha1.Parameter{Name:\"Match\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"kube.*\"}, v1alpha1.Parameter{Name:\"Host\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"elasticsearch-logging-data.kubesphere-logging-system.svc\"}, v1alpha1.Parameter{Name:\"Port\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"9200\"}, v1alpha1.Parameter{Name:\"Logstash_Format\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"On\"}, v1alpha1.Parameter{Name:\"Replace_Dots\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"on\"}, v1alpha1.Parameter{Name:\"Retry_Limit\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"False\"}, v1alpha1.Parameter{Name:\"Type\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"flb_type\"}, v1alpha1.Parameter{Name:\"Time_Key\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"@timestamp\"}, v1alpha1.Parameter{Name:\"Logstash_Prefix\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"ks-logstash-log\"}}}}, Settings:[]v1alpha1.Plugin{v1alpha1.Plugin{Type:\"fluentbit_settings\", Name:\"fluentbit-settings\", Parameters:[]v1alpha1.Parameter{v1alpha1.Parameter{Name:\"Enable\", ValueFrom:(*v1alpha1.ValueFrom)(nil), Value:\"true\"}}}}}, Status:v1alpha1.FluentBitStatus{}}"
time="2020-02-04T09:51:49Z" level=info msg="Generating configuration."
time="2020-02-04T09:51:49Z" level=info msg="Applying service"
time="2020-02-04T09:51:49Z" level=info msg="Applying input"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying filter"
time="2020-02-04T09:51:49Z" level=info msg="Applying output"
time="2020-02-04T09:51:49Z" level=info msg="Applying settings"
time="2020-02-04T09:51:49Z" level=error msg="clusterrolebindings.rbac.authorization.k8s.io \"fluentbit\" already exists"
time="2020-02-04T09:51:49Z" level=error msg="<nil>"
time="2020-02-04T09:51:49Z" level=error msg="configmaps \"fluent-bit-config\" already exists"
time="2020-02-04T09:51:49Z" level=error msg="<nil>"
time="2020-02-04T09:51:49Z" level=info msg="Fluent-bit deployed successfully"

sh-4.2$ curl elasticsearch-logging-data.kubesphere-logging-system.svc:9200/_cat/indices

green open ks-logstash-log-2020.02.04 45sRQisQREaFRcyrqyqqKw 5 1 25832943 0  3.4gb  1.8gb
green open ks-logstash-log-2020.01.22 u093E-G7RfiHdNknoUDnjw 5 1    14444 0 10.9mb  5.4mb
green open ks-logstash-log-2020.01.18 TGfKUprOQSOCpWwg_iFBJQ 5 1    14978 0 11.4mb  5.7mb
green open ks-logstash-log-2020.01.12 3a574FRxSFmf707tXb5kGw 5 1    45968 0 37.2mb 18.7mb
green open ks-logstash-log-2020.02.02 BY_QG50nTwykyciiVv75Gg 5 1    39035 0 23.3mb 11.7mb
green open ks-logstash-log-2020.01.13 7nezOIOjQLGS0DP61g-ypg 5 1    29889 0 19.8mb  9.9mb
green open ks-logstash-log-2020.01.15 TSCqeIW5Q16dWs6q5Fv8iA 5 1    17052 0 13.4mb  6.7mb
green open .kibana_1                  IqsQ0K-RQEa3EKOkesBALA 1 1        0 0   522b   261b
green open ks-logstash-log-2020.01.28 s1vFQFdaSDGfByTRhawq4A 5 1    14350 0 10.8mb  5.3mb
green open ks-logstash-log-2020.01.31 wtUX1BUbSx-Pj5hGFSlyQA 5 1    16729 0 13.1mb  6.6mb
green open ks-logstash-log-2020.01.25 73UCAQlQThWVETVxfDA0wQ 5 1    14846 0 11.4mb  5.7mb
green open ks-logstash-log-2020.01.27 83GWCusaQYKuSesNZl8g9A 5 1    14574 0   11mb  5.5mb
green open ks-logstash-log-2020.01.21 5lOinjmPSiWAeXF6L12acg 5 1    15136 0 11.7mb  5.8mb
green open ks-logstash-log-2020.01.29 Egjh2L93TeqzAsmFEtfXdg 5 1    14580 0   11mb  5.5mb
green open ks-logstash-log-2020.01.26 aqhFiiANQHq2vXvYwGt0OQ 5 1    14849 0 11.2mb  5.6mb
green open ks-logstash-log-2020.02.03 UBiYNCoRSVS5BGo3LrHgtA 5 1    58727 0 37.5mb 18.7mb
green open ks-logstash-log-2020.01.14 aY_kNkNCSI6TQ1FTFNgMLw 5 1    25224 0 16.7mb  8.3mb
green open ks-logstash-log-2020.01.19 BVU4EFGESEOYWv8riKn7lw 5 1    17945 0 14.6mb  7.2mb
green open ks-logstash-log-2020.01.17 qK-gMi-1Tg2V0-XJP4-r5g 5 1    16328 0 12.7mb  6.3mb
green open ks-logstash-log-2020.01.16 5hTTDmybQniQamtK6qBkPw 5 1    14481 0 10.8mb  5.4mb
green open ks-logstash-log-2020.01.20 dCyGdCSOQxy6MUloou84kQ 5 1    14202 0 10.7mb  5.3mb
green open ks-logstash-log-2020.02.01 KTEiRx1XSte23gsAtWBOiA 5 1    19286 0 15.4mb  7.7mb
green open ks-logstash-log-2020.01.23 bGXmhyctTsiTAzsclb1yzQ 5 1    14418 0 10.7mb  5.4mb
green open ks-logstash-log-2020.01.30 P1q2roAqSFuVrGui5lLttg 5 1    16578 0 13.1mb  6.5mb
green open ks-logstash-log-2020.01.24 c1W26dJdTFi7aizboiNYYA 5 1    14461 0 10.7mb  5.4mb

huanggze · 2020年2月11日

rysinal 看一下你的 fluent-bit 镜像版本

rysinal · 2020年2月11日

huanggze KubeSphere 版本 : v2.1.0

      containers:
        - name: fluent-bit
          image: 'kubesphere/fluent-bit:v1.3.2-reload'

huanggze · 2020年2月12日

rysinal

换成 kubespheredev/fluent-bit:v1.3.5-reload 试试，这是fluent bit 的bug，会导致container name hash conflict 使日志元信息丢失

rysinal · 2020年2月12日

huanggze
kubesphere命名空间下找不到1.3.5的镜像

docker pull kubesphere/fluent-bit:v1.3.5-reload
Error response from daemon: manifest for kubesphere/fluent-bit:v1.3.5-reload not found

是否可以修改成官方的 fluent/fluent-bit:1.3.5

docker pull fluent/fluent-bit:1.3.5

huanggze · 2020年2月12日

rysinal

kubespheredev dev dev dev
kubespheredev
kubespheredev

kubespheredev/fluent-bit:v1.3.5-reload

这周发布2.1.1 才会把 dev 去掉

huanggze · 2020年2月12日

rysinal 官方的没有reload功能，不行

rysinal · 2020年2月12日

huanggze 眼拙了。。

已更新了，但是还是一样的问题，业务容器重新部署也没有产生日志，可否远程帮忙调试下，感谢
管理员端（系统容器日志）：

业务账号端（业务容器）：

calvinyu · 2020年2月13日

你们业务应用的日志是写到容器标准输出了吗？还是写到文件里面了？

rysinal · 2020年2月13日

calvinyu 标准输出有的，通过docker logs –tail 10 1564464dcc48 都可以看到业务日志的请求log

huanggze · 2020年2月13日

rysinal

可否把远程连接方式发邮件到 kubesphere@yunify.com ，我们帮你远程看看

我不清楚我在家能不能连上tv，不行微信沟通也行

rysinal · 2020年2月14日

huanggze 已发送微信，谢谢大佬

huanggze · 2020年2月14日

rysinal 好的，我看看

rysinal · 2020年2月14日

huanggze 感谢大佬指点，目前已经解决
问题原因：
master节点设置的软链为：/var/lib/docker -> /data/docker
worker节点设置的软链为：/var/lib/docker -> /work/docker
fluent-bit设置的path为/data/docker
但是业务pod全部分布在worker节点，所以也解释通了为啥只有系统级别（master节点pod）的日志，丢失了业务日志

解决步骤：
尝试在worker节点做软链：/var/lib/docker->/data/docker ->/work/docker 但是重启后仍然没有生效,用以下命令验证

realpath $(docker info -f '{{.DockerRootDir}}') 
/work/docker

所以软链方式不行，只能迁移。

1、systemctl stop docker
2、迁移/work/docker 至 /data/docker
3、确认软链与master节点一致： /var/lib/docker->/data/docker
4、systemctl start docker

打开ks，右下角日志查看，业务日志已收集成功。

再次感谢@huanggze 提供清晰思路

huanggze · 2020年7月17日

补充一个相同问题环境的解决办法：

第一次安装时，把docker放在了扩展盘上，然后做了符号连接到 /var/lib/docker
因此 kubesphere管理程序将/data/docker作为docker的目录传给了日志收集程序 fluentbit
但新加的node未做符号连接（因为没有扩盘），docker的目录就是 /var/lib/docker
因此在新加的节点上 fluentbit 通过 /data/docker这目录去收集日志时，发现是空的，那个目录估计是部署程序生成的，里面就放了一个空的container程序
导致容器中能看到日志，kubesphere WEB页面上通过工作负载搜索不到日志。
修改方法：
kubectl cordon node-xxx
kubectl drain node-xxx –delete-local-data –force -ignore-daemonsets
systemctl stop docker.service

rm -fr /data/docker
mv /var/lib/docker /data/docker
ln -s /data/docker /var/lib/docker

systemctl start dokcer.service
kubectl uncordon node-xxx

检查验证命令

systemctl status docker.service ==>应该是running
kubectl describe node node-xxx
realpath $(docker info -f ‘{{.DockerRootDir}}’ ==> 应该显示 /data/docker

控制台日志查询功能无法打印日志

huanggzeK零SK壹S

rysinalK零S

huanggzeK零SK壹S

rysinalK零S

huanggzeK零SK壹S

rysinalK零S

huanggzeK零SK壹S

huanggzeK零SK壹S

rysinalK零S

calvinyuK零S

rysinalK零S

huanggzeK零SK壹S

rysinalK零S

huanggzeK零SK壹S

rysinalK零S

huanggzeK零SK壹S