huanggze 我现在起了4个data节点，1个master节点，timeout这种报错没了，但是还是前端还是没显示log。看es集群的状态，也没定位出原因来~

      able7

      fluent bit 报错 unavailable_shards_exception。是shard 的有问题？

      curl localhost:9200/_cat/shards

          able7 是的。因为 -2020.03.25 今天的 index 没创建成功

            able7 明天约个时间帮你远程看下？什么时候方便呢？

                再次感谢kubesphere@yunify.com 大佬支持。在此整理下排错思路。
                1、首先根据以下issue查看相关配置是否有误，比如说日志的路径需要根据实际情况修改；我这边就是/data1/docker/containers这个路径：
                https://github.com/kubesphere/kubesphere/issues/680
                2、查看es集群的状态是否正常，符合预期。
                由于我这边k8s内服务非常多，产生的日志量也非常大，1 master/2 data节点部署方式，确实撑不住，一堆GC报错，后面调整成1master/5 data节点，并且修改es集群的相关jvm参数后，集群状态是正常的

                curl http://10.106.215.141:9200/_cluster/health?pretty=true
curl http://10.106.215.141:9200/_cat/indices
curl http://10.106.215.141:9200/_cat/shards
curl http://10.106.215.141:9200/ks-logstash-log-2020.03.27/_search

                通过以上命令我简单查看集群是否ok

                3、我这边确保以上2个问题都得到解决的情况下，前端仍然显示不了日志，然后去查看api的问题：
                因为kubesphere ks-apiserver只是封装es的API接口，我们进入ks-apiserver里面查看
                apk add curl
                curl localhost:9090/kapis/logging.kubesphere.io/v1alpha2/cluster
                接口返回是空的。

                4、后面让开发去单步调试代码，发现开发合ks-apiserver代码有些问题，我们这边针对kubesphere做了一些定制开发，代码合并出现问题，导致ks-apiserver出了问题，现在已解决。

                tips: kubesphere-config cm里面关于logging的配置，也需要仔细检查，不能出现配置错误。
                以上就是排查过程。感谢kubesphere@yunify.com支持

                1 年 后

                huanggze Hi我最近新安装的v3.1.1的，docker路径也是放在/data盘，按照你们说的方法试了一篇还是不行，而且新版的kubersphere和fluent-operator配置上应该有些差异，麻烦帮忙看下!

                fluent-bit日志：

                inputs crd手动改和docker路径（/data/lib/docker/containers）对应的配置：

                fluent-bit挂载的路径

                • zwz 回复了此帖
                    6 个月 后

                    V3.2.1 也遇到同样问题，es日志显示页面为空

                    fluentbit-operator日志

                    [root@Ops-repo ~]# kubectl -n kubesphere-logging-system logs -f fluentbit-operator-745bf5559f-wxrb4
I0308 03:08:37.589870       1 request.go:668] Waited for 1.00297623s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s
2022-03-08T03:08:38.696Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2022-03-08T03:08:38.697Z	INFO	setup	starting manager
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbitconfig	Starting Controller	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig"}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2022-03-08T03:08:39.099Z	INFO	controller-runtime.manager.controller.fluentbit	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBit", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.100Z	INFO	controller-runtime.manager.controller.fluentbit	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBit", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.100Z	INFO	controller-runtime.manager.controller.fluentbit	Starting EventSource	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBit", "source": "kind source: /, Kind="}
2022-03-08T03:08:39.100Z	INFO	controller-runtime.manager.controller.fluentbit	Starting Controller	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBit"}
2022-03-08T03:08:42.497Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "FluentBit.logging.kubesphere.io", "error": "no matches for kind \"FluentBit\" in version \"logging.kubesphere.io/v1alpha2\""}
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/source/source.go:127
2022-03-08T03:08:42.497Z	ERROR	controller-runtime.manager.controller.fluentbit	Could not wait for Cache to sync	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBit", "error": "failed to wait for fluentbit caches to sync: no matches for kind \"FluentBit\" in version \"logging.kubesphere.io/v1alpha2\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:195
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/manager/internal.go:696
2022-03-08T03:08:42.586Z	ERROR	controller-runtime.manager.controller.fluentbitconfig	Could not wait for Cache to sync	{"reconciler group": "logging.kubesphere.io", "reconciler kind": "FluentBitConfig", "error": "failed to wait for fluentbitconfig caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:195
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/manager/internal.go:696
2022-03-08T03:08:42.586Z	ERROR	controller-runtime.manager	error received after stop sequence was engaged	{"error": "failed to wait for fluentbitconfig caches to sync: timed out waiting for cache to be synced"}
2022-03-08T03:08:42.586Z	ERROR	setup	problem running manager	{"error": "failed to wait for fluentbit caches to sync: no matches for kind \"FluentBit\" in version \"logging.kubesphere.io/v1alpha2\""}
main.main
	/workspace/main.go:114
runtime.main
	/usr/local/go/src/runtime/proc.go:225
[root@Ops-repo ~]#