使用外部ES，日志系统能查询全量或时间范围，但条件过滤都查不出来

hwyz_leo

外部ES日志正常写入，通过kibana查询正常

KS的日志查询也能看到日志数，但是容器数是0

今日总共收录了 0 个容器 9644 条日志信息

通过时间范围能看到日志列表

但其他条件过滤都查不到内容，点击日志详情也看不到内容

外部ES 7.10.1，6.8.2，6.4.3版本都是这样的情况

请问这会是什么原因？

hwyz_leo

又安装了事件组件，写入和查询一切正常，所以是日志查询组件有BUG？

wanjunlei

hwyz_leo 把你用kibana查询的数据贴出来看看，看下namespace，pod等信息是不是完整的

DehaoCheng

看一下ks-apiserver有没有什么报错信息。

hwyz_leo

DehaoCheng

W1124 17:10:46.140402 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

W1124 17:10:46.141711 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

W1124 17:10:46.158663 1 options.go:183] ks-apiserver starts without redis provided, it will use in memory cache. This may cause inconsistencies when running ks-apiserver with multiple replicas.

I1124 17:10:46.333079 1 interface.go:60] start helm repo informer

I1124 17:10:46.358537 1 apiserver.go:356] Start cache objects

I1124 17:10:47.226030 1 apiserver.go:562] Finished caching objects

I1124 17:10:47.226067 1 apiserver.go:278] Start listening on :9090

E1124 17:10:59.915358 1 v2.go:105] websocket: close 1005 (no status)

W1124 17:11:07.921764 1 terminal.go:137] 1Process exited

E1124 17:11:59.696526 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

W1124 17:12:07.702882 1 terminal.go:137] 1Process exited

W1124 17:32:25.148602 1 handler.go:86] websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

E1124 17:33:18.151165 1 v2.go:105] websocket: close 1001 (going away)

W1124 17:33:26.157555 1 terminal.go:137] 1Process exited

E1124 17:34:03.210388 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

W1124 17:34:11.216385 1 terminal.go:137] 1Process exited

W1124 17:35:10.261638 1 handler.go:86] websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

E1124 17:35:24.761455 1 v2.go:105] websocket: close 1001 (going away)

W1124 17:35:28.703106 1 handler.go:86] websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

W1124 17:35:32.768058 1 terminal.go:137] 1Process exited

E1124 17:35:36.746813 1 v2.go:105] websocket: close 1001 (going away)

W1124 17:35:44.753288 1 terminal.go:137] 1Process exited

E1124 17:36:03.245315 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

E1124 17:36:03.245834 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

W1124 17:36:11.251663 1 terminal.go:137] 1Process exited

W1124 17:36:11.255816 1 terminal.go:137] 1Process exited

W1124 17:36:36.334462 1 handler.go:86] websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

E1124 17:36:43.072035 1 v2.go:105] websocket: close 1005 (no status)

W1124 17:36:51.078540 1 terminal.go:137] 1Process exited

E1124 17:38:04.064205 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

W1124 17:38:12.070408 1 terminal.go:137] 1Process exited

W1124 17:40:44.724007 1 handler.go:86] websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

E1124 17:40:51.801038 1 v2.go:105] websocket: close 1005 (no status)

W1124 17:40:59.806946 1 terminal.go:137] 1Process exited

E1124 17:42:05.187733 1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF

W1124 17:42:13.194287 1 terminal.go:137] 1Process exited

这些错误貌似请问该怎么分析？

hwyz_leo

wanjunlei

{

"_index": "ks-logstash-log-2021.11.24",

"_type": "_doc",

"_id": "feeeb3da-d1e1-951d-b6e8-152a29ad576c",

"_version": 1,

"_score": null,

"_source": {

"@timestamp": "2021-11-24T09:47:24.429Z",

"log": "2021-11-24T17:47:24+08:00\t\u001b[34minfo\u001b[0m\tingress/controller.go:542\tservice kube-system/tke-eni-ipamd-lock not found\n",

"time": "2021-11-24T09:47:24.429575123Z",

"kubernetes": {

"pod_name": "apisix-ingress-controller-744b748476-gk66s",

"namespace_name": "ingress-apisix",

"container_name": "ingress-controller",

"docker_id": "7d6775f53f7ebb012a2737e4c988e581b46b1539517210035ba6de2e4d052234",

"container_image": "apache/apisix-ingress-controller:1.3.0"

}

},

"fields": {

"@timestamp": [

"2021-11-24T09:47:24.429Z"

],

"time": [

"2021-11-24T09:47:24.429Z"

]

},

"sort": [

1637747244429

]

}

麻烦看下，这个数据有问题吗？我看着貌似都有

wanjunlei

kubectl -n kubesphere-system get cm kubesphere-config -o yaml

确认下kubesphere-system下的cm kubesphere-config中的logging配置是不是正确的

hwyz_leo

wanjunlei

logging:

host: http://10.21.11.7:9200

basicAuth: True

username: "elastic"

password: "***************"

indexPrefix: ks-logstash-log

events:

host: http://10.21.11.7:9200

basicAuth: True

username: "elastic"

password: "*****************"

indexPrefix: ks-logstash-events

log和event的部分，也挺正常的

hwyz_leo

还有人碰到类似的问题吗？

wanjunlei

1.使用内置es是否是正常的？

2.请确认是否是按照官方文档切换的外置es

3.切换外部es后，请确认es中收集的日志是否是实时的。

4.确认日志查询和日志收集是否是使用的相同的配置，可以比较下kubesphere-system下的configmap kubesphere-config 中logging的部分和kubesphere-logging-system下的secret fluent-bit-conf 中的 fluent-bit.conf 中的相关配置

5.尝试重启kubesphere-logging-system 下的daemonset fluent-bit, 并查看是否有异常日志

6.尝试重启kubesphere-system下deployment ks-apiserver，并查看是否有异常日志

7.打开console控制台，确认是未查询到日志，还是查询请求未返回

如果以上操作不能解决你的问题，建议使用kubesphere的工单服务获取进一步的帮助https://kubesphere.cloud/store/service/

hwyz_leo

首先感谢你的回复 😀

1.使用内置es是否是正常的？

正常

2.请确认是否是按照官方文档切换的外置es

是官方文档，就是开启日志功能并配置了外置ES地址

3.切换外部es后，请确认es中收集的日志是否是实时的。

实时的，请求了差不多就能在KIBANA查到

4.确认日志查询和日志收集是否是使用的相同的配置

一致

5.尝试重启kubesphere-logging-system 下的daemonset fluent-bit, 并查看是否有异常日志

没有ERROR，不过会报一个WARNING，每秒会有几次

[http_client] cannot increase buffer: current=512000 requested=544768 max=512000

6.尝试重启kubesphere-system下deployment ks-apiserver，并查看是否有异常日志

看着没啥异常日志

7.打开console控制台，确认是未查询到日志，还是查询请求未返回

在控制台里请问如何确认？而且在日志查询里也不是完全查不到，我尝试查一个日志的全量内容，可以查到对应，比如

I1202 08:19:29.286265 1 leaderelection.go:273] successfully renewed lease kube-system/external-attacher-leader-com-tencent-cloud-csi-cbs

但是我查其中的部分 successfully 或者 * successfully 星号，精确还是模糊，都查不到

但是我查 * ，倒是能查到全是星号的日志，感觉有点奇怪

其他条件选项比如：项目、容器组啥的按实际输也都查不到

poo0054

我现在也是碰见一样的问题，使用内部opensearch在kubesphere日志页面是能够正常显示的，使用了外部的es。我的kubesphere日志功能就无法使用。

今日总共收录了 0 个容器 0 条日志信息

日志和事件都是一样的。在外部es中是能够正常显示出来的。

loki

poo0054

我遇到了同样的问题，外部的 ES 中有数据，通过 kubesphere 的日志页面显示为0，经过检查发现是配置字典 kubesphere-config 中的 indexPrefix 和 output 中的 logstashPrefix 配置不一致导致的

poo0054

我该怎么确定错误

poo0054

kubectl logs -n kubesphere-system ks-installer-6997cdbc7c-lmrsh 日志：


PLAY RECAP *********************************************************************
localhost                  : ok=25   changed=19   unreachable=0    failed=0    skipped=22   rescued=0    ignored=0   
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
Start installing devops
Start installing events
Start installing logging
**************************************************
Waiting for all tasks to be completed ...
task network status is successful  (1/7)
task openpitrix status is successful  (2/7)
task multicluster status is successful  (3/7)
task logging status is successful  (4/7)
task events status is successful  (5/7)
task devops status is successful  (6/7)
task monitoring status is successful  (7/7)
**************************************************
Collecting installation results ...
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.98.11:30880
Account: admin
Password: P@88w0rd
NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     "Cluster Management". If any service is not
     ready, please wait patiently until all components 
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             2023-11-25 12:24:50
#####################################################

ks-apiserver-7b8787c596-6d9j6日志：

[root@k8s-01 data]# kubectl logs  -n kubesphere-system ks-apiserver-7b8787c596-6d9j6  -f
W1125 12:11:19.638628       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1125 12:11:19.642872       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1125 12:11:19.672292       1 metricsserver.go:238] Metrics API not available.
W1125 12:11:19.672384       1 cache.go:64] In-memory cache will be used, this may cause data inconsistencies when running with multiple replicas.
I1125 12:11:19.672535       1 interface.go:50] start helm repo informer
I1125 12:11:19.998998       1 apiserver.go:428] Start cache objects
I1125 12:11:22.731288       1 apiserver.go:619] Finished caching objects
I1125 12:11:22.731344       1 apiserver.go:297] Start listening on :9090
W1125 12:11:23.044282       1 jwt.go:53] token not found in cache
W1125 12:11:23.078451       1 jwt.go:53] token not found in cache
W1125 12:11:23.078714       1 jwt.go:53] token not found in cache
W1125 12:11:23.078452       1 jwt.go:53] token not found in cache
W1125 12:11:23.079445       1 jwt.go:53] token not found in cache

Scarlet-Bell

我升级3.4.0后也是这样的问题 es有日志但是ks的日志查询是空的。而且升级后强制给我开启了opensearch的日志收集，后面尝试关闭es用opensearch发现并不会给我自动部署opensearch的服务，看ks-installer的日志是到opensearch部分直接skip了不知道原因。之后把event和audit模块还有日志收集配置按照官网步骤清理最后删除kubesphere-logging-system的 namespace准备重建日志这块，结果清理完在开启日志收集发现很多crd都没给我自动创建直接整的ks-apiserver服务都起不来了，给我整无语了，还好是测试环境。前后在社区发了两个问题贴都没人处理，搞的我现在生产环境都不敢升级3.4.1了。

poo0054

@loki 太感谢了，现在解决了。logging和事件的都是这里改一下就行了。