• 微服务
  • Jaeger 日志报 all shards failed [type=search_phase_execution_exception]"

创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。

操作系统信息
阿里云K8S 虚拟机,Centos7.5,8C/32G

Kubernetes版本信息
kubectl version 命令执行结果贴在下方

Client Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.4-aliyun.1”, GitCommit:“40ad569”, GitTreeState:"", BuildDate:“2021-06-28T04:01:44Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

Server Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.4-aliyun.1”, GitCommit:“ed12730”, GitTreeState:"", BuildDate:“2021-11-09T01:50:36Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

容器运行时
docker version / crictl version / nerdctl version 结果贴在下方

Client: Docker Engine - Community

Version: 19.03.5

API version: 1.40

Go version: go1.12.12

Git commit: 99c5edce

Built: Thu Jan 9 08:27:14 2020

OS/Arch: linux/amd64

Experimental: false

Server: Docker Engine - Community

Engine:

Version: 19.03.5

API version: 1.40 (minimum version 1.12)

Go version: go1.12.12

Git commit: 99c5edce

Built: Thu Jan 9 08:26:30 2020

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: 1.2.10

GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339

runc:

Version: 1.0.0-rc8+dev

GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657

docker-init:

Version: 0.18.0

GitCommit: fec3683

KubeSphere版本信息
v3.2.0。离线安装还是在线安装。在已有K8s上安装

问题是什么

其它日志,审计,事件查询都正常。只有链路追踪报错。

部署了示例应用,在链路追踪中每刷新一次,jaeger-query pod日志报一次如下错误:
{"level":"error","ts":1638454705.9736323,"caller":"app/http_handler.go:487","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:487\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:236\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/go-stdlib@v1.0.0/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2878\nnet/http.(*conn).serve\n\tnet/http/server.go:1929"}

查询时elastic日志报错

elastic里所有index状态

elastic里logstash-jaeger-span 索引的状态

13 天 后
2 个月 后
8 个月 后