- 已编辑
创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。
操作系统信息
阿里云K8S 虚拟机,Centos7.5,8C/32G
Kubernetes版本信息
将 kubectl version
命令执行结果贴在下方
Client Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.4-aliyun.1”, GitCommit:“40ad569”, GitTreeState:"", BuildDate:“2021-06-28T04:01:44Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.4-aliyun.1”, GitCommit:“ed12730”, GitTreeState:"", BuildDate:“2021-11-09T01:50:36Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}
容器运行时
将 docker version
/ crictl version
/ nerdctl version
结果贴在下方
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 99c5edce
Built: Thu Jan 9 08:27:14 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 99c5edce
Built: Thu Jan 9 08:26:30 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
KubeSphere版本信息
v3.2.0。离线安装还是在线安装。在已有K8s上安装
问题是什么
其它日志,审计,事件查询都正常。只有链路追踪报错。
部署了示例应用,在链路追踪中每刷新一次,jaeger-query pod日志报一次如下错误:
{"level":"error","ts":1638454705.9736323,"caller":"app/http_handler.go:487","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:487\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:236\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/go-stdlib@v1.0.0/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2878\nnet/http.(*conn).serve\n\tnet/http/server.go:1929"}
查询时elastic日志报错
elastic里所有index状态
elastic里logstash-jaeger-span 索引的状态