• 微服务
  • istio-system项目jaeger相关组件启动失败

创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。
发帖前请点击 发表主题 右边的 预览(👀) 按钮,确保帖子格式正确。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。

操作系统信息
物理机,Centos7 ,48C/64G

Kubernetes版本信息
v1.23.10

容器运行时
Docker version 20.10.8, build 3967b7d

KubeSphere版本信息
3.4.0

问题是什么

jaeger相关组件报错,提示未发现es节点, 我们用的是opensearch而不是es

上图中三个容器的报错日志分别为

jaeger-query

 2023/08/29 06:06:57 maxprocs: Leaving GOMAXPROCS=48: CPU quota undefined
 {"level":"info","ts":1693289217.7449164,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
 {"level":"info","ts":1693289217.7450898,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
 {"level":"info","ts":1693289217.7454255,"caller":"flags/admin.go:104","msg":"Mounting health check on admin server","route":"/"}
 {"level":"info","ts":1693289217.745627,"caller":"flags/admin.go:115","msg":"Starting admin HTTP server","http-addr":":16687"}
 {"level":"info","ts":1693289217.7456648,"caller":"flags/admin.go:96","msg":"Admin server started","http.host-port":"[::]:16687","health-status":"unavailable"}
 {"level":"fatal","ts":1693289222.8173027,"caller":"./main.go:101","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available","stacktrace":"main.main.func1\n\t./main.go:101\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\t./main.go:161\nruntime.main\n\truntime/proc.go:255"}

jaeger-collector

2023/08/29 06:09:47 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
 {"level":"info","ts":1693289387.568935,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
 {"level":"info","ts":1693289387.5690646,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
 {"level":"info","ts":1693289387.5693471,"caller":"flags/admin.go:104","msg":"Mounting health check on admin server","route":"/"}
 {"level":"info","ts":1693289387.5694728,"caller":"flags/admin.go:115","msg":"Starting admin HTTP server","http-addr":":14269"}
 {"level":"info","ts":1693289387.5695019,"caller":"flags/admin.go:96","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
 {"level":"fatal","ts":1693289393.30059,"caller":"./main.go:80","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available","stacktrace":"main.main.func1\n\t./main.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

jaeger-collector-headless

 2023/08/29 06:09:47 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
 {"level":"info","ts":1693289387.568935,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
 {"level":"info","ts":1693289387.5690646,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
 {"level":"info","ts":1693289387.5693471,"caller":"flags/admin.go:104","msg":"Mounting health check on admin server","route":"/"}
 {"level":"info","ts":1693289387.5694728,"caller":"flags/admin.go:115","msg":"Starting admin HTTP server","http-addr":":14269"}
 {"level":"info","ts":1693289387.5695019,"caller":"flags/admin.go:96","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
 {"level":"fatal","ts":1693289393.30059,"caller":"./main.go:80","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available","stacktrace":"main.main.func1\n\t./main.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

ks-install yaml中,我们启用了openearch,没用es

请问这个问题的该从哪方面进行排查呀

已okay

查看jaeger中的es配置是否正确

kubectl -n istio-system edit jaeger

之前是

    options:
      es:
        index-prefix: k8s
        server-urls: https://10.8.0.102:9200 #我们的es地址走的是http,不知道为啥生成时变成https了
        tls:
          enabled: true
          skip-host-verify: true
        version: 7
    secretName: jaeger-secret
    type: elasticsearch

修改后

    options:
      es:
        index-prefix: k8s
        server-urls: http://10.8.0.102:9200 #改回http
        tls:
          enabled: false #关闭tls验证
          skip-host-verify: true
        version: 7
    secretName: jaeger-secret
    type: elasticsearch

恢复正常了

3 个月 后

我和你同样的问题,但是我还没有解决。

enabled: false 改过了,url 也改成了http 还是不行。

同时我在容器里使用curl 请求过了,那个地址是可以请求通的。

jaeger-collector 报错启动不了,报错信息:

{

"level": "fatal",

"ts": 1701762571.736633,

"caller": "./main.go:84",

"msg": "Failed to create span writer",

"error": "elastic: Error 503 (Service Unavailable)",

"stacktrace": "main.main.func1\\n\\t./main.go:84\\ngithub.com/spf13/cobra.(\*Command).execute\\n\\tgithub.com/spf13/cobra@v1.2.1/command.go:856\\ngithub.com/spf13/cobra.(\*Command).ExecuteC\\n\\tgithub.com/spf13/cobra@v1.2.1/command.go:974\\ngithub.com/spf13/cobra.(\*Command).Execute\\n\\tgithub.com/spf13/cobra@v1.2.1/command.go:902\\nmain.main\\n\\t./main.go:147\\nruntime.main\\n\\truntime/proc.go:255"

}