服务组件之间调用异常

caixuhui

服务框架是springboot ，使用feign发起远程连接，大部分是正常的，有几个服务调用的时候会报错
no healthy upstream

并且在服务的istio-proxy内可以看到grpc的错误日志
`
2021-01-13T03:23:06.716196Z info PilotSAN []string(nil)

2021-01-13T03:23:06.716230Z info Starting proxy agent

2021-01-13T03:23:06.716266Z info Opening status port 15020

2021-01-13T03:23:06.717994Z info Received new config, creating new Envoy epoch 0

2021-01-13T03:23:06.718692Z info Epoch 0 starting

2021-01-13T03:23:06.718734Z info watching /etc/certs for changes

2021-01-13T03:23:06.723321Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json –restart-epoch 0 –drain-time-s 45 –parent-shutdown-time-s 60 –service-cluster flight-schedule.acdm-mesh –service-node sidecar₁₀.233.112.15_flight-schedule-v1-86cd9c4997-xj627.acdm-mesh_acdm-mesh.svc.cluster.local –max-obj-name-len 189 –local-address-ip-version v4 –log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning –component-log-level misc:error –concurrency 2]

[Envoy (Epoch 0)] [2021-01-13 03:23:06.758][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream

2021-01-13T03:23:08.788792Z info Envoy proxy is ready

[Envoy (Epoch 0)] [2021-01-13 03:30:24.741][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 04:00:43.434][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 04:31:54.473][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 05:03:54.071][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 05:36:30.015][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 06:09:02.489][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 06:37:29.538][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 07:07:21.557][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 07:40:30.195][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 08:07:50.242][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 08:35:52.648][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 09:06:11.251][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

[Envoy (Epoch 0)] [2021-01-13 09:06:12.273][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:06:13.749][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:06:15.617][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:06:21.479][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:06:24.441][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:06:49.393][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:07:06.377][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:07:30.821][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

[Envoy (Epoch 0)] [2021-01-13 09:07:47.741][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
`

zackzhang

caixuhui istio的grpc连接每隔一断时间就会关闭一次，该连接仅仅影响的时控制面的配置下发，不会对服务造成访问故障，因此这个日志是warning级别，这个日志不能说明服务有问题

istio/istio#17383

caixuhui

zackzhang 你好，感谢回复。那前面的问题，那个 no healthy upstream是什么的问题呢？我原本服务没注入sidecar可以正常请求，注入sidecar后就出现了这个问题

zackzhang

caixuhui 你的意思是没注入sidecar可以正常请求，注入sidecar后不能正常请求吗？

caixuhui

zackzhang 是的，原先容器里只有一个xxx服务就可以访问，然后注入后多了istio-proxy 和istio-init就无法访问。重启了服务，在istio-proxy的启动日志有个异常：
[Envoy (Epoch 0)] [2021-01-14 01:13:45.109][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
这个也是no healthy upstream

zackzhang

caixuhui 要看你自己容器的日志有没有报错，应用里面开下debug，或是抓下包，看下请求有没有过来。

caixuhui

zackzhang 找到问题了解决了，是因为应用里的服务组件不是通过点击界面上的【添加组件】来的。
原来istio识别其他服务是从app的配置里面读取service列表的？？我使用流水线创建的service和deploy关联到app上，app的配置内没有更新指定的配置。

zackzhang

caixuhui 你是springboot的微服务，按社区提供的方法，给相应的资源打label及注入sidecar，改造成KubeSphere的微服务的是吧？