部署Bookinfo实例应用后,pod内没有istio-proxy容器。
请问kubesphere如何实现的istio-proxy Injection。

查看istio官网实例,kubectl label namespace default istio-injection=enabled可以启用namespace级别的自动注入。
查看ks创建的project ns没有此lable。

麻烦问下,如何定位我的bookinfo没有注入的问题,多谢。

wnxn 如果想了解一下ks里的怎么和istio做交互的,需要看哪个模块的实现?

你可以看下CRD,kubesphere就是抽象了一个CRD对象,界面创建的都是自定义的资源,在kubesphere里再转换成istio的对象

2 个月 后

istio 自动注入实现方式为:在创建 pod 时,会触发 webhook,修改这个 Pod 的 spec,先创建 istio-init/istio-proxy 两个容器,进行调用 iptables 进行流量拦截,实现自动注入。

webhook 的触发受三个参数的控制 ns lable/default policy/anntations,详见官网

namespaceSelector matchdefault policyPod override annotation sidecar.istio.io/injectSidecar injected?
yesenabledtrue (default)yes
yesenabledfalseno
yesdisabledtrueyes
yesdisabledfalse (default)no
noenabledtrue (default)no
noenabledfalseno
nodisabledtrueno
nodisabledfalse (default)no

下面按照这三个参数一一列举说明:

  1. 对于 ns,三个ns(istio-system/kube-system/kubesphere-system)关闭了自动注入。其余的只要 not in disabled 都会匹配上。包含你说的ns 为 istio-injection=enabled情况。
# ks-installer/roles/ks-istio/tasks/main.yaml
# label istio-system with istio-injection=disabled to avoid sidecarInjector mutatingwebhookconfigurations block pod creation
- name: istio | disable istio-injection
  shell: >
    {{ bin_dir }}/kubectl label ns {{ item }} istio-injection=disabled
  register: import
  failed_when: "import.stderr and 'already' not in import.stderr"
  loop:
    - istio-system
    - kube-system
    - kubesphere-system


# ks-installer/roles/ks-istio/templates/custom-values-istio.yaml.j2
sidecarInjectorWebhook:
  enabled: true
  enableNamespacesByDefault: true
  image: {{ istio_sidecar_injector_image }}

# ks-installer/roles/ks-istio/files/istio/istio/charts/sidecarInjectorWebhook/templates/mutatingwebhook.yaml
{{- if .Values.enableNamespacesByDefault }}
      matchExpressions:
      - key: name
        operator: NotIn
        values:
        - {{ .Release.Namespace }}
      - key: istio-injection
        operator: NotIn
        values:
        - disabled
{{- else }}
      matchLabels:
        istio-injection: enabled
{{- end }}
{{- end }}

2. 从 configMap看 policy 默认值, 可以看到 default policy 为 false:

root@ks-allinone:/root # kubectl get cm/istio-sidecar-injector -n istio-system -o yaml|less
apiVersion: v1
data:
  config: |-
    policy: disabled
    alwaysInjectSelector:
      []
    neverInjectSelector:
      []
      ...  
  1. 我们的 annotaions 的默认值也是 false

annotation默认值

#ks-istio/files/istio/istio/charts/sidecarInjectorWebhook/templates/deployment.yaml
  template:
    metadata:
      labels:
        app: {{ template "sidecar-injector.name" . }}
        chart: {{ template "sidecar-injector.chart" . }}
        heritage: {{ .Release.Service }}
        release: {{ .Release.Name }}
        istio: sidecar-injector
      annotations:
        sidecar.istio.io/inject: "false"
  1. 因此如果要自动注入,则必须显性指明 annotation 为 true。需要在 deployment 指定 anntations
#ks-installer/roles/ks-core/ks-core/templates/sample-bookinfo-configmap.yaml.j2
      template:
        metadata:
          labels:
            app: reviews
            version: v1
            app.kubernetes.io/version: v1
            app.kubernetes.io/name: bookinfo
          annotations:
            sidecar.istio.io/inject: 'true'
  1. 另外看下 mutatingwebhook ,是触发istio-sidecar-injector 这个 service,port 为 443,path /inject

看 mutatingwebhook

root@ks-allinone:/root # kubectl get mutatingwebhookconfigurations/istio-sidecar-injector  -o yaml|less
...
    service:
      name: istio-sidecar-injector
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: kubesphere.io/workspace
      operator: Exists
    - key: istio-injection
      operator: NotIn
      values:
      - disabled
      -   objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30

总结:

  1. kubesphere 如果要实现注入sidecar,需要在pod的annotations里面加上:sidecar.istio.io/inject: "true"
  2. kubesphere关闭了namespace打标签istio-injection的注入方式;
  3. Kubesphere的默认policy为false:
# kubectl -n istio-system get cm istio-sidecar-injector -oyaml
...
apiVersion: v1
data:
  config: |-
    policy: disabled

参考:https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/