Apisix、Service Mesh、Argo-CD-全链路灰度发布

Aailm · 2022年1月25日

KubeSphere版本信息

KubeSphere,v3.2.1
Apisix,2.10.3

Apisix-部署 Istio(通过配置’annotations')

# 关键是下述配置尾部带有'#'部分
kind: Service
apiVersion: v1
metadata:
  name: apisix-gateway
  namespace: ingress-apisix
  labels:
    app: apisix-gateway
  annotations:
    servicemesh.kubesphere.io/enabled: 'true' # istio
spec:
  ports:
    - name: http-9080
      protocol: TCP
      port: 80
      targetPort: 9080
      nodePort: 30099
  selector:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: apisix
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Local # 终端真实IP
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: apisix
  namespace: ingress-apisix
  labels:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: apisix
    app.kubernetes.io/version: 2.10.0
    app.kubesphere.io/instance: apisix
    helm.sh/chart: apisix-0.7.2
  annotations:
    meta.helm.sh/release-name: apisix
    meta.helm.sh/release-namespace: ingress-apisix
    servicemesh.kubesphere.io/enabled: 'true' # istio
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/instance: apisix
      app.kubernetes.io/name: apisix
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: apisix
        app.kubernetes.io/name: apisix
      annotations:
        proxy.istio.io/config: | # istio 正常启动之后启动业务pod
          holdApplicationUntilProxyStarts: true 
        sidecar.istio.io/inject: 'true' # istio

—

Sevice Mesh-EnvoyFiler(可选,主要是转发终端IP及Header处理)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: realip-headers
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH # 禁止增加'server'
          skip_xff_append: false # 转发终端IP
          use_remote_address: true
          generate_request_id: true
          preserve_external_request_id: true
          xff_num_trusted_hops: 1
---           
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: response-remove-headers # 参考"ASM"官方文档配置'header'
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/name: apisix
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      proxy:
        proxyVersion: ^1\.11.*
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value: # lua filter specification
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |-
            function envoy_on_response(response_handle)
                function hasFrameAncestors(rh)
                s = rh:headers():get("Content-Security-Policy");
                delimiter = ";";
                defined = false;
                for match in (s..delimiter):gmatch("(.-)"..delimiter) do
                    match = match:gsub("%s+", "");
                    if match:sub(1, 15)=="frame-ancestors" then
                    return true;
                    end
                end
                return false;
                end
                if not response_handle:headers():get("Content-Security-Policy") then
                csp = "frame-ancestors none;";
                response_handle:headers():add("Content-Security-Policy", csp);
                elseif response_handle:headers():get("Content-Security-Policy") then
                if not hasFrameAncestors(response_handle) then
                    csp = response_handle:headers():get("Content-Security-Policy");
                    csp = csp .. ";frame-ancestors none;";
                    response_handle:headers():replace("Content-Security-Policy", csp);
                end
                end
                if not response_handle:headers():get("X-Frame-Options") then
                response_handle:headers():add("X-Frame-Options", "deny");
                end
                if not response_handle:headers():get("X-XSS-Protection") then
                response_handle:headers():add("X-XSS-Protection", "1; mode=block");
                end
                if not response_handle:headers():get("X-Content-Type-Options") then
                response_handle:headers():add("X-Content-Type-Options", "nosniff");
                end
                if not response_handle:headers():get("Referrer-Policy") then
                response_handle:headers():add("Referrer-Policy", "no-referrer");
                end
                if not response_handle:headers():get("X-Download-Options") then
                response_handle:headers():add("X-Download-Options", "noopen");
                end
                if not response_handle:headers():get("X-DNS-Prefetch-Control") then
                response_handle:headers():add("X-DNS-Prefetch-Control", "off");
                end
                if not response_handle:headers():get("Feature-Policy") then
                response_handle:headers():add("Feature-Policy",
                                                "camera 'none';"..
                                                "microphone 'none';"..
                                                "geolocation 'none';"..
                                                "encrypted-media 'none';"..
                                                "payment 'none';"..
                                                "speaker 'none';"..
                                                "usb 'none';");
                end
                if response_handle:headers():get("x-envoy-upstream-service-time") then
                response_handle:headers():remove("x-envoy-upstream-service-time");
                end
                if response_handle:headers():get("x-envoy-decorator-operation") then
                # 没有效果,无法删除
                response_handle:headers():remove("x-envoy-decorator-operation");
                end
                if response_handle:headers():get("X-Powered-By") then
                response_handle:headers():remove("X-Powered-By");
                end
            end

kubectl apply -f envoy.yaml

—

Argo-CD-增加配置(configmaps/argocd-cm,否则argo处于’OutOfSync’)

data:
  resource.exclusions: | # 排除两种'kind'
    - kinds:
      - "VirtualService"
      - "DestinationRule"

Argo-CD-样例

过程说明:以’Tomcat’为例,模拟发布两个版本

1.初始发布

v1,tomcat:8.5-jdk11-corretto

2.增量发布

v2,tomcat:latest
灰度策略,接收’header’带有’v:2′的流量

# 样例整体组成
│  canary-business.yaml
│
├─mesh-center
│      application.yaml
│      kustomization.yaml
│
└─mesh-service
        deployment.yaml
        kustomization.yaml
        service.yaml
        strategy.yaml

canary-business.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: canary-business
  namespace: argo-cd
spec:
  generators:
  - git:
      repoURL: git/canary-business
      revision: HEAD
      directories:
      - path: '*-center'
      - path: '*-service'
  template:
    metadata:
      name: '{{path.basename}}'
      finalizers:
      - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: git/canary-business
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: kubesphere-sample-dev
#      syncPolicy:
#        automated:
#          prune: false
#          selfHeal: false
#          allowEmpty: true

kustomization.yaml-mesh-center

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - application.yaml

apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
  name: mesh-center
  labels:
    app.kubernetes.io/version: vm1
    app.kubernetes.io/name: mesh-center
  annotations:
    servicemesh.kubesphere.io/enabled: 'true'
spec:
  selector:
    matchLabels:
      app.kubernetes.io/version: vm1
      app.kubernetes.io/name: mesh-center
  addOwnerRef: true
  componentKinds:
    - group: ''
      kind: Service
    - group: apps
      kind: Deployment
    - group: apps
      kind: StatefulSet
    - group: extensions
      kind: Ingress
    - group: servicemesh.kubesphere.io
      kind: Strategy
    - group: servicemesh.kubesphere.io
      kind: ServicePolicy

mesh-service/kustomization.yaml发布-v1

resources:
  - deployment.yaml
  - service.yaml

images:
- name: tomcat
  newName: tomcat
  newTag: 8.5-jdk11-corretto
  
patches:
- patch: |-
    - op: replace
      path: /metadata/name
      value: mesh-web-v1
    - op: add
      path: /metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /metadata/labels/version
      value: v1
    - op: add
      path: /metadata/annotations/servicemesh.kubesphere.io~1enabled
      value: 'true'
    - op: add
      path: /spec/selector/matchLabels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/selector/matchLabels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /spec/selector/matchLabels/version
      value: v1
    - op: add
      path: /spec/template/metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/template/metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /spec/template/metadata/labels/version
      value: v1
    - op: add
      path: /spec/template/metadata/annotations/sidecar.istio.io~1inject
      value: 'true'
    - op: add
      path: /spec/template/metadata/annotations/proxy.istio.io~1config
      value: '{ "holdApplicationUntilProxyStarts": true }'
  target:
    kind: Deployment
    labelSelector: "app=mesh-web"
  options:
    allowNameChange: true  
- patch: |-
    - op: add
      path: /metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /metadata/annotations/kubesphere.io~1workloadType
      value: Deployment
    - op: add
      path: /metadata/annotations/servicemesh.kubesphere.io~1enabled
      value: 'true'
    - op: add
      path: /spec/selector/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/selector/app.kubernetes.io~1version
      value: vm1
  target:
    kind: Service 
    labelSelector: "app=mesh-web"

apisix-上游及路由配置(下述为’dashboard’之中的’json’配置)

{
  "nodes": [
    {
      "host": "mesh-web.kubesphere-sample-dev",
      "port": 80,
      "weight": 1
    }
  ],
  "timeout": {
    "connect": 6,
    "send": 6,
    "read": 6
  },
  "type": "roundrobin",
  "scheme": "http",
  "pass_host": "node", # 
  "name": "mesh-web",
  "keepalive_pool": {
    "idle_timeout": 60,
    "requests": 1000,
    "size": 320
  }
}

{
  "uri": "/mesh-web/*",
  "name": "mesh-web",
  "methods": [
    "GET",
    "POST"
  ],
  "plugins": {
    "proxy-rewrite": {
      "regex_uri": [
        "^/mesh-web/(.*)",
        "/$1"
      ]
    }
  },
  "upstream_id": "391837637637309481", # 此处'id'依据实际情况更改
  "status": 1
}

curl --request GET 'http://apisix-gateway.ingress-apisix/mesh-web/'

mesh-service/kustomization.yaml发布-v2,以金丝雀策略发布

resources:
  - deployment.yaml
  - service.yaml
  - strategy.yaml # 金丝雀策略

images:
- name: tomcat
  newName: tomcat
  newTag: latest
  
patches:
- patch: |-
    - op: replace
      path: /metadata/name
      value: mesh-web-v2 # 将文件之中的v1都改为v2形成新的版本
    - op: add
      path: /metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /metadata/labels/version
      value: v2
    - op: add
      path: /metadata/annotations/servicemesh.kubesphere.io~1enabled
      value: 'true'
    - op: add
      path: /spec/selector/matchLabels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/selector/matchLabels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /spec/selector/matchLabels/version
      value: v2
    - op: add
      path: /spec/template/metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/template/metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /spec/template/metadata/labels/version
      value: v2
    - op: add
      path: /spec/template/metadata/annotations/sidecar.istio.io~1inject
      value: 'true'
    - op: add
      path: /spec/template/metadata/annotations/proxy.istio.io~1config
      value: '{ "holdApplicationUntilProxyStarts": true }'
  target:
    kind: Deployment
    labelSelector: "app=mesh-web"
  options:
    allowNameChange: true  
- patch: |-
    - op: add
      path: /metadata/labels/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /metadata/labels/app.kubernetes.io~1version
      value: vm1
    - op: add
      path: /metadata/annotations/kubesphere.io~1workloadType
      value: Deployment
    - op: add
      path: /metadata/annotations/servicemesh.kubesphere.io~1enabled
      value: 'true'
    - op: add
      path: /spec/selector/app.kubernetes.io~1name
      value: mesh-center
    - op: add
      path: /spec/selector/app.kubernetes.io~1version
      value: vm1
  target:
    kind: Service 
    labelSelector: "app=mesh-web"

apiVersion: v1
items:
- apiVersion: servicemesh.kubesphere.io/v1alpha2
  kind: Strategy
  metadata:
    annotations:
      servicemesh.kubesphere.io/newWorkloadName: mesh-web-v2
      servicemesh.kubesphere.io/oldWorkloadName: mesh-web-v1
      servicemesh.kubesphere.io/workloadReplicas: "1"
      servicemesh.kubesphere.io/workloadType: deployments
    labels:
      app: mesh-web
      app.kubernetes.io/name: mesh-center
      app.kubernetes.io/version: vm1
    name: mesh-canary-strategy
  spec:
    principal: v1
    selector:
      matchLabels:
        app: mesh-web
        app.kubernetes.io/name: mesh-center
        app.kubernetes.io/version: vm1
    strategyPolicy: WaitForWorkloadReady
    template:
      spec:
        hosts:
        - mesh-web
        http:
        - match:
          - headers:
              v:
                exact: "2" # 精确匹配'header'转发流量
          route:
          - destination:
              host: mesh-web
              subset: v2
            weight: 100
        - route:
          - destination:
              host: mesh-web
              subset: v1
            weight: 100
    type: Canary
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

curl --request GET 'http://apisix-gateway.ingress-apisix/mesh-web/',流量转向v1(tomcat版本为8.x)

curl --request GET 'http://apisix-gateway.ingress-apisix/mesh-web/' --header 'v: 2',流量转向v2(tomcat版本为9.x)

效果(观测输出、流量)

遗留问题,如有那位知道如何解决,请指教

istio,http的response之中存在’x-envoy-decorator-operation’,含有技术栈信息,如何删除?
envoy之中固化使用’UTC’时区,造成日志、response的date输出都为’UTC’,如何更改为’Asia/Shanghai’?
apisix 可否直接使用’kubesphere’初始安装的多个节点的etcd,如何配置?

最后

以上如有错漏或是更优、更简方式也请指出

RolandMa1986 · 2022年1月28日

ailm 方案很不错
对于问题1：尝试一下以下 EnvoyFilter， apply 到apisix 的 Namespace。

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: remove-bad-response-headers
spec:
  configPatches:
  - applyTo: HTTP_ROUTE
    patch:
      operation: MERGE
      value:
        decorator:
          propagate: false

问题2：一般建议统一使用 UTC 时区。日志收集之后可以再日志查询的时候做 Localization。需要更改的话，可以尝试修改istio sidecar 的注入模板，挂载时区文件。
问题3：没有这样用过，看一下ApiSix 社区有没有相关说明。

Feynman · 2022年1月28日

新潮玩法

Aailm · 2022年1月29日

RolandMa1986 感谢回复讨论

遗留问题一

调整Sevice Mesh-EnvoyFiler(增加:remove-bad-response-headers)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: realip-headers
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH
          skip_xff_append: false
          use_remote_address: true
          generate_request_id: true
          preserve_external_request_id: true
          xff_num_trusted_hops: 1
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: remove-bad-response-headers
  namespace: ingress-apisix
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/name: apisix
  configPatches:
  - applyTo: HTTP_ROUTE # 清理'x-envoy-decorator-operation'
    patch:
      operation: MERGE
      value:
        decorator:
          propagate: false
---           
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: response-remove-headers
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/name: apisix
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      proxy:
        proxyVersion: ^1\.11.*
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value: # lua filter specification
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |-
            function envoy_on_response(response_handle)
                function hasFrameAncestors(rh)
                s = rh:headers():get("Content-Security-Policy");
                delimiter = ";";
                defined = false;
                for match in (s..delimiter):gmatch("(.-)"..delimiter) do
                    match = match:gsub("%s+", "");
                    if match:sub(1, 15)=="frame-ancestors" then
                    return true;
                    end
                end
                return false;
                end
                if not response_handle:headers():get("Content-Security-Policy") then
                csp = "frame-ancestors none;";
                response_handle:headers():add("Content-Security-Policy", csp);
                elseif response_handle:headers():get("Content-Security-Policy") then
                if not hasFrameAncestors(response_handle) then
                    csp = response_handle:headers():get("Content-Security-Policy");
                    csp = csp .. ";frame-ancestors none;";
                    response_handle:headers():replace("Content-Security-Policy", csp);
                end
                end
                if not response_handle:headers():get("X-Frame-Options") then
                response_handle:headers():add("X-Frame-Options", "deny");
                end
                if not response_handle:headers():get("X-XSS-Protection") then
                response_handle:headers():add("X-XSS-Protection", "1; mode=block");
                end
                if not response_handle:headers():get("X-Content-Type-Options") then
                response_handle:headers():add("X-Content-Type-Options", "nosniff");
                end
                if not response_handle:headers():get("Referrer-Policy") then
                response_handle:headers():add("Referrer-Policy", "no-referrer");
                end
                if not response_handle:headers():get("X-Download-Options") then
                response_handle:headers():add("X-Download-Options", "noopen");
                end
                if not response_handle:headers():get("X-DNS-Prefetch-Control") then
                response_handle:headers():add("X-DNS-Prefetch-Control", "off");
                end
                if not response_handle:headers():get("Feature-Policy") then
                response_handle:headers():add("Feature-Policy",
                                                "camera 'none';"..
                                                "microphone 'none';"..
                                                "geolocation 'none';"..
                                                "encrypted-media 'none';"..
                                                "payment 'none';"..
                                                "speaker 'none';"..
                                                "usb 'none';");
                end
                if response_handle:headers():get("x-envoy-upstream-service-time") then
                response_handle:headers():remove("x-envoy-upstream-service-time");
                end
                if response_handle:headers():get("X-Powered-By") then
                response_handle:headers():remove("X-Powered-By");
                end
            end

访问效果('x-envoy-decorator-operation'已被删除)

HTTP/1.1 200 OK
content-type: text/html;charset=UTF-8
date: Sat, 29 Jan 2022 10:58:15 GMT # 此时已经增加'timezone'配置,但没有改变
content-security-policy: frame-ancestors none;
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: no-referrer
x-download-options: noopen
x-dns-prefetch-control: off
feature-policy: camera 'none';microphone 'none';geolocation 'none';encrypted-media 'none';payment 'none';speaker 'none';usb 'none';
transfer-encoding: chunked

**调整带来的负面问题

问题一:apisix无法重启，出现下述错误

got malformed version message: "" from etcd "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379/version"

问题二:'VirtualService’调整为新的灰度策略没有效果,依然会使用前一次的’VirtualService’配置

 - headers:
                v:
                  exact: '4'
# 增加匹配规则('VirtualService'改变),此时依然使用上述'v=4'的转发规则 
- headers:
                v:
                  exact: '4'
                p:
                  exact: 'android'

**删除’remove-bad-response-headers'之后负面问题消除

遗留问题二

调整istio-proxy(通过注解配置时区及同步主机时间)

        proxy.istio.io/config: >-
          { "holdApplicationUntilProxyStarts":
          true,"proxyMetadata":{"TZ":"Asia/Shanghai"}}
        sidecar.istio.io/inject: 'true'
        sidecar.istio.io/userVolume: '[{"name":"host-time","hostPath":{"path":"/etc/localtime","type":""}}]'
        sidecar.istio.io/userVolumeMount: '[{"mountPath":"/etc/localtime","name":"host-time","readOnly":true}]'

配置效果

访问效果(系统时间为UTC,response时间为UTC)

$ date
Sat Jan 29 11:21:18 Asia 2022
$ env | grep TZ
PROXY_CONFIG={"discoveryAddress":"istiod-1-11-2.istio-system.svc:15012","tracing":{"zipkin":{"address":"jaeger-collector.istio-system.svc:9411"}},"proxyMetadata":{"TZ":"Asia/Shanghai"},"holdApplicationUntilProxyStarts":true}
TZ=Asia/Shanghai
$ ls -la /etc/localtime
-rw-r--r-- 5 root root 556 Jan 26  2021 /etc/localtime

Apisix、Service Mesh、Argo-CD-全链路灰度发布

Aailm

RolandMa1986K零S

FeynmanK零SK贰SK壹S

Aailm

遗留问题一

遗留问题二