kuesphere porter 私有环境LB安装

jyhgit

https://www.yuque.com/docs/share/9625f888-8e88-44a7-bcf1-a6b26b20bfa0?# 《kuesphere porter 私有环境LB安装》

图片见yu que链接

零、参考资料及注意

参考资料

https://github.com/kubesphere/porter/blob/master/doc/install-porter-on-kubesphere.md
https://github.com/kubesphere/porter/blob/master/doc/zh/layer2.md
https://github.com/kubesphere/porter/blob/master/doc/use-porter-in-layer-2-mode.md
https://www.bilibili.com/video/BV17K4y177YG?from=search&seid=17868051001232522796

⚠️可在安装一次后再回来看注意事项

1、不要使用ks console 自带应用商店的porter，版本过老
2、重新安装应用porter后，需要手动删除残留，kubectl get crds | grep network.kubesphere.io 看还有没有bgpconfs,bgppeers,eips残留，有的话先删掉，然后再安装
3、porter及EIP配置完成后修改平台项目网关注释，企业空间–》项目—》项目设置—》高级设置–》网关并修改访问方式为LB
lb.kubesphere.io/v1alpha1:porter
protocol.porter.kubesphere.io/v1alpha1:layer2

一、环境说明

基础vms:vmwares
k8s:1.17.1
docker:1.13
ks:v3.0.0
节点网段：172.31.0.0/22
porter：0.2.1 [0.4.1]
porter配置方式：layer2

二、porter安装

安装方式选择

使用ks console 安装

install

https://github.com/kubesphere/porter/blob/master/doc/install-porter-on-kubesphere.md

delete

安装教程删除后，需要清除残留bgpconfs,bgppeers,eips,否则后续EIP无法找到CRD

[root@namenodemaster ~]# kubectl get crds | grep network.kubesphere.io
bgpconfs.network.kubesphere.io                    2021-02-07T09:30:15Z
bgppeers.network.kubesphere.io                    2021-02-07T09:30:15Z
eips.network.kubesphere.io                        2021-02-07T09:30:15Z
namespacenetworkpolicies.network.kubesphere.io    2021-01-28T09:55:13Z
[root@namenodemaster ~]# kubectl delete crds bgpconfs.network.kubesphere.io bgppeers.network.kubesphere.io eips.network.kubesphere.io

三、配置porter-layer2

由于私有环境不支持BGP所以使用layer2

https://github.com/kubesphere/porter/blob/master/doc/zh/layer2.md
https://github.com/kubesphere/porter/blob/master/doc/use-porter-in-layer-2-mode.md

Enable strictARP for kube-proxy

1、kubectl edit configmap kube-proxy -n kube-system
ipvs:
  strictARP: true
2、Run the following command to restart kube-proxy:
kubectl rollout restart daemonset kube-proxy -n kube-system

配置EIP

• 创建layer2 eip–⚠️必须和节点同一网段且未使用的、网卡名称注意修改，如果多个网卡参考上面第二个链接

apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
  name: porter-layer2-eip
spec:
  address: 172.31.2.200-172.31.2.210
  interface: ens32
  protocol: layer2

• 创建工作负载

apiVersion: apps/v1
kind: Deployment
metadata:
  name: porter-layer2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: porter-layer2
  template:
    metadata:
      labels:
        app: porter-layer2
    spec:
      containers:
        - image: luksa/kubia
          name: kubia
          ports:
            - containerPort: 8080

• 创建service


kind: Service
apiVersion: v1
metadata:
  name: porter-layer2-svc
  annotations:
    lb.kubesphere.io/v1alpha1: porter
    protocol.porter.kubesphere.io/v1alpha1: layer2
spec:
  selector:
    app: porter-layer2
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 8080
  externalTrafficPolicy: Cluster

验证layer2

[namenodemaster]# k get svc porter-layer2-svc
NAME                TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
porter-layer2-svc   LoadBalancer   10.1.37.153   172.31.2.200   80:31060/TCP   16h
mac#ping 172.31.2.201
PING 172.31.2.201 (172.31.2.201): 56 data bytes
92 bytes from 172.31.0.234: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 fe20   0 0000  01  01 fc9c 172.31.99.228  172.31.2.201
 
 [root@namenodemaster porter]# k get no -o wide
NAME             STATUS   ROLES    AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
k8s-node2        Ready    <none>   6d18h   v1.17.1   172.31.0.233   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1
k8s-node3        Ready    <none>   6d18h   v1.17.1   172.31.0.234   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1
namenodemaster   Ready    master   15d     v1.17.1   172.31.1.37    <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1
yarnserver       Ready    <none>   15d     v1.17.1   172.31.1.38    <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1

四、ks 外网访问网关设置

1、给网关添加注释
lb.kubesphere.io/v1alpha1: porter
protocol.porter.kubesphere.io/v1alpha1: layer2
2、正常保存后外网地址会出现任意一个node的ip，如果没有说明有问题

五、创建应用路由验证LB

1、新增应用路由，前提是有svc
2、可以绑定域名到网关地址或者内网dns解析后访问验证

jyhgit

有时候eip 无法ping通，重建项目网关后可以ping通，还会不稳定。
{“level”:“info”,“ts”:1613013813.7893324,“msg”:“setup porter service”,“service”:“kubesphere-controls-system/kubesphere-router-demo-project”}
{“level”:“info”,“ts”:1613013813.7894466,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kubesphere-controls-system/kubesphere-router-demo-project”,“Addr”:"",“Eip”:"",“Protocol”:“layer2”,“Unalloc”:false},“peek”:true,“result”:{“Addr”:“172.31.2.201”,“Eip”:“porter-layer2-eip”,“Protocol”:“layer2”,“Sp”:{}},“err”:null}
{“level”:“error”,“ts”:1613013813.7896729,“logger”:“controller-runtime.controller”,“msg”:“Reconciler error”,“controller”:“LBController”,“request”:“kubesphere-controls-system/kubesphere-router-demo-project”,“error”:“node yarnserver has no nexthop”,“stacktrace”:“github.com/go-logr/zapr.(zapLogger).Error\n\t/home/ubuntu/go-path/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/ubuntu/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/ubuntu/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/ubuntu/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/ubuntu/go-path/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/ubuntu/go-path/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/ubuntu/go-path/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/ubuntu/go-path/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90”}

duanjiong

jyhgit ip addr show ${nic} 看下你yarnserver节点配置的ip地址

370569218

duanjiong EIP在集群里能ping通，在内网无法ping通

jyhgit

duanjiong
集群外部ping eip 一个不通，另外一个时通时不通，这个啥问题，必须同一二层，是指不能经过网关？
[root@yarnserver ~]# ip addr show ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:35:87:63 brd ff:ff:ff:ff:ff:ff
inet 172.31.1.38/22 brd 172.31.3.255 scope global noprefixroute ens32
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe35:8763/64 scope link
valid_lft forever preferred_lft forever
[root@yarnserver ~]# ip r
default via 172.31.0.1 dev ens32 proto static metric 100
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.31.0.0/22 dev ens32 proto kernel scope link src 172.31.1.38 metric 100

集群内部ping 两个eip结果
[root@yarnserver ~]# ping -c 3 172.31.2.201
PING 172.31.2.201 (172.31.2.201) 56(84) bytes of data.
From 172.31.1.38 icmp_seq=1 Destination Host Unreachable
From 172.31.1.38 icmp_seq=2 Destination Host Unreachable
From 172.31.1.38 icmp_seq=3 Destination Host Unreachable

— 172.31.2.201 ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2000ms
pipe 3
[root@yarnserver ~]# ping -c 3 172.31.2.202
PING 172.31.2.202 (172.31.2.202) 56(84) bytes of data.
From 172.31.0.234 icmp_seq=2 Redirect Host(New nexthop: 172.31.2.202)
From 172.31.0.234: icmp_seq=2 Redirect Host(New nexthop: 172.31.2.202)
From 172.31.0.234 icmp_seq=3 Redirect Host(New nexthop: 172.31.2.202)
From 172.31.0.234: icmp_seq=3 Redirect Host(New nexthop: 172.31.2.202)
From 172.31.0.234 icmp_seq=1 Destination Host Unreachable

— 172.31.2.202 ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
pipe 3

外部ping eip

gitboy123

370569218 我也是这个问题，k8s集群节点可以访问eip，一个网段非k8s节点的就访问不到eip了，奇怪

freddie

您好，第四步ks外网访问网关设置，你这边描述的是添加注释保存之后外网地址会出现任意一个node的地址，但是我这边保存之后出现的是eip地址池里面地址

370569218

freddie 本来就是eip池子的ip 与node节点是一个ip段

gitboy123

freddie eip地址池的ip，不是node节点ip

freddie

370569218 好的多谢，可以用了

370569218

gitboy123 我删除重装后就可以了，
你可以先尝试清除残留bgpconfs,bgppeers,eips,后看看不行就重装了，重装前一定要清除残留bgpconfs,bgppeers,eips,

gitboy123

370569218 试过，都删了重装了，依然还是老样子，我在阿里云ecs的专有网络环境上，怀疑跟ecs不让用vip有关，keepalived的虚拟路由冗余协议的vip阿里云的ecs就不支持

zheng1

jyhgit 对，L2模式的客户端必须在同一个二层网络

yyt6200

mark一下