问题描述:

按照simulate_with_bird.md将bird作为模拟路由器,在物理路由器上添加静态路由,将eip范围的请求路由到bird服务器

  • 现象
    • 通过porter成功获取到了ip
    • 在k8s集群和bird服务器上操作网络是通的(可以ping通外部地址,并且curl通过外部地址上的服务)
  • 问题
    • 在局域网中其他PC无法通过外部地址访问服务,可以ping通外部地址

配置信息

  • bird配置如下
router id 192.168.1.106; # bird所在服务器IP

protocol kernel {
	scan time 60;
	import none;
	export all;   # Actually insert routes into the kernel routing table
        merge paths on;
}

protocol device {
	scan time 60;
}

protocol bgp neighbor1 {   
    interface "eno4";
    local as 65001;
    neighbor 192.168.1.118 port 17900 as 65000;  # porter-manager所在服务器
    source address 192.168.1.106; # bird所在服务器IP
    import all;   
    export all;
    enable route refresh off;
    add paths on;
}
  • bgpconf配置如下
apiVersion: network.kubesphere.io/v1alpha2
kind: BgpConf
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"network.kubesphere.io/v1alpha2","kind":"BgpConf","metadata":{"annotations":{},"name":"default"},"spec":{"as":65000,"listenPort":17900,"routerId":"192.168.1.118"}}
  name: default
spec:
  as: 65000
  listenPort: 17900
  routerId: 192.168.1.118
  • bgppeer配置
apiVersion: network.kubesphere.io/v1alpha2
kind: BgpPeer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"network.kubesphere.io/v1alpha2","kind":"BgpPeer","metadata":{"annotations":{},"name":"bgppeer-bird"},"spec":{"conf":{"neighborAddress":"192.168.1.106","peerAs":65001}}}
  name: bgppeer-bird
spec:
  conf:
    neighborAddress: 192.168.1.106
    peerAs: 65001
  • eip配置
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"network.kubesphere.io/v1alpha2","kind":"Eip","metadata":{"annotations":{},"name":"eip-pool-100"},"spec":{"address":"192.168.100.10-192.168.100.255","disable":false,"interface":"ens192","protocol":"bgp"},"status":{"firstIP":"192.168.100.10","lastIP":"192.168.100.255","occupied":false,"poolSize":246,"ready":true,"usage":1,"v4":true}}
  name: eip-pool-100
spec:
  address: 192.168.100.10-192.168.100.255
  interface: ens192
  protocol: bgp

既然你能够ping通, 那你可以先尝试在bird上抓取icmp数据包, 看是否数据包是否到达bird? 然后再检查下bird上的路由

    duanjiong service 信息如下,帮忙分析下哇 谢谢

    [root@node1 anderson]# kubectl describe service spring-k8s-configmap-demo -n a111
    Name:                     spring-k8s-configmap-demo
    Namespace:                a111
    Labels:                   app=spring-k8s-configmap-demo
                              s2ibuilder=spring-k8s-configmap-demo-b2i-n3f8
                              version=v1
    Annotations:              lb.kubesphere.io/v1alpha1: porter
                              protocol.porter.kubesphere.io/v1alpha1: bgp
    Selector:                 app=spring-k8s-configmap-demo,s2ibuilder=spring-k8s-configmap-demo-b2i-n3f8,version=v1
    Type:                     LoadBalancer
    IP:                       10.233.1.17
    LoadBalancer Ingress:     192.168.100.13
    Port:                     http-8080  8080/TCP
    TargetPort:               8080/TCP
    NodePort:                 http-8080  31654/TCP
    Endpoints:                10.233.96.3:8080
    Session Affinity:         None
    External Traffic Policy:  Cluster
    Events:                   <none>
    [root@node1 anderson]# kubectl get service spring-k8s-configmap-demo -n a111
    NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)          AGE
    spring-k8s-configmap-demo   LoadBalancer   10.233.1.17   192.168.100.13   8080:31654/TCP   12d

    bird服务器的ip route

    root@ubuntu:~# ip route
    default via 192.168.1.1 dev eno4 onlink 
    10.42.0.0/24 dev cni0  proto kernel  scope link  src 10.42.0.1 
    10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink 
    10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink 
    172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
    172.18.0.0/16 dev br-f400fe52ff88  proto kernel  scope link  src 172.18.0.1 linkdown 
    172.19.0.0/16 dev br-c017bbc440d4  proto kernel  scope link  src 172.19.0.1 linkdown 
    172.20.0.0/16 dev br-f7149e2e2c8c  proto kernel  scope link  src 172.20.0.1 linkdown 
    172.21.0.0/16 dev br-a5eedf8a78e9  proto kernel  scope link  src 172.21.0.1 linkdown 
    172.22.0.0/16 dev br-67fa49e9bef5  proto kernel  scope link  src 172.22.0.1 linkdown 
    192.168.1.0/24 dev eno4  proto kernel  scope link  src 192.168.1.106 
    192.168.100.13  proto bird 
    	nexthop via 172.20.0.2  dev br-f7149e2e2c8c weight 1 linkdown
    	nexthop via 192.168.1.118  dev eno4 weight 1
    	nexthop via 192.168.1.124  dev eno4 weight 1

    我看你只有两个节点,但是ip route显示nexthop有三条,其中有一条路由是这个nexthop via 172.20.0.2 dev br-f7149e2e2c8c weight 1 linkdown, 可能是这个导致不通, 你这个172.20.0.2哪里来得? 可以得话, 你可以尝试重新创建一个service, 贴下新service得event

      duanjiong 这是一个测试用的集群,172.20.0.2是之前的node,被删掉了

      我重新部署了下刚才的那个service,现在分配到的是两条

      局域网中另一台PC

      那这个service暴露是没问题的, 你需要检查下你bird那台机器ip forward是否开启。 另外, 你需要确认下你windows这台机器ping 192.168.100.13会经过bird, 如果以上步骤都确认完了, 不应该会有问题的。

        duanjiong bird 服务器ip forward检查

        我把eip-pool 从192.168.100.0/24换到10.0.0.0/24就可以了 这是什么原因呢? 我局域网的IP地址范围是192.168.1.0/24

          AndersonYangOh

          还是你的路由问题, 你这种使用方式建议你吧kube-proxy的strictARp给打开然后再用192.168.1.0/24这个网段的eip

            duanjiong 遇到另一个问题咨询下

            • 创建服务,部署落到了主机A,loadbanlance使用porter+bgp做负载均衡器,能正常访问服务
            • 将主机A停止调度后,重新部署该服务,此时外部IP能ping通,但是不能访问服务,报错Connection refused,在bird主机查看ip route也是正常的