metallb 迁移到porter咨询

weekyuan

因为我的master和node节点分别连接不通的路由器，在节点与路由器建立bgp连接的时候需要区分。也就是配置多个邻居并且需要区分不通邻居与节点匹配。如下是我metallb的配置，请问在porter上我该如何配置？

RolandMa1986

Feynman

weekyuan 这个功能快开发的差不多了，社区还在准备文档和测试，下周准备开一个 sig-network 的周例会演示一下新版本的特性，欢迎持续关注

duanjiong

weekyuan 你好， Kubersphere社区最近刚发布Porter新版本，你可以使用kubectl apply -f https://raw.githubusercontent.com/kubesphere/porter/master/deploy/porter.yaml
这个一键安装，然后可以参考这个文档配置neighbor
https://github.com/kubesphere/porter/blob/master/doc/bgp_config.md#specify-nodeselector

duanjiong

您好，这个目前porter release版本暂时还不支持label选择peer，后续版本会支持。

weekyuan

duanjiong 收到，感谢！抽空试验一把

weekyuan

duanjiong 请问可以给一个多个peer的配置示例吗？

duanjiong

weekyuan 你直接参考上面文档就行，创建多个BgpPeer资源就行

weekyuan

duanjiong 今天我测试了下，没有跑通，麻烦帮忙看下哪里的问题。
第一个是无法分配eip：

查看porter-manager有如下错误：
time="2020-12-11T05:46:50Z" level=warning msg="Failed to AcceptTCP" Error="accept tcp4 0.0.0.0:17900: use of closed network connection" Topic=Peer {"level":"error","ts":1607665610.2180018,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"bgpconf","request":"/default","error":"listen tcp6 [::]:17900: socket: address family not supported by protocol","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"} {"level":"info","ts":1607665910.6542802,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"kube-system/kubernetes-dashboard","Addr":"","Eip":"","Protocol":"bgp"},"peek":true,"result":{"Addr":"","Eip":"","Protocol":"","Sp":null},"err":null} {"level":"info","ts":1607665910.6651566,"msg":"AddFinalizer","service":"kube-system/kubernetes-dashboard","finalizer":["finalizer.lb.kubesphere.io/v1alpha1"],"err":null} {"level":"info","ts":1607665910.6736143,"msg":"assignIP update eip","eip":{"usage":1,"poolSize":8192,"used":{"10.145.224.0":"kube-system/kubernetes-dashboard"},"firstIP":"10.145.224.0","lastIP":"10.145.255.255","ready":true,"v4":true}} {"level":"info","ts":1607665910.6736689,"logger":"IPAM","msg":"assignIP","args":{"Key":"kube-system/kubernetes-dashboard","Addr":"","Eip":"","Protocol":"bgp"},"result":{"Addr":"10.145.224.0","Eip":"eip-pool","Protocol":"bgp","Sp":{}},"err":null} {"level":"info","ts":1607665910.6739268,"msg":"callSetLoadBalancer","result":{"Addr":"10.145.224.0","Eip":"eip-pool","Protocol":"bgp","Sp":{}},"announceNodes":["10.142.96.71","10.142.96.135","10.142.96.136","10.142.96.73","10.142.96.72"],"avaliableNodes":["10.142.96.71","10.142.96.135","10.142.96.136","10.142.96.73","10.142.96.72"],"err":"Bgp not ready, please config bgpconf/bgppeer"} {"level":"error","ts":1607665910.673993,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"LBController","request":"kube-system/kubernetes-dashboard","error":"Bgp not ready, please config bgpconf/bgppeer","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

另外eip没有绑定成功，但是地址池eippool显示已经使用，导致无法删除地址池。需要怎么清理已分配的ip的呢？

duanjiong

weekyuan 这个你没配置bgpconf，bgppeer？你kubectl get eip -o yaml看下

duanjiong

weekyuan 你这是配置得ipv6的地址吧，在bgpconf中。麻烦看下你的bgpconf配置文件

weekyuan

duanjiong 都配置了的，刚才我去掉了svc的annotations后yaml中status的used字段自动消失了，之前是显示了used并且与svc绑定的。但现在这样还是无法删除。

`apiVersion: v1
items:

apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“network.kubesphere.io/v1alpha2”,“kind”:“Eip”,“metadata”:{“annotations”:{},“name”:“eip-pool”},“spec”:{“address”:“10.145.248.0/19”,“disable”:false,“interface”:“eth0”,“protocol”:“bgp”}}
creationTimestamp: “2020-12-11T05:35:02Z”
deletionGracePeriodSeconds: 0
deletionTimestamp: “2020-12-11T06:05:27Z”
finalizers:
- finalizer.ipam.kubesphere.io/v1alpha1
  generation: 3
  managedFields:
- apiVersion: network.kubesphere.io/v1alpha2
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:annotations:
  .: {}
  f:kubectl.kubernetes.io/last-applied-configuration: {}
  f:spec:
  .: {}
  f:address: {}
  f:interface: {}
  f:protocol: {}
  manager: kubectl
  operation: Update
  time: “2020-12-11T05:35:02Z”
- apiVersion: network.kubesphere.io/v1alpha2
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:finalizers:
  .: {}
  v:“finalizer.ipam.kubesphere.io/v1alpha1”: {}
  f:status:
  .: {}
  f:firstIP: {}
  f:lastIP: {}
  f:poolSize: {}
  f:ready: {}
  f:v4: {}
  manager: manager
  operation: Update
  time: “2020-12-11T05:51:50Z”
  name: eip-pool
  resourceVersion: “212876”
  selfLink: /apis/network.kubesphere.io/v1alpha2/eips/eip-pool
  uid: 56a41cd5-6396-4f8b-bde6-32546ab379f9
  spec:
  address: 10.145.248.0/19
  interface: eth0
  protocol: bgp
  status:
  firstIP: 10.145.224.0
  lastIP: 10.145.255.255
  poolSize: 8192
  ready: true
  v4: true
  kind: List
  metadata:
  resourceVersion: ""
  selfLink: ""`

weekyuan

duanjiong 需要配置ipv6吗？我集群默认禁用了ipv6的。无法监听ipv6的端口。
apiVersion: network.kubesphere.io/v1alpha2
kind: BgpConf
metadata:
name: default
spec:
as: 65188
listenPort: 17900

weekyuan

weekyuan 是因为porter的CRD定义中去掉了ListenAddresses这个吗？导致默认会监听ipv4和ipv6两个地址端口，然后我的ipv6禁用导致无法监听，程序就判断失败，导致Bgp not ready，从而无法分配eip吗？

duanjiong

weekyuan 估计是你禁用了ipv6地址，然后你没指定监听地址，所以他监听ipv4和ipv6地址，这导致失败了。你可以尝试https://github.com/kubesphere/porter/blob/master/doc/bgp_config.md#specify-gobgp-to-listen-to-ip-addresses 指定一下监听地址为0.0.0.0试试看

weekyuan

duanjiong 是的，绑定问题解决了。这种情况必须要指定监听地址才行。但是地址分配了，只能在集群内访问，无法在集群外访问。我估计是标签选择的我，我配置了指定标签，但这里还是分配了全部节点。

{“level”:“info”,“ts”:1607670439.50396,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kube-system/kubernetes-dashboard”,“Addr”:"",“Eip”:"",“Protocol”:“bgp”},“peek”:true,“result”:{“Addr”:“10.145.224.0”,“Eip”:“eip-pool”,“Protocol”:“bgp”,“Sp”:{}},“err”:null}
{“level”:“info”,“ts”:1607670439.5903332,“msg”:“callSetLoadBalancer”,“result”:{“Addr”:“10.145.224.0”,“Eip”:“eip-pool”,“Protocol”:“bgp”,“Sp”:{}},“announceNodes”:[“10.142.96.73”,“10.142.96.72”,“10.142.96.71”,“10.142.96.135”,“10.142.96.136”],“avaliableNodes”:[“10.142.96.73”,“10.142.96.72”,“10.142.96.71”,“10.142.96.135”,“10.142.96.136”],“err”:null}
{“level”:“info”,“ts”:1607670439.5907962,“logger”:“IPAM”,“msg”:“unAssignIP”,“args”:{“Key”:“kube-system/kubernetes-dashboard”,“Addr”:"",“Eip”:"",“Protocol”:“bgp”},“peek”:true,“result”:{“Addr”:“10.145.224.0”,“Eip”:“eip-pool”,“Protocol”:“bgp”,“Sp”:{}},“err”:null}
{“level”:“info”,“ts”:1607670439.591321,“msg”:“callSetLoadBalancer”,“result”:{“Addr”:“10.145.224.0”,“Eip”:“eip-pool”,“Protocol”:“bgp”,“Sp”:{}},“announceNodes”:[“10.142.96.73”,“10.142.96.72”,“10.142.96.71”,“10.142.96.135”,“10.142.96.136”],“avaliableNodes”:[“10.142.96.73”,“10.142.96.72”,“10.142.96.71”,“10.142.96.135”,“10.142.96.136”],“err”:null}

duanjiong

weekyuan 这个通告的路由你不用管，你得看下你10.142.96.65这台路由器上有没有路由，这才是你能不能外部访问得关键

weekyuan

duanjiong 这个路由很早就创建了，我直接用metallb可以访问。
另外这个eippool分配后，重新改为nodeport，地址池不能回收，地址池始终删不掉。

duanjiong

weekyuan 这里地址回收得问题我记录一下。还有一个问题你需要注意得是你porter部署在哪个节点得，如果你bgppeer配置得nodeselector上没有porter，那路由肯定是同步不了得，你可以给需要部署porter得节点上打个例如porter得标签，然后给porter deployment 加上nodeselect。

weekyuan

duanjiong 我重新尝试将节点打上porter标签，在porter的deployment上添加nodeselect选择porter。重新部署deployment，确保porter节点上的标签能被bgppeer的nodeselector选择到了。还是不能从集群外访问，分配的IP只能集群内访问。