calico网络出现问题，nodeport访问不通，检查发现calico出现问题

xiaosage · 2020年5月13日

klj890 一脸懵。看信息是calico controller连不上apiserver

klj890 · 2020年5月13日

Kubernetes 节点发送的心跳有助于确定节点的可用性。心跳有两种形式：NodeStatus 和 Lease 对象。每个节点在 kube-node-lease命名空间中都有一个关联的 Lease 对象。 Lease 是一种轻量级的资源，可在集群扩展时提高节点心跳机制的性能.
你为什么要删除kube-node-lease？

xiaosage · 2020年5月13日

klj890 她下面什么都没有，我还以为没用

Wwanjunlei · 2020年5月13日

看一下calico-node的日志，kubectl logs

klj890 · 2020年5月13日

root@demo2r01n00:_# kubectl -n kube-node-lease get leases.coordination.k8s.io
NAME HOLDER AGE
demo2r01n00 demo2r01n00 144d
demo2r01n01 demo2r01n01 144d
demo2r01n02 demo2r01n02 144d
demo2r01n04 demo2r01n04 144d
demo2r01n05 demo2r01n05 144d
demo2r01n06 demo2r01n06 144d

不是什么都没有。。。。。。。有节点信息。

xiaosage · 2020年5月13日

wanjunlei

[root@ykj-prod-common-kubesphere-master1 tmp]# kubectl logs -n kube-system calico-node-bcrdm |grep -v INFO
Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
2020-05-13 08:49:04.047 [WARNING][45] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 08:49:04.048 [WARNING][45] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_1_93: Initializing
bird: Mesh_172_16_6_70: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_1_93: Starting
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_70: Starting
bird: Mesh_172_16_6_70: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: Mesh_172_16_6_70: Connected to table master
bird: Mesh_172_16_6_70: State changed to wait
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Mesh_172_16_1_93: Reconfigured
bird: Mesh_172_16_6_70: Reconfigured
bird: Reconfigured
2020-05-13 08:49:04.639 [WARNING][45] int_dataplane.go 384: Failed to cleanup preexisting XDP state error=failed to load XDP program (/tmp/felix-xdp-182611776): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Invalid argument(22). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-xdp-182611776'
Error: failed to load object file

2020-05-13 08:49:04.774 [WARNING][45] ipip_mgr.go 112: Failed to add IPIP tunnel device error=exit status 1
2020-05-13 08:49:04.774 [WARNING][45] ipip_mgr.go 89: Failed configure IPIP tunnel device, retrying... error=exit status 1
bird: Mesh_172_16_1_93: Connected to table master
bird: Mesh_172_16_1_93: State changed to wait
bird: Graceful restart done
bird: Mesh_172_16_1_93: State changed to feed
bird: Mesh_172_16_6_70: State changed to feed
bird: Mesh_172_16_1_93: State changed to up
bird: Mesh_172_16_6_70: State changed to up

xiaosage · 2020年5月13日

wanjunlei

[root@ykj-prod-common-kubesphere-master1 tmp]# kubectl logs -n kube-system calico-node-m8ckx |grep -v INFO
Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
2020-05-13 06:41:45.153 [ERROR][47] resource.go 288: Error from checkcmd: "Hangup\n"
2020-05-13 06:41:45.355 [WARNING][44] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 06:41:45.355 [WARNING][44] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_6_69: Initializing
bird: Mesh_172_16_6_70: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_6_69: Starting
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_70: Starting
bird: Mesh_172_16_6_70: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
... dropped 59 logs ...
bird: Mesh_172_16_6_70: Connected to table master
bird: Mesh_172_16_6_70: State changed to wait
bird: Mesh_172_16_6_69: Connected to table master
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up
bird: Graceful restart done
bird: Mesh_172_16_6_70: State changed to feed
bird: Mesh_172_16_6_70: State changed to up
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up

xiaosage · 2020年5月13日

wanjunlei

Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
2020-05-13 06:40:42.167 [ERROR][48] resource.go 288: Error from checkcmd: "Hangup\n"
2020-05-13 06:40:42.370 [WARNING][50] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 06:40:42.370 [WARNING][50] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
2020-05-13 06:40:42.498 [WARNING][50] int_dataplane.go 384: Failed to cleanup preexisting XDP state error=failed to load XDP program (/tmp/felix-xdp-000481355): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Invalid argument(22). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-xdp-000481355'
Error: failed to load object file

... dropped 7 logs ...
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_1_93: Initializing
bird: Mesh_172_16_6_69: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_1_93: Starting
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_69: Starting
bird: Mesh_172_16_6_69: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
... dropped 4 logs ...
bird: Mesh_172_16_1_93: Connected to table master
bird: Mesh_172_16_1_93: State changed to wait
2020-05-13 06:40:59.715 [WARNING][50] table.go 508: Detected out-of-sync inserts, marking for resync actualRuleIDs=[]string{"", "tVnHkvAo15HuiPy0", "", ""} chainName="OUTPUT" expectedRuleIDs=[]string{"tVnHkvAo15HuiPy0", "", "", ""} ipVersion=0x4 table="filter"
2020-05-13 06:40:59.715 [WARNING][50] table.go 508: Detected out-of-sync inserts, marking for resync actualRuleIDs=[]string{"", "Cz_u1IQiXIMmKD4c", "", ""} chainName="INPUT" expectedRuleIDs=[]string{"Cz_u1IQiXIMmKD4c", "", "", ""} ipVersion=0x4 table="filter"
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_69: Connected to table master
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up
bird: Graceful restart done
bird: Mesh_172_16_1_93: State changed to feed
bird: Mesh_172_16_1_93: State changed to up
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up

xiaosage · 2020年5月13日

klj890
现在我这里也有这个namespace了，里面也有节点数据。但是网络还是不行

Wwanjunlei · 2020年5月13日

把calico-node重启下试试

xiaosage · 2020年5月13日

wanjunlei 不行我重建这个都不行

Wwanjunlei · 2020年5月13日

还是一样的错误？

xiaosage · 2020年5月13日

wanjunlei 还是一样的

xiaosage · 2020年5月13日

wanjunlei 估计得重建了

klj890 · 2020年5月13日

https://github.com/easzlab/kubeasz/issues/66
希望这个对你有帮助。

xiaosage · 2020年5月13日

klj890 好的我看一下谢谢

3344089386 · 2020年12月24日

一样的问题，楼主解决了吗 xiaosage

voldikss · 2023年2月9日

一样的问题，没有删 lease，搭建好就不行

calico网络出现问题，nodeport访问不通，检查发现calico出现问题

xiaosageK零S

klj890K零S

xiaosageK零S

WwanjunleiK零S

klj890K零S

xiaosageK零S

xiaosageK零S

xiaosageK零S

xiaosageK零S

WwanjunleiK零S

xiaosageK零S

WwanjunleiK零S

xiaosageK零S

xiaosageK零S

klj890K零S

xiaosageK零S

3344089386K零S

voldikss