calico网络出现问题，nodeport访问不通，检查发现calico出现问题

最近做的疑似导火索的操作只有这个 kubectl delete namespaces kube-node-lease




这个该怎么排查源头，怎么处理呢？

klj890 一脸懵。看信息是calico controller连不上apiserver

klj890 她下面什么都没有，我还以为没用

wanjunlei

[root@ykj-prod-common-kubesphere-master1 tmp]# kubectl logs -n kube-system calico-node-bcrdm |grep -v INFO
Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
2020-05-13 08:49:04.047 [WARNING][45] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 08:49:04.048 [WARNING][45] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_1_93: Initializing
bird: Mesh_172_16_6_70: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_1_93: Starting
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_70: Starting
bird: Mesh_172_16_6_70: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: Mesh_172_16_6_70: Connected to table master
bird: Mesh_172_16_6_70: State changed to wait
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Mesh_172_16_1_93: Reconfigured
bird: Mesh_172_16_6_70: Reconfigured
bird: Reconfigured
2020-05-13 08:49:04.639 [WARNING][45] int_dataplane.go 384: Failed to cleanup preexisting XDP state error=failed to load XDP program (/tmp/felix-xdp-182611776): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Invalid argument(22). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-xdp-182611776'
Error: failed to load object file

2020-05-13 08:49:04.774 [WARNING][45] ipip_mgr.go 112: Failed to add IPIP tunnel device error=exit status 1
2020-05-13 08:49:04.774 [WARNING][45] ipip_mgr.go 89: Failed configure IPIP tunnel device, retrying... error=exit status 1
bird: Mesh_172_16_1_93: Connected to table master
bird: Mesh_172_16_1_93: State changed to wait
bird: Graceful restart done
bird: Mesh_172_16_1_93: State changed to feed
bird: Mesh_172_16_6_70: State changed to feed
bird: Mesh_172_16_1_93: State changed to up
bird: Mesh_172_16_6_70: State changed to up

wanjunlei

[root@ykj-prod-common-kubesphere-master1 tmp]# kubectl logs -n kube-system calico-node-m8ckx |grep -v INFO
Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
2020-05-13 06:41:45.153 [ERROR][47] resource.go 288: Error from checkcmd: "Hangup\n"
2020-05-13 06:41:45.355 [WARNING][44] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 06:41:45.355 [WARNING][44] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_6_69: Initializing
bird: Mesh_172_16_6_70: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_6_69: Starting
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_70: Starting
bird: Mesh_172_16_6_70: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
... dropped 59 logs ...
bird: Mesh_172_16_6_70: Connected to table master
bird: Mesh_172_16_6_70: State changed to wait
bird: Mesh_172_16_6_69: Connected to table master
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up
bird: Graceful restart done
bird: Mesh_172_16_6_70: State changed to feed
bird: Mesh_172_16_6_70: State changed to up
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][47] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][44] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up

wanjunlei

Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
2020-05-13 06:40:42.167 [ERROR][48] resource.go 288: Error from checkcmd: "Hangup\n"
2020-05-13 06:40:42.370 [WARNING][50] int_dataplane.go 354: Failed to query VXLAN device error=Link not found
2020-05-13 06:40:42.370 [WARNING][50] int_dataplane.go 366: Can't enable XDP acceleration. error=kernel is too old (have: 3.10.0 but want at least: 4.16.0)
2020-05-13 06:40:42.498 [WARNING][50] int_dataplane.go 384: Failed to cleanup preexisting XDP state error=failed to load XDP program (/tmp/felix-xdp-000481355): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Invalid argument(22). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-xdp-000481355'
Error: failed to load object file

... dropped 7 logs ...
bird: device1: Initializing
bird: direct1: Initializing
bird: device1: Starting
bird: device1: Initializing
bird: direct1: Initializing
bird: Mesh_172_16_1_93: Initializing
bird: Mesh_172_16_6_69: Initializing
bird: device1: Starting
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Graceful restart started
bird: Graceful restart done
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
bird: device1: Connected to table master
bird: device1: State changed to feed
bird: direct1: Starting
bird: direct1: Connected to table master
bird: direct1: State changed to feed
bird: Mesh_172_16_1_93: Starting
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_69: Starting
bird: Mesh_172_16_6_69: State changed to start
bird: Graceful restart started
bird: Started
bird: device1: State changed to up
bird: direct1: State changed to up
... dropped 4 logs ...
bird: Mesh_172_16_1_93: Connected to table master
bird: Mesh_172_16_1_93: State changed to wait
2020-05-13 06:40:59.715 [WARNING][50] table.go 508: Detected out-of-sync inserts, marking for resync actualRuleIDs=[]string{"", "tVnHkvAo15HuiPy0", "", ""} chainName="OUTPUT" expectedRuleIDs=[]string{"tVnHkvAo15HuiPy0", "", "", ""} ipVersion=0x4 table="filter"
2020-05-13 06:40:59.715 [WARNING][50] table.go 508: Detected out-of-sync inserts, marking for resync actualRuleIDs=[]string{"", "Cz_u1IQiXIMmKD4c", "", ""} chainName="INPUT" expectedRuleIDs=[]string{"Cz_u1IQiXIMmKD4c", "", "", ""} ipVersion=0x4 table="filter"
bird: Mesh_172_16_1_93: State changed to start
bird: Mesh_172_16_6_69: Connected to table master
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up
bird: Graceful restart done
bird: Mesh_172_16_1_93: State changed to feed
bird: Mesh_172_16_1_93: State changed to up
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:30.573 [ERROR][48] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.614 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
2020-05-13 08:46:54.615 [ERROR][50] watcher.go 115: Watch channel error error=etcdserver: mvcc: required revision has been compacted
bird: Mesh_172_16_6_69: State changed to start
bird: Mesh_172_16_6_69: State changed to feed
bird: Mesh_172_16_6_69: State changed to up

klj890
现在我这里也有这个namespace了，里面也有节点数据。但是网络还是不行

wanjunlei 不行 我重建这个都不行

wanjunlei 还是一样的

wanjunlei 估计得重建了

klj890 好的 我看一下 谢谢