flannel vxlan, check sum

集群内pod无法dig mysql.middleware 修改主机dns nameserver为集群core dns地址，主机上可以dig mysql.middleware

一、问题分析

从pod内发出的请求数据包到flannel网卡时出现问题

二、抓包分析

环境
类型地址网卡 flannel地址 docker0地址
宿主机 192.168.189.5 eth0 172.30.17.0/32 172.30.17.0/24
宿主机 192.168.189.3 eth0 172.30.44.0/32 172.30.44.0/24

类型	地址	网卡	flannel地址	docker0地址
宿主机	192.168.189.5	eth0	172.30.17.0/32	172.30.17.0/24
宿主机	192.168.189.3	eth0	172.30.44.0/32	172.30.44.0/24

修改192.168.189.5主机的nameserver，主机上，dig mysql.middleware.svc.cluster.local

在192.168.189.3的eth0上抓包

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# tcpdump -i eth0 -nnvvS  udp and   host 192.168.189.5 and  host 192.168.189.3

10:00:45.171729 IP (tos 0x0, ttl 64, id 25021, offset 0, flags [none], proto UDP (17), length 153)
    192.168.189.5.37578 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 64, id 56488, offset 0, flags [none], proto UDP (17), length 103)
    172.30.17.0.6570 > 172.30.44.4.53: [udp sum ok] 30379+ [1au] A? mysql.middleware.svc.cluster.local. ar: . OPT UDPsize=4096 [COOKIE 8dd7b1a036b9e96b] (75)

10:00:45.172148 IP (tos 0x0, ttl 64, id 11850, offset 0, flags [none], proto UDP (17), length 203)
    192.168.189.3.39446 > 192.168.189.5.8472: [bad udp cksum 0x7427 -> 0x6287!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 61091, offset 0, flags [DF], proto UDP (17), length 153)
    172.30.44.4.53 > 172.30.17.0.6570: [bad udp cksum 0x95d7 -> 0x8437!] 30379*- q: A? mysql.middleware.svc.cluster.local. 1/0/1 mysql.middleware.svc.cluster.local. A 10.254.83.174 ar: . OPT UDPsize=2048 DO [COOKIE 8dd7b1a036b9e96b] (125)

在192.168.189.3的flannel.1上抓包

1
2
3
4
5
# tcpdump -i flannel.1 -nnvvS  host 172.30.44.4  or  172.30.44.0 and  host 172.30.17.0

172.30.17.0.28706 > 172.30.44.4.53: [udp sum ok] 20554+ [1au] A? mysql.middleware.svc.cluster.local. ar: . OPT UDPsize=4096 [COOKIE 02885f0fcec139cb] (75)
10:14:09.100234 IP (tos 0x0, ttl 63, id 25272, offset 0, flags [DF], proto UDP (17), length 153)
    172.30.44.4.53 > 172.30.17.0.28706: [bad udp cksum 0x95d7 -> 0xcda2!] 20554*- q: A? mysql.middleware.svc.cluster.local. 1/0/1 mysql.middleware.svc.cluster.local. A 10.254.83.174 ar: . OPT UDPsize=4096 [COOKIE 02885f0fcec139cb] (125)

注意：请求来的时候sum ok，回的时候 bad

在192.168.189.5的一个pod内，dig mysql.middleware.svc.cluster.local

在192.168.189.3的eth0上抓包

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# tcpdump -i eth0 -nnvvS  udp and   host 192.168.189.5 and  host 192.168.189.3

10:20:11.021851 IP (tos 0x0, ttl 64, id 18100, offset 0, flags [none], proto UDP (17), length 153)
    192.168.189.5.54947 > 192.168.189.3.8472: [bad udp cksum 0x694b -> 0x37cc!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 29991, offset 0, flags [none], proto UDP (17), length 103)
    172.30.17.0.42230 > 172.30.44.4.53: [udp sum ok] 38007+ [1au] A? mysql.middleware.svc.cluster.local. ar: . OPT UDPsize=1232 [COOKIE 8313fa6ef3e826cf] (75)

10:20:16.026185 IP (tos 0x0, ttl 64, id 22564, offset 0, flags [none], proto UDP (17), length 153)
    192.168.189.5.29594 > 192.168.189.3.8472: [bad udp cksum 0x3c0a -> 0x9ad5!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 1772, offset 0, flags [none], proto UDP (17), length 103)
    172.30.17.0.60334 > 172.30.44.4.53: [udp sum ok] 38007+ [1au] A? mysql.middleware.svc.cluster.local. ar: . OPT UDPsize=1232 [COOKIE 8313fa6ef3e826cf] (75)

10:20:16.115807 IP (tos 0x0, ttl 64, id 22619, offset 0, flags [none], proto UDP (17), length 78)
    192.168.189.5.58541 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28

10:20:17.139797 IP (tos 0x0, ttl 64, id 22876, offset 0, flags [none], proto UDP (17), length 78)
    192.168.189.5.58541 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28

10:20:18.163773 IP (tos 0x0, ttl 64, id 23179, offset 0, flags [none], proto UDP (17), length 78)
    192.168.189.5.58541 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28

在192.168.189.3的flannel.1上抓包

1
2
3
4
5
# tcpdump -i flannel.1 -nnvvS  host 172.30.44.4  or  172.30.44.0 and  host 172.30.17.0

10:20:16.115829 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28
10:20:17.139812 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28
10:20:18.163795 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.30.44.4 tell 172.30.17.0, length 28

注意：dig的udp请求来的时候sum bad了，似乎被内核网络丢弃了 pod内发送的udp数据包不能正常到达

在192.168.189.5的一个pod内，curl 172.30.44.4:53

在192.168.189.3的eth0上抓包

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# tcpdump -i eth0 -nnvvS  udp and   host 192.168.189.5 and  host 192.168.189.3

10:25:36.561653 IP (tos 0x0, ttl 64, id 27101, offset 0, flags [none], proto UDP (17), length 110)
    192.168.189.5.60121 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 57662, offset 0, flags [DF], proto TCP (6), length 60)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [S], cksum 0x3e44 (correct), seq 1333049522, win 64860, options [mss 1410,sackOK,TS val 115443102 ecr 0,nop,wscale 7], length 0

10:25:36.561757 IP (tos 0x0, ttl 64, id 10781, offset 0, flags [none], proto UDP (17), length 110)
    192.168.189.3.50170 > 192.168.189.5.8472: [bad udp cksum 0x4a95 -> 0x80d1!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [S.], cksum 0x956f (incorrect -> 0xcbab), seq 2031558188, ack 1333049523, win 64308, options [mss 1410,sackOK,TS val 3396801268 ecr 115443102,nop,wscale 7], length 0

10:25:36.561993 IP (tos 0x0, ttl 64, id 27102, offset 0, flags [none], proto UDP (17), length 102)
    192.168.189.5.60121 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 57663, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [.], cksum 0xf37e (correct), seq 1333049523, ack 2031558189, win 507, options [nop,nop,TS val 115443103 ecr 3396801268], length 0

10:25:36.562083 IP (tos 0x0, ttl 64, id 27103, offset 0, flags [none], proto UDP (17), length 179)
    192.168.189.5.60121 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 57664, offset 0, flags [DF], proto TCP (6), length 129)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [P.], cksum 0x9dd2 (correct), seq 1333049523:1333049600, ack 2031558189, win 507, options [nop,nop,TS val 115443103 ecr 3396801268], length 77 [prefix length(18245) != length(75)] (invalid)

10:25:36.562109 IP (tos 0x0, ttl 64, id 10782, offset 0, flags [none], proto UDP (17), length 102)
    192.168.189.3.50170 > 192.168.189.5.8472: [bad udp cksum 0x4a9d -> 0xa86b!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 4244, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [.], cksum 0x9567 (incorrect -> 0xf335), seq 2031558189, ack 1333049600, win 502, options [nop,nop,TS val 3396801269 ecr 115443103], length 0

10:25:38.562312 IP (tos 0x0, ttl 64, id 12090, offset 0, flags [none], proto UDP (17), length 102)
    192.168.189.3.50170 > 192.168.189.5.8472: [bad udp cksum 0x4a9d -> 0xa09a!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 4245, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [F.], cksum 0x9567 (incorrect -> 0xeb64), seq 2031558189, ack 1333049600, win 502, options [nop,nop,TS val 3396803269 ecr 115443103], length 0

10:25:38.562435 IP (tos 0x0, ttl 64, id 28447, offset 0, flags [none], proto UDP (17), length 102)
    192.168.189.5.60121 > 192.168.189.3.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 57665, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [F.], cksum 0xe38e (correct), seq 1333049600, ack 2031558190, win 507, options [nop,nop,TS val 115445103 ecr 3396803269], length 0

10:25:38.562519 IP (tos 0x0, ttl 64, id 12091, offset 0, flags [none], proto UDP (17), length 102)
    192.168.189.3.50170 > 192.168.189.5.8472: [bad udp cksum 0x4a9d -> 0x98c9!] OTV, flags [I] (0x08), overlay 0, instance 1
IP (tos 0x0, ttl 63, id 4246, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [.], cksum 0x9567 (incorrect -> 0xe393), seq 2031558190, ack 1333049601, win 502, options [nop,nop,TS val 3396803269 ecr 115445103], length 0

在192.168.189.3的flannel.1上抓包

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# tcpdump -i flannel.1 -nnvvS  host 172.30.44.4  or  172.30.44.0 and  host 172.30.17.0

三次握手
10:25:36.561678 IP (tos 0x0, ttl 63, id 57662, offset 0, flags [DF], proto TCP (6), length 60)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [S], cksum 0x3e44 (correct), seq 1333049522, win 64860, options [mss 1410,sackOK,TS val 115443102 ecr 0,nop,wscale 7], length 0
10:25:36.561744 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [S.], cksum 0x956f (incorrect -> 0xcbab), seq 2031558188, ack 1333049523, win 64308, options [mss 1410,sackOK,TS val 3396801268 ecr 115443102,nop,wscale 7], length 0
10:25:36.561998 IP (tos 0x0, ttl 63, id 57663, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [.], cksum 0xf37e (correct), seq 1333049523, ack 2031558189, win 507, options [nop,nop,TS val 115443103 ecr 3396801268], length 0

四次挥手
10:25:36.562088 IP (tos 0x0, ttl 63, id 57664, offset 0, flags [DF], proto TCP (6), length 129)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [P.], cksum 0x9dd2 (correct), seq 1333049523:1333049600, ack 2031558189, win 507, options [nop,nop,TS val 115443103 ecr 3396801268], length 77 [prefix length(18245) != length(75)] (invalid)

10:25:36.562105 IP (tos 0x0, ttl 63, id 4244, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [.], cksum 0x9567 (incorrect -> 0xf335), seq 2031558189, ack 1333049600, win 502, options [nop,nop,TS val 3396801269 ecr 115443103], length 0
10:25:38.562279 IP (tos 0x0, ttl 63, id 4245, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [F.], cksum 0x9567 (incorrect -> 0xeb64), seq 2031558189, ack 1333049600, win 502, options [nop,nop,TS val 3396803269 ecr 115443103], length 0

10:25:38.562447 IP (tos 0x0, ttl 63, id 57665, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.17.0.57972 > 172.30.44.4.53: Flags [F.], cksum 0xe38e (correct), seq 1333049600, ack 2031558190, win 507, options [nop,nop,TS val 115445103 ecr 3396803269], length 0
10:25:38.562512 IP (tos 0x0, ttl 63, id 4246, offset 0, flags [DF], proto TCP (6), length 52)
    172.30.44.4.53 > 172.30.17.0.57972: Flags [.], cksum 0x9567 (incorrect -> 0xe393), seq 2031558190, ack 1333049601, win 502, options [nop,nop,TS val 3396803269 ecr 115445103], length 0

pod内发送的tcp数据包可以正常发送

三、解决方案，关闭外层udp数据包检测

要在Linux中设置网卡或内核参数以允许数据包不进行校验和计算，可以使用ethtool工具或sysctl进行配置。

2.1、使用ethtool工具（推荐）

首先，确保你已经安装了ethtool工具。如果没有安装，可以通过包管理器进行安装，比如在Ubuntu上可以使用以下命令：
1
sudo apt-get install ethtool
然后，使用ethtool来设置网卡参数。例如，要禁用网卡的校验和计算，可以使用以下命令：
1
sudo ethtool -K eth0 tx off rx off
这将禁用eth0网卡的发送和接收校验和计算。

2.2、使用sysctl设置内核参数（测试了这种方式似乎不生效）

打开文件/etc/sysctl.conf：
1
sudo nano /etc/sysctl.conf
在文件末尾添加以下行以禁用校验和：
1
net.ipv4.tcp_checksum = 0
保存并关闭文件后，执行以下命令使更改生效：
1
sudo sysctl -p

Table of Contents

一、问题分析

二、抓包分析

三、解决方案，关闭外层udp数据包检测

2.1、使用ethtool工具（推荐）

2.2、使用sysctl设置内核参数（测试了这种方式似乎不生效）