ftp客户端来连lvs节点，lvs节点panic

【标题描述】
ftp客户端如FileZilla用21端口来建连，经过lvs节点(nat模式)转发到后端（ftp服务端），某些场景下会导致lvs节点panic重启。

【环境信息】
硬件信息：
与具体硬件无关

软件信息：
组网：ftp客户端（如PC版的：FileZilla）===>lvs节点===>后端（ftp服务端）

```
# cat /etc/os-release 
NAME="openEuler"
VERSION="20.03 (LTS-SP3)"
ID="openEuler"
VERSION_ID="20.03"
PRETTY_NAME="openEuler 20.03 (LTS-SP3)"
ANSI_COLOR="0;31"
```

```
# uname -a
Linux openEuler 4.19.90-2112.8.0.0131.oe1.aarch64 #1 SMP Fri Dec 31 19:53:20 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
```

【问题复现步骤】
1、组网：
ftp客户端（如PC版的：FileZilla）===>lvs节点===>后端节点（ftp服务端）

2、lvs节点配置：（nat模式）
（1）关闭firewalld，关闭selinux，避免不必要的干扰
systemctl stop firewalld
setenforce 0

（2）装ipvsadm的rpm包
yum install ipvsadm

（3）还需配置另外一个ip-A（简称虚拟ip）用于跟ftp客户端通信（备注：原来的ip-B跟后端ftp服务端节点通信，简称local-ip）
配置虚拟ip-A：若在eth0网卡上，掩码16位。
`ip a add $ip-A/16 dev eth0:1`

（4）配置lvs-nat模式转发
配置虚拟服务：
`ipvsadm -A   -t  $ip-A:21   -s  rr`
配置真实的服务器：
`ipvsadm -a   -t  $ip-A:21   -r  $ftp服务端ip:21    -m`

(5)把ip_forward转发打开
sysctl -w net.ipv4.ip_forward=1

(6)加载ip_vs_ftp.ko
modprobe ip_vs_ftp

备注：lvs节点需未加载nf_nat_ipv4.ko

3、后端节点（ftp服务端）配置：
（1）关firewalld，关闭selinux
systemctl stop firewalld.service
setenforce 0

（2）配置gateway为上述lvs节点的ip-B，将报文回给lvs节点。
原理：将ftp-client的地址（如client-ip）回给lvs节点的local-ip上（ip-B）。
若网口为eth0，掩码为24位：
ip route add   client-ip网段/24  via $ip-B  dev eth0

（3）安装vsftpd包并开启ftp服务端服务
yum install vsftpd
systemctl restart vsftpd

（4）新增用户ftptest且设置密码（不能用已有的root，因为vsftpd服务默认禁用了root）
useradd ftptest
passwd ftptest

4、用FileZilla来连接ftpd服务端
![输入图片说明](https://foruda.gitee.com/images/1690337943383421629/6c334f20_13243789.png "屏幕截图")

出现概率：必现

【预期结果】
ftp功能正常，lvs节点正常。

【实际结果】
lvs节点panic重启，ftp功能异常

【附件信息】
[ 1967.469504] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[ 1967.470334] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[ 1967.471834] IPVS: ipvs loaded.
[ 1967.480075] IPVS: [rr] scheduler registered.
[ 2057.855925] IPVS: ftp: loaded support on port[0] = 21
[ 3514.485173] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
[ 3514.486388] Mem abort info:
[ 3514.486761]   ESR = 0x96000005
[ 3514.487155]   Exception class = DABT (current EL), IL = 32 bits
[ 3514.487908]   SET = 0, FnV = 0
[ 3514.488301]   EA = 0, S1PTW = 0
[ 3514.488702] Data abort info:
[ 3514.489084]   ISV = 0, ISS = 0x00000005
[ 3514.489574]   CM = 0, WnR = 0
[ 3514.489957] user pgtable: 64k pages, 48-bit VAs, pgdp = 00000000c5a493e7
[ 3514.490815] [0000000000000028] pgd=0000000000000000, pud=0000000000000000
[ 3514.491678] Internal error: Oops: 96000005 [#1] SMP
[ 3514.492304] Modules linked in: ip_vs_ftp nf_nat binfmt_misc ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 fuse msdos xfs devlink tun scsi_transport_iscsi bridge stp llc libcrc32c rfkill ip_set nfnetlink sunrpc vfat fat aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce sg sch_fq_codel ext4 mbcache jbd2 sd_mod virtio_scsi virtio_gpu virtio_console virtio_net net_failover failover virtio_pci virtio_mmio virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
[ 3514.498897] Process swapper/3 (pid: 0, stack limit = 0x0000000033a46a05)
[ 3514.499818] CPU: 3 PID: 0 Comm: swapper/3 Kdump: loaded Not tainted 4.19.90-2112.8.0.0131.oe1.aarch64 #1
[ 3514.501105] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 3514.502037] pstate: 80000005 (Nzcv daif -PAN -UAO)
[ 3514.502717] pc : __nf_nat_mangle_tcp_packet+0xb4/0x1c0 [nf_nat]
[ 3514.503533] lr : __nf_nat_mangle_tcp_packet+0xb4/0x1c0 [nf_nat]
[ 3514.504349] sp : ffff00000806f690
[ 3514.504817] x29: ffff00000806f690 x28: 0000000000000045 
[ 3514.505550] x27: 0000000000000003 x26: 0000000000000001 
[ 3514.506279] x25: ffff800024dc0d80 x24: 0000000000000001 
[ 3514.507023] x23: ffff800025c9dc62 x22: 0000000000000046 
[ 3514.507757] x21: 0000000000000013 x20: 0000000000000012 
[ 3514.508483] x19: ffff8000fdfdb600 x18: 0000000000000000 
[ 3514.509212] x17: 0000000000000001 x16: 0000000000000000 
[ 3514.509943] x15: ffff000008e44b48 x14: ffffffffffffffff 
[ 3514.510673] x13: 0000000000000018 x12: 0101010101010101 
[ 3514.511413] x11: 7f7fffffffff7f7f x10: 4909d3f0f5da60ff 
[ 3514.512149] x9 : ffffff7f7f7f7f7f x8 : 2c3634322c30312c 
[ 3514.512882] x7 : 3930322c32382c38 x6 : ffff800025c9dca4 
[ 3514.513612] x5 : 0000000000000013 x4 : 000000000000005a 
[ 3514.514340] x3 : 0000000003a20008 x2 : ffff800025c9dc62 
[ 3514.515105] x1 : ffff000001650258 x0 : 0000000000000000 
[ 3514.515848] Call trace:
[ 3514.516197]  __nf_nat_mangle_tcp_packet+0xb4/0x1c0 [nf_nat]
[ 3514.516969]  ip_vs_ftp_out.part.2+0x270/0x410 [ip_vs_ftp]
[ 3514.517709]  ip_vs_ftp_out+0x70/0x80 [ip_vs_ftp]
[ 3514.518349]  ip_vs_app_pkt_out+0x17c/0x250 [ip_vs]
[ 3514.519020]  tcp_snat_handler+0x2f4/0x3d8 [ip_vs]
[ 3514.519669]  handle_response+0x68/0x260 [ip_vs]
[ 3514.520296]  ip_vs_out.part.7+0x184/0x658 [ip_vs]
[ 3514.520945]  ip_vs_out+0x88/0xc8 [ip_vs]
[ 3514.521486]  ip_vs_reply4+0x34/0x40 [ip_vs]
[ 3514.522062]  nf_hook_slow+0x50/0x100
[ 3514.522553]  ip_forward+0x460/0x508
[ 3514.523052]  ip_rcv_finish+0x90/0xb0
[ 3514.523543]  ip_rcv+0x64/0xd8
[ 3514.523953]  __netif_receive_skb_one_core+0x6c/0x90
[ 3514.524628]  __netif_receive_skb+0x28/0x80
[ 3514.525210]  netif_receive_skb_internal+0x50/0xd8
[ 3514.525856]  napi_gro_receive+0xf8/0x170
[ 3514.526408]  receive_buf+0x2b0/0x6c8 [virtio_net]
[ 3514.527072]  virtnet_poll+0x170/0x370 [virtio_net]
[ 3514.527741]  net_rx_action+0x188/0x408
[ 3514.528269]  __do_softirq+0x128/0x344
[ 3514.528787]  irq_exit+0x11c/0x128
[ 3514.529249]  __handle_domain_irq+0x6c/0xc0
[ 3514.529826]  gic_handle_irq+0x78/0x180
[ 3514.530356]  el1_irq+0xb8/0x140
[ 3514.530817]  arch_cpu_idle+0x38/0x1d0
[ 3514.531348]  default_idle_call+0x30/0x68
[ 3514.531908]  do_idle+0x1ec/0x2c8
[ 3514.532370]  cpu_startup_entry+0x2c/0x88
[ 3514.532919]  secondary_start_kernel+0x1a4/0x1e8
[ 3514.533549] Code: b9408264 3940cb20 4b160096 97fffd4a (f9401406) 
[ 3514.534394] SMP: stopping secondary CPUs
[ 3514.537157] Starting crashdump kernel...
[ 3514.537714] Bye!

备注：
分析：
__nf_nat_mangle_tcp_packet()函数里之前是函数指针查找，如下，若环境未加载nf_nat_ipv4.ko则l3proto为空指针，故panic。

```
l3proto = __nf_nat_l3proto_find(nf_ct_l3num(ct));
l3proto->csum_recalc(skb, IPPROTO_TCP, tcph, &tcph->check,
		     datalen, oldlen);
```

社区高版本现改成了直接调用nf_nat_csum_recalc()函数：
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_nat_helper.c

请合入patch：（需要适配）
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dac3fe72596f91011afc649a9d181b18466dd895
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6bac76db1da3cb162c425d58ae421486f8e43955

GVP openEuler/kernel

内容风险标识

评论 (1)

GVPopenEuler/kernel

内容风险标识