代码拉取完成，页面将自动刷新
【特性描述】
支持hns roce根据申请的页面的实际情况去动态计算PBL表的page size以及PBL表的hopnum。
【特性竞争力】
该特性能够减小RoCEE硬件访问MR时的寻址级数，从而减少时延。
【硬件架构】
鲲鹏板载网卡ROCEE
【特性约束】
不支持FRMR类型的MR，其余操作都支持
【涉及仓库】
driver/inifiband/hw/hns
Hi hellotcc, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: Kernel, and any of the maintainers: @yangyingliang , @gatieme , @jiaoff , @zhengzengkai , @LiuYongQiang0816 , @wangxiongfeng , @kevinzhu1 , @jentlestea , @lujialin2 , @wuxu_buque , @xukuohai , @lengchao , @alvin-ling , @yuzenghui , @yuehaibing , @juntianlinux , @chenguangli , @gasonchen , @whoisxxx , @koulihong , @kailiu42 , @hanjun-guo , @woqidaideshi , @chiqijun , @wkfxxx , @thundertown , @guohaocs2c , @kylin-mayukun , @oskernel0719 , @liuxinux , @newbeats , @zhujianwei001 , @zhenpengzheng , @SuperSix173 , @colyli , @zhangyi089 , @htforge , @qiuuuuu , @xiexiuqi
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
涉及基础功能进行压力，测试脚本：
#!/bin/bash
set -x
set -e
while [ 1 ]
do
a=$RANDOM
b=$RANDOM
pg_sz=`expr $a \* $b`
pg_sz=`expr $pg_sz % 2147483648`
sleep 2
ib_send_bw -d hns_0 -s $pg_sz &
sleep 2
ib_send_bw -d hns_0 -s $pg_sz 192.168.100.100
sleep 2
ib_write_bw -d hns_0 -s $pg_sz &
sleep 2
ib_write_bw -d hns_0 -s $pg_sz 192.168.100.100
sleep 2
ib_read_bw -d hns_0 -s $pg_sz &
sleep 2
ib_read_bw -d hns_0 -s $pg_sz 192.168.100.100
done
压力测试部分日志：
+ ib_write_bw -d hns_0 -s 56541943
************************************
* Waiting for client to connect... *
************************************
+ ib_write_bw -d hns_0 -s 56541943 192.168.100.100
-------------------------------------------------------------------- -------------------
RDMA_Write BW Test
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
-------------------------------------------------------------------- -------------------
local address: LID 0000 QPN 0x03d8 PSN 0x6283be RKey 0x000200 VAddr 0x00ffffb16402f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
-------------------------------------------------------------------- -------------------
RDMA_Write BW Test
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
-------------------------------------------------------------------- -------------------
local address: LID 0000 QPN 0x03d9 PSN 0x3745ae RKey 0x000300 VAddr 0x00ffff8d3c62f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x03d8 PSN 0x6283be RKey 0x000200 VAdd r 0x00ffffb16402f7
remote address: LID 0000 QPN 0x03d9 PSN 0x3745ae RKey 0x000300 VAdd r 0x00ffff8d3c62f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
-------------------------------------------------------------------- -------------------
-------------------------------------------------------------------- -------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] M sgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] M sgRate[Mpps]
56541943 5000 2717.02 2714.78 0.000050
-------------------------------------------------------------------- -------------------
56541943 5000 2717.02 2714.78 0.000050
-------------------------------------------------------------------- -------------------
查看自适应的效果，利用perftest测试有概率触发page size的变化。利用rdma tool进行查询，不同的mr的drv_buf_pg_shift可能不一样
root:/root/$ ib_write_bw -d hns_0 -s 1 --run_infinite &
[1] 14995
root:/root/$
************************************
* Waiting for client to connect... *
************************************
root:/root/$ ib_write_bw -d hns_0 -s 1 --run_infinite 192.168.100.100 &
[2] 14996
root:/root/$
root:/root/$ ---------------------------------------------------------------------------------------
RDMA_Write BW Test
---------------------------------------------------------------------------------------
Dual-port : OFF Device : hns_0
RDMA_Write BW Test
Number of qps : 1 Transport type : IB
Dual-port : OFF Device : hns_0
Connection type : RC Using SRQ : OFF
Number of qps : 1 Transport type : IB
TX depth : 128
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
CQ Moderation : 100
Mtu : 1024[B]
Mtu : 1024[B]
Link type : Ethernet
Link type : Ethernet
GID index : 2
GID index : 2
Max inline data : 0[B]
Max inline data : 0[B]
rdma_cm QPs : OFF
rdma_cm QPs : OFF
Data ex. method : Ethernet
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x5b80 PSN 0xdd83b8 RKey 0x000300 VAddr 0x00aaab08f64000
local address: LID 0000 QPN 0x5b71 PSN 0x618f77 RKey 0x000200 VAddr 0x00aaaadacc5000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x5b80 PSN 0xdd83b8 RKey 0x000300 VAddr 0x00aaab08f64000
remote address: LID 0000 QPN 0x5b71 PSN 0x618f77 RKey 0x000200 VAddr 0x00aaaadacc5000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
root:/root/$ rdma res show mr -ddd -j -p
[ {
"ifindex": 0,
"ifname": "hns_0",
"mrn": 5851,
"rkey": "200",
"lkey": "200",
"mrlen": 8192,
"pdn": 5852,
"pid": 14995,
"comm": "ib_write_bw",{
"drv_pbl_hop_num": 1,
"drv_ba_pg_shift": 14,
"drv_buf_pg_shift": 12
}
},{
"ifindex": 0,
"ifname": "hns_0",
"mrn": 5852,
"rkey": "300",
"lkey": "300",
"mrlen": 8192,
"pdn": 5853,
"pid": 14996,
"comm": "ib_write_bw",{
"drv_pbl_hop_num": 1,
"drv_ba_pg_shift": 14,
"drv_buf_pg_shift": 14
}
} ]
大页测试，大页测试时，hopnum变为0
[root@localhost ~]# echo 1000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
[root@localhost ~]# mkdir -p /mnt/huge
[root@localhost ~]#
[root@localhost ~]# mount -t hugetlbfs nodev /mnt/huge
[root@localhost ~]#
[root@localhost ~]# cat /proc/meminfo
MemTotal: 129875076 kB
MemFree: 123508852 kB
MemAvailable: 123254328 kB
Buffers: 26080 kB
Cached: 289552 kB
SwapCached: 0 kB
Active: 84452 kB
Inactive: 307200 kB
Active(anon): 4120 kB
Inactive(anon): 83824 kB
Active(file): 80332 kB
Inactive(file): 223376 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 76020 kB
Mapped: 52532 kB
Shmem: 11924 kB
KReclaimable: 173228 kB
Slab: 923648 kB
SReclaimable: 173228 kB
SUnreclaim: 750420 kB
KernelStack: 25152 kB
PageTables: 2208 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 68107836 kB
Committed_AS: 348420 kB
VmallocTotal: 135290159040 kB
VmallocUsed: 156600 kB
VmallocChunk: 0 kB
Percpu: 157184 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 32768 kB
CmaFree: 3344 kB
HugePages_Total: 1000
HugePages_Free: 1000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 2048000 kB
[root@localhost ~]# ib_send_bw -d hns_0 -s 1 --use_hugepages --run_infinite &
[1] 24612
[root@localhost ~]#
************************************
* Waiting for client to connect... *
************************************
[root@localhost ~]# ib_send_bw -d hns_0 -s 1 --use_hugepages --run_infinite 192.168.100.100 &
[2] 24613
[root@localhost ~]# ---------------------------------------------------------------------------------------
Send BW Test
---------------------------------------------------------------------------------------
Dual-port : OFF Device : hns_0
Send BW Test
Number of qps : 1 Transport type : IB
Dual-port : OFF Device : hns_0
Connection type : RC Using SRQ : OFF
Number of qps : 1 Transport type : IB
RX depth : 512
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
TX depth : 128
Mtu : 1024[B]
CQ Moderation : 100
Link type : Ethernet
Mtu : 1024[B]
GID index : 2
Link type : Ethernet
Max inline data : 0[B]
GID index : 2
rdma_cm QPs : OFF
Max inline data : 0[B]
Data ex. method : Ethernet
rdma_cm QPs : OFF
---------------------------------------------------------------------------------------
Data ex. method : Ethernet
local address: LID 0000 QPN 0x0020 PSN 0xa737f0
---------------------------------------------------------------------------------------
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
local address: LID 0000 QPN 0x0021 PSN 0xd82fac
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x0020 PSN 0xa737f0
remote address: LID 0000 QPN 0x0021 PSN 0xd82fac
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
[root@localhost ~]# 1 20551800 0.00 3.92 4.110358
[root@localhost ~]# rdma res show mr -ddd -j -p
[ {
"ifindex": 14,
"ifname": "hns_0",
"mrn": 4,
"rkey": "200",
"lkey": "200",
"mrlen": 8192,
"pdn": 5,
"pid": 24613,
"comm": "ib_send_bw",{
"drv_pbl_hop_num": 0,
"drv_ba_pg_shift": 13,
"drv_buf_pg_shift": 13
}
},{
"ifindex": 14,
"ifname": "hns_0",
"mrn": 5,
"rkey": "300",
"lkey": "300",
"mrlen": 8192,
"pdn": 6,
"pid": 24612,
"comm": "ib_send_bw",{
"drv_pbl_hop_num": 0,
"drv_ba_pg_shift": 13,
"drv_buf_pg_shift": 13
}
} ]
登录 后才可以发表评论
FileDragTip