代码拉取完成，页面将自动刷新
【特性描述】
支持hns roce根据申请的页面的实际情况去动态计算PBL表的page size以及PBL表的hopnum。
【特性竞争力】
该特性能够减小RoCEE硬件访问MR时的寻址级数，从而减少时延。
【硬件架构】
鲲鹏板载网卡ROCEE
【特性约束】
不支持FRMR类型的MR，其余操作都支持
【涉及仓库】
driver/inifiband/hw/hns
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
涉及基础功能进行压力，测试脚本：
#!/bin/bash
set -x
set -e
while [ 1 ]
do
a=$RANDOM
b=$RANDOM
pg_sz=`expr $a \* $b`
pg_sz=`expr $pg_sz % 2147483648`
sleep 2
ib_send_bw -d hns_0 -s $pg_sz &
sleep 2
ib_send_bw -d hns_0 -s $pg_sz 192.168.100.100
sleep 2
ib_write_bw -d hns_0 -s $pg_sz &
sleep 2
ib_write_bw -d hns_0 -s $pg_sz 192.168.100.100
sleep 2
ib_read_bw -d hns_0 -s $pg_sz &
sleep 2
ib_read_bw -d hns_0 -s $pg_sz 192.168.100.100
done
压力测试部分日志：
+ ib_write_bw -d hns_0 -s 56541943
************************************
* Waiting for client to connect... *
************************************
+ ib_write_bw -d hns_0 -s 56541943 192.168.100.100
-------------------------------------------------------------------- -------------------
RDMA_Write BW Test
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
-------------------------------------------------------------------- -------------------
local address: LID 0000 QPN 0x03d8 PSN 0x6283be RKey 0x000200 VAddr 0x00ffffb16402f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
-------------------------------------------------------------------- -------------------
RDMA_Write BW Test
Dual-port : OFF Device : hns_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
-------------------------------------------------------------------- -------------------
local address: LID 0000 QPN 0x03d9 PSN 0x3745ae RKey 0x000300 VAddr 0x00ffff8d3c62f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x03d8 PSN 0x6283be RKey 0x000200 VAdd r 0x00ffffb16402f7
remote address: LID 0000 QPN 0x03d9 PSN 0x3745ae RKey 0x000300 VAdd r 0x00ffff8d3c62f7
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
-------------------------------------------------------------------- -------------------
-------------------------------------------------------------------- -------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] M sgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] M sgRate[Mpps]
56541943 5000 2717.02 2714.78 0.000050
-------------------------------------------------------------------- -------------------
56541943 5000 2717.02 2714.78 0.000050
-------------------------------------------------------------------- -------------------
查看自适应的效果，利用perftest测试有概率触发page size的变化。利用rdma tool进行查询，不同的mr的drv_buf_pg_shift可能不一样
root:/root/$ ib_write_bw -d hns_0 -s 1 --run_infinite &
[1] 14995
root:/root/$
************************************
* Waiting for client to connect... *
************************************
root:/root/$ ib_write_bw -d hns_0 -s 1 --run_infinite 192.168.100.100 &
[2] 14996
root:/root/$
root:/root/$ ---------------------------------------------------------------------------------------
RDMA_Write BW Test
---------------------------------------------------------------------------------------
Dual-port : OFF Device : hns_0
RDMA_Write BW Test
Number of qps : 1 Transport type : IB
Dual-port : OFF Device : hns_0
Connection type : RC Using SRQ : OFF
Number of qps : 1 Transport type : IB
TX depth : 128
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
CQ Moderation : 100
Mtu : 1024[B]
Mtu : 1024[B]
Link type : Ethernet
Link type : Ethernet
GID index : 2
GID index : 2
Max inline data : 0[B]
Max inline data : 0[B]
rdma_cm QPs : OFF
rdma_cm QPs : OFF
Data ex. method : Ethernet
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x5b80 PSN 0xdd83b8 RKey 0x000300 VAddr 0x00aaab08f64000
local address: LID 0000 QPN 0x5b71 PSN 0x618f77 RKey 0x000200 VAddr 0x00aaaadacc5000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x5b80 PSN 0xdd83b8 RKey 0x000300 VAddr 0x00aaab08f64000
remote address: LID 0000 QPN 0x5b71 PSN 0x618f77 RKey 0x000200 VAddr 0x00aaaadacc5000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
root:/root/$ rdma res show mr -ddd -j -p
[ {
"ifindex": 0,
"ifname": "hns_0",
"mrn": 5851,
"rkey": "200",
"lkey": "200",
"mrlen": 8192,
"pdn": 5852,
"pid": 14995,
"comm": "ib_write_bw",{
"drv_pbl_hop_num": 1,
"drv_ba_pg_shift": 14,
"drv_buf_pg_shift": 12
}
},{
"ifindex": 0,
"ifname": "hns_0",
"mrn": 5852,
"rkey": "300",
"lkey": "300",
"mrlen": 8192,
"pdn": 5853,
"pid": 14996,
"comm": "ib_write_bw",{
"drv_pbl_hop_num": 1,
"drv_ba_pg_shift": 14,
"drv_buf_pg_shift": 14
}
} ]
大页测试，大页测试时，hopnum变为0
[root@localhost ~]# echo 1000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
[root@localhost ~]# mkdir -p /mnt/huge
[root@localhost ~]#
[root@localhost ~]# mount -t hugetlbfs nodev /mnt/huge
[root@localhost ~]#
[root@localhost ~]# cat /proc/meminfo
MemTotal: 129875076 kB
MemFree: 123508852 kB
MemAvailable: 123254328 kB
Buffers: 26080 kB
Cached: 289552 kB
SwapCached: 0 kB
Active: 84452 kB
Inactive: 307200 kB
Active(anon): 4120 kB
Inactive(anon): 83824 kB
Active(file): 80332 kB
Inactive(file): 223376 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 76020 kB
Mapped: 52532 kB
Shmem: 11924 kB
KReclaimable: 173228 kB
Slab: 923648 kB
SReclaimable: 173228 kB
SUnreclaim: 750420 kB
KernelStack: 25152 kB
PageTables: 2208 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 68107836 kB
Committed_AS: 348420 kB
VmallocTotal: 135290159040 kB
VmallocUsed: 156600 kB
VmallocChunk: 0 kB
Percpu: 157184 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 32768 kB
CmaFree: 3344 kB
HugePages_Total: 1000
HugePages_Free: 1000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 2048000 kB
[root@localhost ~]# ib_send_bw -d hns_0 -s 1 --use_hugepages --run_infinite &
[1] 24612
[root@localhost ~]#
************************************
* Waiting for client to connect... *
************************************
[root@localhost ~]# ib_send_bw -d hns_0 -s 1 --use_hugepages --run_infinite 192.168.100.100 &
[2] 24613
[root@localhost ~]# ---------------------------------------------------------------------------------------
Send BW Test
---------------------------------------------------------------------------------------
Dual-port : OFF Device : hns_0
Send BW Test
Number of qps : 1 Transport type : IB
Dual-port : OFF Device : hns_0
Connection type : RC Using SRQ : OFF
Number of qps : 1 Transport type : IB
RX depth : 512
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
TX depth : 128
Mtu : 1024[B]
CQ Moderation : 100
Link type : Ethernet
Mtu : 1024[B]
GID index : 2
Link type : Ethernet
Max inline data : 0[B]
GID index : 2
rdma_cm QPs : OFF
Max inline data : 0[B]
Data ex. method : Ethernet
rdma_cm QPs : OFF
---------------------------------------------------------------------------------------
Data ex. method : Ethernet
local address: LID 0000 QPN 0x0020 PSN 0xa737f0
---------------------------------------------------------------------------------------
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
local address: LID 0000 QPN 0x0021 PSN 0xd82fac
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
remote address: LID 0000 QPN 0x0020 PSN 0xa737f0
remote address: LID 0000 QPN 0x0021 PSN 0xd82fac
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:100
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
[root@localhost ~]# 1 20551800 0.00 3.92 4.110358
[root@localhost ~]# rdma res show mr -ddd -j -p
[ {
"ifindex": 14,
"ifname": "hns_0",
"mrn": 4,
"rkey": "200",
"lkey": "200",
"mrlen": 8192,
"pdn": 5,
"pid": 24613,
"comm": "ib_send_bw",{
"drv_pbl_hop_num": 0,
"drv_ba_pg_shift": 13,
"drv_buf_pg_shift": 13
}
},{
"ifindex": 14,
"ifname": "hns_0",
"mrn": 5,
"rkey": "300",
"lkey": "300",
"mrlen": 8192,
"pdn": 6,
"pid": 24612,
"comm": "ib_send_bw",{
"drv_pbl_hop_num": 0,
"drv_ba_pg_shift": 13,
"drv_buf_pg_shift": 13
}
} ]
登录 后才可以发表评论
FileDragTip