# storprototrace **Repository Path**: openeuler/storprototrace ## Basic Information - **Project Name**: storprototrace - **Description**: Storprototrace (storage protocol trace) is a tracing function for IO events entering the iscsi protocol driver layer based on libbpf. - **Primary Language**: Unknown - **License**: MulanPSL-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 2 - **Forks**: 6 - **Created**: 2024-11-25 - **Last Updated**: 2025-01-22 ## Categories & Tags **Categories**: Uncategorized **Tags**: storage ## README # **storprototrace简介** storprototrace是基于libbpf实现的对进入iscsi协议驱动层的I/O事件的追踪功能，相比于 blktrace，本工具主要用于统计I/O进入iscsi协议驱动层后的各阶段时延，主要包括： + 队列排队等待时间 + I/O 发送时间 + I/O传输完成时间 实现了I/O在iscsi协议驱动层中不同阶段的时延汇总记录，可以更细致地了解I/O请求进入iscsi协议驱动层的各阶段时延，识别性能瓶颈，并为问题定位和系统优化提供方向。 # iscsi协议驱动层新增统计项详细描述 + 队列排队等待阶段 描述：统计I/O请求在设备队列中等待执行的时间，即从iscsi协议驱动层接收到请求到开始处理请求的时间差 目的：识别设备内部队列的排队延迟，优化排队时间 + I/O发送阶段 描述：统计设备实际处理I/O请求的时间，从开始处理请求到发送I/O之间的耗时 目的：计算设备处理I/O请求的时延，优化设备处理效率 + I/O传输完成阶段 描述：统计I/O请求实际处理完成的时间，从I/O请求发送到接收到应答之前的耗时 目的：计算I/O请求发送到完成处理的时延，分析传输过程是否异常 # 设计实现流程图 细化I/O请求在iscsi协议驱动层的处理过程 ![](https://yueryoufeng.oss-cn-beijing.aliyuncs.com/img_public/%E5%9B%BE%E7%89%871.png) 使用ebpf去监控I/O请求在iscsi协议驱动层各阶段的处理时延 ![](https://yueryoufeng.oss-cn-beijing.aliyuncs.com/img_public/%E5%9B%BE%E7%89%872.png) # ISCSI环境部署安装 安装两台机器，安装系统阶段将主机名分别设置为tester和target，tester用于客户端，target用于提供iscsi资源。 分别关闭两台机器的firewalld。 ``` [root@target ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@tester ~]# systemctl disable --now firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. ``` 为target添加一块2G大小的scsi磁盘。 可以看到新添加的磁盘设备名为/dev/sdb。 ``` [root@target ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 29G 0 part ├─klas-root 253:0 0 26G 0 lvm / └─klas-swap 253:1 0 3G 0 lvm [SWAP] sdb 8:16 0 2G 0 disk sr0 11:0 1 1024M 0 rom ``` 在tester上安装open-iscsi软件包。 ``` [root@tester ~]# dnf install -y open-iscsi ``` 在tester上修改/etc/iscsi/initiatorname.iscsi文件。 ``` [root@tester ~]# vim /etc/iscsi/initiatorname.iscsi [root@tester ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2024-03.cn.eu:611562999633 ``` 在target上安装targetcli软件包。 ``` [root@target ~]# dnf install targetcli ``` ``` [root@target ~]# targetcli Warning: Could not load preferences file /root/.targetcli/prefs.bin. targetcli shell version 2.1.54 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0] /> /backstores/block create iscsitest /dev/sdb Created block storage object iscsitest using /dev/sdb. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iscsitest ...................................................................... [/dev/sdb (2.0GiB) write-thru deactivated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0] /> /iscsi create iqn.2024-03.cn.eu Created target iqn.2024-03.cn.eu. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iscsitest ...................................................................... [/dev/sdb (2.0GiB) write-thru deactivated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2024-03.cn.eu ............................................................................................ [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 0] | o- luns .......................................................................................................... [LUNs: 0] | o- portals .................................................................................................... [Portals: 1] | o- 0.0.0.0:3260 ..................................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0] ``` 将iqn和设备关联在一起。 ``` /> /iscsi/iqn.2024-03.cn.eu/tpg1/luns create /backstores/block/iscsitest Created LUN 0. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iscsitest ........................................................................ [/dev/sdb (2.0GiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2024-03.cn.eu ............................................................................................ [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 0] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 .................................................................. [block/iscsitest (/dev/sdb) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- 0.0.0.0:3260 ..................................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0] ``` 为tester增加访问权限。 ``` /> /iscsi/iqn.2024-03.cn.eu/tpg1/acls create iqn.2024-03.cn.eu:611562999633 Created Node ACL for iqn.2024-03.cn.eu:611562999633 Created mapped LUN 0. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iscsitest ........................................................................ [/dev/sdb (2.0GiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2024-03.cn.eu ............................................................................................ [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.2024-03.cn.eu:611562999633 .................................................................. [Mapped LUNs: 1] | | o- mapped_lun0 ............................................................................. [lun0 block/iscsitest (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 .................................................................. [block/iscsitest (/dev/sdb) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- 0.0.0.0:3260 ..................................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0] ``` 基于lun0为tester创建第2个mapped lun。 ``` /> cd /iscsi/iqn.2024-03.cn.eu/tpg1/acls/iqn.2024-03.cn.eu:611562999633/ /iscsi/iqn.20...:611562999633> ls o- iqn.2024-03.cn.eu:611562999633 ............................................................................ [Mapped LUNs: 1] o- mapped_lun0 ....................................................................................... [lun0 block/iscsitest (rw)] /iscsi/iqn.20...:611562999633> create 1 lun0 Warning: TPG LUN 0 already mapped to this NodeACL Created Mapped LUN 1. /iscsi/iqn.20...:611562999633> ls o- iqn.2024-03.cn.eu:611562999633 ............................................................................ [Mapped LUNs: 2] o- mapped_lun0 ....................................................................................... [lun0 block/iscsitest (rw)] o- mapped_lun1 ....................................................................................... [lun0 block/iscsitest (rw)] ``` 退出targetcli配置。 ``` /iscsi/iqn.20...:611562999633> exit Global pref auto_save_on_exit=true Configuration saved to /etc/target/saveconfig.json ``` 为了防止target机器重启后targetcli配置丢失，需要enable target服务。 ``` [root@target ~]# systemctl enable target Created symlink /etc/systemd/system/multi-user.target.wants/target.service → /usr/lib/systemd/system/target.service. ``` 在tester启动并enable iscsid服务。 ``` [root@tester ~]# systemctl enable --now iscsid Created symlink /etc/systemd/system/multi-user.target.wants/iscsid.service → /usr/lib/systemd/system/iscsid.service. ``` 在tester登录iscsi。 ``` [root@tester ~]# iscsiadm -m discovery -p 192.168.1.43 -t sendtargets 192.168.1.43:3260,1 iqn.2024-03.cn.eu [root@tester ~]# iscsiadm -m node -T iqn.2024-03.cn.eu -l Logging in to [iface: default, target: iqn.2024-03.cn.eu, portal: 192.168.1.43,3260] Login to [iface: default, target: iqn.2024-03.cn.eu, portal: 192.168.1.43,3260] successful. ``` 192.168.1.43为target的ip地址。 可以看到tester上新增的磁盘设备为/dev/sdb和/dev/sdc,且其wwid相同，即sdb，sdc对应同一个实际存储块设备。 ``` [root@tester ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 600M 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 28.4G 0 part ├─klas-root 253:0 0 25.4G 0 lvm / └─klas-swap 253:1 0 3G 0 lvm [SWAP] sdb 8:16 0 2G 0 disk sdc 8:32 0 2G 0 disk sr0 11:0 1 1024M 0 rom [root@tester ~]# /lib/udev/scsi_id -g -u /dev/sdb 360014050df36420411f4766b22193573 [root@tester ~]# /lib/udev/scsi_id -g -u /dev/sdc 360014050df36420411f4766b22193573 ``` # 代码编译、安装、运行 首次编译时，需要获取clang和libbpf等依赖，可通过运行以下命令解决编译依赖问题 ./install-dep.sh 可通过运行以下命令进行编译安装 mkdir build cd build cmake .. make 可通过以下命令进行运行 ./storprototrace ``` [root@localhost build]# ./storprototrace BPF program loaded and attached successfully. Connect | RW | Toal Interval(ns) | Max Interval(ns) sid cid | Count total | Waiting Sending Complete | Waiting Sending Complete 1 0 | 1 8192 | 70035(1) 972210(1) 1042245(1) | 70035 972210 1042245 1 0 | 4 36864 | 25012(4) 597588(4) 622600(4) | 40967 1159767 1195784 1 0 | 3 32768 | 35049(3) 1052350(2) 1087399(3) | 18446740847831477010 1087001 1776127 1 0 | 1 8192 | 31413(1) 829833(1) 861246(1) | 31413 829833 861246 ``` # future works + 支持统计指定LUN的时延 + 支持统计SID、CID + 支持统计target + 支持统计initiator + 支持统计读写操作