SATA盘NCQ场景下发多个IO，其中一个IO出错后，出现群体慢盘问题优化

【标题描述】SATA盘NCQ场景下发多个IO，其中一个IO出错后，出现群体慢盘问题优化

【环境信息】
硬件信息：
1） 鲲鹏920
软件信息：
[root@localhost 0000:b4:02.0]# cat /etc/euleros-latest
eulerversion=EulerOS_Server_V200R008C00SPC300B630
compiletime=2019-12-27-10-58-38
kernelversion=4.19.36-vhulk1907.1.0.h619
[root@localhost 0000:b4:02.0]# uname -a
Linux localhost.localdomain 4.19.36-vhulk1907.1.0.h619.eulerosv2r8.aarch64 #1 SMP Mon Jul 22 00:00:00 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
【问题复现步骤】
1、对SATA盘注入UNC故障
2、对盘跑压测
3、磁盘框接入多块盘
【预期结果】
故障盘出问题时，对其他盘无明显影响
【实际结果】
其他盘IO卡住30秒左右
【问题根因】
NCQ场景IO出错是面向盘的报错，盘只会回一个tag全零的SDB帧。盘上的IO均不会返回。基于开源社区的IO错误处理机制，驱动处理单个异常IO后，内核会锁住host，等待所有IO超时失败后，才进错误处理（Error Handler)，并执行完EH后再对host解锁。导致host锁住时长接近下发IO时设置的超时时间，引发慢盘现象。

【解决方案】
NCQ场景出错后，类似ahci驱动，调用libata接口ata_link_abort，对盘链路IO执行abort处理，加速进入错误处理，消除群体慢盘问题，并通过内核EH机制给盘下发read log ext命令，获取错误详情，立即返回给用户。(同时解决群体慢盘和无法获取盘故障信息的问题）

本方案相比之前的解决方案：NCQ场景出错后，驱动将盘未返回的IO以出错的形式快速返回给内核，加速进入错误处理。 降低NCQ降级的可能性，对sata盘性能更佳，且修改量更小。（https://gitee.com/openeuler/kernel/issues/I5CG2F）

【附件信息】
验证结果符合预期，无群体慢盘现象，UNC场景能内核能立即将盘的故障信息通过块设备层传递给用户，日志如下：
[26736.715314] hisi_sas_v3_hw 0000:b4:02.0: erroneous completion disk err dev id=2 sas_addr=0x5000000000000605 CQ hdr: 0x400903 0x200e5 0x0 0x80470000
[26736.781599] sas: Enter sas_scsi_recover_host busy: 32 failed: 32
[26736.781602] sas: trying to find task 0x00000000bd841516
[26736.781603] sas: sas_scsi_find_task: aborting task 0x00000000bd841516
[26736.781766] sas: sas_scsi_find_task: task 0x00000000bd841516 is aborted
[26736.781767] sas: sas_eh_handle_sas_errors: task 0x00000000bd841516 is aborted
[26736.781768] sas: trying to find task 0x000000001045137a
[26736.781769] sas: sas_scsi_find_task: aborting task 0x000000001045137a
[26736.781918] sas: sas_scsi_find_task: task 0x000000001045137a is aborted
[26736.781919] sas: sas_eh_handle_sas_errors: task 0x000000001045137a is aborted
[26736.781920] sas: trying to find task 0x000000007e5beec4
[26736.781920] sas: sas_scsi_find_task: aborting task 0x000000007e5beec4
[26736.782066] sas: sas_scsi_find_task: task 0x000000007e5beec4 is aborted
[26736.782067] sas: sas_eh_handle_sas_errors: task 0x000000007e5beec4 is aborted
[26736.782068] sas: trying to find task 0x00000000c79baf80
[26736.782069] sas: sas_scsi_find_task: aborting task 0x00000000c79baf80
[26736.782215] sas: sas_scsi_find_task: task 0x00000000c79baf80 is aborted
[26736.782216] sas: sas_eh_handle_sas_errors: task 0x00000000c79baf80 is aborted
[26736.782217] sas: trying to find task 0x000000006d25d7ad
[26736.782218] sas: sas_scsi_find_task: aborting task 0x000000006d25d7ad
[26736.782362] sas: sas_scsi_find_task: task 0x000000006d25d7ad is aborted
[26736.782363] sas: sas_eh_handle_sas_errors: task 0x000000006d25d7ad is aborted
[26736.782364] sas: trying to find task 0x00000000cf03a281
[26736.782365] sas: sas_scsi_find_task: aborting task 0x00000000cf03a281
[26736.782511] sas: sas_scsi_find_task: task 0x00000000cf03a281 is aborted
[26736.782512] sas: sas_eh_handle_sas_errors: task 0x00000000cf03a281 is aborted
[26736.782513] sas: trying to find task 0x00000000f946a49f
[26736.782514] sas: sas_scsi_find_task: aborting task 0x00000000f946a49f
[26736.782659] sas: sas_scsi_find_task: task 0x00000000f946a49f is aborted
[26736.782660] sas: sas_eh_handle_sas_errors: task 0x00000000f946a49f is aborted
[26736.782661] sas: trying to find task 0x00000000a76e098f
[26736.782662] sas: sas_scsi_find_task: aborting task 0x00000000a76e098f
[26736.782808] sas: sas_scsi_find_task: task 0x00000000a76e098f is aborted
[26736.782809] sas: sas_eh_handle_sas_errors: task 0x00000000a76e098f is aborted
[26736.782810] sas: trying to find task 0x00000000ce59ac01
[26736.782811] sas: sas_scsi_find_task: aborting task 0x00000000ce59ac01
[26736.782956] sas: sas_scsi_find_task: task 0x00000000ce59ac01 is aborted
[26736.782957] sas: sas_eh_handle_sas_errors: task 0x00000000ce59ac01 is aborted
[26736.782958] sas: trying to find task 0x000000003df3dde0
[26736.782959] sas: sas_scsi_find_task: aborting task 0x000000003df3dde0
[26736.783105] sas: sas_scsi_find_task: task 0x000000003df3dde0 is aborted
[26736.783106] sas: sas_eh_handle_sas_errors: task 0x000000003df3dde0 is aborted
[26736.783107] sas: trying to find task 0x000000003bd7c876
[26736.783108] sas: sas_scsi_find_task: aborting task 0x000000003bd7c876
[26736.783253] sas: sas_scsi_find_task: task 0x000000003bd7c876 is aborted
[26736.783254] sas: sas_eh_handle_sas_errors: task 0x000000003bd7c876 is aborted
[26736.783255] sas: trying to find task 0x00000000339cf767
[26736.783256] sas: sas_scsi_find_task: aborting task 0x00000000339cf767
[26736.783402] sas: sas_scsi_find_task: task 0x00000000339cf767 is aborted
[26736.783403] sas: sas_eh_handle_sas_errors: task 0x00000000339cf767 is aborted
[26736.783403] sas: trying to find task 0x00000000a3bfd272
[26736.783404] sas: sas_scsi_find_task: aborting task 0x00000000a3bfd272
[26736.783549] sas: sas_scsi_find_task: task 0x00000000a3bfd272 is aborted
[26736.783550] sas: sas_eh_handle_sas_errors: task 0x00000000a3bfd272 is aborted
[26736.783551] sas: trying to find task 0x00000000cbbf2c64
[26736.783552] sas: sas_scsi_find_task: aborting task 0x00000000cbbf2c64
[26736.783698] sas: sas_scsi_find_task: task 0x00000000cbbf2c64 is aborted
[26736.783699] sas: sas_eh_handle_sas_errors: task 0x00000000cbbf2c64 is aborted
[26736.783700] sas: trying to find task 0x00000000fec98799
[26736.783701] sas: sas_scsi_find_task: aborting task 0x00000000fec98799
[26736.783847] sas: sas_scsi_find_task: task 0x00000000fec98799 is aborted
[26736.783848] sas: sas_eh_handle_sas_errors: task 0x00000000fec98799 is aborted
[26736.783849] sas: trying to find task 0x000000008eb2fb8c
[26736.783850] sas: sas_scsi_find_task: aborting task 0x000000008eb2fb8c
[26736.783997] sas: sas_scsi_find_task: task 0x000000008eb2fb8c is aborted
[26736.783998] sas: sas_eh_handle_sas_errors: task 0x000000008eb2fb8c is aborted
[26736.783999] sas: trying to find task 0x000000009670fa55
[26736.783999] sas: sas_scsi_find_task: aborting task 0x000000009670fa55
[26736.784145] sas: sas_scsi_find_task: task 0x000000009670fa55 is aborted
[26736.784146] sas: sas_eh_handle_sas_errors: task 0x000000009670fa55 is aborted
[26736.784147] sas: trying to find task 0x00000000d7d9e375
[26736.784147] sas: sas_scsi_find_task: aborting task 0x00000000d7d9e375
[26736.784295] sas: sas_scsi_find_task: task 0x00000000d7d9e375 is aborted
[26736.784296] sas: sas_eh_handle_sas_errors: task 0x00000000d7d9e375 is aborted
[26736.784297] sas: trying to find task 0x00000000c6bc9612
[26736.784298] sas: sas_scsi_find_task: aborting task 0x00000000c6bc9612
[26736.784444] sas: sas_scsi_find_task: task 0x00000000c6bc9612 is aborted
[26736.784444] sas: sas_eh_handle_sas_errors: task 0x00000000c6bc9612 is aborted
[26736.784445] sas: trying to find task 0x00000000933c52fd
[26736.784446] sas: sas_scsi_find_task: aborting task 0x00000000933c52fd
[26736.784593] sas: sas_scsi_find_task: task 0x00000000933c52fd is aborted
[26736.784594] sas: sas_eh_handle_sas_errors: task 0x00000000933c52fd is aborted
[26736.784594] sas: trying to find task 0x0000000085f003b7
[26736.784595] sas: sas_scsi_find_task: aborting task 0x0000000085f003b7
[26736.784741] sas: sas_scsi_find_task: task 0x0000000085f003b7 is aborted
[26736.784742] sas: sas_eh_handle_sas_errors: task 0x0000000085f003b7 is aborted
[26736.784743] sas: trying to find task 0x00000000edbaec7c
[26736.784743] sas: sas_scsi_find_task: aborting task 0x00000000edbaec7c
[26736.784890] sas: sas_scsi_find_task: task 0x00000000edbaec7c is aborted
[26736.784890] sas: sas_eh_handle_sas_errors: task 0x00000000edbaec7c is aborted
[26736.784891] sas: trying to find task 0x00000000d95d5699
[26736.784892] sas: sas_scsi_find_task: aborting task 0x00000000d95d5699
[26736.785037] sas: sas_scsi_find_task: task 0x00000000d95d5699 is aborted
[26736.785038] sas: sas_eh_handle_sas_errors: task 0x00000000d95d5699 is aborted
[26736.785039] sas: trying to find task 0x0000000023a01da8
[26736.785040] sas: sas_scsi_find_task: aborting task 0x0000000023a01da8
[26736.785187] sas: sas_scsi_find_task: task 0x0000000023a01da8 is aborted
[26736.785187] sas: sas_eh_handle_sas_errors: task 0x0000000023a01da8 is aborted
[26736.785188] sas: trying to find task 0x000000007e64f0dc
[26736.785189] sas: sas_scsi_find_task: aborting task 0x000000007e64f0dc
[26736.785343] sas: sas_scsi_find_task: task 0x000000007e64f0dc is aborted
[26736.785345] sas: sas_eh_handle_sas_errors: task 0x000000007e64f0dc is aborted
[26736.785349] sas: trying to find task 0x000000005bceddf7
[26736.785349] sas: sas_scsi_find_task: aborting task 0x000000005bceddf7
[26736.785525] sas: sas_scsi_find_task: task 0x000000005bceddf7 is aborted
[26736.785526] sas: sas_eh_handle_sas_errors: task 0x000000005bceddf7 is aborted
[26736.785528] sas: trying to find task 0x00000000df0682b5
[26736.785529] sas: sas_scsi_find_task: aborting task 0x00000000df0682b5
[26736.785692] sas: sas_scsi_find_task: task 0x00000000df0682b5 is aborted
[26736.785693] sas: sas_eh_handle_sas_errors: task 0x00000000df0682b5 is aborted
[26736.785695] sas: trying to find task 0x00000000b914f450
[26736.785695] sas: sas_scsi_find_task: aborting task 0x00000000b914f450
[26736.785852] sas: sas_scsi_find_task: task 0x00000000b914f450 is aborted
[26736.785853] sas: sas_eh_handle_sas_errors: task 0x00000000b914f450 is aborted

[26736.785855] sas: trying to find task 0x0000000000770c6f
[26736.785856] sas: sas_scsi_find_task: aborting task 0x0000000000770c6f
[26736.786010] sas: sas_scsi_find_task: task 0x0000000000770c6f is aborted
[26736.786011] sas: sas_eh_handle_sas_errors: task 0x0000000000770c6f is aborted
[26736.786013] sas: trying to find task 0x00000000f28de410
[26736.786014] sas: sas_scsi_find_task: aborting task 0x00000000f28de410
[26736.786168] sas: sas_scsi_find_task: task 0x00000000f28de410 is aborted
[26736.786169] sas: sas_eh_handle_sas_errors: task 0x00000000f28de410 is aborted
[26736.786171] sas: trying to find task 0x00000000656515a8
[26736.786172] sas: sas_scsi_find_task: aborting task 0x00000000656515a8
[26736.786325] sas: sas_scsi_find_task: task 0x00000000656515a8 is aborted
[26736.786326] sas: sas_eh_handle_sas_errors: task 0x00000000656515a8 is aborted
[26736.786327] sas: trying to find task 0x00000000c7375662
[26736.786328] sas: sas_scsi_find_task: aborting task 0x00000000c7375662
[26736.786481] sas: sas_scsi_find_task: task 0x00000000c7375662 is aborted
[26736.786482] sas: sas_eh_handle_sas_errors: task 0x00000000c7375662 is aborted
[26736.786493] sas: ata1974: end_device-6:1: cmd error handler
[26736.786521] sas: ata1973: end_device-6:0: dev error handler
[26736.786542] sas: ata1974: end_device-6:1: dev error handler
[26736.786602] ata1974.00: exception Emask 0x0 SAct 0xffffffff SErr 0x0 action 0x6
[26736.786606] ata1974.00: failed command: READ FPDMA QUEUED
[26736.786609] ata1974.00: cmd 60/60:00:d1:26:00/00:00:00:00:00/40 tag 25 ncq dma 49152 in
                        res 41/40:60:10:27:00/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[26736.786610] ata1974.00: status: { DRDY ERR }
[26736.786611] ata1974.00: error: { UNC }
[26736.786617] ata1974: hard resetting link
[26736.789866] hisi_sas_v3_hw 0000:b4:02.0: phydown: phy5 phy_state=0x4
[26736.789869] hisi_sas_v3_hw 0000:b4:02.0: ignore flutter phy5 down
[26736.946149] hisi_sas_v3_hw 0000:b4:02.0: phyup: phy5 link_rate=10(sata)
[26736.946158] sas: sas_form_port: phy5 belongs to port1 already(1)!
[26737.105793] ata1974.00: configured for UDMA/133
[26737.105876] sd 6:0:1:0: [sdb] tag#229 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[26737.105878] sd 6:0:1:0: [sdb] tag#229 Sense Key : Medium Error [current]
[26737.105880] sd 6:0:1:0: [sdb] tag#229 Add. Sense: Unrecovered read error - auto reallocate failed
[26737.105883] sd 6:0:1:0: [sdb] tag#229 CDB: Read(10) 28 00 00 00 26 d1 00 00 60 00
[26737.105884] print_req_error: I/O error, dev sdb, sector 10000
[26737.105902] ata1974: EH complete
[26737.105909] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 32 tries: 1

GVP openEuler/kernel

内容风险标识

评论 (1)

GVPopenEuler/kernel

内容风险标识