[openEuler 1.0-LTS 2003 lts sp3][iscsi]UAF宕机

【标题描述】能够简要描述问题：说明什么场景下，做了什么操作，出现什么问题（尽量使用正向表达方式）
iscsi依赖iscsid，iscsi遇到错误时是通过netlink通知iscsid的。机制不可靠。
【环境信息】
硬件信息：
1）鲲鹏920，qume虚拟机等可出现
软件信息：
1） 2003 sp3最新内核
如果有特殊组网，请提供网络拓扑图
【问题复现步骤】
具体操作步骤
出现概率(是否必现，概率性错误)
必先，使用如下测试代码，iscsi建立后，通过echo 50 > /sys/module/iscsi_tcp/parameters/random_fault
然后通过iscsiadm 进行 logout login操作。就会触发问题

--- a/drivers/scsi/iscsi_tcp.c
+++ b/drivers/scsi/iscsi_tcp.c
@@ -256,6 +256,11 @@ iscsi_sw_tcp_conn_restore_callbacks(struct iscsi_conn *conn)
        write_unlock_bh(&sk->sk_callback_lock);
 }
 
+
+u32 random_fault = 0;
+EXPORT_SYMBOL(random_fault);
+module_param(random_fault, uint, 0644);
+
 /**
  * iscsi_sw_tcp_xmit_segment - transmit segment
  * @tcp_conn: the iSCSI TCP connection
@@ -270,7 +275,7 @@ iscsi_sw_tcp_conn_restore_callbacks(struct iscsi_conn *conn)
  * it will retrieve the hash value and send it as well.
  */
 static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn,
-                                    struct iscsi_segment *segment)
+                                    struct iscsi_segment *segment, struct iscsi_task *task)
 {
        struct iscsi_sw_tcp_conn *tcp_sw_conn = tcp_conn->dd_data;
        struct socket *sk = tcp_sw_conn->sock;
@@ -289,6 +294,11 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn,
                if (segment->total_copied + segment->size < segment->total_size)
                        flags |= MSG_MORE;
 
+               if (task->sc && ((get_random_long() % 100) < random_fault)) {
+                       iscsi_tcp_segment_unmap(segment);
+                       return -1;
+               }



 void iscsi_conn_error_event(struct iscsi_cls_conn *conn, enum iscsi_err error)
 {
        struct nlmsghdr *nlh;
@@ -2432,6 +2440,11 @@ void iscsi_conn_error_event(struct iscsi_cls_conn *conn, enum iscsi_err error)
        if (!priv)
                return;
 
+       if ((get_random_long() % 100) < random_fault) {
+               printk("gogogo\n");
+               return ;
+       }
+

【预期结果】
上游有相关patch解决此类问题

0ab710458da1 2020-02-10 22:46:54 -0500 scsi: iscsi: Perform connection failure entirely in kernel space
82b8cf40bfe1 2020-03-26 21:59:20 -0400 scsi: iscsi: Report connection state in sysfs
7e7cd796f277 2020-05-26 21:20:24 -0400 scsi: iscsi: Fix deadlock on recovery path during GFP_IO reclaim
9e67600ed6b8 2021-03-29 21:17:45 -0400 scsi: iscsi: Fix race condition between login and sync thread
0dcf8febcb7b 2021-04-07 21:30:59 -0400 scsi: iscsi: Fix iSCSI cls conn state
891e2639deae 2021-06-02 01:28:19 -0400 scsi: iscsi: Stop queueing during ep_disconnect
9e5fe1700896 2021-06-02 01:28:20 -0400 scsi: iscsi: Rel ref after iscsi_lookup_endpoint()
23d6fefbb3f6 2021-06-02 01:28:20 -0400 scsi: iscsi: Fix in-kernel conn failure handling

【实际结果】
描述出问题的结果
【附件信息】
比如系统message日志/组件日志、dump信息、图片等

Hi ***HDER001***, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at **[Here](https://gitee.com/openeuler/community/blob/master/en/sig-infrastructure/command.md)**.
If you have any questions, please contact the SIG: [Kernel](https://gitee.com/openeuler/community/tree/master/sig/Kernel), and any of the maintainers: @yangyingliang , @gatieme , @jiaoff , @zhengzengkai , @LiuYongQiang0816 , @wangxiongfeng , @kevinzhu1 , @jentlestea , @lujialin2 , @wuxu_buque , @xukuohai , @lengchao , @alvin-ling , @yuzenghui , @juntianlinux , @chenguangli , @gasonchen , @whoisxxx , @koulihong , @kailiu42 , @hanjun-guo , @woqidaideshi , @chiqijun , @wkfxxx , @thundertown , @guohaocs2c , @kylin-mayukun , @oskernel0719 , @liuxinux , @newbeats , @zhujianwei001 , @zhenpengzheng , @SuperSix173 , @colyli , @zhangyi089 , @htforge , @qiuuuuu , @yuehaibing_planb , @haochengxie , @guzitao , @CTC-XiboWang , @Hongchen_Zhang , @chenwei_kernel , @x56Jason , @gouhao2022 , @duanqiangwen , @georgeguo , @mao-hongbo , @allen-shi , @xiexiuqi

测试代码中的 iscsi_sw_tcp_xmit_segment调用的地方没有跟随更改

命令: iscsiadm -m node -u

问题栈:
// 注入故障无法发送小小给客户端通知task失败
iscsi_xmitworker
iscsi_data_xmit
iscsi_xmit_task
iscsi_tcp_task_xmit
iscsi_sw_tcp_pdu_xmit
iscsi_sw_tcp_xmit
iscsi_sw_tcp_xmit_segment //error
iscsi_conn_failure // error handling
iscsi_conn_error_event // stop

// 等待360s*3 无法收到消息，触发unbind_session
__iscsi_unbind_session
scsi_remove_target
scsi_remove_device
__scsi_remove_device
blk_cleanup_queue
blk_exit_queue
elevator_exit
blk_mq_exit_sched
blk_mq_sched_tags_teardown
blk_mq_sched_free_tags
blk_mq_free_rqs
free_page // 释放rq
blk_mq_free_rq_map
kfree(tags->rqs);
kfree(tags->static_rqs);
// 遍历未执行完毕的task，继续执行，问题发生
___sys_sendmsg
sock_sendmsg
netlink_sendmsg
netlink_unicast
iscsi_if_rx
iscsi_if_recv_msg
iscsi_sw_tcp_conn_stop
iscsi_conn_stop
iscsi_start_session_recovery
fail_scsi_tasks
fail_scsi_task
sc = task->sc;
sc->result = err << 16; // uaf
修复:
0ab710458da1 ("scsi: iscsi: Perform connection failure entirely in kernel space")
在错误路径发送error_event的时候先执行conn->transport->stop_conn(iscsi_sw_tcp_conn_stop), 避免在释放rq之后执行失败的task

修复补丁会引入kabi变更无法规避

命令: iscsiadm -m node -u

// 如果有两个target，删除完第一个target之后会下发task，然后删除第二个target
问题栈: 
// 触发删除target
__iscsi_unbind_session
  scsi_remove_target
    scsi_remove_device
      __scsi_remove_device
        device_del // 先删除device
          bus_remove_device
            device_release_driver
              device_release_driver_internal
                __device_release_driver
                  sd_remove
                    sd_shutdown
                      sd_sync_cache// 关闭sd设备之前先完成未完成的rq, 如果rq->result不等于0会循环执行三次
                       __scsi_execute
                        blk_execute_rq
                         1)blk_execute_rq_nowait
                          blk_mq_sched_insert_request
                            blk_mq_run_hw_queue
                             __blk_mq_delay_run_hw_queue
                              __blk_mq_run_hw_queue
                               blk_mq_sched_dispatch_requests
                                __blk_mq_sched_dispatch_requests
                                 blk_mq_dispatch_rq_list
                                  scsi_queue_rq
                                   scsi_dispatch_cmd
                                    iscsi_queuecommand
                                     // 注入故障 无法发送消息给用户态通知task失败 
                                      iscsi_xmitworker
                                       iscsi_data_xmit
                                         iscsi_prep_scsi_cmd_pdu
                                         iscsi_xmit_task
                                          iscsi_tcp_task_xmit
                                           iscsi_sw_tcp_pdu_xmit
                                             iscsi_sw_tcp_xmit
                                              iscsi_sw_tcp_xmit_segment //error
                                               iscsi_conn_failure        // error handling
                                                session->state = ISCSI_STATE_FAILED;
                                                iscsi_conn_error_event // stop
                        2)wait_for_completion_io // 等待io完成 超时 走到超时流程
        blk_cleanup_queue
          blk_exit_queue
            elevator_exit
              blk_mq_exit_sched
                blk_mq_sched_tags_teardown
                  blk_mq_sched_free_tags
                    blk_mq_free_rqs
                      free_page // 释放rq
                    blk_mq_free_rq_map
                      kfree(tags->rqs);
                      kfree(tags->static_rqs);
// 超时，会requeue
// rq超时 走超时流程完成request
blk_mq_timeout_work
  blk_mq_queue_tag_busy_iter
    bt_for_each
      sbitmap_for_each_set
        __sbitmap_for_each_set
         fn (bt_iter)
          bt_iter
           iter_data->fn (blk_mq_check_expired)
             blk_mq_check_expired
               blk_mq_rq_timed_out
                 req->q->mq_ops->timeout
                   scsi_timeout
                     scsi_times_out
                       iscsi_eh_cmd_timed_out
                         rc = BLK_EH_DONE; // 超时直接rc = BLK_EH_DONE
                       scsi_abort_command // 触发abort cmd，将失败的cmd丢弃
                         queue_delayed_work
                           scmd_eh_abort_handler
                             scsi_try_to_abort_cmd
                               iscsi_eh_abort // session->state != ISCSI_STATE_LOGGED_IN 什么都不做 结束
                             scsi_eh_scmd_add
                               scsi_eh_inc_host_failed
                                scsi_eh_wakeup
                                  wake_up_process(shost->ehandler); // scsi_error_handler
                 blk_add_timer(req) // rc!=BLK_EH_DONE 加入到time_outwork继续执行
scsi_error_handler
  scsi_unjam_host
    scsi_eh_ready_devs // reset target
      scsi_eh_stu // nothing 没有打印 shost_for_each_device(sdev, shost) 为空
      scsi_eh_bus_device_reset // nothing 没有打印 shost_for_each_device(sdev, shost) 为空
      scsi_eh_target_reset // target reset
        scsi_try_target_reset
          iscsi_eh_recover_target
            iscsi_eh_target_reset // FAIL执行iscsi_eh_session_reset
            iscsi_eh_session_reset // session reset 
              session = cls_session->dd_data; // 重新设置session 
      scsi_eh_bus_reset // bus reset
        return list_empty(work_q)
      scsi_eh_host_reset // host reset
        scsi_try_host_reset
          hostt->eh_host_reset_handler // 返回FAIL
        shost_printk(KERN_INFO, shost,"%s: HRST failed\n",current->comm));
      scsi_eh_offline_sdevs // 判断offline的状态
        scsi_device_set_state
          illegal:
            "Illegal state transition %s->%s",
    scsi_eh_flush_done_q // 完成rq
        "%s: flush retry cmd\n"
        __scsi_queue_insert
          blk_mq_requeue_request
           __blk_mq_requeue_request // requeue 这个request 重新下发到驱动
             rq_qos_requeue
    scsi_restart_operations
      scsi_run_host_queues // shost_for_each_device(sdev, shost) 为空 
blk_mq_run_work_fn
  __blk_mq_run_hw_queue
    blk_mq_sched_dispatch_requests
      __blk_mq_sched_dispatch_requests
        blk_mq_dispatch_rq_list
          scsi_queue_rq
            scsi_dispatch_cmd
              iscsi_queuecommand
                scsi_mq_done
                  blk_mq_complete_request
                    blk_mq_force_complete_rq // 加入软中断 尝试完成request
// 软中断完成request
run_ksoftirqd
  __do_softirq
    blk_done_softirq
      scsi_softirq_done
        scsi_decide_disposition // 判断命令是否完成
        scsi_queue_insert // 未完成需要加到队列里面重新执行
        scsi_finish_command // 如果还未完成且等待超过wait_for的时间，执行这个强制往下走
          scsi_io_completion
            scsi_io_completion_action
              scsi_end_request
                __blk_mq_end_request
// 删除完target之后，用户态下发消息触发stop_conn, 会遍历未执行完毕的task， 继续执行，问题发生
___sys_sendmsg
  sock_sendmsg
    netlink_sendmsg
      netlink_unicast
        iscsi_if_rx
          iscsi_if_recv_msg
            iscsi_sw_tcp_conn_stop
              iscsi_conn_stop
                iscsi_start_session_recovery
                 fail_scsi_tasks
                   fail_scsi_task
                    sc = task->sc;
                    sc->result = err << 16; // uaf

GVP openEuler/kernel

内容风险标识

评论 (5)

GVPopenEuler/kernel

内容风险标识