离线业务设置cpu.cfs_quota_us进行压力测试，出现kernel panic

【标题描述】离线业务设置cpu.cfs_quota_us进行压力测试，出现kernel panic
【环境信息】
qemu虚拟机部署openeuler版本
【问题复现步骤】

git clone https://github.com/indeedeng/fibtest.git
cd fibtest && make
运行两个fibtest, 其中一个设置为离线,并且配置cpu bandwidth, 线程数设置为200.
【预期结果】
正常执行，不出现hardlockup和panic
【实际结果】
出现panic:

[  693.506255] BUG: unable to handle kernel paging request at 000000a2a2127317
[  693.511679] PGD 0 P4D 0
[  693.513769] Oops: 0002 [#1] SMP PTI
[  693.517412] CPU: 24 PID: 0 Comm: swapper/24 Kdump: loaded Tainted:
[  693.524828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
[  693.530401] RIP: 0010:_raw_spin_lock+0xc/0x20
[  693.532069] Code: 01 00 00 75 05 48 89 d8 5b c3 e8 df 67 87 ff 48 89 d8 5b c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 02 f3 c3 89 c6 e8 93 52 87 ff 66 90 c3 0f 1f
[  693.541483] RSP: 0000:ffff94d71ec03e90 EFLAGS: 00010046
[  693.543441] RAX: 0000000000000000 RBX: 000000a2a2127317 RCX: 0000000000000830
[  693.546120] RDX: 0000000000000001 RSI: 00000000000000fd RDI: 000000a2a2127317
[  693.548910] RBP: 00000000170ef11a R08: ffff94d0094a9200 R09: 0000000000000000
[  693.551196] R10: 0000000000100000 R11: 0000000000099865 R12: ffff94d012423b30
[  693.553436] R13: ffff94d71ea23e70 R14: 0000000000000001 R15: ffff94d71ea24000
[  693.555936] FS:  0000000000000000(0000) GS:ffff94d71ec00000(0000) knlGS:0000000000000000
[  693.558857] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  693.560897] CR2: 000000a2a2127317 CR3: 000000077b80a005 CR4: 00000000003606e0
[  693.563724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  693.566424] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  693.568973] Call Trace:
[  693.569887]  <IRQ>
[  693.570577]  distribute_cfs_runtime+0x66/0x130
[  693.572288]  sched_cfs_period_timer+0x103/0x210
[  693.573929]  ? sched_cfs_slack_timer+0xc0/0xc0
[  693.575681]  __hrtimer_run_queues+0x125/0x2f0
[  693.577202]  ? kvm_clock_get_cycles+0xd/0x10
[  693.578670]  hrtimer_interrupt+0xe5/0x240
[  693.580074]  smp_apic_timer_interrupt+0x6a/0x130
[  693.581670]  apic_timer_interrupt+0xf/0x20
[  693.583088]  </IRQ>
[  693.583860] RIP: 0010:native_safe_halt+0xe/0x10
[  693.585509] Code: eb bd 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d f6 75 56 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d e6 75 56 00 fb f4 <c3> 90 0f 1f 44 00 00 41 54 55 53 e8 42 74 89 ff 65 8b 2d ab d8 96

【附件信息】
比如系统message日志/组件日志、dump信息、图片等

Hi zqiao216, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: Kernel, and any of the maintainers: @YangYingliang , @pi3orama , @成坚 (CHENG Jian) , @jiaoff , @刘勇强 , @Xie XiuQi

1.定位分析:

经过对vmcore进行分析，在distribute_cfs_runtime()中对cfs_b->throttled_cfs_rq中的cfs_rq遍历过程中，cfs_rq->throttled_list指向了自己，在此情况下，list_for_each_entry_rcu()会陷入死循环，导致hardlockup问题。

crash> bt
    [exception RIP: _raw_spin_lock]
    RIP: ffffffff97ea1fc0  RSP: ffff947dde803e98  RFLAGS: 00000087
    RAX: ffff9476dea14b90  RBX: ffff947dde6232c0  RCX: 0000000000000001
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff947dde6232c0
    RBP: 000000001da2852e   R8: 0000000000000008   R9: 0000000000000000
    R10: 0000000000100000  R11: 00000000000161e2  R12: ffff9476c634de30
    R13: 000000001dcd6500  R14: 0000000000000001  R15: ffff9476dea14a00   // R15为当前遍历到的cfs_rq地址
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
#13 [ffff947dde803e98] _raw_spin_lock at ffffffff97ea1fc0
#14 [ffff947dde803e98] distribute_cfs_runtime at ffffffff976fcecd
#15 [ffff947dde803ed0] sched_cfs_period_timer at ffffffff976fd143
#16 [ffff947dde803f08] __hrtimer_run_queues at ffffffff97744ab5
#17 [ffff947dde803f80] hrtimer_interrupt at ffffffff97745295
#18 [ffff947dde803fd8] smp_apic_timer_interrupt at ffffffff980027aa
#19 [ffff947dde803ff0] apic_timer_interrupt at ffffffff98001d0f

crash> struct cfs_rq ffff9476dea14a00
struct cfs_rq {
....
 throttled_list = {
    next = 0xffff9476dea14b90,     // throttled_list已被初始化，正常情况下，应该指向其他cfs_rq
    prev = 0xffff9476dea14b90
  },
....

结合代码，throttled_list被初始化只有在unthrottle_qos_cfs_rq()存在，说明在我们遍历过程中，有cpu调用了throttle_qos_cfs_rq和unthrottle_qos_cfs_rq()，导致出错。
问题的原因为, 在distribute_cfs_runtime()对某个cfs_rq(cfs_rq_curr)进行unthrottle_cfs_rq()调用后，会调用rq_unlock()释放锁。在循环取下一个cfs_rq(cfs_rq_next)节点前，存在其他核cpu对cfs_rq_curr进行了qos throttle，改变了cfs_rq_curr->throttled_list.next指针，而后导致在distribute_cfs_runtime（）中遍历链表出现问题。
输入图片说明

2.复现证实

为了证实上面的分析结果，cfs_rq->throttled_list节点在distribute_cfs_runtime()中发生了改变,可加入以下几行代码，进行复现。
输入图片说明
因为unthrottle_qos_cfs_rq()调用前需要持锁，所以在rq锁释放前提前获取next_cfs_rq, 与后面实际获取的值进行比较，结果可以得到如下日志，证实cfs_rq在rq_unlock()后发生了改变（此问题在物理机很难复现，虚拟机内极易复现，因为虚拟机在rq_unlock()时，vcpu线程可能会被切出）。
输入图片说明