[OLK-6.6] SCHED_WARN_ON(cfs_rq->throttled_clock) in throttle_cfs_rq

**【缺陷描述】**
长稳环境出现Hard Lockup问题

**一、缺陷信息**

**【缺陷所属的os版本】**
OS：
NAME="openEuler"
VERSION="22.03 LTS"
ID="openEuler"
VERSION_ID="22.03"
PRETTY_NAME="openEuler 22.03 LTS"
ANSI_COLOR="0;31"

**【内核版本】**
kernel:
6.6

**【缺陷所属软件及版本号】**

**【环境信息】**
硬件信息

- Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz

软件信息

网络信息

**【问题复现步骤】：请描述具体的操作步骤**
运行调度领域级长稳测试

**【实际结果】**
出现 hard LOCKUP问题

**【期望结果】**
系统无 hard LOCKUP问题

**【其他相关附件信息】**
[11365.856035] smpboot: CPU 7 is now offline
[11366.493712] smpboot: Booting Node 0 Processor 7 APIC 0x10
[11381.393479] ------------[ cut here ]------------
[11413.243022] watchdog: Watchdog detected hard LOCKUP on cpu 82
[11413.243027] Modules linked in: my_hrtimer(OE) nfsv3 nfs_acl nfs lockd grace fscache netfs tls dns_resolver brd ntfs exfat vfat fat btrfs xor raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_mangle iptable_nat xt_addrtype ebtable_filter ebtables xt_conntrack nf_nat ip6table_filter ip6_tables nf_conntrack iptable_filter ip_tables nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c tun br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common rfkill isst_if_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm rpcrdma irqbypass sunrpc rdma_ucm rapl ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser ib_umad intel_cstate ib_ipoib rdma_cm iw_cm libiscsi ib_cm scsi_transport_iscsi mlx5_ib ib_uverbs iTCO_wdt mei_me iTCO_vendor_support ipmi_ssif ib_core intel_uncore sg pcspkr joydev acpi_ipmi
[11413.243181]  i2c_i801 mei ioatdma i2c_smbus ipmi_si lpc_ich intel_pch_thermal ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad fuse ext4 mbcache jbd2 sd_mod t10_pi ast crct10dif_pclmul crc64_rocksoft_generic crc32_pclmul crc64_rocksoft ahci crc32c_intel crc64 drm_shmem_helper drm_kms_helper libahci ghash_clmulni_intel igb mlx5_core drm megaraid_sas sha512_ssse3 dca libata i2c_algo_bit mlxfw pci_hyperv_intf wmi
[11413.243243] irq event stamp: 130227
[11413.243245] hardirqs last  enabled at (130227): [<ffffffffacc241be>] cpuidle_idle_call+0x19e/0x1e0
[11413.243258] hardirqs last disabled at (130226): [<ffffffffacc242c6>] do_idle+0xb6/0x110
[11413.243264] softirqs last  enabled at (129802): [<ffffffffacb6f38c>] handle_softirqs+0x3bc/0x4c0
[11413.243274] softirqs last disabled at (129797): [<ffffffffacb6f86a>] irq_exit_rcu+0xea/0x120
[11413.243282] CPU: 82 PID: 0 Comm: swapper/82 Kdump: loaded Tainted: G S         OE      6.6.0-9ded7a40d8e3 #10
[11413.243289] Hardware name: Powerleader PR2715P3/T1DM-E2, BIOS OEM57.22 11/23/2021
[11413.243292] RIP: 0010:native_queued_spin_lock_slowpath+0x48a/0x580
[11413.243304] Code: 1e eb 02 f3 90 4c 89 ff e8 43 b2 05 ff 8b 43 08 85 c0 74 ef 48 89 df e8 84 b3 05 ff 4c 8b 33 4d 85 f6 74 08 41 0f 0d 0e eb 02 <f3> 90 48 89 ef e8 1c b2 05 ff 8b 45 00 66 85 c0 75 ee 89 c2 66 31
[11413.243310] RSP: 0018:ffff88febb809c00 EFLAGS: 00000002
[11413.243315] RAX: 0000000001380101 RBX: ffff88febba09880 RCX: dffffc0000000000
[11413.243319] RDX: 0000000000000003 RSI: 0000000000000002 RDI: ffff88feb3e087c0
[11413.243323] RBP: ffff88feb3e087c0 R08: ffffffffae045b34 R09: ffffed1fd67c10f8
[11413.243327] R10: ffff88feb3e087c3 R11: ffff88febb809ff8 R12: 00000000014c0000
[11413.243331] R13: 1ffff11fd7701381 R14: 0000000000000000 R15: ffff88febba09888
[11413.243335] FS:  0000000000000000(0000) GS:ffff88febb800000(0000) knlGS:0000000000000000
[11413.243339] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11413.243343] CR2: 00007f98419bd9e0 CR3: 00000004db932006 CR4: 00000000007706e0
[11413.243346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11413.243349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11413.243352] PKRU: 55555554
[11413.243354] Call Trace:
[11413.243357]  <NMI>
[11413.243359]  ? watchdog_hardlockup_check.part.0+0xa4/0x170
[11413.243368]  ? __perf_event_overflow+0x1d0/0x310
[11413.243379]  ? handle_pmi_common+0x2a8/0x470
[11413.243389]  ? __pfx_handle_pmi_common+0x10/0x10
[11413.243397]  ? __pfx_native_set_fixmap+0x10/0x10
[11413.243405]  ? apei_read+0x115/0x120
[11413.243414]  ? __pfx_apei_read+0x10/0x10
[11413.243418]  ? memcpy_fromio+0x22/0xd0
[11413.243429]  ? ghes_copy_tofrom_phys+0x74/0x130
[11413.243438]  ? __ghes_peek_estatus.isra.0+0x8b/0xf0
[11413.243445]  ? ghes_in_nmi_queue_one_entry.constprop.0+0x98/0x3f0
[11413.243452]  ? rcu_is_watching+0x1f/0x40
[11413.243459]  ? lock_acquire+0xc7/0x3b0
[11413.243469]  ? intel_bts_interrupt+0x7d/0x1b0
[11413.243476]  ? intel_pmu_handle_irq+0x158/0x320
[11413.243483]  ? perf_event_nmi_handler+0x39/0x60
[11413.243490]  ? nmi_handle+0x100/0x300
[11413.243501]  ? default_do_nmi+0x45/0x110
[11413.243508]  ? exc_nmi+0x100/0x190
[11413.243513]  ? end_repeat_nmi+0x16/0x67
[11413.243524]  ? native_queued_spin_lock_slowpath+0x494/0x580
[11413.243531]  ? native_queued_spin_lock_slowpath+0x48a/0x580
[11413.243538]  ? native_queued_spin_lock_slowpath+0x48a/0x580
[11413.243546]  ? native_queued_spin_lock_slowpath+0x48a/0x580
[11413.243553]  </NMI>
[11413.243555]  <IRQ>
[11413.243557]  ? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
[11413.243564]  ? rcu_is_watching+0x1f/0x40
[11413.243568]  ? lock_acquire+0xc7/0x3b0
[11413.243576]  do_raw_spin_lock+0x19b/0x1b0
[11413.243582]  ? __pfx_do_raw_spin_lock+0x10/0x10
[11413.243587]  ? __pfx_do_raw_spin_trylock+0x10/0x10
[11413.243595]  _raw_spin_lock_nested+0x61/0x80
[11413.243601]  raw_spin_rq_lock_nested+0x11/0x20
[11413.243609]  distribute_cfs_runtime+0xc8/0x560
[11413.243620]  sched_cfs_period_timer+0x1a4/0x360
[11413.243630]  __run_hrtimer+0x2a8/0x490
[11413.243637]  ? __pfx_sched_cfs_period_timer+0x10/0x10
[11413.243643]  ? __pfx_lock_acquire+0x10/0x10
[11413.243650]  __hrtimer_run_queues+0x108/0x190
[11413.243657]  ? __pfx___hrtimer_run_queues+0x10/0x10
[11413.243662]  ? rcu_is_watching+0x1f/0x40
[11413.243666]  ? lock_release+0x4b/0x160
[11413.243673]  ? ktime_get_update_offsets_now+0x5e/0x1d0
[11413.243680]  ? ktime_get_update_offsets_now+0xd9/0x1d0
[11413.243688]  hrtimer_interrupt+0x1a0/0x360
[11413.243694]  ? lock_release+0x4b/0x160
[11413.243703]  __sysvec_apic_timer_interrupt+0x8b/0x230
[11413.243711]  sysvec_apic_timer_interrupt+0x6d/0x90
[11413.243718]  </IRQ>
[11413.243720]  <TASK>
[11413.243722]  asm_sysvec_apic_timer_interrupt+0x16/0x20
[11413.243728] RIP: 0010:cpuidle_enter_state+0x12f/0x570
[11413.243735] Code: 8b 14 24 48 0f a3 15 40 2f ff 01 0f 82 eb 02 00 00 31 ff e8 e3 26 c0 fe 45 84 ff 0f 85 bb 02 00 00 e8 b5 0e dc fe fb 45 85 e4 <0f> 88 28 02 00 00 49 63 c4 4c 2b 6c 24 08 48 8d 14 40 48 8d 14 90
[11413.243748] RSP: 0018:ffff88c046f37da0 EFLAGS: 00000202
[11413.243752] RAX: 0000000000000000 RBX: ffffe8fffc60c400 RCX: dffffc0000000000
[11413.243756] RDX: 0000000000000003 RSI: 0000000000000008 RDI: ffffffffae0217fb
[11413.243759] RBP: ffffffffafca9f60 R08: 0000000000000000 R09: fffffbfff60028e5
[11413.243762] R10: ffffffffb001472f R11: 0000000000000000 R12: 0000000000000002
[11413.243766] R13: 00000a59efec1784 R14: 0000000000000002 R15: 0000000000000000
[11413.243772]  ? cpuidle_enter_state+0x12b/0x570
[11413.243780]  ? __pfx___schedule+0x10/0x10
[11413.243790]  cpuidle_enter+0x3c/0x60
[11413.243800]  cpuidle_idle_call+0x168/0x1e0
[11413.243805]  ? __pfx_cpuidle_idle_call+0x10/0x10
[11413.243810]  ? lock_release+0x20/0x160
[11413.243816]  ? sched_clock+0xc/0x20
[11413.243822]  ? sched_clock_cpu+0x11/0x130
[11413.243829]  ? tsc_verify_tsc_adjust+0x134/0x170
[11413.243834]  ? rcu_nocb_flush_deferred_wakeup+0x28/0xc0
[11413.243841]  do_idle+0xbd/0x110
[11413.243846]  cpu_startup_entry+0x30/0x40
[11413.243851]  start_secondary+0x1a4/0x1d0
[11413.243861]  ? __pfx_start_secondary+0x10/0x10
[11413.243870]  secondary_startup_64_no_verify+0x18f/0x19b
[11413.243883]  </TASK>
[11413.243886] Kernel panic - not syncing: Hard LOCKUP
[11413.243889] CPU: 82 PID: 0 Comm: swapper/82 Kdump: loaded Tainted: G S         OE      6.6.0-9ded7a40d8e3 #10
[11413.243895] Hardware name: Powerleader PR2715P3/T1DM-E2, BIOS OEM57.22 11/23/2021
[11413.243898] Call Trace:
[11413.243900]  <NMI>
[11413.243902]  dump_stack_lvl+0x47/0x80
[11413.243911]  panic+0x20a/0x410
[11413.243917]  ? __pfx_panic+0x10/0x10
[11413.243925]  ? nmi_panic+0x27/0x70
[11413.243931]  nmi_panic+0x5a/0x70
[11413.243936]  watchdog_hardlockup_check.part.0+0x15f/0x170
[11413.243944]  __perf_event_overflow+0x1d0/0x310
[11413.243952]  handle_pmi_common+0x2a8/0x470
[11413.243961]  ? __pfx_handle_pmi_common+0x10/0x10
[11413.243969]  ? __pfx_native_set_fixmap+0x10/0x10
[11413.243975]  ? apei_read+0x115/0x120
[11413.243980]  ? __pfx_apei_read+0x10/0x10
[11413.243984]  ? memcpy_fromio+0x22/0xd0
[11413.243991]  ? ghes_copy_tofrom_phys+0x74/0x130
[11413.243998]  ? __ghes_peek_estatus.isra.0+0x8b/0xf0
[11413.244006]  ? ghes_in_nmi_queue_one_entry.constprop.0+0x98/0x3f0
[11413.244013]  ? rcu_is_watching+0x1f/0x40
[11413.244017]  ? lock_acquire+0xc7/0x3b0
[11413.244024]  ? intel_bts_interrupt+0x7d/0x1b0
[11413.244031]  intel_pmu_handle_irq+0x158/0x320
[11413.244038]  perf_event_nmi_handler+0x39/0x60
[11413.244045]  nmi_handle+0x100/0x300
[11413.244053]  default_do_nmi+0x45/0x110
[11413.244058]  exc_nmi+0x100/0x190
[11413.244063]  end_repeat_nmi+0x16/0x67
[11413.244068] RIP: 0010:native_queued_spin_lock_slowpath+0x48a/0x580
[11413.244075] Code: 1e eb 02 f3 90 4c 89 ff e8 43 b2 05 ff 8b 43 08 85 c0 74 ef 48 89 df e8 84 b3 05 ff 4c 8b 33 4d 85 f6 74 08 41 0f 0d 0e eb 02 <f3> 90 48 89 ef e8 1c b2 05 ff 8b 45 00 66 85 c0 75 ee 89 c2 66 31
[11413.244080] RSP: 0018:ffff88febb809c00 EFLAGS: 00000002
[11413.244083] RAX: 0000000001380101 RBX: ffff88febba09880 RCX: dffffc0000000000
[11413.244087] RDX: 0000000000000003 RSI: 0000000000000002 RDI: ffff88feb3e087c0
[11413.244090] RBP: ffff88feb3e087c0 R08: ffffffffae045b34 R09: ffffed1fd67c10f8
[11413.244094] R10: ffff88feb3e087c3 R11: ffff88febb809ff8 R12: 00000000014c0000
[11413.244097] R13: 1ffff11fd7701381 R14: 0000000000000000 R15: ffff88febba09888
[11413.244103]  ? native_queued_spin_lock_slowpath+0x494/0x580
[11413.244112]  ? native_queued_spin_lock_slowpath+0x48a/0x580
[11413.244119]  ? native_queued_spin_lock_slowpath+0x48a/0x580
[11413.244126]  </NMI>
[11413.244128]  <IRQ>
[11413.244130]  ? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
[11413.244137]  ? rcu_is_watching+0x1f/0x40
[11413.244142]  ? lock_acquire+0xc7/0x3b0
[11413.244149]  do_raw_spin_lock+0x19b/0x1b0
[11413.244155]  ? __pfx_do_raw_spin_lock+0x10/0x10
[11413.244160]  ? __pfx_do_raw_spin_trylock+0x10/0x10
[11413.244168]  _raw_spin_lock_nested+0x61/0x80
[11413.244174]  raw_spin_rq_lock_nested+0x11/0x20
[11413.244179]  distribute_cfs_runtime+0xc8/0x560
[11413.244189]  sched_cfs_period_timer+0x1a4/0x360
[11413.244199]  __run_hrtimer+0x2a8/0x490
[11413.244204]  ? __pfx_sched_cfs_period_timer+0x10/0x10
[11413.244211]  ? __pfx_lock_acquire+0x10/0x10
[11413.244218]  __hrtimer_run_queues+0x108/0x190
[11413.244224]  ? __pfx___hrtimer_run_queues+0x10/0x10
[11413.244229]  ? rcu_is_watching+0x1f/0x40
[11413.244234]  ? lock_release+0x4b/0x160
[11413.244240]  ? ktime_get_update_offsets_now+0x5e/0x1d0
[11413.244246]  ? ktime_get_update_offsets_now+0xd9/0x1d0
[11413.244254]  hrtimer_interrupt+0x1a0/0x360
[11413.244260]  ? lock_release+0x4b/0x160
[11413.244268]  __sysvec_apic_timer_interrupt+0x8b/0x230
[11413.244274]  sysvec_apic_timer_interrupt+0x6d/0x90
[11413.244280]  </IRQ>
[11413.244282]  <TASK>
[11413.244284]  asm_sysvec_apic_timer_interrupt+0x16/0x20
[11413.244289] RIP: 0010:cpuidle_enter_state+0x12f/0x570
[11413.244295] Code: 8b 14 24 48 0f a3 15 40 2f ff 01 0f 82 eb 02 00 00 31 ff e8 e3 26 c0 fe 45 84 ff 0f 85 bb 02 00 00 e8 b5 0e dc fe fb 45 85 e4 <0f> 88 28 02 00 00 49 63 c4 4c 2b 6c 24 08 48 8d 14 40 48 8d 14 90
[11413.244300] RSP: 0018:ffff88c046f37da0 EFLAGS: 00000202
[11413.244304] RAX: 0000000000000000 RBX: ffffe8fffc60c400 RCX: dffffc0000000000
[11413.244308] RDX: 0000000000000003 RSI: 0000000000000008 RDI: ffffffffae0217fb
[11413.244311] RBP: ffffffffafca9f60 R08: 0000000000000000 R09: fffffbfff60028e5
[11413.244314] R10: ffffffffb001472f R11: 0000000000000000 R12: 0000000000000002
[11413.244317] R13: 00000a59efec1784 R14: 0000000000000002 R15: 0000000000000000
[11413.244324]  ? cpuidle_enter_state+0x12b/0x570
[11413.244332]  ? __pfx___schedule+0x10/0x10
[11413.244338]  cpuidle_enter+0x3c/0x60
[11413.244344]  cpuidle_idle_call+0x168/0x1e0
[11413.244349]  ? __pfx_cpuidle_idle_call+0x10/0x10
[11413.244354]  ? lock_release+0x20/0x160
[11413.244359]  ? sched_clock+0xc/0x20
[11413.244364]  ? sched_clock_cpu+0x11/0x130
[11413.244369]  ? tsc_verify_tsc_adjust+0x134/0x170
[11413.244373]  ? rcu_nocb_flush_deferred_wakeup+0x28/0xc0
[11413.244379]  do_idle+0xbd/0x110
[11413.244384]  cpu_startup_entry+0x30/0x40
[11413.244389]  start_secondary+0x1a4/0x1d0
[11413.244396]  ? __pfx_start_secondary+0x10/0x10
[11413.244404]  secondary_startup_64_no_verify+0x18f/0x19b
[11413.244415]  </TASK>
[11413.246054] kexec: Bye!

**【缺陷详情及分析指导参考链接】**

GVP openEuler/kernel

内容风险标识

评论 (1)

GVPopenEuler/kernel

内容风险标识