BUG: soft lockup in __mmap_region

syzkaller测试，出现perf_mmap的softlockup问题：
[ 169.095303] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [syz-executor.15:4042]
[ 169.095599] Modules linked in:
[ 169.095853] CPU: 3 PID: 4042 Comm: syz-executor.15 Tainted: G L 4.19.90+ #2
[ 169.096206] Hardware name: linux,dummy-virt (DT)
[ 169.096454] pstate: 60000005 (nZCv daif -PAN -UAO)
[ 169.096672] pc : __ll_sc___cmpxchg_case_acq_8+0x0/0x20
[ 169.096860] lr : mutex_lock+0x28/0x58
[ 169.096986] sp : ffff00000f1e3bb0
[ 169.097099] x29: ffff00000f1e3bb0 x28: ffff0000097bbec0
[ 169.097252] x27: ffff000009870650 x26: 0000000000000001
[ 169.097412] x25: 0000000000000000 x24: 0000000000000002
[ 169.097583] x23: ffff8000d3f8aa90 x22: ffff8000d3f8a800
[ 169.097761] x21: ffff8000c85ef1d0 x20: 0000000000000000
[ 169.097980] x19: ffff8000d3f8aa90 x18: 0000000000000000
[ 169.098244] x17: 0000000000000000 x16: ffff8000d1382200
[ 169.098578] x15: 0000000000000000 x14: 0000000000000000
[ 169.098873] x13: 0000000000000000 x12: 0000000000000000
[ 169.099178] x11: 0000000000000000 x10: 0000000000000d00
[ 169.099547] x9 : ffff00000f1e3a30 x8 : ffff8000d1382f60
[ 169.100064] x7 : 0000000000000000 x6 : 00000005cc69ca47
[ 169.100380] x5 : 0000000000011704 x4 : ffff00000f1e3b68
[ 169.100627] x3 : dead000000000200 x2 : ffff8000d1382200
[ 169.100843] x1 : 0000000000000000 x0 : ffff8000d3f8aa90
[ 169.101062] Call trace:
[ 169.101188] __ll_sc___cmpxchg_case_acq_8+0x0/0x20
[ 169.101377] perf_mmap+0x218/0x5b0
[ 169.101533] __mmap_region+0x3e8/0x658
[ 169.101683] do_mmap+0x388/0x470
[ 169.101859] vm_mmap_pgoff+0xf0/0x128
[ 169.102006] ksys_mmap_pgoff+0xb8/0x348
[ 169.102166] __arm64_sys_mmap+0x34/0x48
[ 169.102327] el0_svc_handler+0x94/0x260
[ 169.102482] el0_svc+0x10/0x380
[ 169.102639] Sending NMI from CPU 3 to CPUs 0-2:
[ 169.103347] NMI backtrace for cpu 2
[ 169.103366] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 4.19.90+ #2
[ 169.103373] Hardware name: linux,dummy-virt (DT)
[ 169.103379] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 169.103384] pc : arch_cpu_idle+0x38/0x240
[ 169.103391] lr : arch_cpu_idle+0x30/0x240
[ 169.103396] sp : ffff00000aa03ec0
[ 169.103401] x29: ffff00000aa03ec0 x28: ffff0000097ccc40
[ 169.103416] x27: 0000000000000000 x26: ffff0000097ccba0
[ 169.103429] x25: ffff000009211878 x24: ffff000009049000
[ 169.103441] x23: 0000000000000000 x22: ffff000009782000
[ 169.103453] x21: 0000000000000004 x20: ffff000009781b58
[ 169.103465] x19: ffff000009204000 x18: 0000000000000000
[ 169.103477] x17: 00000000000000a8 x16: 0000000000000000
[ 169.103489] x15: 0000000000000000 x14: 0000000000000000
[ 169.103500] x13: 0000000000000000 x12: 0000000000000000
[ 169.103512] x11: 0000000000000000 x10: 0000000000000d00
[ 169.103524] x9 : ffff00000aa03e50 x8 : ffff8000c0bc8d60
[ 169.103536] x7 : ffff8000fffb7cc0 x6 : 0000000000000002
[ 169.103548] x5 : ffff00000aa03e70 x4 : 00008000f6da5000
[ 169.103562] x3 : 00008000f6da5000 x2 : ffff00000aa03ec0
[ 169.103574] x1 : ffff8000c0bc8000 x0 : 00000000000000e0
[ 169.103586] Call trace:
[ 169.103592] arch_cpu_idle+0x38/0x240
[ 169.103597] default_idle_call+0x30/0x70
[ 169.103601] do_idle+0x210/0x340
[ 169.103607] cpu_startup_entry+0x2c/0x88
[ 169.103612] secondary_start_kernel+0x1e4/0x310
[ 169.103629] NMI backtrace for cpu 1
[ 169.103681] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G L 4.19.90+ #2
[ 169.103698] Hardware name: linux,dummy-virt (DT)
[ 169.103841] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 169.103861] pc : arch_cpu_idle+0x38/0x240
[ 169.103905] lr : arch_cpu_idle+0x30/0x240
[ 169.103981] sp : ffff00000a9fbec0
[ 169.103996] x29: ffff00000a9fbec0 x28: ffff0000097ccc40
[ 169.104142] x27: 0000000000000000 x26: ffff0000097ccba0
[ 169.104174] x25: ffff000009211878 x24: ffff000009049000
[ 169.104266] x23: 0000000000000000 x22: ffff000009782000
[ 169.104297] x21: 0000000000000002 x20: ffff000009781b58
[ 169.104365] x19: ffff000009204000 x18: 0000000000000000
[ 169.104421] x17: 00000000000000a8 x16: 0000000000000000
[ 169.104457] x15: 0000000000000000 x14: 0000000000000000
[ 169.104525] x13: 0000000000000000 x12: 0000000000000000
[ 169.104573] x11: 0000000000000000 x10: 0000000000000d00
[ 169.104626] x9 : ffff00000a9fbe50 x8 : ffff8000c0bcc060
[ 169.104725] x7 : ffff8000fff9dcc0 x6 : 0000000000000002
[ 169.104859] x5 : ffff00000a9fbe70 x4 : 0000000000000000
[ 169.104958] x3 : 00008000f6d8b000 x2 : ffff00000a9fbec0
[ 169.105045] x1 : ffff8000c0bcb300 x0 : 00000000000000e0
[ 169.105119] Call trace:
[ 169.105130] arch_cpu_idle+0x38/0x240
[ 169.105219] default_idle_call+0x30/0x70
[ 169.105235] do_idle+0x210/0x340
[ 169.105252] cpu_startup_entry+0x28/0x88
[ 169.105268] secondary_start_kernel+0x1e4/0x310
[ 169.105294] NMI backtrace for cpu 0
[ 169.105347] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 4.19.90+ #2
[ 169.105417] Hardware name: linux,dummy-virt (DT)
[ 169.105431] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 169.105484] pc : arch_cpu_idle+0x38/0x240
[ 169.105519] lr : arch_cpu_idle+0x30/0x240
[ 169.105565] sp : ffff000009773e70
[ 169.105583] x29: ffff000009773e70 x28: ffff0000097ccc40
[ 169.105763] x27: 0000000000000000 x26: ffff0000097ccba0
[ 169.105890] x25: ffff000009211878 x24: ffff000009049000
[ 169.105940] x23: 0000000000000000 x22: ffff000009782000
[ 169.105987] x21: 0000000000000001 x20: ffff000009781b58
[ 169.106057] x19: ffff000009204000 x18: 0000000000000000
[ 169.106119] x17: 00000000000000a8 x16: 0000000000000000
[ 169.106156] x15: 0000000000000000 x14: 0000000000000000
[ 169.106199] x13: 0000000000000000 x12: 0000000000000000
[ 169.106311] x11: 0000000000000000 x10: 0000000000000d00
[ 169.106402] x9 : ffff000009773e00 x8 : ffff00000978e3e0
[ 169.106492] x7 : ffff8000fff83cc0 x6 : ffff000009833bf8
[ 169.106564] x5 : 000000254db593f0 x4 : 0000000000000000
[ 169.106644] x3 : 00008000f6d71000 x2 : ffff000009773e70
[ 169.106674] x1 : ffff00000978d680 x0 : 00000000000000e0
[ 169.106821] Call trace:
[ 169.106855] arch_cpu_idle+0x38/0x240
[ 169.106902] default_idle_call+0x30/0x70
[ 169.106924] do_idle+0x210/0x340
[ 169.106964] cpu_startup_entry+0x28/0x88
[ 169.106995] rest_init+0xe0/0xf0
[ 169.107032] start_kernel+0x544/0x570

Hi yangjihong2021, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: Kernel, and any of the maintainers: @xiexiuqi , @yangyingliang , @gatieme , @jiaoff , @guohaocs2c , @hanjun-guo , @woqidaideshi , @zhengzengkai , @newbeats , @zhangyi089 , @colyli , @thundertown , @htforge , @chiqijun , @lengchao , @zhujianwei001 , @kylin-mayukun , @wangxiongfeng , @wkfxxx , @SuperSix173 , @jentlestea , @oskernel0719 , @lujialin2 , @gasonchen , @kailiu42 , @whoisxxx , @wuxu_buque , @koulihong , @liuxinux , @kevinzhu1 , @xukuohai , @alvin-ling , @juntianlinux , @chenguangli , @yuehaibing , @zhenpengzheng , @LiuYongQiang0816 , @yuzenghui

perf_mmap_close和ioctl存在条件竞争，导致在perf_mmap时进入死循环

CPU1 CPU2

perf_mmap_close(e2)
if (atomic_dec_and_test(&e2->rb->mmap_count)) // 1 - > 0
detach_rest = true

ioctl(e1, IOC_SET_OUTPUT, e2)
perf_event_set_output(e1, e2)

...
              list_for_each_entry_rcu(e, &e2->rb->event_list, rb_entry)
                ring_buffer_attach(e, NULL);
                // e1 isn't yet added and
                // therefore not detached

ring_buffer_attach(e1, e2->rb)
list_add_rcu(&e1->rb_entry,
&e2->rb->event_list)

After this; e1 is attached to an unmapped rb and a subsequent
perf_mmap() will loop forever more:

again:
                    mutex_lock(&e->mmap_mutex);
                    if (event->rb) {
                            ...
                            if (!atomic_inc_not_zero(&e->rb->mmap_count)) {
                                    ...
                                    mutex_unlock(&e->mmap_mutex);
                                    goto again;
                            }
                    }

回合主线修复补丁：68e3c69803dada336893640110cb87221bb01dcf

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 80782cddb1da..d2b354991bf5 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -6253,10 +6253,10 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)

                if (!atomic_inc_not_zero(&event->rb->mmap_count)) {
                        /*
-                        * Raced against perf_mmap_close() through
-                        * perf_event_set_output(). Try again, hope for better
-                        * luck.
+                        * Raced against perf_mmap_close(); remove the
+                        * event and try again.
                         */
+                       ring_buffer_attach(event, NULL);
                        mutex_unlock(&event->mmap_mutex);
                        goto again;
                }
@@ -11825,14 +11825,25 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr,
        goto out;
 }

+static void mutex_lock_double(struct mutex *a, struct mutex *b)
+{
+       if (b < a)
+               swap(a, b);
+
+       mutex_lock(a);
+       mutex_lock_nested(b, SINGLE_DEPTH_NESTING);
+}
+
 static int
 perf_event_set_output(struct perf_event *event, struct perf_event *output_event)
 {
        struct perf_buffer *rb = NULL;
        int ret = -EINVAL;

-       if (!output_event)
+       if (!output_event) {
+               mutex_lock(&event->mmap_mutex);
                goto set;
+       }

在perf_mmap时进入循环时，将ringbuffer attch成NULL，以跳出循环

合入修复补丁后，问题不再复现：
2022/07/25 09:29:55 parsed 270 programs
2022/07/25 09:29:55 executed programs: 0
2022/07/25 09:30:27 executed programs: 270
2022/07/25 09:30:33 executed programs: 540
2022/07/25 09:30:40 executed programs: 810
2022/07/25 09:30:46 executed programs: 1080
2022/07/25 09:30:53 executed programs: 1350
2022/07/25 09:30:59 executed programs: 1620
2022/07/25 09:31:06 executed programs: 1890
2022/07/25 09:31:12 executed programs: 2160
2022/07/25 09:31:24 executed programs: 2430
2022/07/25 09:31:35 executed programs: 2700
2022/07/25 09:31:46 executed programs: 2970
2022/07/25 09:31:59 executed programs: 3240
2022/07/25 09:32:12 executed programs: 3510
2022/07/25 09:32:24 executed programs: 3780
2022/07/25 09:32:36 executed programs: 4050
2022/07/25 09:32:52 executed programs: 4320
2022/07/25 09:33:08 executed programs: 4590
2022/07/25 09:33:23 executed programs: 4860
2022/07/25 09:33:36 executed programs: 5130
2022/07/25 09:33:48 executed programs: 5400
2022/07/25 09:33:56 executed programs: 5670
2022/07/25 09:34:02 executed programs: 5940
2022/07/25 09:34:09 executed programs: 6210
2022/07/25 09:34:16 executed programs: 6480
2022/07/25 09:34:22 executed programs: 6750
2022/07/25 09:34:28 executed programs: 7020
2022/07/25 09:34:35 executed programs: 7290
2022/07/25 09:34:42 executed programs: 7560
2022/07/25 09:34:48 executed programs: 7830
2022/07/25 09:34:55 executed programs: 8100
2022/07/25 09:35:02 executed programs: 8370
2022/07/25 09:35:08 executed programs: 8640
2022/07/25 09:35:15 executed programs: 8910
2022/07/25 09:35:21 executed programs: 9180
2022/07/25 09:35:28 executed programs: 9450
2022/07/25 09:35:34 executed programs: 9720
2022/07/25 09:35:40 executed programs: 9990
2022/07/25 09:35:46 executed programs: 10260
2022/07/25 09:35:53 executed programs: 10530
2022/07/25 09:36:00 executed programs: 10800
2022/07/25 09:36:06 executed programs: 11070
2022/07/25 09:36:13 executed programs: 11340
2022/07/25 09:36:20 executed programs: 11610
2022/07/25 09:36:26 executed programs: 11880
2022/07/25 09:36:33 executed programs: 12150
2022/07/25 09:36:39 executed programs: 12420
2022/07/25 09:36:46 executed programs: 12690
2022/07/25 09:36:52 executed programs: 12960
2022/07/25 09:36:59 executed programs: 13230
2022/07/25 09:37:05 executed programs: 13500
2022/07/25 09:37:12 executed programs: 13770
2022/07/25 09:37:18 executed programs: 14040
2022/07/25 09:37:24 executed programs: 14310
2022/07/25 09:37:30 executed programs: 14580
2022/07/25 09:37:37 executed programs: 14850
2022/07/25 09:37:43 executed programs: 15120
2022/07/25 09:37:49 executed programs: 15390
2022/07/25 09:37:55 executed programs: 15660
2022/07/25 09:38:02 executed programs: 15930
2022/07/25 09:38:08 executed programs: 16200
2022/07/25 09:38:15 executed programs: 16470
2022/07/25 09:38:21 executed programs: 16740
2022/07/25 09:38:27 executed programs: 17010
2022/07/25 09:38:34 executed programs: 17280
2022/07/25 09:38:40 executed programs: 17550
2022/07/25 09:38:47 executed programs: 17820
2022/07/25 09:38:53 executed programs: 18090
2022/07/25 09:39:00 executed programs: 18360
2022/07/25 09:39:06 executed programs: 18630
2022/07/25 09:39:13 executed programs: 18900
2022/07/25 09:39:19 executed programs: 19170
2022/07/25 09:39:26 executed programs: 19440
2022/07/25 09:39:32 executed programs: 19710
2022/07/25 09:39:39 executed programs: 19980
2022/07/25 09:39:46 executed programs: 20250
2022/07/25 09:39:53 executed programs: 20520
2022/07/25 09:39:59 executed programs: 20790
2022/07/25 09:40:05 executed programs: 21060
2022/07/25 09:40:12 executed programs: 21330
2022/07/25 09:40:18 executed programs: 21600
2022/07/25 09:40:25 executed programs: 21870
2022/07/25 09:40:31 executed programs: 22140
2022/07/25 09:40:38 executed programs: 22410
2022/07/25 09:40:44 executed programs: 22680
2022/07/25 09:40:50 executed programs: 22950
2022/07/25 09:40:57 executed programs: 23220
2022/07/25 09:41:04 executed programs: 23490
2022/07/25 09:41:10 executed programs: 23760
2022/07/25 09:41:17 executed programs: 24030
2022/07/25 09:41:23 executed programs: 24300
2022/07/25 09:41:29 executed programs: 24570
2022/07/25 09:41:36 executed programs: 24840
2022/07/25 09:41:42 executed programs: 25110
2022/07/25 09:41:49 executed programs: 25380
2022/07/25 09:41:55 executed programs: 25650
2022/07/25 09:42:02 executed programs: 25920
2022/07/25 09:42:08 executed programs: 26190
2022/07/25 09:42:14 executed programs: 26460
2022/07/25 09:42:21 executed programs: 26730
2022/07/25 09:42:27 executed programs: 27000
2022/07/25 09:42:34 executed programs: 27270
2022/07/25 09:42:40 executed programs: 27540
2022/07/25 09:42:47 executed programs: 27810
2022/07/25 09:42:53 executed programs: 28080
2022/07/25 09:43:00 executed programs: 28350
2022/07/25 09:43:06 executed programs: 28620
2022/07/25 09:43:13 executed programs: 28890
2022/07/25 09:43:20 executed programs: 29160
2022/07/25 09:43:26 executed programs: 29430
2022/07/25 09:43:33 executed programs: 29700
2022/07/25 09:43:39 executed programs: 29970
2022/07/25 09:43:46 executed programs: 30240
2022/07/25 09:43:53 executed programs: 30510
2022/07/25 09:43:59 executed programs: 30780
2022/07/25 09:44:06 executed programs: 31050
2022/07/25 09:44:12 executed programs: 31320
2022/07/25 09:44:18 executed programs: 31590
2022/07/25 09:44:25 executed programs: 31860
2022/07/25 09:44:31 executed programs: 32130
2022/07/25 09:44:38 executed programs: 32400
2022/07/25 09:44:44 executed programs: 32670
2022/07/25 09:44:50 executed programs: 32940
2022/07/25 09:44:56 executed programs: 33210
2022/07/25 09:45:03 executed programs: 33480
2022/07/25 09:45:09 executed programs: 33750
2022/07/25 09:45:15 executed programs: 34020
2022/07/25 09:45:22 executed programs: 34290
2022/07/25 09:45:28 executed programs: 34560
2022/07/25 09:45:35 executed programs: 34830
2022/07/25 09:45:41 executed programs: 35100
2022/07/25 09:45:48 executed programs: 35370
2022/07/25 09:45:56 executed programs: 35640
2022/07/25 09:46:04 executed programs: 35910
2022/07/25 09:46:11 executed programs: 36180
2022/07/25 09:46:18 executed programs: 36450
2022/07/25 09:46:24 executed programs: 36720
2022/07/25 09:46:31 executed programs: 36990
2022/07/25 09:46:37 executed programs: 37260
2022/07/25 09:46:44 executed programs: 37530
2022/07/25 09:46:50 executed programs: 37800
2022/07/25 09:46:57 executed programs: 38070
2022/07/25 09:47:03 executed programs: 38340
2022/07/25 09:47:10 executed programs: 38610
2022/07/25 09:47:16 executed programs: 38880
2022/07/25 09:47:23 executed programs: 39150
2022/07/25 09:47:29 executed programs: 39420
2022/07/25 09:47:35 executed programs: 39690
2022/07/25 09:47:41 executed programs: 39960
2022/07/25 09:47:47 executed programs: 40230
2022/07/25 09:47:54 executed programs: 40500
2022/07/25 09:48:01 executed programs: 40770
2022/07/25 09:48:07 executed programs: 41040
2022/07/25 09:48:14 executed programs: 41310
2022/07/25 09:48:20 executed programs: 41580
2022/07/25 09:48:26 executed programs: 41850
2022/07/25 09:48:33 executed programs: 42120
2022/07/25 09:48:39 executed programs: 42390
2022/07/25 09:48:45 executed programs: 42660
2022/07/25 09:48:51 executed programs: 42930
2022/07/25 09:48:58 executed programs: 43200
2022/07/25 09:49:05 executed programs: 43470
2022/07/25 09:49:11 executed programs: 43740
2022/07/25 09:49:17 executed programs: 44010
2022/07/25 09:49:24 executed programs: 44280
2022/07/25 09:49:31 executed programs: 44550
2022/07/25 09:49:37 executed programs: 44820
2022/07/25 09:49:44 executed programs: 45090
2022/07/25 09:49:50 executed programs: 45360
2022/07/25 09:49:57 executed programs: 45630
2022/07/25 09:50:04 executed programs: 45900
2022/07/25 09:50:10 executed programs: 46170
2022/07/25 09:50:17 executed programs: 46440
2022/07/25 09:50:24 executed programs: 46710
2022/07/25 09:50:30 executed programs: 46980
2022/07/25 09:50:36 executed programs: 47250
2022/07/25 09:50:43 executed programs: 47520
2022/07/25 09:50:49 executed programs: 47790
2022/07/25 09:50:55 executed programs: 48060
2022/07/25 09:51:02 executed programs: 48330
2022/07/25 09:51:08 executed programs: 48600
2022/07/25 09:51:15 executed programs: 48870
2022/07/25 09:51:21 executed programs: 49140
2022/07/25 09:51:27 executed programs: 49410
2022/07/25 09:51:34 executed programs: 49680
2022/07/25 09:51:40 executed programs: 49950
2022/07/25 09:51:47 executed programs: 50220
2022/07/25 09:51:53 executed programs: 50490
2022/07/25 09:52:00 executed programs: 50760
2022/07/25 09:52:07 executed programs: 51030
2022/07/25 09:52:13 executed programs: 51300
2022/07/25 09:52:20 executed programs: 51570
2022/07/25 09:52:26 executed programs: 51840
2022/07/25 09:52:33 executed programs: 52110
2022/07/25 09:52:39 executed programs: 52380
2022/07/25 09:52:46 executed programs: 52650
2022/07/25 09:52:53 executed programs: 52920
2022/07/25 09:52:59 executed programs: 53190
2022/07/25 09:53:06 executed programs: 53460
2022/07/25 09:53:12 executed programs: 53730
2022/07/25 09:53:19 executed programs: 54000
2022/07/25 09:53:25 executed programs: 54270
2022/07/25 09:53:31 executed programs: 54540
2022/07/25 09:53:38 executed programs: 54810
2022/07/25 09:53:44 executed programs: 55080
2022/07/25 09:53:50 executed programs: 55350
2022/07/25 09:53:57 executed programs: 55620
2022/07/25 09:54:04 executed programs: 55890
2022/07/25 09:54:10 executed programs: 56160
2022/07/25 09:54:16 executed programs: 56430
2022/07/25 09:54:23 executed programs: 56700
2022/07/25 09:54:30 executed programs: 56970
2022/07/25 09:54:37 executed programs: 57240
2022/07/25 09:54:43 executed programs: 57510
2022/07/25 09:54:49 executed programs: 57780
2022/07/25 09:54:56 executed programs: 58050
2022/07/25 09:55:02 executed programs: 58320
2022/07/25 09:55:08 executed programs: 58590
2022/07/25 09:55:15 executed programs: 58860
2022/07/25 09:55:21 executed programs: 59130
2022/07/25 09:55:27 executed programs: 59400
2022/07/25 09:55:34 executed programs: 59670
2022/07/25 09:55:41 executed programs: 59940
2022/07/25 09:55:48 executed programs: 60210
2022/07/25 09:55:55 executed programs: 60480
2022/07/25 09:56:01 executed programs: 60750

Resolved

src-openEuler/kernel

内容风险标识

评论 (5)

src-openEuler/kernel .gitee-modal { width: 500px !important; }

内容风险标识