【olk-5.10】nfs4_callback_compound hungtask

INFO: task NFSv4 callback:93188 blocked for more than 1310 seconds.
      Not tainted 5.10.0-00574-gc310db27923f-dirty #16
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:NFSv4 callback  state:D stack:    0 pid:93188 ppid:     2 flags:0x00000208
Call trace:
 __switch_to+0x98/0xdc arch/arm64/kernel/process.c:639
 context_switch kernel/sched/core.c:4055 [inline]
 __schedule+0x690/0xf0c kernel/sched/core.c:5003
 schedule+0xd8/0x220 kernel/sched/core.c:5139
 schedule_timeout+0x390/0x42c kernel/time/timer.c:2126
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common+0x148/0x240 kernel/sched/completion.c:117
 wait_for_completion+0x20/0x30 kernel/sched/completion.c:138
 kthread_stop+0xf8/0x39c kernel/kthread.c:671
 svc_stop_kthreads+0x138/0x260 net/sunrpc/svc.c:797
 svc_set_num_threads+0xc8/0xe4 net/sunrpc/svc.c:818
 nfs_callback_down+0x120/0x200 fs/nfs/callback.c:325
 nfs4_destroy_callback fs/nfs/nfs4client.c:279 [inline]
 nfs4_shutdown_client+0x138/0x150 fs/nfs/nfs4client.c:287
 nfs4_free_client+0x20/0x40 fs/nfs/nfs4client.c:303
 nfs_put_client+0x238/0x340 fs/nfs/client.c:273
 nfs4_callback_compound+0x2cc/0x630 fs/nfs/callback_xdr.c:981
 nfs_callback_dispatch+0x44/0x6c fs/nfs/callback_xdr.c:995
 svc_process_common+0x9a0/0xdf0 net/sunrpc/svc.c:1400
 svc_process+0x138/0x1c4 net/sunrpc/svc.c:1542
 nfs4_callback_svc+0x58/0x90 fs/nfs/callback.c:89
 kthread+0x1e0/0x220 kernel/kthread.c:328
 ret_from_fork+0x10/0x18 arch/arm64/kernel/entry.S:1727
INFO: task umount.nfs4:93544 blocked for more than 1310 seconds.
      Not tainted 5.10.0-00574-gc310db27923f-dirty #16
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:umount.nfs4     state:D stack:    0 pid:93544 ppid:     1 flags:0x00000209
Call trace:
 __switch_to+0x98/0xdc arch/arm64/kernel/process.c:639
 context_switch kernel/sched/core.c:4055 [inline]
 __schedule+0x690/0xf0c kernel/sched/core.c:5003
 schedule+0xd8/0x220 kernel/sched/core.c:5139
 schedule_preempt_disabled+0x44/0x74 kernel/sched/core.c:5198
 __mutex_lock_common kernel/locking/mutex.c:1086 [inline]
 __mutex_lock+0x3b8/0xc40 kernel/locking/mutex.c:1148
 mutex_lock_nested+0x2c/0x3c kernel/locking/mutex.c:1163
 nfs_callback_down+0x64/0x200 fs/nfs/callback.c:319
 nfs4_destroy_callback fs/nfs/nfs4client.c:279 [inline]
 nfs4_shutdown_client+0x138/0x150 fs/nfs/nfs4client.c:287
 nfs4_free_client+0x20/0x40 fs/nfs/nfs4client.c:303
 nfs_put_client+0x238/0x340 fs/nfs/client.c:273
 nfs_free_server+0x78/0x13c fs/nfs/client.c:964
 nfs_kill_super+0x48/0x64 fs/nfs/super.c:1336
 deactivate_locked_super+0x84/0x124 fs/super.c:341
 deactivate_super+0xac/0xcc fs/super.c:372
 cleanup_mnt+0x184/0x240 fs/namespace.c:1143
 __cleanup_mnt+0x1c/0x30 fs/namespace.c:1150
 task_work_run+0xc8/0x14c kernel/task_work.c:187
 tracehook_notify_resume include/linux/tracehook.h:188 [inline]
 do_notify_resume+0x23c/0x24c arch/arm64/kernel/signal.c:728
 work_pending+0xc/0xa0
INFO: task mount.nfs:93730 blocked for more than 1310 seconds.
      Not tainted 5.10.0-00574-gc310db27923f-dirty #16
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mount.nfs       state:D stack:    0 pid:93730 ppid:     1 flags:0x00000209
Call trace:
 __switch_to+0x98/0xdc arch/arm64/kernel/process.c:639
 context_switch kernel/sched/core.c:4055 [inline]
 __schedule+0x690/0xf0c kernel/sched/core.c:5003
 schedule+0xd8/0x220 kernel/sched/core.c:5139
 schedule_preempt_disabled+0x44/0x74 kernel/sched/core.c:5198
 __mutex_lock_common kernel/locking/mutex.c:1086 [inline]
 __mutex_lock+0x3b8/0xc40 kernel/locking/mutex.c:1148
 mutex_lock_nested+0x2c/0x3c kernel/locking/mutex.c:1163
 nfs_callback_up+0x58/0x280 fs/nfs/callback.c:280
 nfs4_init_callback fs/nfs/nfs4client.c:323 [inline]
 nfs4_init_client_minor_version fs/nfs/nfs4client.c:406 [inline]
 nfs4_init_client+0x154/0x260 fs/nfs/nfs4client.c:427
 nfs_get_client+0x3a8/0x450 fs/nfs/client.c:429
 nfs4_set_client+0x168/0x250 fs/nfs/nfs4client.c:902
 nfs4_init_server+0x1f4/0x3f0 fs/nfs/nfs4client.c:1120
 nfs4_create_server+0x94/0x130 fs/nfs/nfs4client.c:1168
 nfs4_try_get_tree+0x50/0x14c fs/nfs/nfs4super.c:226
 nfs_get_tree+0x7c/0xcc fs/nfs/fs_context.c:1359
 vfs_get_tree+0x54/0x180 fs/super.c:1576
 do_new_mount+0x1a8/0x220 fs/namespace.c:2950
 path_mount+0x248/0x344 fs/namespace.c:3283
 do_mount fs/namespace.c:3296 [inline]
 __do_sys_mount fs/namespace.c:3507 [inline]
 __se_sys_mount fs/namespace.c:3481 [inline]
 __arm64_sys_mount+0x19c/0x1dc fs/namespace.c:3481
 __invoke_syscall arch/arm64/kernel/syscall.c:36 [inline]
 invoke_syscall arch/arm64/kernel/syscall.c:48 [inline]
 el0_svc_common.constprop.0+0xac/0x244 arch/arm64/kernel/syscall.c:155
 do_el0_svc+0x8c/0xbc arch/arm64/kernel/syscall.c:217
 el0_svc+0x20/0x30 arch/arm64/kernel/entry-common.c:385
 el0_sync_handler+0xb0/0xbc arch/arm64/kernel/entry-common.c:417
 el0_sync+0x160/0x180 arch/arm64/kernel/entry.S:1190

Showing all locks held in the system:
1 lock held by khungtaskd/57:
 #0: fffff2c5499cdfa0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x44/0x268 kernel/locking/lockdep.c:6396
1 lock held by systemd-journal/218:
1 lock held by NFSv4 callback/93188:
 #0: fffff2c549de1ec8 (nfs_callback_mutex){+.+.}-{3:3}, at: nfs_callback_down+0x64/0x200 fs/nfs/callback.c:319
1 lock held by umount.nfs4/93544:
 #0: fffff2c549de1ec8 (nfs_callback_mutex){+.+.}-{3:3}, at: nfs_callback_down+0x64/0x200 fs/nfs/callback.c:319
1 lock held by mount.nfs/93730:
 #0: fffff2c549de1ec8 (nfs_callback_mutex){+.+.}-{3:3}, at: nfs_callback_up+0x58/0x280 fs/nfs/callback.c:280

=============================================

Hi yang-erkun, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: Kernel, and any of the maintainers.

nfs的callback线程在调用nfs4_callback_compound过程中，会首先find client，然后put client，如果此次put client是最后一次，那么其会一直调用到nfs_callback_down，该函数会发送信号给所有的callback线程，然后等对应线程退出，从而形成死锁，当前思路是通过异步方式做该逻辑。

INFO: task NFSv4 callback:93188 blocked for more than 1310 seconds.
Not tainted 5.10.0-00574-gc310db27923f-dirty #16
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
task:NFSv4 callback state:D stack: 0 pid:93188 ppid: 2 flags:0x00000208
Call trace:
__switch_to+0x98/0xdc arch/arm64/kernel/process.c:639
context_switch kernel/sched/core.c:4055 [inline]
__schedule+0x690/0xf0c kernel/sched/core.c:5003
schedule+0xd8/0x220 kernel/sched/core.c:5139
schedule_timeout+0x390/0x42c kernel/time/timer.c:2126
do_wait_for_common kernel/sched/completion.c:85 [inline]
__wait_for_common kernel/sched/completion.c:106 [inline]
wait_for_common+0x148/0x240 kernel/sched/completion.c:117
wait_for_completion+0x20/0x30 kernel/sched/completion.c:138
kthread_stop+0xf8/0x39c kernel/kthread.c:671
svc_stop_kthreads+0x138/0x260 net/sunrpc/svc.c:797
svc_set_num_threads+0xc8/0xe4 net/sunrpc/svc.c:818
nfs_callback_down+0x120/0x200 fs/nfs/callback.c:325
nfs4_destroy_callback fs/nfs/nfs4client.c:279 [inline]
nfs4_shutdown_client+0x138/0x150 fs/nfs/nfs4client.c:287
nfs4_free_client+0x20/0x40 fs/nfs/nfs4client.c:303
nfs_put_client+0x238/0x340 fs/nfs/client.c:273
nfs4_callback_compound+0x2cc/0x630 fs/nfs/callback_xdr.c:981
nfs_callback_dispatch+0x44/0x6c fs/nfs/callback_xdr.c:995
svc_process_common+0x9a0/0xdf0 net/sunrpc/svc.c:1400
svc_process+0x138/0x1c4 net/sunrpc/svc.c:1542
nfs4_callback_svc+0x58/0x90 fs/nfs/callback.c:89
kthread+0x1e0/0x220 kernel/kthread.c:328
ret_from_fork+0x10/0x18 arch/arm64/kernel/entry.S:1727

GVP openEuler/kernel

内容风险标识

评论 (2)

GVPopenEuler/kernel

内容风险标识