代码拉取完成，页面将自动刷新
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
谢谢反馈
请内核团队的同事了解
coding
5年前
coding
5年前
[root@pc-openeuler-1 log]#
Message from syslogd@pc-openeuler-1 at Apr 21 13:30:35 ...
kernel:[344259.358524] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dbus-daemon:1296]
我看每次报错都有 multipathd[1236]: sda: unusable path 的错误陆续报。
还有报 soft lockup 的地方，每次都是 virtio 先报。
Apr 20 04:00:40 pc-openeuler-1 chronyd[1307]: Can't synchronise: no selectable sources
Apr 20 04:00:44 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:00:49 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:00:55 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:01:01 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:01:09 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:02:04 pc-openeuler-1 kernel: [223721.182851] watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [kworker/3:0:11867]
Apr 20 04:02:05 pc-openeuler-1 kernel: [223724.162697] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.712488] dm_region_hash dm_log dm_mod
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.714635] CPU: 3 PID: 11867 Comm: kworker/3:0 Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.718953] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.721133] Workqueue: events virtio_gpu_fb_dirty_work [virtio_gpu]
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.722885] pstate: 00000005 (nzcv daif -PAN -UAO)
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.724391] pc : vp_notify+0x28/0x38 [virtio_pci]
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.726326] lr : virtqueue_kick+0x3c/0x78 [virtio_ring]
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.728013] sp : ffff0000110cfae0
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.729224] x29: ffff0000110cfae0 x28: 0000000000000000
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.730811] x27: ffff0000110cfc20 x26: 0000000000000001
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.733028] x25: 0000000000480020 x24: 0000000000000001
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.735203] x23: ffff800161c61140 x22: ffff80016cafa448
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.737428] x21: 0000000000000000 x20: ffff80016c6f6000
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.741074] x19: ffff80016c6f6000 x18: 0000000000000000
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.743481] x17: 0000000000000000 x16: 0000000000000000
Apr 20 04:02:06 pc-openeuler-1 kernel: [223724.744935] x15: 0000000000000000 x14: 0000000000000000
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.746376] x13: 0000000000000000 x12: 0000000000000000
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.747828] x11: 0000000000000000 x10: 0000000000000b80
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.749280] x9 : ffff0000110cfd40 x8 : ffff0000110cfbf8
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.751135] x7 : 00000000000011c0 x6 : ffff7fe000587180
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.752596] x5 : 000000000034e2bb x4 : ffff8001ff720ba0
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.754056] x3 : 0000000000000001 x2 : 0000000000000040
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.756162] x1 : ffff00000fda3000 x0 : 0000000000000000
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.758529] Call trace:
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.761243] vp_notify+0x28/0x38 [virtio_pci]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.763495] virtqueue_kick+0x3c/0x78 [virtio_ring]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.766302] virtio_gpu_queue_ctrl_buffer_locked+0x180/0x248 [virtio_gpu]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.768560] virtio_gpu_queue_fenced_ctrl_buffer+0xdc/0x160 [virtio_gpu]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.775386] virtio_gpu_cmd_transfer_to_host_2d+0xa4/0xd0 [virtio_gpu]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.777164] virtio_gpu_dirty_update+0x194/0x218 [virtio_gpu]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.778933] virtio_gpu_fb_dirty_work+0x3c/0x48 [virtio_gpu]
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.796026] process_one_work+0x1b0/0x448
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.800202] worker_thread+0x54/0x468
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.804346] kthread+0x134/0x138
Apr 20 04:02:09 pc-openeuler-1 kernel: [223724.808260] ret_from_fork+0x10/0x18
Apr 20 04:02:09 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:02:14 pc-openeuler-1 multipathd[1236]: sda: unusable path
multipathd应该没关系，关闭这个服务就好了。目前不支持多路径，后续已经默认关闭了这个服务
我觉得重点是看一下virtio调用栈，可能的原因
freesky-edward
5年前
coding
5年前
coding
5年前
coding
5年前
[root@pc-openeuler-1 ~]#
Message from syslogd@pc-openeuler-1 at Apr 21 16:00:31 ...
kernel:[353237.984311] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [kworker/1:2:15437]
Message from syslogd@pc-openeuler-1 at Apr 21 16:00:31 ...
kernel:[353254.729207] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:15454]
Message from syslogd@pc-openeuler-1 at Apr 21 16:01:31 ...
kernel:[353309.845852] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/0:1:18880]
[root@pc-openeuler-1 ~]# cat /proc/sys/kernel/watchdog
1
[root@pc-openeuler-1 ~]# cat /proc/sys/kernel/watchdog_cpumask
0-3
[root@pc-openeuler-1 ~]# cat /proc/sys/kernel/watchdog_thresh
5
[root@pc-openeuler-1 ~]#
soft lockup这个bug没有让系统彻底死机，但是若干个进程（或者kernel thread）被锁死在了某个状态（一般在内核区域），很多情况下这个是由于内核锁的使用的问题。
[root@pc-openeuler-1 ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
kernel.sysrq=0
net.ipv4.ip_forward=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.default.secure_redirects=0
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.tcp_syncookies=1
kernel.dmesg_restrict=1
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
[root@pc-openeuler-1 ~]#
查看top情况
[root@pc-openeuler-1 ~]# top
top - 17:07:03 up 4 days, 3:14, 3 users, load average: 0.96, 0.77, 0.67
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.1 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5251.5 free, 426.6 used, 1135.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5390.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1424 root 20 0 215872 4864 2688 S 19.3 0.1 123:40.06 crond
21156 root 20 0 218432 6144 3392 R 0.3 0.1 0:00.77 top
1 root 20 0 185600 20288 9024 S 0.0 0.3 210:32.94 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:34.45 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.06 mm_percpu_wq
9 root 20 0 0 0 0 S 0.0 0.0 4:14.29 ksoftirqd/0
10 root 20 0 0 0 0 I 0.0 0.0 7:20.88 rcu_sched
11 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh
12 root rt 0 0 0 0 S 0.0 0.0 6:54.58 migration/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
15 root rt 0 0 0 0 S 0.0 0.0 3:23.77 migration/1
16 root 20 0 0 0 0 S 0.0 0.0 4:48.34 ksoftirqd/1
18 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/1:0H-kblockd
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2
20 root rt 0 0 0 0 S 0.0 0.0 5:44.07 migration/2
21 root 20 0 0 0 0 S 0.0 0.0 36:36.95 ksoftirqd/2
23 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/2:0H
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/3
25 root rt 0 0 0 0 S 0.0 0.0 5:44.36 migration/3
26 root 20 0 0 0 0 S 0.0 0.0 4:00.66 ksoftirqd/3
28 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/3:0H-kblockd
29 root 20 0 0 0 0 S 0.0 0.0 0:00.23 kdevtmpfs
30 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
31 root 20 0 0 0 0 S 0.0 0.0 0:01.36 kauditd
34 root 20 0 0 0 0 S 0.0 0.0 9:38.33 khungtaskd
35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
36 root 0 -20 0 0 0 I 0.0 0.0 0:00.65 writeback
37 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
38 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
39 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged
41 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 crypto
42 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kintegrityd
43 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
44 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 md
45 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 edac-poller
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
49 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
100 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kthrotld
101 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/41-pciehp
102 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/42-pciehp
103 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/43-pciehp
104 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/44-pciehp
105 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/45-pciehp
106 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/47-pciehp
107 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/50-pciehp
查看iotop
[root@pc-openeuler-1 ~]# iotop
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 16
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H-kblockd]
8 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq]
9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
10 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
11 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
13 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/0]
14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/1]
15 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
16 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
18 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H-kblockd]
19 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/2]
20 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/2]
21 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/2]
23 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/2:0H]
24 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/3]
25 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/3]
26 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/3]
28 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/3:0H-kblockd]
29 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs]
30 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns]
31 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kauditd]
34 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khungtaskd]
35 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [oom_reaper]
36 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [writeback]
37 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kcompactd0]
38 be/5 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksmd]
39 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khugepaged]
41 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [crypto]
42 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kintegrityd]
43 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd]
44 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md]
45 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [edac-poller]
46 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdogd]
49 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kswapd0]
100 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthrotld]
101 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/41-pciehp]
102 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/42-pciehp]
103 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/43-pciehp]
104 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/44-pciehp]
105 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/45-pciehp]
106 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/47-pciehp]
107 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/50-pciehp]
108 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/53-pciehp]
109 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/55-pciehp]
110 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/57-pciehp]
111 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/59-pciehp]
112 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/61-pciehp]
113 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [irq/63-pciehp]
[root@pc-openeuler-1 ~]# uname -a
Linux pc-openeuler-1 4.19.90-vhulk2001.1.0.0026.aarch64 #1 SMP Fri Feb 7 04:09:58 UTC 2020 aarch64 GNU/Linux
[root@pc-openeuler-1 ~]# ps -ef | grep find
root 21305 15560 0 17:14 pts/0 00:00:00 grep find
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# ps -ef | grep find
Message from syslogd@pc-openeuler-1 at Apr 21 17:15:21 ...
kernel:[357748.172879] watchdog: BUG: soft lockup - CPU#3 stuck for 13s! [kworker/3:2:21034]
[root@pc-openeuler-1 ~]# ls -l /proc/
total 0
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 1
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 10
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 100
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 101
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 102
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 103
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 104
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 105
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 106
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 107
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 108
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 109
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 11
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 110
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 111
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 112
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 113
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 115
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 116
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 117
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 12
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1222
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1232
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1233
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1259
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1260
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1269
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1270
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1277
dr-xr-xr-x. 9 avahi avahi 0 Apr 17 13:53 1294
dr-xr-xr-x. 9 dbus dbus 0 Apr 17 13:53 1296
dr-xr-xr-x. 9 polkitd polkitd 0 Apr 17 13:53 1299
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 13
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1303
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1306
dr-xr-xr-x. 9 chrony chrony 0 Apr 17 13:53 1307
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1315
dr-xr-xr-x. 9 avahi avahi 0 Apr 17 13:57 1338
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1374
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1376
dr-xr-xr-x. 9 systemd-network systemd-network 0 Apr 17 13:53 1377
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1396
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 14
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1400
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1424
dr-xr-xr-x. 9 root root 0 Apr 21 07:57 14981
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 15
dr-xr-xr-x. 9 root root 0 Apr 21 09:16 15150
dr-xr-xr-x. 9 root root 0 Apr 21 09:21 15157
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15437
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15544
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15551
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15553
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15559
dr-xr-xr-x. 9 root root 0 Apr 21 12:16 15560
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 16
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 1735
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 1763
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 18
dr-xr-xr-x. 9 root root 0 Apr 21 15:56 18880
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 19
dr-xr-xr-x. 9 root root 0 Apr 21 16:18 19866
dr-xr-xr-x. 9 root root 0 Apr 21 16:16 19877
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 2
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 20
dr-xr-xr-x. 9 root root 0 Apr 21 16:25 20200
dr-xr-xr-x. 9 root root 0 Apr 21 16:26 20204
dr-xr-xr-x. 9 root root 0 Apr 21 16:26 20205
dr-xr-xr-x. 9 root root 0 Apr 21 16:36 20448
dr-xr-xr-x. 9 root root 0 Apr 21 16:36 20452
dr-xr-xr-x. 9 root root 0 Apr 21 16:36 20453
dr-xr-xr-x. 9 root root 0 Apr 21 16:43 20795
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 21
dr-xr-xr-x. 9 root root 0 Apr 21 17:06 21031
dr-xr-xr-x. 9 root root 0 Apr 21 17:06 21034
dr-xr-xr-x. 9 root root 0 Apr 21 17:06 21035
dr-xr-xr-x. 9 root root 0 Apr 21 17:06 21036
dr-xr-xr-x. 9 root root 0 Apr 21 17:06 21082
dr-xr-xr-x. 9 root root 0 Apr 21 17:10 21209
dr-xr-xr-x. 9 root root 0 Apr 21 17:14 21303
dr-xr-xr-x. 9 root root 0 Apr 21 17:18 21448
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 23
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 24
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 25
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 26
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 28
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 29
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 3
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 30
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 31
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 339
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 34
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 340
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 341
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 348
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 35
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 36
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 37
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 38
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 39
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 4
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 403
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 41
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 415
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 42
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 421
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 422
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 43
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 44
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 45
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 46
dr-xr-xr-x. 9 root root 0 Apr 17 13:53 486
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 49
dr-xr-xr-x. 9 root root 0 Apr 17 13:57 498
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 6
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 8
dr-xr-xr-x. 9 root root 0 Apr 17 13:52 9
-r--r--r--. 1 root root 0 Apr 21 17:17 buddyinfo
dr-xr-xr-x. 4 root root 0 Apr 17 13:53 bus
-r--r--r--. 1 root root 0 Apr 17 13:52 cgroups
-r--r--r--. 1 root root 0 Apr 17 13:52 cmdline
-r--r--r--. 1 root root 0 Apr 21 17:17 consoles
-r--r--r--. 1 root root 0 Apr 17 13:53 cpuinfo
-r--r--r--. 1 root root 0 Apr 21 17:17 crypto
-r--r--r--. 1 root root 0 Apr 17 13:52 devices
lrwxrwxrwx. 1 root root 29 Apr 18 00:31 device-tree -> /sys/firmware/devicetree/base
-r--r--r--. 1 root root 0 Apr 21 16:03 diskstats
dr-xr-xr-x. 2 root root 0 Apr 18 14:25 driver
-r--r--r--. 1 root root 0 Apr 21 17:17 execdomains
-r--r--r--. 1 root root 0 Apr 21 17:17 fb
-r--r--r--. 1 root root 0 Apr 17 13:52 filesystems
dr-xr-xr-x. 5 root root 0 Apr 17 13:53 fs
-r--r--r--. 1 root root 0 Apr 17 13:53 interrupts
-r--r--r--. 1 root root 0 Apr 17 13:53 iomem
-r--r--r--. 1 root root 0 Apr 21 17:17 ioports
dr-xr-xr-x. 52 root root 0 Apr 17 13:53 irq
-r--r--r--. 1 root root 0 Apr 17 13:53 kallsyms
-r--------. 1 root root 140746078355456 Apr 17 13:52 kcore
-r--r--r--. 1 root root 0 Apr 21 17:17 keys
-r--r--r--. 1 root root 0 Apr 21 17:17 key-users
-r--------. 1 root root 0 Apr 21 17:17 kmsg
-r--------. 1 root root 0 Apr 21 17:17 kpagecgroup
-r--------. 1 root root 0 Apr 21 17:17 kpagecount
-r--------. 1 root root 0 Apr 21 17:17 kpageflags
dr-xr-xr-x. 2 root root 0 Apr 18 14:25 livepatch
-r--r--r--. 1 root root 0 Apr 17 13:53 loadavg
-r--r--r--. 1 root root 0 Apr 21 17:17 locks
-r--r--r--. 1 root root 0 Apr 21 17:17 mdstat
-r--r--r--. 1 root root 0 Apr 17 13:52 meminfo
-r--r--r--. 1 root root 0 Apr 17 13:52 misc
-r--r--r--. 1 root root 0 Apr 17 13:53 modules
lrwxrwxrwx. 1 root root 11 Apr 17 13:52 mounts -> self/mounts
-r--r--r--. 1 root root 0 Apr 21 17:17 mtd
lrwxrwxrwx. 1 root root 8 Apr 17 13:52 net -> self/net
-r--------. 1 root root 0 Apr 21 17:17 pagetypeinfo
-r--r--r--. 1 root root 0 Apr 21 17:17 partitions
-r--r--r--. 1 root root 0 Apr 21 15:43 sched_debug
-r--r--r--. 1 root root 0 Apr 21 17:17 schedstat
dr-xr-xr-x. 3 root root 0 Apr 17 13:53 scsi
lrwxrwxrwx. 1 root root 0 Jan 1 1970 self -> 21448
-r--------. 1 root root 0 Apr 21 17:17 slabinfo
-r--r--r--. 1 root root 0 Apr 21 17:17 softirqs
-r--r--r--. 1 root root 0 Apr 17 13:53 stat
-r--r--r--. 1 root root 0 Apr 17 13:52 swaps
dr-xr-xr-x. 1 root root 0 Apr 17 13:52 sys
--w-------. 1 root root 0 Apr 17 13:53 sysrq-trigger
dr-xr-xr-x. 2 root root 0 Apr 18 14:25 sysvipc
lrwxrwxrwx. 1 root root 0 Jan 1 1970 thread-self -> 21448/task/21448
-r--------. 1 root root 0 Apr 21 17:17 timer_list
dr-xr-xr-x. 4 root root 0 Apr 17 13:57 tty
-r--r--r--. 1 root root 0 Apr 17 13:57 uptime
-r--r--r--. 1 root root 0 Apr 21 17:17 version
-r--------. 1 root root 0 Apr 21 17:17 vmallocinfo
-r--r--r--. 1 root root 0 Apr 21 17:10 vmstat
-r--r--r--. 1 root root 0 Apr 21 17:17 zoneinfo
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# cd /sys/kernel/debug/tracing/
[root@pc-openeuler-1 tracing]# ls
available_events hwlat_detector saved_cmdlines_size stack_max_size trace_pipe
available_filter_functions instances saved_tgids stack_trace tracing_cpumask
available_tracers kprobe_events set_event stack_trace_filter tracing_max_latency
buffer_size_kb kprobe_profile set_event_pid synthetic_events tracing_on
buffer_total_size_kb max_graph_depth set_ftrace_filter timestamp_mode tracing_thresh
current_tracer options set_ftrace_notrace trace uprobe_events
dyn_ftrace_total_info per_cpu set_ftrace_pid trace_clock uprobe_profile
enabled_functions printk_formats set_graph_function trace_marker
events README set_graph_notrace trace_marker_raw
free_buffer saved_cmdlines snapshot trace_options
[root@pc-openeuler-1 tracing]#
[root@pc-openeuler-1 tracing]# cat available_tracers
hwlat blk function_graph wakeup_dl wakeup_rt wakeup function nop
[root@pc-openeuler-1 tracing]#
[root@pc-openeuler-1 tracing]# cat current_tracer
nop
[root@pc-openeuler-1 tracing]# echo function > current_tracer
[root@pc-openeuler-1 tracing]# cat current_tracer
function
[root@pc-openeuler-1 tracing]# cat trace | head -n 20
# tracer: function
#
# entries-in-buffer/entries-written: 210704/1172169 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
irqbalance-1306 [002] .... 358490.143816: show_ipi_list <-arch_show_interrupts
irqbalance-1306 [002] .... 358490.143816: seq_printf <-show_ipi_list
irqbalance-1306 [002] .... 358490.143816: seq_vprintf <-seq_printf
irqbalance-1306 [002] .... 358490.143817: seq_printf <-show_ipi_list
irqbalance-1306 [002] .... 358490.143817: seq_vprintf <-seq_printf
irqbalance-1306 [002] .... 358490.143818: seq_printf <-show_ipi_list
irqbalance-1306 [002] .... 358490.143818: seq_vprintf <-seq_printf
irqbalance-1306 [002] .... 358490.143819: seq_printf <-show_ipi_list
irqbalance-1306 [002] .... 358490.143819: seq_vprintf <-seq_printf
[root@pc-openeuler-1 tracing]#
[root@pc-openeuler-1 tracing]# echo function_graph > current_tracer
[root@pc-openeuler-1 tracing]# cat trace | head -n 20
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
2) | tick_nohz_idle_stop_tick() {
2) 0.360 us | can_stop_idle_tick.isra.1();
2) | tick_nohz_next_event() {
2) 0.380 us | rcu_needs_cpu();
2) | get_next_timer_interrupt() {
2) 0.480 us | __next_timer_interrupt();
2) 0.600 us | hrtimer_get_next_event();
2) 2.800 us | }
2) 0.340 us | timekeeping_max_deferment();
2) 5.600 us | }
2) | tick_nohz_stop_tick() {
2) 0.340 us | calc_load_nohz_start();
2) 0.340 us | cpu_load_update_nohz_start();
2) 0.340 us | quiet_vmstat();
2) | hrtimer_start_range_ns() {
2) 0.360 us | lock_hrtimer_base.isra.1();
[root@pc-openeuler-1 tracing]#
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 18:00:39 ...
kernel:[360450.162283] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [systemd:1]
Message from syslogd@pc-openeuler-1 at Apr 21 18:30:37 ...
kernel:[362252.300305] watchdog: BUG: soft lockup - CPU#3 stuck for 15s! [kworker/3:2:21034]
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 18:00:39 ...
kernel:[360450.162283] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [systemd:1]
Message from syslogd@pc-openeuler-1 at Apr 21 18:30:37 ...
kernel:[362252.300305] watchdog: BUG: soft lockup - CPU#3 stuck for 15s! [kworker/3:2:21034]
^C
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:25 ...
kernel:[364043.641048] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [in:imjournal:1327]
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:33 ...
kernel:[364044.161561] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [crond:22012]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:47 ...
kernel:[364107.280553] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/u8:0:22015]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:48 ...
kernel:[364113.221898] watchdog: BUG: soft lockup - CPU#2 stuck for 19s! [crond:22021]
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 18:00:39 ...
kernel:[360450.162283] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [systemd:1]
Message from syslogd@pc-openeuler-1 at Apr 21 18:30:37 ...
kernel:[362252.300305] watchdog: BUG: soft lockup - CPU#3 stuck for 15s! [kworker/3:2:21034]
^C
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:25 ...
kernel:[364043.641048] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [in:imjournal:1327]
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:33 ...
kernel:[364044.161561] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [crond:22012]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:47 ...
kernel:[364107.280553] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/u8:0:22015]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:48 ...
kernel:[364113.221898] watchdog: BUG: soft lockup - CPU#2 stuck for 19s! [crond:22021]
^C
[root@pc-openeuler-1 tracing]#
Message from syslogd@pc-openeuler-1 at Apr 21 19:20:25 ...
kernel:[365244.886399] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [kworker/0:0:21036]
Message from syslogd@pc-openeuler-1 at Apr 21 19:50:56 ...
kernel:[367073.585774] watchdog: BUG: soft lockup - CPU#3 stuck for 12s! [kworker/3:2:21034]
Message from syslogd@pc-openeuler-1 at Apr 21 19:51:02 ...
kernel:[367073.585780] watchdog: BUG: soft lockup - CPU#0 stuck for 17s! [kworker/0:0:21036]
Message from syslogd@pc-openeuler-1 at Apr 21 19:51:04 ...
kernel:[367083.948643] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [kworker/1:2:15437]
Message from syslogd@pc-openeuler-1 at Apr 21 20:01:31 ...
kernel:[367708.053723] watchdog: BUG: soft lockup - CPU#2 stuck for 11s! [swapper/2:0]
[root@pc-openeuler-1 tracing]# ps -aux | grep watchdogd
root 46 0.0 0.0 0 0 ? S Apr17 0:00 [watchdogd]
root 22113 35.0 0.0 214016 1600 pts/0 S+ 20:07 0:00 grep watchdogd
[root@pc-openeuler-1 tracing]# ps -aux | grep multipathd
root 22127 12.0 0.0 214016 1536 pts/0 S+ 20:07 0:00 grep multipathd
[root@pc-openeuler-1 tracing]#
[root@pc-openeuler-1 ~]# ps -aux | grep watchdogd
root 46 0.0 0.0 0 0 ? S Apr17 0:00 [watchdogd]
root 22187 84.0 0.0 214016 1536 pts/0 S+ 20:36 0:00 grep watchdogd
[root@pc-openeuler-1 ~]# ps -aux | grep multipathd
root 22201 0.0 0.0 214016 1536 pts/0 S+ 20:36 0:00 grep multipathd
[root@pc-openeuler-1 ~]# top -cbp 46
top - 20:37:52 up 4 days, 6:45, 1 user, load average: 0.66, 0.54, 0.49
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.3 us, 4.5 sy, 0.0 ni, 93.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:37:55 up 4 days, 6:45, 1 user, load average: 0.66, 0.54, 0.49
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:37:58 up 4 days, 6:45, 1 user, load average: 0.61, 0.53, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:01 up 4 days, 6:45, 1 user, load average: 0.61, 0.53, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:04 up 4 days, 6:45, 1 user, load average: 0.56, 0.52, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:07 up 4 days, 6:45, 1 user, load average: 0.52, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:10 up 4 days, 6:45, 1 user, load average: 0.52, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:13 up 4 days, 6:45, 1 user, load average: 0.47, 0.50, 0.47
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:21 up 4 days, 6:45, 1 user, load average: 0.60, 0.53, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.4 us, 3.3 sy, 0.0 ni, 59.6 id, 0.0 wa, 19.1 hi, 15.6 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:25 up 4 days, 6:45, 1 user, load average: 0.55, 0.52, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 1.0 sy, 0.0 ni, 93.7 id, 0.0 wa, 3.4 hi, 0.4 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:28 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:28 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 1.6 sy, 0.0 ni, 96.8 id, 0.0 wa, 0.0 hi, 1.6 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:28 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 50.0 sy, 0.0 ni, 50.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:29 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:29 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 4.8 sy, 0.0 ni, 95.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:32 up 4 days, 6:45, 1 user, load average: 0.50, 0.51, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:35 up 4 days, 6:45, 1 user, load average: 0.46, 0.50, 0.47
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 95.1 id, 0.0 wa, 4.6 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:38 up 4 days, 6:45, 1 user, load average: 0.43, 0.49, 0.47
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.3 sy, 0.0 ni, 95.4 id, 0.0 wa, 3.6 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:46 up 4 days, 6:45, 1 user, load average: 1.19, 0.65, 0.52
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.4 us, 7.5 sy, 0.0 ni, 0.7 id, 0.0 wa, 61.6 hi, 19.8 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:50 up 4 days, 6:45, 1 user, load average: 1.50, 0.72, 0.55
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.9 us, 0.4 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.5 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:53 up 4 days, 6:46, 1 user, load average: 1.38, 0.71, 0.54
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:56 up 4 days, 6:46, 1 user, load average: 1.38, 0.71, 0.54
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:38:59 up 4 days, 6:46, 1 user, load average: 1.27, 0.70, 0.54
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:39:02 up 4 days, 6:46, 1 user, load average: 1.17, 0.69, 0.54
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
top - 20:39:05 up 4 days, 6:46, 1 user, load average: 1.17, 0.69, 0.54
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5260.9 free, 409.9 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5406.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]^C
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# ps -ef | grep watchdogd
root 46 2 0 Apr17 ? 00:00:00 [watchdogd]
root 22254 15560 46 20:41 pts/0 00:00:07 grep watchdogd
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# top -cbp 46
top - 20:42:34 up 4 days, 6:49, 1 user, load average: 2.06, 1.50, 0.89
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 3.1 sy, 0.0 ni, 96.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5262.8 free, 408.1 used, 1143.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5408.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]^C
^C
[root@pc-openeuler-1 ~]# strace -cp 46
strace: attach: ptrace(PTRACE_SEIZE, 46): Operation not permitted
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# cat /proc/46/stack
Message from syslogd@pc-openeuler-1 at Apr 21 21:30:32 ...
kernel:[373045.191273] watchdog: BUG: soft lockup - CPU#2 stuck for 12s! [crond:23098]
Message from syslogd@pc-openeuler-1 at Apr 21 21:30:49 ...
kernel:[373045.843894] watchdog: BUG: soft lockup - CPU#0 stuck for 13s! [sshd:15559]
Message from syslogd@pc-openeuler-1 at Apr 21 21:30:53 ...
kernel:[373046.632763] watchdog: BUG: soft lockup - CPU#1 stuck for 13s! [systemd-network:1377]
又报Message了
coding
5年前
[root@pc-openeuler-1 ~]# ps -ef | grep syslogd
root 1303 1 8 Apr17 ? 09:16:58 /usr/sbin/rsyslogd -n -iNONE
root 23133 15560 0 21:33 pts/0 00:00:00 grep syslogd
[root@pc-openeuler-1 ~]# ps -fp 1303
UID PID PPID C STIME TTY TIME CMD
root 1303 1 8 Apr17 ? 09:17:07 /usr/sbin/rsyslogd -n -iNONE
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# cat /proc/46/stack
[<0>] __switch_to+0xe4/0x148
[<0>] kthread_worker_fn+0x208/0x240
[<0>] kthread+0x134/0x138
[<0>] ret_from_fork+0x10/0x18
[<0>] 0xffffffffffffffff
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# cat /proc/1303/stack
[<0>] __switch_to+0xe4/0x148
[<0>] do_select+0x440/0x658
[<0>] core_sys_select+0x200/0x438
[<0>] __arm64_sys_pselect6+0x290/0x2e8
[<0>] el0_svc_common+0x78/0x130
[<0>] el0_svc_handler+0x38/0x78
[<0>] el0_svc+0x8/0xc
[<0>] 0xffffffffffffffff
[root@pc-openeuler-1 ~]# cat /proc/1303/syscall
72 0x1 0x0 0x0 0x0 0xffffcb39a5d8 0x0 0xffffcb39a580 0xfffc2fb521c0
[root@pc-openeuler-1 ~]# grep 72 /usr/include/asm/unistd.h
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# grep 72 /usr/include/asm/unistd.h
[root@pc-openeuler-1 ~]#
Message from syslogd@pc-openeuler-1 at Apr 21 21:41:08 ...
kernel:[373679.994219] watchdog: BUG: soft lockup - CPU#3 stuck for 11s! [kworker/3:2:21034]
Message from syslogd@pc-openeuler-1 at Apr 21 21:41:13 ...
kernel:[373680.317773] watchdog: BUG: soft lockup - CPU#0 stuck for 19s! [kworker/0:0:21036]
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]#
还是继续报Message
coding
5年前
[root@pc-openeuler-1 ~]# ps -ef | grep watchdogd
root 46 2 0 Apr17 ? 00:00:00 [watchdogd]
root 23585 15560 70 21:50 pts/0 00:00:02 grep watchdogd
[root@pc-openeuler-1 ~]# top -H -p 46
top - 21:51:35 up 4 days, 7:58, 1 user, load average: 0.88, 1.04, 1.45
Threads: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 6813.8 total, 5235.6 free, 410.4 used, 1167.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5404.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
top - 21:53:01 up 4 days, 8:00, 1 user, load average: 0.19, 0.76, 1.31
Threads: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5235.6 free, 410.4 used, 1167.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5404.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
[root@pc-openeuler-1 ~]# ps -ef | grep watchdogd
root 46 2 0 Apr17 ? 00:00:00 [watchdogd]
root 23585 15560 70 21:50 pts/0 00:00:02 grep watchdogd
[root@pc-openeuler-1 ~]# top -H -p 46
top - 21:52:24 up 4 days, 7:59, 1 user, load average: 0.38, 0.87, 1.37
Threads: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 6813.8 total, 5235.6 free, 410.4 used, 1167.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5404.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
46 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
[root@pc-openeuler-1 ~]# lsmod
Module Size Used by
binfmt_misc 262144 1
ip6t_rpfilter 262144 1
ip6t_REJECT 262144 2
nf_reject_ipv6 262144 1 ip6t_REJECT
ipt_REJECT 262144 2
nf_reject_ipv4 262144 1 ipt_REJECT
xt_conntrack 262144 13
ebtable_nat 262144 1
ip6table_nat 262144 1
nf_nat_ipv6 262144 1 ip6table_nat
ip6table_mangle 262144 1
ip6table_raw 262144 1
ip6table_security 262144 1
iptable_nat 262144 1
nf_nat_ipv4 262144 1 iptable_nat
nf_nat 262144 2 nf_nat_ipv6,nf_nat_ipv4
iptable_mangle 262144 1
iptable_raw 262144 1
iptable_security 262144 1
nf_conntrack 327680 4 xt_conntrack,nf_nat,nf_nat_ipv6,nf_nat_ipv4
nf_defrag_ipv6 262144 1 nf_conntrack
nf_defrag_ipv4 262144 1 nf_conntrack
libcrc32c 262144 2 nf_conntrack,nf_nat
ip_set 262144 0
nfnetlink 262144 1 ip_set
ebtable_filter 262144 1
ebtables 262144 2 ebtable_nat,ebtable_filter
ip6table_filter 262144 1
ip6_tables 262144 7 ip6table_filter,ip6table_raw,ip6table_nat,ip6table_mangle,ip6table_security
iptable_filter 262144 1
vfat 262144 1
fat 262144 1 vfat
dm_multipath 262144 0
aes_ce_blk 262144 0
crypto_simd 262144 1 aes_ce_blk
cryptd 262144 1 crypto_simd
aes_ce_cipher 262144 1 aes_ce_blk
ghash_ce 262144 0
sha2_ce 262144 0
sha256_arm64 262144 1 sha2_ce
sha1_ce 262144 0
ofpart 262144 0
cmdlinepart 262144 0
sg 262144 0
virtio_input 262144 0
cfi_cmdset_0001 262144 2
virtio_balloon 262144 0
cfi_probe 262144 0
cfi_util 262144 2 cfi_probe,cfi_cmdset_0001
gen_probe 262144 1 cfi_probe
physmap_of 262144 0
chipreg 262144 2 physmap_of,cfi_probe
uio_pdrv_genirq 262144 0
mtd 262144 4 cmdlinepart,physmap_of,ofpart
uio 262144 1 uio_pdrv_genirq
sch_fq_codel 262144 2
ip_tables 262144 5 iptable_filter,iptable_security,iptable_raw,iptable_nat,iptable_mangle
ext4 851968 3
mbcache 262144 1 ext4
jbd2 262144 1 ext4
sd_mod 262144 4
virtio_net 262144 0
net_failover 262144 1 virtio_net
virtio_scsi 262144 3
virtio_gpu 262144 1
failover 262144 1 net_failover
virtio_mmio 262144 0
virtio_pci 262144 0
virtio_ring 262144 7 virtio_mmio,virtio_balloon,virtio_scsi,virtio_input,virtio_gpu,virtio_pci,virtio_net
virtio 262144 7 virtio_mmio,virtio_balloon,virtio_scsi,virtio_input,virtio_gpu,virtio_pci,virtio_net
dm_mirror 262144 0
dm_region_hash 262144 1 dm_mirror
dm_log 262144 2 dm_region_hash,dm_mirror
dm_mod 327680 9 dm_multipath,dm_log,dm_mirror
[root@pc-openeuler-1 ~]#
[346062.993562] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:0:14989]
[346063.090304] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
[346066.664181] dm_region_hash dm_log dm_mod
[346067.055975] CPU: 0 PID: 14989 Comm: kworker/0:0 Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
[346067.076928] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[346067.090846] Workqueue: events_freezable update_balloon_stats_func [virtio_balloon]
[346067.103875] pstate: 00000005 (nzcv daif -PAN -UAO)
[346067.120772] pc : vp_notify+0x28/0x38 [virtio_pci]
[346067.135242] lr : virtqueue_kick+0x3c/0x78 [virtio_ring]
[346067.145279] sp : ffff00000e2cfd00
[373060.164367] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
[373063.760164] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
[373063.834630] dm_region_hash dm_log dm_mod
[373063.920678] dm_region_hash dm_log dm_mod
[373063.923125] CPU: 2 PID: 23098 Comm: crond Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
[373063.944445] CPU: 0 PID: 15559 Comm: sshd Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
[373063.950905] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[373063.963750] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[373063.967359] pstate: 00000005 (nzcv daif -PAN -UAO)
[373063.978235] pstate: 80000005 (Nzcv daif -PAN -UAO)
[373063.980572] pc : sched_clock+0x40/0x88
[373063.991197] pc : flush_work+0x0/0x38
[373063.993862] lr : sched_clock+0x40/0x88
[373064.004405] lr : tty_buffer_flush_work+0x20/0x30
[373064.006914] sp : ffff00000804f600
[373064.017473] sp : ffff00000c44f800
[373064.019953] x29: ffff00000804f600 x28: 000000000000fa80
[373064.030509] x29: ffff00000c44f800 x28: 0000000000000000
[373064.033074] x27: ffff80016d88e800 x26: 0000000000000048
[373064.043441] x27: 0000000000000001 x26: 0000000000000000
[373064.045961] x25: 0000000000000000 x24: 0000000000000028
[373064.056242] x25: 0000000000002000 x24: 0000000000000000
[373064.058715] x23: ffff0000092b70c8 x22: 00000000000000ac
[373064.068988] x23: ffff800162e8f000 x22: ffff00000c44f9a8
[373064.071389] x21: 0000000000000000 x20: ffff0000092b70c0
[373064.081727] x21: ffff800162e8f000 x20: ffff00000c44f9a8
[373064.104275] x19: ffff0000092b70c8 x18: 0000000000000000
[373064.184870] x19: ffff8001668efe00 x18: 0000000000000000
[373064.187345] x17: 0000000000000000 x16: 0000000000000000
[373064.197661] x17: 0000000000000000 x16: 0000000000000000
[373064.201186] x15: 0000000000000000 x14: cb73e42f88b7a687
[373064.212928] x15: 0000000000000000 x14: 0000000000000000
[373064.215816] x13: b3c5100000000000 x12: 3dcf040218507684
[373064.226698] x13: 0000000000000000 x12: 0000000000000000
[373064.229237] x11: ffff000002520600 x10: 0000000000005b68
[373064.239772] x11: 0000000000000000 x10: 00000000000072c0
[373064.242083] x9 : ffff800160047c00 x8 : 0000000000000018
[373064.252136] x9 : ffff800160040e00 x8 : 0000000000000018
[373064.254427] x7 : ffff800166625b78 x6 : ffff000009273708
[373064.264612] x7 : ffff8001708a72d0 x6 : 0000000000000001
[373064.266787] x5 : 0000000000000000 x4 : ffff000009273708
[373064.276421] x5 : 00008001f6730000 x4 : 00000000000072c0
[373064.278858] x3 : 0000000000000000 x2 : 0000000000000023
[373064.288841] x3 : 00000000000072d8 x2 : 008c36560de8ef00
[373064.291273] x1 : ffff00000804f5f0 x0 : 0000116722f5099f
[373064.301321] x1 : 0000000000000000 x0 : ffff8001668efe08
[373064.303749] Call trace:
[373064.313780] Call trace:
[373064.316218] sched_clock+0x40/0x88
[373064.326262] flush_work+0x0/0x38
[373064.328685] trace_clock_local+0xc/0x18
[373064.338715] n_tty_poll+0x13c/0x1d0
[373064.341119] function_graph_enter+0x80/0x190
[373064.351100] tty_poll+0x90/0xd0
[373064.353516] prepare_ftrace_return+0x28/0x58
[373064.363449] do_select+0x2b0/0x658
[373064.365809] ftrace_graph_caller+0x1c/0x24
[373064.375689] core_sys_select+0x200/0x438
[373064.378079] iptable_mangle_hook+0x20/0x128 [iptable_mangle]
[373064.387938] __arm64_sys_pselect6+0x290/0x2e8
[373064.390338] nf_hook_slow+0x50/0x100
[373064.400205] el0_svc_common+0x78/0x130
[373064.402556] ip_output+0x11c/0x130
[373064.412261] el0_svc_handler+0x38/0x78
[373064.414551] ip_local_out+0x58/0x68
[373064.424202] el0_svc+0x8/0xc
[373064.426448] __ip_queue_xmit+0x12c/0x368
[373064.480996] ip_queue_xmit+0x10/0x18
[373064.482692] __tcp_transmit_skb+0x4f0/0xa28
[373064.484490] __tcp_send_ack.part.14+0xb4/0x130
[373064.486321] tcp_send_ack+0x34/0x40
[373064.488015] __tcp_ack_snd_check+0x50/0x1d0
[373064.489793] tcp_rcv_established+0x2f4/0x698
[373064.491597] tcp_v4_do_rcv+0x178/0x228
[373064.493281] tcp_v4_rcv+0xc8c/0xde0
[373064.494935] ip_local_deliver_finish+0x84/0x288
[373064.496730] ip_local_deliver+0x68/0x118
[373064.498473] ip_rcv_finish+0x90/0xb0
[373064.500215] ip_rcv+0x64/0xd8
[373064.501820] __netif_receive_skb_one_core+0x68/0x90
[373064.503772] __netif_receive_skb+0x28/0x80
[373064.505543] netif_receive_skb_internal+0x54/0x100
[373064.507438] napi_gro_receive+0xf8/0x170
[373064.509187] receive_buf+0x15c/0x500 [virtio_net]
[373064.511051] virtnet_poll+0x170/0x338 [virtio_net]
[373064.512923] net_rx_action+0x178/0x400
[373064.514641] __do_softirq+0x11c/0x31c
[373064.516332] irq_exit+0x11c/0x128
[373064.517956] __handle_domain_irq+0x6c/0xc0
[373064.519708] gic_handle_irq+0x6c/0x170
[373064.521386] el1_irq+0xb8/0x140
[373064.522972] __arch_copy_to_user+0x180/0x21c
[373064.524728] copy_page_to_iter+0xd0/0x320
[373064.526443] generic_file_buffered_read+0x254/0x740
[373064.528296] generic_file_read_iter+0x114/0x190
[373064.530118] ext4_file_read_iter+0x5c/0x140 [ext4]
[373064.531972] __vfs_read+0x11c/0x188
[373064.533614] vfs_read+0x94/0x150
[373064.535221] ksys_read+0x74/0xf0
[373064.536813] __arm64_sys_read+0x24/0x30
[373064.538498] el0_svc_common+0x78/0x130
[373064.540173] el0_svc_handler+0x38/0x78
[373064.541840] el0_svc+0x8/0xc
[373679.994219] watchdog: BUG: soft lockup - CPU#3 stuck for 11s! [kworker/3:2:21034]
[373680.317773] watchdog: BUG: soft lockup - CPU#0 stuck for 19s! [kworker/0:0:21036]
[373686.483476] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
[373687.655553] Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_multipath aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce ofpart cmdlinepart sg virtio_input cfi_cmdset_0001 virtio_balloon cfi_probe cfi_util gen_probe physmap_of chipreg uio_pdrv_genirq mtd uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod virtio_net net_failover virtio_scsi virtio_gpu failover virtio_mmio virtio_pci virtio_ring virtio dm_mirror
[373687.851687] dm_region_hash dm_log dm_mod
[373688.245240] dm_region_hash dm_log dm_mod
[373688.254562] CPU: 3 PID: 21034 Comm: kworker/3:2 Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
[373688.263482] CPU: 0 PID: 21036 Comm: kworker/0:0 Kdump: loaded Tainted: G C L 4.19.90-vhulk2001.1.0.0026.aarch64 #1
[373688.282701] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[373688.302453] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[373688.312720] Workqueue: events virtio_gpu_fb_dirty_work [virtio_gpu]
[373688.323287] Workqueue: events_freezable update_balloon_stats_func [virtio_balloon]
[373688.333455] pstate: 80000005 (Nzcv daif -PAN -UAO)
[373688.343978] pstate: 80000005 (Nzcv daif -PAN -UAO)
[373688.353791] pc : vp_notify+0x28/0x38 [virtio_pci]
[373688.363714] pc : vp_notify+0x28/0x38 [virtio_pci]
[373688.373441] lr : vp_notify+0x18/0x38 [virtio_pci]
[373688.383294] lr : vp_notify+0x18/0x38 [virtio_pci]
[373688.393043] sp : ffff000013a6fb50
[373688.403165] sp : ffff000010b8fd00
[373688.412829] x29: ffff000013a6fb50 x28: 0000000000000000
[373688.422477] x29: ffff000010b8fd00 x28: 0000000000000000
[373688.432407] x27: ffff000013a6fc90 x26: 0000000000000001
[373688.442585] x27: 0000000000000000 x26: ffff000009296300
[373688.533883] x25: 0000000000480020 x24: 0000000000000001
[373688.544419] x25: 0000000000000000 x24: ffff80016deca028
[373688.554674] x23: ffff800161c68a00 x22: 0000000000000010
[373688.565065] x23: ffff80016deca000 x22: 0000000000000032
[373688.575399] x21: 0000000000000000 x20: ffff80016c6f6000
[373688.586122] x21: ffff80016ddd5000 x20: ffff80016deca020
[373688.596567] x19: ffff80016c6f6000 x18: 0000000000000000
[373688.607270] x19: ffff80016ddd5000 x18: 0000000000000000
[373688.617882] x17: 0000000000000000 x16: 0000000000000000
[373688.628566] x17: 0000000000000000 x16: 0000000000000000
[373688.639147] x15: 0000000000000000 x14: 00000000000007ff
[373688.649748] x15: ffff800169216f00 x14: 0000000000000000
[373688.660042] x13: 0000000000000000 x12: 0000000000000000
[373688.670628] x13: ffff000008a6e0e8 x12: 0000000000000000
[373688.680740] x11: ffff000008a6e0c0 x10: 000000000000d508
[373688.690962] x11: ffff000008a6e0c0 x10: 0000000000000ac8
[373688.700985] x9 : ffff800160048200 x8 : 0000000000000018
[373688.711121] x9 : ffff800160040e00 x8 : 0000000000000018
[373688.721080] x7 : ffff800162e1d518 x6 : 0000000000000001
[373688.731093] x7 : ffff800166a60ad8 x6 : 0000000000000001
[373688.740815] x5 : 00008001f67c0000 x4 : 000000000000d508
[373688.750576] x5 : 00008001f6730000 x4 : 0000000000000ac8
[373688.760251] x3 : 000000000000d520 x2 : 008c36560de8ef00
[373688.770207] x3 : 0000000000000ae0 x2 : 008c36560de8ef00
[373688.779837] x1 : ffff00000fda3000 x0 : 0000000000000000
[373688.789628] x1 : ffff00000fd03008 x0 : 0000000000000002
[373688.799182] Call trace:
[373688.809100] Call trace:
[373688.818187] vp_notify+0x28/0x38 [virtio_pci]
[373688.827224] vp_notify+0x28/0x38 [virtio_pci]
[373688.836343] virtqueue_kick+0x3c/0x78 [virtio_ring]
[373688.845569] virtqueue_kick+0x3c/0x78 [virtio_ring]
[373688.854498] virtio_gpu_queue_ctrl_buffer_locked+0x180/0x248 [virtio_gpu]
[373688.863407] update_balloon_stats_func+0x90/0xc0 [virtio_balloon]
[373688.872522] virtio_gpu_queue_ctrl_buffer+0x50/0x78 [virtio_gpu]
[373688.881825] process_one_work+0x1b0/0x448
[373688.890876] virtio_gpu_cmd_resource_flush+0x8c/0xb0 [virtio_gpu]
[373688.899759] worker_thread+0x54/0x468
[373688.946220] virtio_gpu_dirty_update+0x1b0/0x218 [virtio_gpu]
[373688.955306] kthread+0x134/0x138
[373688.964455] virtio_gpu_fb_dirty_work+0x3c/0x48 [virtio_gpu]
[373688.973295] ret_from_fork+0x10/0x18
[373688.982442] process_one_work+0x1b0/0x448
[373689.001666] worker_thread+0x54/0x468
[373689.010761] kthread+0x134/0x138
[373689.019700] ret_from_fork+0x10/0x18
[root@pc-openeuler-1 ~]# strace -fp 46
strace: attach: ptrace(PTRACE_SEIZE, 46): Operation not permitted
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# fdisk -l
GPT PMBR size mismatch (125829119 != 167772159) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Disk /dev/sda: 80 GiB, 85899345920 bytes, 167772160 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 47AE6BAF-E4DA-48D8-AF88-9800C5702C9C
Device Start End Sectors Size Type
/dev/sda1 2048 411647 409600 200M EFI System
/dev/sda2 411648 2508799 2097152 1G Linux filesystem
/dev/sda3 2508800 125827071 123318272 58.8G Linux LVM
Disk /dev/mapper/openeuler-root: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/openeuler-home: 38.82 GiB, 41662021632 bytes, 81371136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# ifconfig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 15.0.0.12 netmask 255.255.255.0 broadcast 15.0.0.255
ether fa:16:3e:e1:28:5c txqueuelen 1000 (Ethernet)
RX packets 224540 bytes 48437128 (46.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 77928 bytes 47811934 (45.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 6 bytes 504 (504.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6 bytes 504 (504.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@pc-openeuler-1 ~]# ethtool -i enp1s0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# dmesg |grep stuck
[128870.694299] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:3:8429]
[128870.694305] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [in:imjournal:1327]
[130067.792545] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:1:9346]
[130076.289237] watchdog: BUG: soft lockup - CPU#3 stuck for 42s! [kworker/3:3:8429]
[133473.581850] watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [kworker/3:3:8429]
[133473.582078] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [crond:1424]
[135359.002369] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd-journal:9255]
[139790.355979] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:2:9415]
[139802.211050] watchdog: BUG: soft lockup - CPU#3 stuck for 35s! [kworker/3:2:10258]
[148299.325471] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [in:imjournal:1327]
[148299.325659] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:2:10258]
[151125.634696] watchdog: BUG: soft lockup - CPU#3 stuck for 40s! [kworker/3:2:10258]
[151667.052727] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [gdbus:1345]
[153793.452831] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:9535]
[155857.012259] watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [kworker/0:2:9535]
[156808.255160] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:2:10258]
[156859.640637] watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [systemd-journal:9255]
[157443.036563] watchdog: BUG: soft lockup - CPU#3 stuck for 48s! [kworker/3:2:10258]
[158857.885466] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:2:9535]
[159169.649188] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [kworker/3:2:10258]
[159579.779217] watchdog: BUG: soft lockup - CPU#3 stuck for 35s! [kworker/3:2:10258]
[162514.502111] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [irqbalance:1306]
[162519.488658] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [gmain:1387]
[162567.495478] watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [irqbalance:1306]
[163783.836495] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:2:10258]
[164286.582186] watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [kworker/3:2:10258]
[167886.034996] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:2:10258]
[168243.650181] watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [kworker/3:2:10258]
[168491.682829] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:10716]
[168495.335455] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [gmain:1387]
[169097.929006] watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [kworker/3:0:10716]
[172058.327343] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [jbd2/dm-0-8:421]
[172380.784948] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:10716]
[174746.710358] watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [kworker/0:2:9535]
[174746.710410] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:0:10716]
[175318.907551] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:10716]
[175477.460468] watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [tuned:1856]
[176050.228361] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:10716]
[176082.402569] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:0:10716]
[176514.627397] watchdog: BUG: soft lockup - CPU#3 stuck for 42s! [kworker/3:0:10716]
[177278.778754] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:10716]
[177278.779115] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [in:imjournal:1327]
[178315.482262] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:9535]
[179157.323750] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [systemd-journal:9255]
[179426.257803] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:2:9535]
[180631.651655] watchdog: BUG: soft lockup - CPU#3 stuck for 63s! [kworker/3:0:10716]
[181829.343981] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:0:10935]
[182781.566385] watchdog: BUG: soft lockup - CPU#3 stuck for 36s! [kworker/3:0:10716]
[182981.314382] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [multipathd:1239]
[183011.080375] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:10716]
[183018.243739] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:0:10862]
[183033.956106] watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [multipathd:1239]
[184624.937194] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd-journal:9255]
[185454.118628] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:10935]
[185759.707002] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [gmain:1387]
[185862.303287] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0:10862]
[185872.473180] watchdog: BUG: soft lockup - CPU#3 stuck for 35s! [kworker/3:0:10716]
[185873.590524] watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [kworker/0:0:10935]
[186144.235752] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:10716]
[186334.951796] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:0:10716]
[186567.624788] watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [kworker/3:0:10716]
[187160.984935] watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [kworker/0:0:10935]
[187162.059751] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:0:10716]
[187814.231070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0:10862]
[187819.887867] watchdog: BUG: soft lockup - CPU#0 stuck for 31s! [kworker/0:0:10935]
[187819.897111] watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [kworker/3:0:10716]
[187874.230950] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0:10862]
[187883.970423] watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [kworker/3:0:10716]
[188454.072544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:11103]
[188566.618771] watchdog: BUG: soft lockup - CPU#3 stuck for 41s! [kworker/3:1:11103]
[188566.619047] watchdog: BUG: soft lockup - CPU#0 stuck for 56s! [kworker/0:2:11097]
[189029.661174] watchdog: BUG: soft lockup - CPU#3 stuck for 29s! [kworker/3:1:11103]
[189381.451840] watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [kworker/3:1:11103]
[189617.022590] watchdog: BUG: soft lockup - CPU#3 stuck for 54s! [kworker/3:1:11103]
[189617.022609] watchdog: BUG: soft lockup - CPU#0 stuck for 40s! [systemd-journal:11102]
[189850.692760] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:1:11103]
[190210.798148] watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [kworker/0:2:11097]
[190306.246783] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
[190435.146785] watchdog: BUG: soft lockup - CPU#3 stuck for 60s! [kworker/3:1:11103]
[190642.376525] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:11103]
[191346.853680] watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [kworker/3:1:11103]
[192376.198777] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [systemd:1]
[192462.217530] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:0:10862]
[192475.332515] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [kworker/3:1:11103]
[194138.214367] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0:10862]
[194140.432350] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:1:11103]
[194625.750293] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [multipathd:1239]
[194878.897237] watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [jbd2/dm-0-8:421]
[199888.592950] watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [kworker/3:1:11103]
[199888.593060] watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [kworker/0:0:11255]
[201475.037098] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:1:11103]
[202493.337472] watchdog: BUG: soft lockup - CPU#3 stuck for 29s! [kworker/3:1:11103]
[203005.156440] watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [kworker/3:1:11103]
[204917.086088] watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [kworker/3:1:11103]
[207458.402133] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [crond:1424]
[208284.072103] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:2:11417]
[210602.171495] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:2:11419]
[212152.703107] watchdog: BUG: soft lockup - CPU#3 stuck for 36s! [kworker/3:2:11417]
[212950.237931] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [bash:11790]
[212954.660038] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/0:0:11255]
[214738.160242] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:11779]
[214749.389826] watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [kworker/3:2:11417]
[215316.624207] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [multipathd:1239]
[215431.254970] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:2:11417]
[217486.155822] watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [kworker/3:0:11867]
[217579.023600] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:11867]
[218251.541736] watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [kworker/3:0:11867]
[218259.413299] watchdog: BUG: soft lockup - CPU#0 stuck for 42s! [kworker/0:0:11255]
[218540.328702] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:0:11255]
[220436.716664] watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [kworker/3:0:11867]
[220436.716727] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [kworker/0:0:11255]
[220685.086569] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [jbd2/dm-0-8:421]
[220794.661758] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:0:11867]
[221051.822463] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:11867]
[223721.182851] watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [kworker/3:0:11867]
[226946.264203] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [dnf:12072]
[227283.923013] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:11867]
[227329.042977] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [swapper/0:0]
[229502.572246] watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [jbd2/dm-0-8:421]
[229514.535201] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:20]
[229821.789426] watchdog: BUG: soft lockup - CPU#0 stuck for 73s! [gmain:1387]
[229821.968490] watchdog: BUG: soft lockup - CPU#3 stuck for 59s! [kworker/3:0:11867]
[234531.199273] watchdog: BUG: soft lockup - CPU#3 stuck for 35s! [kworker/3:0:11867]
[235064.122732] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd-journal:11102]
[235097.398378] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd-journal:11102]
[236592.312114] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:11867]
[238289.253720] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [jbd2/dm-0-8:421]
[238756.029303] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:3:12015]
[239970.094922] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:11779]
[239971.426464] watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [kworker/0:0:11255]
[239971.426552] watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [kworker/3:3:12015]
[241705.377841] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [swapper/0:0]
[244002.189839] watchdog: BUG: soft lockup - CPU#3 stuck for 51s! [kworker/3:3:12015]
[244050.083863] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:11779]
[245257.248524] watchdog: BUG: soft lockup - CPU#3 stuck for 29s! [kworker/3:3:12015]
[245257.265565] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:0:11255]
[252458.061986] watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:2:12465]
[252474.310382] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [kworker/3:3:12015]
[258484.584783] watchdog: BUG: soft lockup - CPU#0 stuck for 43s! [kworker/0:1:12464]
[258484.643666] watchdog: BUG: soft lockup - CPU#3 stuck for 39s! [kworker/3:0:12608]
[259677.110212] watchdog: BUG: soft lockup - CPU#3 stuck for 36s! [kworker/3:0:12608]
[260125.177085] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [kworker/0:1:12464]
[260185.312565] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:1:12464]
[260245.571511] watchdog: BUG: soft lockup - CPU#3 stuck for 55s! [kworker/3:0:12608]
[260319.282342] watchdog: BUG: soft lockup - CPU#0 stuck for 31s! [kworker/0:1:12464]
[260423.132452] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:0:12608]
[260423.132538] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [kworker/0:1:12464]
[260605.996061] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [kworker/0:1:12464]
[260606.045055] watchdog: BUG: soft lockup - CPU#3 stuck for 49s! [kworker/3:0:12608]
[260675.490227] watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [kworker/3:0:12608]
[261326.037859] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:2:12465]
[261336.241710] watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [kworker/3:0:12608]
[261466.468141] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:0:12608]
[261710.038310] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:2:12465]
[261717.045323] watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [kworker/3:0:12608]
[261717.045327] watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [irqbalance:1306]
[262378.657488] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:12608]
[263092.577248] watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [kworker/0:1H:348]
[264170.122250] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [jbd2/dm-0-8:421]
[267501.682759] watchdog: BUG: soft lockup - CPU#3 stuck for 48s! [kworker/3:0:12608]
[267505.075842] watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [systemd-network:1377]
[267508.627874] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:1:12464]
[268060.520928] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd:1]
[268060.678706] watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [chronyd:1307]
[268060.832732] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:12711]
[268076.817436] watchdog: BUG: soft lockup - CPU#3 stuck for 47s! [kworker/3:0:12608]
[276582.647298] watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [kworker/0:3:12897]
[278281.523785] watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [kworker/3:0:12999]
[281367.153803] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ksoftirqd/2:21]
[281881.619504] watchdog: BUG: soft lockup - CPU#0 stuck for 44s! [irqbalance:1306]
[282618.082931] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:0:12999]
[282649.804723] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [in:imjournal:1327]
[282650.015663] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:12897]
[282650.294629] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:25]
[282802.062919] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0]
[283080.710904] watchdog: BUG: soft lockup - CPU#0 stuck for 46s! [crond:13510]
[283164.615539] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [migration/0:12]
[283218.537473] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:0:12999]
[283425.141632] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [auditd:1277]
[284914.567248] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [multipathd:1239]
[284977.507167] watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [irqbalance:1306]
[284980.253554] watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [kworker/3:0:12999]
[284980.255455] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [sh:13585]
[285447.056229] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [sshd:13334]
[285680.684896] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [kworker/0:0:13498]
[285695.898677] watchdog: BUG: soft lockup - CPU#2 stuck for 28s! [NetworkManager:1376]
[285696.792136] watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [in:imjournal:1327]
[285755.478715] watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [kworker/1:1:13501]
[285756.051611] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [jbd2/dm-0-8:421]
[285758.119680] watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:13520]
[285758.489484] watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [kworker/3:0:12999]
[286149.594963] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [multipathd:1239]
[286728.360225] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:0:13498]
[286782.695193] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:1:13738]
[286842.796319] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:0:13498]
[287877.309063] watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [systemd-journal:13810]
[288551.160744] watchdog: BUG: soft lockup - CPU#0 stuck for 54s! [swapper/0:0]
[289681.063337] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [kworker/0:1:14135]
[290779.739085] watchdog: BUG: soft lockup - CPU#0 stuck for 31s! [kworker/0:1:14135]
[290824.256353] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:1:14135]
[290990.264792] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [kworker/0:1:14135]
[291133.536777] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:0]
[291508.593883] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [tuned:1856]
[292106.788463] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [multipathd:1239]
[292166.486342] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [systemd-udevd:13327]
[292170.103246] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [multipathd:1240]
[292173.013452] watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [in:imjournal:1327]
[292194.153504] watchdog: BUG: soft lockup - CPU#0 stuck for 46s! [irqbalance:1306]
[292664.932470] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [chronyd:1307]
[292844.076609] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [tuned:1856]
[295118.304311] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [in:imjournal:1327]
[295119.286204] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [gmain:1387]
[295120.572802] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:3:14233]
[298055.608630] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [multipathd:1239]
[298106.997405] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [auditd:1277]
[298270.427633] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-journal:14378]
[298684.420041] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [multipathd:1239]
[298847.587060] watchdog: BUG: soft lockup - CPU#0 stuck for 46s! [in:imjournal:1327]
[299333.468769] watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [multipathd:1236]
[299334.425976] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:25]
[299334.460984] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [crond:1424]
[299349.593055] watchdog: BUG: soft lockup - CPU#0 stuck for 56s! [kworker/0:0:14287]
[301057.171637] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:3:14233]
[301057.200076] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [in:imjournal:1327]
[301506.303404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:3:14233]
[301564.402609] watchdog: BUG: soft lockup - CPU#0 stuck for 41s! [jbd2/dm-0-8:421]
[301688.371032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:3:14233]
[301735.545321] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [irqbalance:1306]
[301842.572314] watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [kworker/3:3:14233]
[302649.197038] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irqbalance:1306]
[302649.695680] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [in:imjournal:1327]
[302654.281508] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [gmain:1387]
[303210.363763] watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [kworker/u8:2:13855]
[303453.760247] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:14496]
[303837.260184] watchdog: BUG: soft lockup - CPU#0 stuck for 55s! [swapper/0:0]
[304810.669507] watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [crond:1424]
[305181.193161] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [kworker/0:0:14287]
[305330.128073] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [multipathd:1239]
[306470.927763] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:0:14287]
[306716.557214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:14496]
[306745.312876] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [migration/0:12]
[311321.640447] watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [kworker/0:1:14491]
[313683.065199] watchdog: BUG: soft lockup - CPU#3 stuck for 46s! [kworker/3:1:14496]
[313708.535288] watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [migration/2:20]
[313711.627287] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [gmain:1387]
[313803.402766] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:1:14377]
[313806.334229] watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:1:14496]
[313820.351686] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [systemd:1]
[313888.811532] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [auditd:1277]
[316123.944561] watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [jbd2/dm-0-8:421]
[317266.108915] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:1:14377]
[317398.795688] watchdog: BUG: soft lockup - CPU#1 stuck for 34s! [crond:14825]
[317407.764676] watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [kworker/0:1H:348]
[317511.682669] watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [crond:1424]
[317854.560515] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [migration/0:12]
[319566.438018] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [tuned:1856]
[320278.442703] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [jbd2/dm-0-8:421]
[320870.860909] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:14861]
[320972.886780] watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [gmain:1387]
[321078.766553] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [irqbalance:1306]
[321084.108837] watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [in:imjournal:1327]
[321092.008514] watchdog: BUG: soft lockup - CPU#3 stuck for 36s! [kworker/3:0:14768]
[321438.678316] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:14768]
[323021.513176] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-journal:14919]
[324011.994033] watchdog: BUG: soft lockup - CPU#0 stuck for 31s! [irqbalance:1306]
[324063.390582] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [systemd-journal:14919]
[324118.410748] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/0:2:14861]
[324203.901426] watchdog: BUG: soft lockup - CPU#0 stuck for 50s! [systemd-journal:14919]
[324482.761460] watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [kworker/0:2:14861]
[325741.996911] watchdog: BUG: soft lockup - CPU#0 stuck for 88s! [kworker/0:2:14861]
[325797.482123] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [kworker/0:2:14861]
[325800.111634] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [gmain:1387]
[325800.572637] watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [systemd-journal:14919]
[326377.870443] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:14915]
[326377.912311] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:14768]
[326461.102135] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd-network:1377]
[326525.275584] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:2:14861]
[327224.736393] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:2:14861]
[327907.358616] watchdog: BUG: soft lockup - CPU#0 stuck for 35s! [kworker/0:2:14861]
[328110.449018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:0:14768]
[328137.966011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [irqbalance:1306]
[328138.114008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [tuned:1856]
[328138.602006] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [in:imjournal:1327]
[328184.472259] watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [kworker/3:0:14768]
[328188.377630] watchdog: BUG: soft lockup - CPU#1 stuck for 32s! [kworker/1:0:14915]
[328192.675804] watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [multipathd:1239]
[328198.502586] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [irqbalance:1306]
[328369.864290] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:1:15078]
[328374.194376] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:14768]
[328430.558425] watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [kworker/0:0:14989]
[328453.445212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [in:imjournal:1327]
[328458.095589] watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [gmain:1387]
[328592.256335] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [dbus-daemon:1296]
[328658.016372] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [multipathd:1240]
[328668.024007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:20]
[328668.313987] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:2:15095]
[328710.435741] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:0:14989]
[328875.225318] watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [systemd-logind:14506]
[328876.000812] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:14989]
[329034.452844] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [(coredump):15121]
[329127.988885] watchdog: BUG: soft lockup - CPU#0 stuck for 35s! [kworker/0:0:14989]
[329260.050923] watchdog: BUG: soft lockup - CPU#0 stuck for 61s! [migration/0:12]
[329279.192063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [chronyd:1307]
[330357.707226] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [jbd2/dm-0-8:421]
[340090.834176] watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [kworker/3:0:15454]
[340091.129561] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [systemd-network:1377]
[344259.358524] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dbus-daemon:1296]
[346062.993562] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:0:14989]
[346109.818760] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:2:15437]
[346121.510088] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [kworker/3:0:15454]
[353237.984311] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [kworker/1:2:15437]
[353254.729207] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/3:0:15454]
[353309.845852] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/0:1:18880]
[355678.501557] watchdog: BUG: soft lockup - CPU#0 stuck for 12s! [swapper/0:0]
[357748.172879] watchdog: BUG: soft lockup - CPU#3 stuck for 13s! [kworker/3:2:21034]
[360450.162283] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [systemd:1]
[362252.300305] watchdog: BUG: soft lockup - CPU#3 stuck for 15s! [kworker/3:2:21034]
[364043.641048] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [in:imjournal:1327]
[364044.161561] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [crond:22012]
[364107.280553] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/u8:0:22015]
[364113.221898] watchdog: BUG: soft lockup - CPU#2 stuck for 19s! [crond:22021]
[365244.886399] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [kworker/0:0:21036]
[367073.585774] watchdog: BUG: soft lockup - CPU#3 stuck for 12s! [kworker/3:2:21034]
[367073.585780] watchdog: BUG: soft lockup - CPU#0 stuck for 17s! [kworker/0:0:21036]
[367083.948643] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [kworker/1:2:15437]
[367708.053723] watchdog: BUG: soft lockup - CPU#2 stuck for 11s! [swapper/2:0]
[370645.938502] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [kworker/1:2:15437]
[370646.990510] watchdog: BUG: soft lockup - CPU#3 stuck for 12s! [kworker/3:2:21034]
[371308.844919] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [run-parts:22456]
[373045.191273] watchdog: BUG: soft lockup - CPU#2 stuck for 12s! [crond:23098]
[373045.843894] watchdog: BUG: soft lockup - CPU#0 stuck for 13s! [sshd:15559]
[373046.632763] watchdog: BUG: soft lockup - CPU#1 stuck for 13s! [systemd-network:1377]
[373679.994219] watchdog: BUG: soft lockup - CPU#3 stuck for 11s! [kworker/3:2:21034]
[373680.317773] watchdog: BUG: soft lockup - CPU#0 stuck for 19s! [kworker/0:0:21036]
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# cat /proc/cpuinfo |grep processor
processor : 0
processor : 1
processor : 2
processor : 3
[root@pc-openeuler-1 ~]# ps -eo ppid,pid,user,args |grep watchdog
2 46 root [watchdogd]
15560 24185 root grep watchdog
Message from syslogd@pc-openeuler-1 at Apr 21 22:30:22 ...
kernel:[376640.304555] watchdog: BUG: soft lockup - CPU#3 stuck for 11s! [kworker/3:1:23736]
[root@pc-openeuler-1 ~]#
[root@pc-openeuler-1 ~]# ps -eo ppid,pid,user,args |grep watchdog
2 46 root [watchdogd]
15560 24219 root grep watchdog
[root@pc-openeuler-1 ~]#
Apr 20 03:19:13 pc-openeuler-1 rsyslogd[1303]: [origin software="rsyslogd" swVersion="8.1907.0" x-pid="1303" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr 20 03:19:14 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:19:20 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:19:22 pc-openeuler-1 rhsmd[11976]: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Apr 20 03:20:03 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:05 pc-openeuler-1 chronyd[1307]: Forward time jump detected!
Apr 20 03:20:07 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:35 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:40 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:45 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:50 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:55 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:00 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:05 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:11 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:16 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:21 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:32 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:37 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:42 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:47 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:52 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:57 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:11 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:16 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:21 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:32 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:37 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:42 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:47 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:52 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:57 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:02 pc-openeuler-1 multipathd[1236]: sda: unusable path
/var/log/messages...skipping...
Apr 20 03:19:13 pc-openeuler-1 rsyslogd[1303]: [origin software="rsyslogd" swVersion="8.1907.0" x-pid="1303" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr 20 03:19:14 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:19:20 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:19:22 pc-openeuler-1 rhsmd[11976]: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Apr 20 03:20:03 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:05 pc-openeuler-1 chronyd[1307]: Forward time jump detected!
Apr 20 03:20:07 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:35 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:40 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:45 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:50 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:20:55 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:00 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:05 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:11 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:16 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:21 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:32 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:37 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:42 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:47 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:52 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:21:57 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:11 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:16 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:21 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:32 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:37 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:42 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:47 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:52 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:22:57 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:02 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:07 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:12 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:14 pc-openeuler-1 chronyd[1307]: Selected source 199.182.204.197
Apr 20 03:23:17 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:22 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:27 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:33 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:38 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:43 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:48 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:23:55 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:03 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:08 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:13 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:18 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:18 pc-openeuler-1 chronyd[1307]: Selected source 203.107.6.88
Apr 20 03:24:24 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:24:29 pc-openeuler-1 multipathd[1236]: sda: unusable path
[root@pc-openeuler-1 ~]# ps -ef | grep multipathd
root 24271 15560 41 22:36 pts/0 00:00:00 grep multipathd
[root@pc-openeuler-1 ~]# ps -fp 24271
UID PID PPID C STIME TTY TIME CMD
[root@pc-openeuler-1 ~]#
Message from syslogd@pc-openeuler-1 at Apr 18 23:58:30 ...
kernel:[122698.773329] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:1]
Message from syslogd@pc-openeuler-1 at Apr 18 23:59:55 ...
kernel:[122823.508244] watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [kworker/3:0:9130]
Message from syslogd@pc-openeuler-1 at Apr 19 00:07:50 ...
kernel:[123195.779527] watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [sshd:7789]
Message from syslogd@pc-openeuler-1 at Apr 19 00:13:59 ...
kernel:[123655.217188] watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [kworker/0:1:8930]
Message from syslogd@pc-openeuler-1 at Apr 19 00:35:29 ...
kernel:[124937.671514] watchdog: BUG: soft lockup - CPU#3 stuck for 74s! [kworker/3:3:8429]
Message from syslogd@pc-openeuler-1 at Apr 19 00:35:29 ...
kernel:[124937.671720] watchdog: BUG: soft lockup - CPU#0 stuck for 55s! [in:imjournal:1327]
Message from syslogd@pc-openeuler-1 at Apr 19 01:40:44 ...
kernel:[128870.694305] watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [in:imjournal:1327]
Message from syslogd@pc-openeuler-1 at Apr 21 17:15:21 ...
kernel:[357748.172879] watchdog: BUG: soft lockup - CPU#3 stuck for 13s! [kworker/3:2:21034]
Message from syslogd@pc-openeuler-1 at Apr 21 18:00:39 ...
kernel:[360450.162283] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [systemd:1]
Message from syslogd@pc-openeuler-1 at Apr 21 18:30:37 ...
kernel:[362252.300305] watchdog: BUG: soft lockup - CPU#3 stuck for 15s! [kworker/3:2:21034]
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:25 ...
kernel:[364043.641048] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [in:imjournal:1327]
Message from syslogd@pc-openeuler-1 at Apr 21 19:00:33 ...
kernel:[364044.161561] watchdog: BUG: soft lockup - CPU#1 stuck for 11s! [crond:22012]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:47 ...
kernel:[364107.280553] watchdog: BUG: soft lockup - CPU#0 stuck for 14s! [kworker/u8:0:22015]
Message from syslogd@pc-openeuler-1 at Apr 21 19:01:48 ...
Message from syslogd@pc-openeuler-1 at Apr 21 23:02:50 ...
kernel:[378557.385893] watchdog: BUG: soft lockup - CPU#2 stuck for 13s! [crond:1424]
Message from syslogd@pc-openeuler-1 at Apr 21 23:30:18 ...
kernel:[380239.974362] watc
仔细看ssh刚远程时报的Message，刚好是从4月18号开始的，一直到21号也就是今天，其实最初我是2020-04-17 11:12:29 在鹏城实验室平台申请的openEuler测试机，最早发现这个问题是4月17号11:47，在ssh远程登录测试机以后，正常终端模式下，突然开始报Message
coding
5年前
coding
5年前
coding
5年前
网上查找资料，发现引发CPU死锁的原因有很多种：
服务器电源供电不足，导致CPU电压不稳导致CPU死锁
https://ubuntuforums.org/showthread.php?t=2205211
I bought a small (500W) new power supply made by what I feel is a reputable company and made the swap.
GREAT NEWS: After replacing the power supply, the crashes completely stopped!
I wanted to wait a while just to be sure, but it is now a few weeks since the new powersupply went in, and I haven't had a single crash since.
The power supply is not something that I would normally worry about, but in this case it totally fixed my problem.
Thanks to those who read my post, and especially to those who responded.
vcpus超过物理cpu cores
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
虚机所在的宿主机的CPU太忙或磁盘IO太高
虚机的的CPU太忙或磁盘IO太高
https://www.centos.org/forums/viewtopic.php?t=60087
BIOS KVM开启以后的相关bug，关闭KVM可解决，但关闭以后物理机不支持虚拟化
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
VM网卡驱动存在bug，处理高水位流量时存在bug导致CPU死锁
BIOS开启了超频，导致超频时电压不稳，容易出现CPU死锁
https://ubuntuforums.org/showthread.php?t=2205211
Linux kernel存在bug
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
KVM存在bug
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
clocksource tsc unstable on CentOS and cloud Linux with Hyper-V Virtualisation
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
通过设置clocksource=jiffies可解决
BIOS Intel C-State开启导致，关闭可解决
https://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds
https://support.citrix.com/article/CTX127395
http://blog.sina.com.cn/s/blog_906d892d0102vn26.html
BIOS spread spectrum开启导致
当主板上的时钟震荡发生器工作时，脉冲的尖峰会产生emi（电磁干扰）。spread spectrum（频展）设定功能可以降低脉冲发生器所产生的电磁干扰，脉冲波的尖峰会衰减为较为平滑的曲线。
如果我们没有遇到电磁干扰问题，建议将此项设定为disabled，这栏可以优化系统的性能表现和稳定性；
否则应该将此项设定为enabled。 如果对cpu进行超频，必须将此项禁用。因为即使是微小的脉冲值漂移也会导致超频运行的cpu锁死。
再次强调：CPU超频时，SPREAD SPECTRUM必须关闭，否则容易出现锁死cpu的情况。
coding
5年前
coding
5年前
coding
5年前
Apr 20 03:53:37 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:53:39 pc-openeuler-1 systemd[1]: Starting dnf makecache...
Apr 20 03:53:42 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 03:53:44 pc-openeuler-1 dnf[12011]: EulerOS-2.0SP8 base 3.9 kB/s | 3.8 kB 00:00
Apr 20 03:53:44 pc-openeuler-1 dnf[12011]: Metadata cache created.
Apr 20 03:53:45 pc-openeuler-1 systemd[1]: dnf-makecache.service: Succeeded.
Apr 20 03:53:45 pc-openeuler-1 systemd[1]: Started dnf makecache.
Apr 20 03:53:45 pc-openeuler-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
Apr 20 03:53:45 pc-openeuler-1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a
ddr=? terminal=? res=success'
Apr 20 03:53:47 pc-openeuler-1 multipathd[1236]: sda: unusable path
coding
5年前
coding
5年前
pr 20 04:48:08 pc-openeuler-1 chronyd[1307]: Selected source 111.230.189.174
Apr 20 04:48:09 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:15 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:20 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:33 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:36 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:36 pc-openeuler-1 chronyd[1307]: Selected source 84.16.73.33
Apr 20 04:48:41 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:46 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:51 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:48:56 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:49:01 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 04:49:02 pc-openeuler-1 chronyd[1307]: Selected source 111.230.189.174
Apr 20 04:49:04 pc-openeuler-1 chronyd[1307]: Source 111.230.189.174 replaced with 144.76.76.107
Apr 20 04:49:06 pc-openeuler-1 multipathd[1236]: sda: unusable path
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
Apr 20 05:55:36 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 05:55:40 pc-openeuler-1 systemd[1]: Starting dnf makecache...
Apr 20 05:55:41 pc-openeuler-1 multipathd[1236]: sda: unusable path
Apr 20 05:55:43 pc-openeuler-1 dnf[12134]: Metadata cache refreshed recently.
Apr 20 05:55:43 pc-openeuler-1 systemd[1]: dnf-makecache.service: Succeeded.
Apr 20 05:55:43 pc-openeuler-1 systemd[1]: Started dnf makecache.
Apr 20 05:55:43 pc-openeuler-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
Apr 20 05:55:43 pc-openeuler-1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? a
ddr=? terminal=? res=success'
Apr 20 05:55:46 pc-openeuler-1 multipathd[1236]: sda: unusable path
coding
5年前
wangxiongfeng
5年前
会用到gpu吗 一直是virtio-gpu这个模块在报错 不加载下面这个ko试试，看看是不是这个ko引起的问题
drivers/gpu/drm/virtio/virtio-gpu.ko
coding
5年前
coding
5年前
[root@pc-openeuler-1 log]#
Message from syslogd@pc-openeuler-1 at Apr 22 20:01:57 ...
kernel:[72937.931100] watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [kworker/0:0:6057]
[root@pc-openeuler-1 ~]#
Socket error Event: 32 Error: 10053.
Connection closing...Socket close.
Connection closed by foreign host.
Disconnected from remote host at 23:30:29.
位置：俄罗斯圣彼得堡
位置：美国
[root@pc-openeuler-1 log]# tail -n 20 messages-20200422
Apr 22 03:01:59 pc-openeuler-1 kernel: [11809.107186] kthread+0x134/0x138
Apr 22 03:01:59 pc-openeuler-1 kernel: [11809.108846] ret_from_fork+0x10/0x18
Apr 22 03:01:59 pc-openeuler-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:01:59 pc-openeuler-1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:01:59 pc-openeuler-1 systemd[1]: sysstat-collect.service: Succeeded.
Apr 22 03:01:59 pc-openeuler-1 systemd[1]: Started system activity accounting tool.
Apr 22 03:01:59 pc-openeuler-1 chronyd[1306]: Forward time jump detected!
Apr 22 03:01:59 pc-openeuler-1 chronyd[1306]: Can't synchronise: no selectable sources
Apr 22 03:05:13 pc-openeuler-1 chronyd[1306]: Selected source 185.209.85.222
Apr 22 03:09:33 pc-openeuler-1 chronyd[1306]: Selected source 162.159.200.123
Apr 22 03:10:36 pc-openeuler-1 systemd[1]: Starting system activity accounting tool...
Apr 22 03:10:36 pc-openeuler-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:10:36 pc-openeuler-1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:10:36 pc-openeuler-1 systemd[1]: sysstat-collect.service: Succeeded.
Apr 22 03:10:37 pc-openeuler-1 systemd[1]: Started system activity accounting tool.
Apr 22 03:20:32 pc-openeuler-1 systemd[1]: Starting system activity accounting tool...
Apr 22 03:20:42 pc-openeuler-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:20:42 pc-openeuler-1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 22 03:20:42 pc-openeuler-1 systemd[1]: sysstat-collect.service: Succeeded.
Apr 22 03:20:42 pc-openeuler-1 systemd[1]: Started system activity accounting tool.
[root@pc-openeuler-1 log]#
[root@pc-openeuler-1 log]#
Message from syslogd@pc-openeuler-1 at Apr 23 00:10:57 ...
kernel:[87932.939881] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:0:7246]
报的调用栈还是 virtio-gpu.ko 里面吗
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
Fred_Li
5年前
如果您要上传附件，有两种方式：
1）在google doc 或者 其他类似平台上共享，在此贴链接。
2）编辑issue的内容（就是原帖，不是comments），应该可以增加附件。
谢谢
wangxiongfeng
5年前
调用栈的信息发一下 soft lockup打印看出不什么来
wangxiongfeng
5年前
dmesg 里面应该有soft lockup的调用栈
当前正在做host排查，目前发现cpu为916，内部排查916环境没有发现类似问题。当前正在做的事情，
1.了解鹏城使用宿主机虚拟化系统（内部测试未发现该问题），对比分析两者的情况
2.协调鹏城的同事在同一台宿主机上新建虚拟机是否能重现，
3.如果能重现，直接改小硬件狗时间，产生core文件，继续分析。
zhanghailiang
5年前
coding
5年前
coding
5年前
coding
5年前
今天上午11:54朋友在鹏城实验室平台上申请的centos7虚拟机上跑docker也突然报系统内核软死锁的bug了
coding
5年前
本人考虑是否是鹏城实验室平台虚拟化那一块存在问题？
zhanghailiang
5年前
coding
5年前
coding
5年前
zhanghailiang
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
zhanghailiang
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
coding
5年前
zhanghailiang
5年前
已经跟鹏程实验室确定，非openeuler问题，而是host OS的问题，host os使用了中标麒麟，版本较老，鹏程实验室已经推送麒麟方面定位，此问题单关闭
登录 后才可以发表评论
FileDragTip