【419x86】【稳定性】ltp压力测试系统卡着，bmc控制台一直打印systemd[1]：Failed to Journal Service

**【缺陷描述】：请补充详细的缺陷问题现象描述**
在intel-2288H V7 机器上执行ltp长稳测试1天左右，系统卡着,远程连接不上，bmc控制台输入命令没有反应，无法登陆系统，bmc控制台一直打印systemd[1]：Failed to Journal Service。
注：仅-2288H V7机器上出现问题，在2288H V6上测试时系统没有卡着，没有出现此问题。

**一、缺陷信息**
大量重复出现 systemd[1]: Failed to start Journal Service. ，说明系统的 Journal Service（日志服务，用于记录系统、服务等运行日志）启动失败
**【缺陷所属的os版本】（如openEuler-22.03-LTS，参考命令"cat /etc/os-release"结果）**
#cat /etc/os-release 
NAME="openEuler"
VERSION="20.03 (LTS-SP4)"
ID="openEuler"
VERSION_ID="20.03"
PRETTY_NAME="openEuler 20.03 (LTS-SP4)"
ANSI_COLOR="0;31"

**【内核版本】（如kernel-5.10.0-60.138.0.165，参考命令"uname -r"结果）**
#uname -r
4.19.90-2312.1.0.0255.oe2003sp4.x86_64

**【缺陷所属软件及版本号】（如kernel-5.10.0-60.138.0.165，参考命令"rpm -q 包名"结果）**
#rpm -q systemd
systemd-243-67.oe2003sp4.x86_64
[root@ip70-189-155-36 ~]# rpm -ql systemd|grep d/systemd-journald
/usr/lib/systemd/systemd-journald

**【环境信息】**
硬件信息

intel-2288H V7物理机, Intel(R) Xeon(R) Gold 6454S	2200 MHz	32 cores/64 threads	2560/65536/61440 KB
512G内存大小

软件信息
- 无

网络信息
- 无

**【问题复现步骤】：请描述具体的操作步骤**
一、设置系统参数
echo 1 > /proc/sys/kernel/panic
echo 1 > /proc/sys/kernel/hardlockup_panic
echo 1 > /proc/sys/kernel/softlockup_panic
echo 60 > /proc/sys/kernel/watchdog_thresh
echo 1 > /proc/sys/kernel/hung_task_panic
echo 1200 > /proc/sys/kernel/hung_task_timeout_secs
二、执行稳定性测试：
1、下载 ltp-20240129
2、编译安装ltp
yum install -y gcc make automake screen
cd /home/ltp/ltp-20240129/
make autotools
./configure
make -j
make install

3、剔除部分用例
cd /opt/ltp
cat > skip.list << EOF
min_free_kbytes
oom01
oom02
oom03
oom04
oom05
EOF

4、配置压力
mem_stress=$(($(free | grep Mem | awk '{print $2}') * 1024 * 8 / 100))
cpu_stress=$(($(cat /proc/cpuinfo | grep "processor" | wc -l) * 8 / 10))

5、执行ltp压力测试
screen -dmS test /opt/ltp/runltp -c "$cpu_stress" -m 10,1,"$mem_stress",1 -i 2 -S /opt/ltp/skip.list -p -l /home/ltp_test/result.log -d /home/ltp_test -o /home/ltp_test/ltpstress.screen -t 168h

6、写入oom分数，防止加压进程因oom被杀
proclist=$(ps -aux | grep genload | grep -v grep | awk '{print $2}')
for ltpproc in $proclist; do
    echo '-1000' > "/proc/$ltpproc/oom_score_adj"
done

概率性重现（按照上面的步骤测试3次，出现2次卡着）

**【实际结果】**，请描述出问题的结果和影响
长稳测试执行7d，系统正常运行不发生异常复位。
**【期望结果】**，请描述出期望的结果和影响
机器上执行1天左右，系统卡着连接不上，bmc控制台输入密码没有反应，无法登陆系统，bmc控制台一直打印systemd[1]：Failed to Journal Service。
注：仅在intel-2288H V7机器上出现问题，在2288H V6 上测试时系统没有卡着，没有出现此问题。

**【其他相关附件信息】**
比如系统message日志/组件日志、dump信息、图片等
dump信息:
    DUMPFILE: vmcore [PARTIAL DUMP]
        CPUS: 128
        DATE: Sun Sep 29 22:08:28 2024
      UPTIME: 1 days, 21:22:33
LOAD AVERAGE: 112.16, 111.32, 127.51
       TASKS: 143
     VERSION: #1 SMP Thu Sep 5 10:04:12 UTC 2024
     MACHINE: x86_64 (2200 Mhz)
      MEMORY: 255.6 GB
       PANIC: "Kernel panic - not syncing: hung_task: blocked tasks"
         PID: 667
     COMMAND: "khungtaskd"
        TASK: ff1d5d806623da00 [THREAD_INFO: ff1d5d806623da00]
         CPU: 44
       STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 667 TASK: ff1d5d806623da00 CPU: 44 COMMAND: "khungtaskd"
 #0 [ff4d28800e3afcf0] machine_kexec at ffffffff8325558f
 #1 [ff4d28800e3afd48] __crash_kexec at ffffffff83359e41
 #2 [ff4d28800e3afe08] panic at ffffffff832ae9a5
 #3 [ff4d28800e3afe90] watchdog at ffffffff8338daa8
 #4 [ff4d28800e3aff10] kthread at ffffffff832d31d3

---

vmcore-dmesg.txt信息:
[55376.480179] pps pps0: removed
[55376.480477] pps pps0: new PPS source pty_slave4
[55376.480576] pps pps0: source "/dev/pts4" added
[55376.480674] pps pps0: removed
[55376.480985] pps pps0: new PPS source pty_slave4
[55376.481089] pps pps0: source "/dev/pts4" added
[55376.481186] pps pps0: removed
[55376.481471] pps pps0: new PPS source pty_slave4
[55376.481572] pps pps0: source "/dev/pts4" added
[55376.481661] pps pps0: removed
[55376.481991] pps pps0: new PPS source pty_slave4
[55376.482098] pps pps0: source "/dev/pts4" added
[55376.482195] pps pps0: removed
[55376.482480] pps pps0: new PPS source pty_slave4
[55376.482578] pps pps0: source "/dev/pts4" added
[55376.482660] pps pps0: removed
[55376.482960] pps pps0: new PPS source pty_slave4
[55376.483053] pps pps0: source "/dev/pts4" added
[55376.483150] pps pps0: removed
[55376.483452] pps pps0: new PPS source pty_slave4
[55376.483561] pps pps0: source "/dev/pts4" added
[55376.483648] pps pps0: removed
[55376.483964] pps pps0: new PPS source pty_slave4
[55376.484068] pps pps0: source "/dev/pts4" added
[55376.484168] pps pps0: removed
[55376.484461] pps pps0: new PPS source pty_slave4
[55376.484564] pps pps0: source "/dev/pts4" added
[55376.484642] pps pps0: removed
[55376.484941] pps pps0: new PPS source pty_slave4
[55376.485043] pps pps0: source "/dev/pts4" added
....
[55535.563623] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55535.563702] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55535.563773] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55535.563853] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55535.563937] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55535.564022] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55539.408425] systemd-udevd[1917076]: slcan0: Could not set offload features, ignoring: No such device
[55540.432430] systemd-udevd[1917051]: sl0: Could not set offload features, ignoring: No such device
[55540.564347] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564426] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564500] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564581] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564658] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564731] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564807] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564884] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.564961] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55540.565031] systemd-journald[1917256]: /dev/kmsg buffer overrun, some messages lost.
[55543.506043] INFO: task request_key03:1853122 blocked for more than 1200 seconds.
[55543.506190]       Tainted: G        W  OE     4.19.90-2312.1.0.0255.oe2003sp4.x86_64 #1
[55543.510587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[55543.510719] request_key03   D    0 1853122      1 0x00000084
[55543.514957] Call Trace:
[55543.514997]  ? __schedule+0x290/0x760
[55543.519253]  schedule+0x29/0xc0
[55543.519296]  schedule_timeout+0x1ee/0x3a0
[55543.519355]  ? check_preempt_curr+0x75/0x80
[55543.519410]  ? ttwu_do_wakeup+0x19/0x150
[55543.519461]  ? try_to_wake_up+0x54/0x540
[55543.519509]  wait_for_completion+0x126/0x190
[55543.519600]  ? wake_up_q+0x70/0x70
[55543.535506]  call_usermodehelper_exec+0x13d/0x170
[55543.535585]  call_sbin_request_key+0x2a9/0x340
[55543.535652]  request_key_and_link+0x4ee/0x650
[55543.535719]  ? keyring_alloc+0x80/0x80
[55543.535765]  ? key_default_cmp+0x20/0x20
[55543.535814]  __x64_sys_request_key+0xf7/0x190
[55543.535923]  do_syscall_64+0x5f/0x240
[55543.553495]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[55543.553640] Kernel panic - not syncing: hung_task: blocked tasks
[55543.553802] CPU: 30 PID: 667 Comm: khungtaskd Kdump: loaded Tainted: G        W  OE     4.19.90-2312.1.0.0255.oe2003sp4.x86_64 #1
[55543.553904] Hardware name: 2288H V7/BC15MBSA, BIOS 2.00.57 03/30/2023
[55543.553965] Call Trace:
[55543.554004]  dump_stack+0x66/0x8b
[55543.554051]  panic+0xe4/0x2a2
[55543.554088]  ? entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[55543.554156]  ? _raw_spin_lock_irq+0x27/0x27
[55543.554214]  watchdog+0x2a8/0x5a0
[55543.554262]  ? unregister_hungtask_control+0x30/0x30
[55543.554339]  kthread+0x113/0x130
[55543.554385]  ? kthread_create_worker_on_cpu+0x70/0x70
[55543.554453]  ret_from_fork+0x1f/0x40
[55543.554506] kernel fault(0x5) notification starting on CPU 30
[55543.610383] kernel fault(0x5) notification finished on CPU 30

**【缺陷详情及分析指导参考链接】**
由于压力大导致jounrald进程复位重启失败,没有复位. 最近发现好多长稳测试用例都是v7的,v7的默认频率不是最大,已经调整性能并重新测试了.
**二、缺陷分析结构反馈**
影响性分析说明：
systemd-jounrnald服务启动失败导致不能正常记录日志, 系统不能正常登录
缺陷严重等级:(High)

缺陷根因说明:

受影响版本排查(受影响/不受影响):
openEuler-20.03-LTS-SP4

修复是否涉及abi变化(是/否):
openEuler-20.03-LTS-SP4

src-openEuler/systemd

内容风险标识

评论 (5)

src-openEuler/systemd .gitee-modal { width: 500px !important; }

内容风险标识