[openEuler 20.03-LTS SP3]openEuler开启crash_kexec_post_notifiers后，panic通知链无法完全遍历

【标题描述】在cmdline配置crash_kexec_post_notifiers=1后，x86虚拟机hung死，无法正常生成vmcore，部分注册过panic通知链的模块无法正常收到虚拟机内核panic事件的通知
【环境信息】
虚拟机配置

<domain type='kvm'>
  <name>openEuler</name>
  <memory unit='GiB'>8</memory>
  <currentMemory unit='GiB'>8</currentMemory>
  <memtune>
    <hard_limit unit='GiB'>8</hard_limit>
  </memtune>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>1024</shares>
    <period>100000</period>
    <quota>-1</quota>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='yes' timeout='0'/>
    <bios useserial='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
    <!--numa>
      <cell id='0' cpus='0' memory='4194304' unit='KiB'/>
    </numa-->
  </cpu>
  <clock offset='variable' adjustment='0' basis='utc'>
    <timer name='rtc' track='guest'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>preserve</on_crash>
  <on_lockfailure>poweroff</on_lockfailure>
  <devices>
<emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
        <source file='/Images/openEuler-20.03-LTS-SP3-x86_64-dvd.iso'/>
        <target dev='hda' bus='ide'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/Images/xxxxx.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <!--boot order='1'/-->
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
     <controller type='usb' index='1' model='pci-ohci'>
       <alias name='usb1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
     </controller>
     <controller type='ide' index='0'>
       <alias name='ide'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
     </controller>
     <controller type='scsi' index='0' model='virtio-scsi'>
       <alias name='scsi0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
     </controller>
     <controller type='virtio-serial' index='0'>
       <alias name='virtio-serial0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
     </controller>
     <controller type='pci' index='0' model='pci-root'>
       <alias name='pci.0'/>
     </controller>
     <controller type='pci' index='1' model='pci-bridge'>
       <model name='pci-bridge'/>
       <target chassisNr='1'/>
       <alias name='pci.1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
     </controller>
     <interface type='bridge'>
       <mac address='00:16:3e:bd:xx:xx'/>
       <source bridge='br0'/>
       <target dev='vm-openeuler'/>
       <model type='virtio'/>
       <alias name='net0'/>
     </interface>
     <serial type='pty'>
       <source path='/dev/pts/14'/>
       <target port='0'/>
       <alias name='serial0'/>
     </serial>
     <console type='pty' tty='/dev/pts/14'>
       <source path='/dev/pts/14'/>
       <target type='serial' port='0'/>
       <alias name='serial0'/>
     </console>
     <input type='tablet' bus='usb'>
       <alias name='input0'/>
       <address type='usb' bus='0' port='1'/>
     </input>
     <input type='mouse' bus='ps2'>
       <alias name='input1'/>
     </input>
     <input type='keyboard' bus='ps2'>
       <alias name='input2'/>
     </input>
 <input type='mouse' bus='ps2'/>
     <graphics type='vnc' port='5905' autoport='yes' listen='0.0.0.0' keymap='en-us'>
       <listen type='address' address='0.0.0.0'/>
     </graphics>
     <video>
       <model type='cirrus' vram='16384' heads='1' primary='yes'/>
       <alias name='video0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
     </video>
     <memballoon model='virtio'>
       <alias name='balloon0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
     </memballoon>
     <panic model='isa'>
       <address type='isa' iobase='0x505'/>
     </panic>
   </devices>
   <!--seclabel type='none' model='none'/>
   <seclabel type='dynamic' model='dac' relabel='yes'>
     <label>+0:+0</label>
     <imagelabel>+0:+0</imagelabel>
   </seclabel-->
 </domain>

软件信息：
1） OS版本及分支：openEuler-20.03-LTS-SP3
2）内核信息：kernel-4.19.90-2112.8.0.0131.oe1.x86_64
3）发现问题的组件版本信息：kernel-4.19.90-2112.8.0.0131.oe1.x86_64
【问题复现步骤】
1、cmdline配置crash_kexec_post_notifiers=1 crashkernel=512M参数
2、执行sysctl -w kernel.softlockup_panic=1 和systemctl start kdump使能软狗和kdump服务
3、通过ko触发softlockup，进入panic流程，ko代码可参考https://blog.csdn.net/rikeyone/article/details/105798469
出现概率：必现
【预期结果】
虚拟机复位，生成vmcore，已注册过panic通知链的所有模块收到通知（目前已知链表头上挂有以下5通知链trace_panic_handler/pvpanic_panic_notify/rcu_panic/hung_task_panic/dump_kernel_offset）
【实际结果】
虚拟机hung住，即使修改虚拟机xml配置文件中的on_crash选项为coredump-destroy、coredump-restart也无法由虚拟机自身生成vmcore，5条通知链中仅有trace_panic_handler/pvpanic_panic_notify收到通知。
【附件信息】

【目前为止的定位信息】
1、crash_kexec_post_notifiers和kdump服务开启后，panic流程将执行atomic_notifier_call_chain通知panic通知链上的模块；

2、当通知链执行到pvpanic模块时，pvpanic_panic_notify回调将向qemu发送PVPANIC_PANICKED事件

static int pvpanic_panic_notify(struct notifier_block *nb, unsigned long code,
                     void *unused)
{
        pvpanic_send_event(PVPANIC_PANICKED);
        return NOTIFY_DONE;
}

3、qemu接收事件进行处理，当检测事件为PVPANIC_PANICKED时，qemu将对guest事件进行响应

static void handle_event(int event)
{
    static bool logged;

    if (event & ~PVPANIC_PANICKED && !logged) {
        qemu_log_mask(LOG_GUEST_ERROR, "pvpanic: unknown event %#x.\n", event);
        logged = true;
    }

    if (event & PVPANIC_PANICKED) {
        qemu_system_guest_panicked(NULL);
        return;
    }
}

4、qemu根据虚拟机xml的配置决定crash的响应动作：

<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>preserve</on_crash>
<on_lockfailure>poweroff</on_lockfailure>
<devices>

5、由于当前虚拟机配置为preserve，qemu将保留虚拟机当前crash的状态
6、而虚拟机中atomic_notifier_call_chain执行流程被阻塞在pvpanic_panic_notify回调中，后续通知链无法继续收到通知，panic流程也无法进入生成vmcore的阶段

当前kernel主线已合入相关补丁来，增加PVPANIC_CRASH_LOADED事件来应对此种场景：https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.17-rc5&id=191941692a3d1b6a9614502b279be062926b70f5
qemu也提交了两个合入适配了对应修改：https://github.com/qemu/qemu/commit/7dc58deea79a343ac3adc5cadb97215086054c86
https://github.com/qemu/qemu/commit/b1b0393c3c58c0e96c7c44e2e98baa252d6c6813

Hi chenmaodong, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: Kernel, and any of the maintainers: @YangYingliang , @pi3orama , @成坚 (CHENG Jian) , @Qiuuuuu , @zhengzengkai , @gogooo , @Xie XiuQi

已合入，关闭

GVP openEuler / kernel

Content Risk Flag

Comments (2)

GVPopenEuler / kernel

Content Risk Flag