94 Star 261 Fork 91

泰晓科技 / RISCV-Linux

 / 详情

【老师提案】 RISC-V 虚拟化技术调研与分析

Done
member
Opened this issue  
2022-06-25 23:30

项目标题:RISC-V虚拟化技术调研与分析
项目描述:RISC-V虚拟化技术的调研和实现分析
项目难度:进阶
项目社区导师:@walimis
导师联系方式:walimis@walimis.org
合作导师联系方式:
项目产出要求:

学习运行RISC-V KVM:

  1. 通过qemu/spike运行Host Linux
  2. 通过kvmtool运行Guest Linux

通过阅读规范,产出如下文档:

  1. RISC-V Hypervisor Extension v1.0 规范解读

通过阅读kvm/qemu/kvmtool/spike代码,产出如下文档:

  1. vCPU虚拟化在RISC-V的实现
  2. 内存虚拟化在RISC-V的实现
  3. 中断虚拟化在RISC-V的实现
  4. timer虚拟化在RISC-V的实现
  5. 设备虚拟化在RISC-V的实现
  6. Host和Guest性能评测和比较

通过学习代码,可以提交一些patch到riscv kvm upstream
项目技术要求:

  • Linux 基本操作
  • 熟悉 C 语言
  • 了解算法基础知识
  • 了解处理器基本工作原理
  • 了解虚拟化和KVM的基本工作原理

实习计划:

  • 阅读指定书籍和资料,对整体虚拟化有个大概的理解和认识
  • RISC-V如何进行虚拟化模式切换
    • 学习规范8.1和8.2
      • 理解virtualization mode,理解V=1和V=0
      • 学习Hypervisor and Virtual Supervisor CSRs
      • 如何从V=0到V=1进行切换
    • 研究qemu代码,找到如何从V=0到V=1进行切换的代码并分析
      • 文件:target/riscv/op_helper.c
      • 重点函数:helper_sret
    • 研究spike代码,找到如何从V=0到V=1进行切换的代码并分析
      • 文件:riscv/insns/sret.h
    • 研究Linux RISC-V KVM模块,如何从V=0到V=1进行切换
      • 文件:arch/riscv/kvm/vcpu.c,函数kvm_arch_vcpu_create()
      • 文件:arch/riscv/kvm/vcpu_switch.S
  • vCPU虚拟化在RISC-V的实现
    • 学习指定书第四章,理解CPU虚拟化
    • 研究kvmtool中的vCPU操作
      • 文件:riscv/kvm-cpu.c
    • 研究Linux kvm中RISC-V vCPU的实现
      • 文件:arch/riscv/kvm/vcpu.c
    • RISC-V Host/Guest切换
      • 如何切换;切换上下文切换
      • 文件:arch/riscv/kvm/vcpu_switch.S
  • 内存虚拟化在RISC-V的实现
    • 学习指定书第五章,理解内存虚拟化
    • 学习规范8.5 Two-Stage Address Translation,理解RISC-V如何支持2阶段地址转换
    • 研究kvmtool中的内存虚拟化操作
    • 研究Linux kvm中RISC-V内存虚拟化的实现
      • 文件:arch/riscv/kvm/mmu.c, arch/riscv/kvm/tlb.c
  • 中断虚拟化在RISC-V的实现
    • 学习指定书第五章,理解中断虚拟化
    • 学习规范8.4 Machine-Level CSRs和8.6 Traps,理解RISC-V如何中断虚拟化
    • 研究kvmtool中的中断注入的操作
    • 研究Linux kvm中RISC-V中断虚拟化的实现
      • 文件:arch/riscv/kvm/vcpu.c, arch/riscv/kvm/vcpu_exit.c
  • timer虚拟化在RISC-V的实现
    • 研究Linux kvm中RISC-V timer虚拟化的实现
      • 文件:arch/riscv/kvm/vcpu_timer.c
  • MMIO虚拟化在RISC-V的实现
    • 研究Linux kvm中MMIO虚拟化的实现
      • 文件:arch/riscv/kvm/vcpu.c, arch/riscv/kvm/vcpu_exit.c
    • 使用kvmtool跑基于virtio net/virtio block的Guest os的实例

参考书籍:

  • 《QEMU/KVM 源码解析与应用》

参考资料:

相关的开源软件仓库列表:

Comments (41)

walimis created任务
walimis set assignee to walimis
walimis changed description
walimis changed title
falcon changed description
walimis changed description
walimis changed description
walimis changed description
falcon changed title
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description
Expand operation logs

考核任务,写一份中文指南:

  1. 参考文档:https://github.com/kvm-riscv/howto/wiki
  2. 基于ubuntu 20.04平台,使用最新qemu/spike/kvmtools/linux代码,附上对应的commit
  3. 运行qemu/spike跑起来Host Linux
  4. 加载kvm模块
  5. 运行kvmtools跑起来Guest Linux
  6. 卸载kvm模块(可能需要改代码)

[Daily Update]0705-2022(0706补)

参照wiki boot Host/Guest Linux

  1. 已完成部分(基于WSL2下的Ubuntu20.04发行版)
    • RISCV toolchain, QEMU(commit 409b951ba6621f2f115aebddfb15ce2dd78ec24f), OpenSBI, Linux内核(Linux 5.19-rc5)下载编译
    • 上述流程的文字记录与流程撰写
  2. 正在进行部分
    • kvmtools编译,make lkvm-static出现问题Makefile:357: *** No libfdt found. Please install libfdt-dev package. Stop., 但不论是通过手动编译还是apt下载安装都无法使之检测到libfdt。

[daily update]0706-2022

尝试找出昨天出错的问题,在运行Ubuntu操作系统的机器上进行昨日的步骤。

  • debug: WSL2中运行的发行版不支持虚拟化,昨日No libfdt found.错误或因此导致。
    $ egrep -c '(vmx|svm)' /proc/cpuinfo
    0 
    
  • 尝试解决:在运行Ubuntu的机器上进行实验
    • 进度:代码全部下载完毕,riscv toolchain编译完成
    • 剩余任务
      • 其余编译、运行流程(spike/qemu+kvm run host/guest linux)
      • kvm kernel module uninstall

WSL2 支持不支持嵌套虚拟化跟 RISC-V 都没有关系。RISC-V 的虚拟化是跑在 QEMU 模拟的带有 H-extension 的硬件上的。

[daily update]0707-2022

  • 进度
    • 成功在Ubuntu20.04 container上用QEMU+KVM boot Host/Guest Linux
    • 修改linux/arch/riscv/kvm/main.c代码支持在虚拟机内以rmmod kvm指令卸载kvm module
  • 当日工作
    • 尝试在WSL2和Ubuntu本机进行整个流程
      • 通过apt install gcc-riscv64-linux-gnu解决了libfdt无法用于kvmtool编译的问题(后续qemu x-h=true问题解决,应该可以正常运行,还未尝试)
      • 在Ubuntu本机环境检验了apt install gcc-riscv64-linux-gnu安装工具链的可用性。推测或许因为桌面环境以及已有配置较为复杂,apt安装和本地编译并不能完全支持后续步骤:
        • 仅使用apt install gcc-riscv64-linux-gnu安装的工具链缺乏ld,无法链接程序
        • 手动编译的riscv64-unknown-linux-gnu-gcc足够齐全,但是头文件配置出现问题(busybox build rootFS step)
    • 成功在docker中创建Ubuntu20.04 container完成所有流程(组件准备、qemu运行(载入、卸载))
  • TODO:文档整理

ld 在 binutils 中,可以试试:apt install binutils-riscv64-linux-gnu

Linux Lab 中已经完整支持所有的工具链的,也是一种候选的方式,一开始就是推荐搭建统一使用这个环境,对齐后续实验的环境。Ubuntu 20.04 + Qemu v6.2.0 + RISC-V toolchains + Linux Kernel 源码 + riscv64/virt 虚拟开发板,全部是一应俱全的。Linux Lab Disk 则连安装都不需要,直接可用。

好的,后续可以让xiakai再使用linux Lab来做一下。

如下是我ubuntu上安装的riscv相关的编译软件:
binutils-riscv64-linux-gnu
cpp-9-riscv64-linux-gnu
cpp-riscv64-linux-gnu
gcc-9-riscv64-linux-gnu
gcc-9-riscv64-linux-gnu-base:amd64
gcc-riscv64-linux-gnu
libatomic1-riscv64-cross
libc6-dev-riscv64-cross
libc6-riscv64-cross
libgcc-9-dev-riscv64-cross
libgcc-s1-riscv64-cross
libgomp1-riscv64-cross
linux-libc-dev-riscv64-cross

[daily update]0708-2022(0709 补)

  • 进度
    • docker 环境下完成全部运行任务
    • 产出文档已经提交 RP

[daily update]0711-2022(0712 补)

  • 阅读《QEMU/KVM 源码解析与应用》第一章:QEMU 与 KVM 概述

[daily update]0712-2022

  • 阅读《QEMU/KVM 源码解析与应用》第二、三章
    • QEMU 基本组件:内部运行机制(事件循环、线程模型、CPU 等模型的 OO 的 C 实现)
    • 主板与固件模拟:主板、BIOS 的模拟与初始化
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description
walimis changed description

[daily update]0713-2022

  • 阅读《QEMU/KVM 源码解析与应用》第4,5章(至5.4)

[daily update]0714-2022

  • 阅读《QEMU/KVM 源码解析与应用》第 5-8 章(5.4-8.4)
    • KVM 对于内存虚拟化的实现
    • QEMU 终端虚拟化的机制与实现
    • 略读(7,8 章):设备虚拟化、虚拟机迁移、虚拟化安全等

[daily update]0717-2022

  • 阅读riscv-privileged8.1-8.2.1

[daily update]0719-2022(0720 补)

  • 进度:RISC-V 如何进行虚拟化模式切换[Half Done]
    • 整理特权指令级规范中关于虚拟模式切换的论述(特权级对应的虚拟化模式、特殊 CSR、trap 如何触发特权级切换及其具体行为)
    • 阅读如下代码,初步捋清代码内部虚拟化模式切换机制及其对应寄存器(mstatus, hstatus, sstatus),产出原始文档
      • qemu/target/riscv/op_helper.c
      • spike: riscv-isa-sim/riscv/insns/sret.h
      • linux/arch/riscv/kvm/vcpu.c, linux/arch/riscv/kvm/vcpu_switch.S

[daily update]0724-2022

提交分析文章 20220723-virt-mode.md Pull Request.

[daily update]0726-2022

  • 根据审阅意见修改文章 (20220723-virt-mode.md)
  • 阅读《QEMU/KVM 源码解析与应用》及如下网络文章,理清 kvm 整体的虚拟化的架构
  • 澄清了一个个人疑点:虚拟机维护的寄存器数据仅用于保存独立的数据,方便后续恢复运行;各种 CSR 均对应硬件中的寄存器单元(QEMU 等仿真器除外)。虚拟机与硬件的寄存器并不是互相补全的关系,而是数据备份与物理载体的关系。

[daily update]0729-2022

提交 20220729-kvm-hello-world.md

@XiakaiPan @walimis @trueptolemy 类似这样的新特性也可以撰写文章分析一下:https://lwn.net/ml/linux-kernel/20220722230241.1944655-1-avagin@google.com/

[PATCH 0/5] KVM/x86: add a new hypercall to execute host system

There is a class of applications that use KVM to manage multiple address
spaces rather than use it as an isolation boundary. In all other terms,
they are normal processes that execute system calls, handle signals,
etc. Currently, each time when such a process needs to interact with the
operation system, it has to switch to host and back to guest. Such
entire switches are expensive and significantly increase the overhead of
system calls. The new hypercall reduces this overhead by more than two
times.

The new hypercall runs system calls on the host.  As for native system
calls, seccomp filters are executed before system calls. It takes one
argument that is a pointer to a pt_regs structure in the host address
space. It provides registers to execute a system call according to the
calling convention. Arguments are passed in %rdi, %rsi, %rdx, %r10, %r8
and %r9 and a return code is stored in %rax. 

The hypercall returns 0 if a system call has been executed. Otherwise,
it returns an error code.

This series introduces a new capability that has to be set to enable the
hypercall. The new hypercall is a backdoor for regular virtual machines,
so it is disabled by default. There is another standard way to allow
hypercalls via cpuid. It has not been used because one of the common
ways to manage them is to request all available features and let them
all together. In this case, it is a hard requirement that the new
hypercall can be enabled only intentionally.

[daily update]0812-2022

Update 20220729-kvm-user-app.md with kvmtool and QEMU analysis.

Commit PR.

[daily update]0815-2022

DONE:

  • RISC-V 两级地址转换机制解析
  • SFENCE, HFENCE 指令解析
  • mstatus, hstatus, sstatus, vsstatus 对应关系解析

HALF DONE:

  • status CSR 比较(field 划分及其在两级地址转换中的功能)
  • atp CSR 比较(satphgatp, vsatp对应关系)
S Status H Status INVAL Status CSRs Status
SFENCE DONE HFENCE DONE INVAL... TODO status Ongoing
Basic Page-Based Address Translation TODO 2-Stage Ongoing atp Ongoing
Simulator Implementation TODO Simulator Implementation TODO Simulator Implementation TODO trap TODO
interrupt TODO
exception TODO
time TODO
config/performance/security TODO

[daily update]0816-2022

RISC-V 虚拟化调研

Half Done:Basic Page-Based Address Translation in S-Mode

[daily update]0817-2022

DONE:CSR xstatus
Ongoing: Basic Page-Based Address Translation, 2-Stage AT

[daily update]0822-2022

Reviewed 20220802-kvm-user-app.md.
PR of 20220812-mem-virt-rv-1.md.
Half done 20220812-mem-virt-rv-2.md

Status Address Translation 2-Stage AT
Done Mechanism Flowchart
Ongoing Spike implementation analysis G-Stage

[daily update]0823-2022

Collaboration

Update 20220802-kvm-user-app.md and 20220812-mem-virt-rv-1.md.

Verify the method of inserting URL.

Optimize the commit log of PR.

Virtualization

Ongoing: Implementation of PTW in Spike.

Done: Sinval extension.

这个虚拟化的专栏可以关注一下:https://zhuanlan.zhihu.com/p/565787442

[daily update]0922-2022

Modify last article about memory management.

Finish part of h-extension trap mechanism analysis (basic CSR, instructions).

TODO:

  1. How extra H-extension CSRs provide trap handling functions based on S-Extension.
  2. How Spike accomplish H trap handling.

[daily update]0923-2022

Figure out the function of MIP/MIE, SIP/SIE, HVIP/HIP/HIE and HGEIP/HGEIE.

Analyse how H-Ext. CSR HIP/HIE work with SIP/SIE to finish S-level trap handling.

TODO:

The hardware difference between machines that have H-Support and not. (QEMU)

Simulator implementation of extra hypervisor related trap. (Spike)

[daily update]0925-2022

Done

Figure out the function of hardware, OS (including KVM module), kvmtool and guest OS in the process of 2-stage address translation.

TODO

Figure out the function of hardware, OS (including KVM module), kvmtool and guest OS in the process of trap handling.

  1. Where trap can be brought in, what about the types?
  2. Which mode should the trap be processed (CSR read/write privilege of hypervisor, guest OS)?

[daily update]0927-2022

Done

Modification of last article about Memory Management of H-Mode. Submit PR.

Finish an analysis article of trap handling. Submit PR.

[daily update]0929-2022

Try tinycorrect to auto review articles, find out several shortcuts:

  1. Maybe some environment configuration tutorial should be provided, especially for beginner whose experience is almost not related to Web, such as nodejs. I met this problem on my Windows machine, I try to use docker, Git bash and WSL to run tinycorrect.sh, but failed because of:
    1. endline format (dos2unix): solved
    2. terminate /bin/bash when execute cd path/to/tico && . tinycorrect.sh: to be solved
  2. Flowchart image generation:
    1. Only links of generated images from flowcharts are inserted into articles, but no images are converted and saved into images/ directory: to be solved.
    2. About the saving place of generated images, I assume it is images/riscv-kvm/mem-virt-2/mermaid-riscv-kvm-mem-virt-2-1.png for article named riscv-kvm-mem-virt-2.md. I think it's necessary to validate the specific partition mechanism: to be solved.

[daily update]1011-2022

Future work plan: KVM Implementation of

  1. G-Stage Address Translation.
  2. Trap Handling between Guests and Hypervisor.
  3. Timer and MMIO management in KVM.

Today update:

  1. Comb through code related to G-Stage memory allocation for VMs.

[daily update]1012-2022

  • Today Update:

Go through the inner function calling situation in KVM.

  • Tomorrow:

Refer to memory allocation implementation of Linux Buddy memory manager.

[daily update]1013-1016

  • Update:
    • Memory deallocation/free in Buddy System
    • G-Stage Page Fault Handling
      • gstage remap process

[daily update]1017-1018

  • Update
    • Finish code analysis of page fault handling in KVM.
  • Next
    • HGATP update logic.

[daily update]1030-2022

  • Done: KVM implementation of memory virtualization in H-Ext.
  • Half Done: KVM implementation of exception handling in H-Ext.
  • Others: analysis towards interrupt handling will be delayed due to many related content about VFIO, VIRTIO, PCI, PLIC, etc., which is out of my current knowledge and needs further study.

[Month Plan] November

RISC-V 中断虚拟化在 KVM 中的实现

  • KVM 对 RISC-V Specification 中有关 Guest 中断判断与处理机制的一般实现(arch/riscv/kvm/)
    • RISC-V Interrupt Definition: include/asm/csr.h
    • WFI: vcpu_insn.c
    • set/unset/has/flush/sync/_interrupt: vcpu.c
  • Timer 虚拟化及其中断处理
    • Guest Timer: vcpu.c, vcpu_timer.c, include/asm/kvm_vcpu_timer.h
    • Kernel Timer: kernel/sbi.c, include/linux/hrtimer.h, etc.
  • 外部中断:MMIO(Memory-Mapped Input/Output)
    • caller: vcpu_exit.c, vcpu.c
    • callee: kvm_riscv_vcpu_mmio_load/store/return in vcpu_insn.c

kvmtool 中断注入

kvmtool/riscv/

  • PLIC: plic.c
  • PCI: pci.c
  • virtio: virtio_console__inject_interrupt in virtio/console.c
  • IRQ: irq.c, include/kvm/irq.h
walimis changed description

[daily update]1113-2022

Week Update

  • Framework of interrupts handling in KVM and kvmtool has been figured out.
  • The outline of the article is already listed.
walimis changed description
falcon changed assignee from walimis to XiakaiPan
walimis changed description

@XiakaiPan 第一阶段看看什么时候收尾,收尾后记得改为 “已完成” 状态。

XiakaiPan changed issue state from 进行中 to 已完成

Sign in to comment

Status
Assignees
Milestones
Pull Requests
Successfully merging a pull request will close this issue.
Branches
Planed to start   -   Planed to end
-
Top level
Priority
参与者(5)
5069540 wuzhangjin 1594531572
1
https://gitee.com/tinylab/riscv-linux.git
git@gitee.com:tinylab/riscv-linux.git
tinylab
riscv-linux
RISCV-Linux

Search