| Platform | XPU | Shim | Level-1 | Level-2 | Level-3 |
|---|---|---|---|---|---|
| CUDA | NVIDIA Ampere GPUs (sm86) | ✅ | ✅ | 🚧 | 🚧 |
| NVIDIA Volta GPUs (sm70) | ✅ | ✅ | |||
| NVIDIA Kepler GPUs (sm35) | ✅ | ❌ | |||
| Other NVIDIA GPUs | 🔘 | 🔘 | |||
| HIP | AMD GPUs | ✅ | ✅ | 🔘 | 🔘 |
| LevelZero | Intel GPUs | ✅ | ✅ | 🔘 | 🔘 |
| Intel Integrated NPUs | ✅ | ❌ | |||
| OpenCL | NVIDIA GPUs | ✅ | ✅ | 🔘 | 🔘 |
| AMD GPUs | 🔘 | 🔘 | |||
| Intel GPUs | 🔘 | 🔘 | |||
| Xillinx FPGAs | 🔘 | ❌ | |||
| AscendCL | Ascend NPUs | ✅ | ✅ | 🔘 | ❌ |
| cuDLA | NVIDIA DLA | ✅ | ✅ | ❌ | ❌ |
| VPI | NVIDIA OFA | ✅ | ✅ | ❌ | ❌ |
| NVIDIA PVA | ❌ | ❌ |
XSched consists of four key components: XPU shim (XShim), XPU task preemption module (XPreempt), XPU hardware adapter layer (XAL), and an XScheduler. XShim, XPreempt, and XAL are three dynamically linked libraries that are preloaded into each XPU application process, while XScheduler runs as a centric system service daemon.
- **XShim:** named as `shim` in the code, intercepts XPU driver API calls and redirects commands to the XQueue ①, allowing applications to run on XSched without modifications (transparency).
- **[XPreempt](preempt):** named as `preempt` in the code, implements XQueue interfaces based on the multi-level hardware model ②. Contains an [agent](preempt/src/sched/agent.cpp) that watches the state of XQueue (e.g., ready or idle) and generates scheduling events to notify the XScheduler via IPC ③. Also responsible for applying the scheduling operations (e.g., suspend or resume an XQueue) received from the XScheduler ⑤.
- **XAL:** named as `hal` in the code, implements the multi-level hardware model interfaces by calling XPU driver APIs.
- **[XScheduler](service/server):** named as `xserver` in the code, coordinates all XQueues from different processes, monitors global XQueue status through agent-reported events ③, and invokes the scheduling policy to make decisions when status changes. Decisions are enforced by sending scheduling operations to agents ④. The policy is modular and customizable to suit various workloads.
- **[XCLI](service/cli):** a command-line tool that can monitor XQueue status, change the policy, or give scheduling hints (e.g., priority) ⑥.
## Development Plan
We will continue to support XSched on more OSes and platforms, and improve the performance of XSched. Please stay tuned!
- [x] Integrated into LLM serving systems (e.g., llama.cpp, vLLM)
- [x] Support Windows
- [ ] Support MacOS
- [ ] Install as system daemon
## Contributing
XSched is designed to be extensible and flexible.
We welcome contributions:
- Support more platforms, or a higher preemption level on existing platforms. See [guide](platforms/example/README.md)
- Implement a new scheduling policy. See [guide](sched/README.md)
- Integrate XSched into AI-powered applications.
- Report or fix issues.
## Citation
If you use XSched for your research, please cite our [paper](docs/xsched-osdi25.pdf):
```bibtex
@inproceedings{Shen2025xsched,
title = {{XSched}: Preemptive Scheduling for Diverse {XPU}s},
author = {Weihang Shen and Mingcong Han and Jialong Liu and Rong Chen and Haibo Chen},
booktitle = {19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25)},
year = {2025},
isbn = {978-1-939133-47-2},
address = {Boston, MA},
pages = {671--692},
url = {https://www.usenix.org/conference/osdi25/presentation/shen-weihang},
publisher = {USENIX Association},
month = jul
}
```
The artifacts of XSched is published on [Github](https://github.com/XpuOS/xsched-artifacts) and [Zenodo](https://doi.org/10.5281/zenodo.15327992).
## Contact Us
- For technical questions and feature requests, please submit via GitHub [Issues](https://github.com/XpuOS/xsched/issues)
- For collaborations and partnerships, please reach out at [rongchen@sjtu.edu.cn](mailto:rongchen@sjtu.edu.cn)