GPU（Nvidia）容器实例设备节点权限偶发丢失问题

版本：openeuler-22.03sp1 + systemd-249
描述：初步分析在Systemd 刷新所有unit的时候（daemon-reload、SetUnitProperties等操作），会丢失GPU设备节点权限信息（刷新覆盖原有GPU写入的正确权限）
问题：GPU容器实例内部存在设备节点，但是无法调用，返回Failed to initialize NVML: Unknown Error错误：
Eg:
```
# nvidia-smi
# Failed to initialize NVML: Unknown Error
```
查看容器实例Cgroup目录下的devices.list，发现实例缺失 Nvidia 设备权限。
异常：
```
# cat /sys/fs/cgroup/devices/kubepods/kubepods-burstable.slice/***/device.list
b *:* m
c *:* m
```
正常：
```
cat device.list
b *:* m
c *:* m
c 555:0 rw
c 555:1 rw
```

由于故障偶发，监控发现，systemd会修改devices.list的程序
```
systemd called devegroup_access_write Cgroup path: kubepods-pod_uuid. slice/cri-containerd-uuid/devices.allow Input data: c 5:2  ram
systemd called devegroup_access_write Cgroup path: kubepods-pod_uuid. slice/cri-containerd-uuid/devices.deny Input data: a
```

查看 Documentation/cgroup-v1/devices.txt 文档，
```
devices.list（只读）
devices.allow（只写）
devices.deny（只写）
三个文件之间的关系:
 通过｜devices.allow｜写入 -> ｜devices.list(存放设备节点信息)| <- 删除｜devices.deny｜通过
```
devices.list的实现来看，只有操作devices.deny才会删除devices.list的内容。
所以这里的排查重点在有devices.deny操作的Systemd。Systemd操作cgroup devices.list的逻辑是这样的，先禁止所有设备节点的权限，再写入白名单设备节点：
vim systemd-249/src/core/cgroup.c
```
static void cgroup_context_apply()
    -> cgroup_apply_devices(Unit *u)
        -> r = cg_set_attribute("devices", path, "devices.deny", "a");
        -> LIST_FOREACH(device_allow, a, c->device_allow) 
            -> r = bpf_devices_allow_list_device(prog, path, a->path, acc);
```
![输入图片说明](https://foruda.gitee.com/images/1743994580859489406/8940b5de_12680603.png "屏幕截图")
![输入图片说明](https://foruda.gitee.com/images/1743994513559535883/046e4c15_12680603.png "屏幕截图")

分析Systemd的Devices行为：首先排查可能性最大的路径：Systemd在重写cgroup devices.list的时候，丢失了GPU相关的设备节点。
runc创建调用dbus接口通过systemd会创建一个名字52c2148_b8f0_4e40_92d7_a0522bb6c8f8, 并指定 cgroup路径 /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52c2148_b8f0_4e40_92d7_a0522bb6c8f8.slice/{memory,cpu}的限制：

![输入图片说明](https://foruda.gitee.com/images/1743994674304170782/581c1c9c_12680603.png "屏幕截图")

在机器上执行systemd daemon-reload，
容器内执行 nvidia-smi ，立刻失去Nvidia设备的访问权限：
![输入图片说明](https://foruda.gitee.com/images/1743995308575775540/7dd85530_12680603.png "屏幕截图")

同时，同一台机器上其他GPU容器实例的设备节点也全部丢失：
```
# cat /sys/fs/cgroup/devices/kubepods/kubepods-burstable.slice/***/device.list
b *:* m
c *:* m
```

**由上面构建的验证环境可知，系统上执行了Systemd重新加载所有unit，会导致GPU软件丢失。
同时，排查CentOS的机器上Dbus消息里面，同样有非常多命令会涉及devices.list操作，为什么GPU丢失设备节点的问题从来没有在CentOS上发生。**

**分析CentOS和OpenEuler 的不同行为** ：
CentOS-219版本的Systemd创建的unit默认是static状态，在reload/mask等流程中会被跳过：
```
systemctl status cri-containeed-***.scope
    Load: loaded (/run/systemd/system/cri-containerd-***.scope; static; vendor preset: disabled)
    Drop-In: /run/systemd/system/cri-containerd-***.scope.d
            └─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-CPUShares.conf
```
OpenEuler版本的Systemd-249创建unit是  **transient** （https://systemd.io/TRANSIENT-SETTINGS/），无法被设置为static/disable状态：
```
systemctl status cri-containeed-***.scope
    Load: loaded (/run/systemd/system/cri-containerd-***.scope; transient)
Transient: yes
    Drop-In: /run/systemd/transient/cri-containerd-***.scope.d
            └─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-CPUShares.conf
```

```
# systemctl cat cri-containerd-***.scope
# /run/systemd/transient/cri-containerd-***.scope
# This is a transient unit file, created programmatically via the systemd API. Do not edit.
[Unit]
Description=libcontainer container ***

[Scope]
Slice=kubepods-burstable-pod***.slice
Delegate=yes
MemoryAccounting=yes
CPUAccounting=yes
BlockIOAccounting=yes
TasksAccounting=yes

[Unit]
DefaultDependencies=no

# /run/systemd/transient/cri-containerd-***.scope.d/50-CPUShares.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
CPUShares=1

# /run/systemd/transient/cri-containerd-***.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

# /run/systemd/transient/cri-containerd-***.scope.d/50-DevicePolicy.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DevicePolicy=strict
```

src-openEuler/systemd
关闭

内容风险标识

评论 (8)

src-openEuler/systemd关闭 .gitee-modal { width: 500px !important; }

内容风险标识