Files

2.5 KiB

LXC GPU Device Bind Rules — Quick Reference

AMD ROCm (APU / dGPU)

# /etc/pve/lxc/NNN.conf additions
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 511:0 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file

Verify host devices before adding:

ls -la /dev/dri/ /dev/kfd

NVIDIA CUDA

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.cgroup2.devices.allow: c 235:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file

Intel Arc / OneAPI

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 10:122 rwm    # /dev/dri/renderD128 via misc major
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

Device Number Reference

Path Major Minor Group
/dev/dri/card0 226 0 video
/dev/dri/renderD128 226 128 render
/dev/dri/renderD129 226 129 render
/dev/kfd 511 0 compute
/dev/nvidia0 195 0 nvidia
/dev/nvidiactl 195 255 nvidia
/dev/nvidia-uvm 243 0 nvidia

Dynamic Device Discovery Script

Run on host to auto-generate LXC mount entries:

#!/bin/bash
# generate_lxc_gpu_mounts.sh
DEV_ALLOW=""
MOUNTS=""

# AMD kfd
if [ -e /dev/kfd ]; then
    DEV_ALLOW+="lxc.cgroup2.devices.allow: c 511:0 rwm\n"
    MOUNTS+="lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file\n"
fi

# DRI devices
for dev in /dev/dri/card* /dev/dri/renderD*; do
    [ -e "$dev" ] || continue
    MAJ=$(stat -c '%t' "$dev")
    MIN=$(stat -c '%T' "$dev")
    MAJ_DEC=$((16#$MAJ))
    MIN_DEC=$((16#$MIN))
    DEV_ALLOW+="lxc.cgroup2.devices.allow: c ${MAJ_DEC}:${MIN_DEC} rwm\n"
    BASENAME=$(basename "$dev")
    MOUNTS+="lxc.mount.entry: $dev dev/dri/$BASENAME none bind,optional,create=file\n"
done

echo "# Device allow rules:"
echo -e "$DEV_ALLOW"
echo "# Mount entries:"
echo -e "$MOUNTS"