Forwarding the GPU with Proxmox and LXC

I got my hands on a machine with an RTX 6000 GPU and wanted to splice the GPU across multiple runtime contexts.

I heard that this is possible through LXC which does not require to have dedicated access to the GPU ¹.

To do this, one needs to configure the exact same GPU driver on the host and the container. As Proxmox is based on Debian, it makes sense to use Debian as a LXC system as well.

Host preparation

We need to install the nvidia driver on the host.

1. add non-free to /etc/apt/sources.list
2. apt install nvidia-driver pve-firmware nvidia-kernel-support
3. reboot
4. check nvidia-smi

This should give a valid output.

Now create an LXC based on the Debian 12 template in Proxmox.

You need to add

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

to /etc/pve/lxc/<YOUR LXC ID>.conf

Now start the container and install the nvidia driver in the LXC as follows.

Container

1. add non-free to /etc/apt/sources.list
2. apt install nvidia-driver firmware-misc-nonfree
3. reboot
4. check nvidia-smi
5. this should give a valid output

This looks good. We can even do this a second time, or create a LXC template from that. It works to slice the GPU, just like we would have different processes running on GPU on a host.

Install VNC and test

I then tried starting VNC and steam ².

sudo apt-get install tightvncserver
tightvncserver
# starts a new vnc session

# install steam
wget -O ~/steam.deb http://media.steampowered.com/client/installer/steam.deb && apt install ~/steam.deb && rm ~/steam.deb
# run steam
steam

Though steam does not run as root ³, which is the case in an LXC, so I had to tinker a little further, but did not get this running.

There is also https://github.com/games-on-whales/gow which I did not try.

Major drawback

Yet, I experienced a major drawback when trying this setup. While it allows to split the GPU just fine, the host does always allocate the GPU, even if it is not used. This removes the possibility to pass the GPU through to a proper QEMU VM.

The only way to fix this was to uninstall the nvidia-driver on the host again, which disables the possibility to run the nvidia-driver in the LXC. So on the long run, I am going with QEMU VM with passthrough GPU.

If the requirement is to split the usage of the GPU, one can still use OCI/docker containers inside the VM to split the GPU ressources.

References