Arch Linux

JATothrim

Member

Registered: 2019-01-31

Posts: 21

Setup of /dev/dri in systemd-nspawn container

Hi! I'm having trouble getting vulkaninfo (amdgpu) to work in a systemd-nspawn container that I'm setting up. I'm setting this up on Wayland/KDE Plasma. I'm currently stuck with:

cat /etc/systemd/nspawn/box.nspawn
[Exec]
Boot=true
PrivateUsers=pick
[Network]
VirtualEthernet=on
Bridge=vbr0
[Files]
# Bind host wayland (works, graphical apps launch, if I export WAYLAND_DISPLAY=/tmp/wayland.sock)
Bind=/run/user/1000/wayland-0:/tmp/wayland.sock
# GPU Access. Not working.
Bind=/dev/dri

None of my attempts with fiddling with /etc/subgid or trying to idmap allowed the container access /dev/dri/* device files, and they show up as:

ls -la /dev/dri
drwxr-xr-x  3 nobody nobody      100 Jul 24 00:37 .
drwxr-xr-x  8 root   root        460 Jul 24 01:05 ..
drwxr-xr-x  2 nobody nobody       80 Jul 24 00:37 by-path
crw-rw----+ 1 nobody nobody 226,   1 Jul 24 00:37 card1
crw-rw-rw-  1 nobody nobody 226, 128 Jul 24 00:37 renderD12

And vulkaninfo thus fails with:
'DISPLAY' environment variable not set... skipping surface info
WARNING: [../mesa-26.1.5/src/amd/vulkan/radv_physical_device.c:2393] Code 0 : Could not open device /dev/dri/renderD128: Operation not permitted (VK_ERROR_INCOMPATIBLE_DRIVER)
ERROR: [Loader Message] Code 0 : setup_loader_term_phys_devs:  Failed to detect any valid GPUs in the current config
ERROR at /usr/src/debug/vulkan-tools/Vulkan-Tools/vulkaninfo/./vulkaninfo.h:249:vkEnumeratePhysicalDevices failed with ERROR_INITIALIZATION_FAILED
drm_info:
/dev/dri/card1: Permission denied
Failed to retrieve information from /dev/dri/card1

I have done several hours of googling at this point, and trying to understand the nspawn documentation hasn't gotten me anywhere yet. What I'm missing?
Fully isolating the container from the host is not too critical.

---

If it ain't broken yet, some time passes and I'm doomed to learn its secrets.
Now it is half broken but twice as fast than before.

jonno2002

Member

Registered: 2016-11-21

Posts: 887

Re: Setup of /dev/dri in systemd-nspawn container

i dont use wayland but this is my config which works with intel graphics, should get you on the right track, and yes when i set this up years ago info was hard to find. i use it for playing games using wine and lib32 without cluttering my main system, works well

cat '/etc/systemd/system/systemd-nspawn@.service.d/override.conf' 
[Service]
DeviceAllow=/dev/dri/card0 rw
DeviceAllow=/dev/dri/renderD128 rw
DeviceAllow=char-usb_device rwm 
DeviceAllow=char-input rwm 
DeviceAllow=char-alsa rwm 


cat /etc/systemd/nspawn/container.nspawn
[Exec]
Boot=1
PrivateUsers=no

[Files]
BindReadOnly=/tmp/.X11-unix
Bind=/dev/dri
Bind=/dev/shm

# input
Bind=/dev/input

# pulseaudio
Bind=/run/user/1000/pulse:/run/user/host/pulse

# alsa
Bind=/dev/snd
Bind=/dev/dsp

[Network]
VirtualEthernet=no

JATothrim

Member

Registered: 2019-01-31

Posts: 21

Re: Setup of /dev/dri in systemd-nspawn container

[Service]
DeviceAllow=/dev/dri/card0 rw
DeviceAllow=/dev/dri/renderD128 rw
...
[Exec]
PrivateUsers=no

In the container the device file perms are correct according to stat, it's just that the owner/group UID/GID mapping is wrong.

EDIT:

systemctl set-property systemd-nspawn@box.service DeviceAllow="/dev/dri/renderD128 rw"
systemctl set-property systemd-nspawn@box.service DeviceAllow="/dev/dri/card1 rw"

Created the override.conf and vulkaninfo now runs. However, the /tmp/wayland.sock owner/group is nobody and thus graphical apps fail.

I do want user isolation, and PrivateUsers=no setting would disable this. The old host had some configuration done in /etc/subgid. I'm unsure how this plays with PrivateUsers=pick or yes.

EDIT2: I cannot change to PrivateUsers=no anymore, as this would break the container user folders. It appears that PrivateUsers=pick might be incompatible what I'm trying to do. Only promising clues I have are "rootidmap" and "owneridmap" from  https://github.com/systemd/systemd/pull … 293667bbba

EDIT3: I found a solution which is not perfect:

[Files]
Bind=/run/user/1000/wayland-0:/opt/host/wayland-0:owneridmap
# Above maps the socket as the root user in container so:
export WAYLAND_DISPLAY=/opt/host/wayland-0
sudo -E vkcube # works!

SOLVED: /opt/host/ and /opt/host/wayland-0 must have the right owner before the container is launched. So, this won't work with volatile destination file systems well. With owneridmap the user id mapping works, and I can now run graphical apps without sudo!

Last edited by JATothrim (2026-07-24 17:12:57)

---

If it ain't broken yet, some time passes and I'm doomed to learn its secrets.
Now it is half broken but twice as fast than before.

jonno2002

Member

Registered: 2016-11-21

Posts: 887

Re: Setup of /dev/dri in systemd-nspawn container

glad you figured it out, yea im sure i went down that path of private users but it breaks the ability to use devices on the host as you found out, its been so long since ive touched it ive forgotten all the ins and outs thats why i just posted my config to help you out

JATothrim

Member

Registered: 2019-01-31

Posts: 21

Re: Setup of /dev/dri in systemd-nspawn container

I need to give a warning about changing PrivateUsers=pick to off. This setting should not be changed afterwards, because the container will likely break. I had to restore several directories owner/group manually, and I suspect there is still some breakage from booting the container with PrivateUsers=off.

The /dev/dri/renderD128 /dev/dri/card1 owner and group still aren't correct (shown as nobody), so drm_info does not work. vulkaninfo, vkcube however do work, which is what matters to me. I would like to still know how to bind the /dev/dri correctly into the container. It cannot use the trick I just discovered, because /dev/ is a special volatile filesystem.

---

If it ain't broken yet, some time passes and I'm doomed to learn its secrets.
Now it is half broken but twice as fast than before.

Toolybird

Member

Registered: 2017-09-30

Posts: 91

Re: Setup of /dev/dri in systemd-nspawn container

drm_info works fine here. I just keep it really simple (with args):

--bind=/dev/dri --property=DeviceAllow='char-drm rw'