NVIDIA

NVIDIA Toolkit Breaking On Fedora Silverblue After Update

Updating NVIDIA drivers broke the NVIDIA container toolkit. I fixed this by re-installing the NVIDIA container toolkit in rpm-ostree by uninstalling, rebooting, installing, rebooting. Yes, that actually somehow changes things, but it works now.

In short, run this on your host machine, in order:

 export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
 sudo rpm-ostree uninstall \
   nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
   
 systemctl reboot
 

After the reboot...

 export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
 sudo rpm-ostree install \
   nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
   libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
   
 systemctl reboot

Source

Edit: It seems that, at least for Podman, it's easier to just regenerate the CDI, which is what Podman containers use to find the proper libraries on the host machine:

 sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

NVIDIA Toolkit plus SELinux plus Docker

Requires either privileged or:

 security_opt:
   - label=disable

... for CUDA to actually work in the Docker containers. I think it's the same reason you need ":z" on all the bind mounts when you run Docker on an SELinux-enabled system.