How to Run Nvidia Container in NixOS WSL

[Fr4nk1inCs]

I'd like to run some ML projects using docker with Nvidia GPU enabled, but I don't know how to set up nvidia-container-toolkit in NixOS-WSL.

My docker-related config is

virtualisation.docker = {
  enable = true;
  rootless = {
    enable = true;
    setSocketVariable = true;
    daemon.settings = {
      default-runtime = "nvidia";
      runtimes.nvidia.path = "${pkgs.nvidia-docker}/bin/nvidia-container-runtime";
    };
  };
};
hardware.nvidia-container-toolkit.enable = true;

Here's the error message when I try running a simple nvidia-smi using docker:

$ docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

[flaviu-toader]

I see the same on my NixOS, my configuration looks like @Fr4nk1inCs's config above. Running on a Dell 5560 with Nvidia dGPU and Intel iGPU and with Prime sync configured.

[shikanime]

Didn't had the time to check, have you tried to enable the CDI feature in the docker daemon settings ? As I saw that it wasn't enabled by default in the docker documentation, wsl provide a Nvidia CDI as far I remember.

[Fr4nk1inCs]

Thanks @shikanime, CDI is the solution:

$ sudo docker run --rm --device nvidia.com/gpu=all --security-opt=label=disable ubuntu nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4070 SUPER (UUID: ...)
$ # rootless
$ docker run --rm --device nvidia.com/gpu=all --security-opt=label=disable ubuntu nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4070 SUPER (UUID: ...)

Configuration:

virtualisation.docker = {
  enable = true;
  rootless = {
    enable = true;
    setSocketVariable = true;
    daemon.settings = {
      features.cdi = true;
      cdi-spec-dirs = ["/home/${username}/.cdi"];
    };
  };
  daemon.settings = {
    features.cdi = true;
  };
};
hardware = {
  nvidia = {
    modesetting.enable = true;
    nvidiaSettings = false;
    open = false;
  };
  nvidia-container-toolkit.enable = true;
};
services.xserver.videoDrivers = ["nvidia"];

The cdi-spec-dirs in rootless configuration is due to NVIDIA/nvidia-container-toolkit#434 (comment)

After rebuild, we need to run the nvidia-ctk (available in pkgs.nvidia-docker) to generate configuration in /etc/cdi and $HOME/.cdi:

$ nvidia-ctk cdi generate --output="$HOME/nvidia.yaml"
...
$ sudo nvidia-ctk cdi generate --output="/etc/cdi/nvidia.yaml"
...

[Scalamando]

Docker Compose

In case anyone also faces this problem:

For me this worked perfectly for docker run commands but failed with docker compose:

# compose.yml
services:
  ollama:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

The above compose.yml would produce the following error message when used with docker compose up -d:

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

Solution

The solution was to simply use the cdi driver instead:

# ...
deploy:
  resources:
    reservations:
      devices:
        - driver: cdi
          device_ids:
            - nvidia.com/gpu=all