NVIDIA CUDA in Docker: GPU Containers for City-Wide Video Surveillance

Overview

Our client runs 900+ video cameras spread all over a city. The idea of the project was to collect the recorded video, store it on dedicated GPU servers and run analysis scripts against it. The infrastructure had to scale, and the development process had to be fully automated, so the team could change the analysis scripts at any time.

So the project focused on dockerizing, orchestrating and automating software that uses video cards and GPUs for its calculations. Every analysis service was a microservice built on NVIDIA CUDA (Compute Unified Device Architecture).

What we built

CI/CD with one-click rollback. We set up Continuous Integration & Continuous Delivery so that an updated script rolls out to all production servers automatically. If the new version has a bug, the previous one is one click away.

Orchestration. We orchestrated the containers with Rancher. At the time, GPU access in Docker went through the nvidia-docker plugin, and Rancher didn't work correctly with it. We built a custom solution that handled the problem. Both tools from that setup are retired now: nvidia-docker is deprecated and archived, and its job is done by the NVIDIA Container Toolkit, which works with plain Docker, Compose and Kubernetes. The setup below is how we'd do it today.

Monitoring. We added monitoring with a visual interface for server and component metrics, from hardware stats like GPU temperature and power draw to software stats like interface load and process state.

How NVIDIA GPUs reach a Docker container

A common misconception is that the CUDA driver lives inside the container. It doesn't. The split looks like this:

• Host: the NVIDIA GPU driver. It's installed once on the server, the same way as without Docker.
• NVIDIA Container Toolkit: also on the host. When a container asks for a GPU, it mounts the GPU devices and the driver's user-space libraries (libcuda, NVML, video codec libraries) into the container.
• Container image: your application plus the CUDA runtime and libraries it needs, for example from the official nvidia/cuda images. No driver in the image.

This is why one GPU server can run containers built for different CUDA versions side by side. The only condition is that the host driver is new enough for the CUDA version in each image. Official CUDA images enforce it: if the driver is too old, the container doesn't start.

How to run a CUDA container on a GPU host

The steps below are for Ubuntu with Docker Engine already installed. They follow the NVIDIA Container Toolkit install guide and Docker's GPU access docs. Written for NVIDIA Container Toolkit 1.20.1 on Ubuntu 22.04 and 24.04.

Step 1. Check the driver on the host

shell
1nvidia-smi

You should see a table with your GPU, the driver version and a "CUDA Version" field. That field is the newest CUDA version the driver supports. If the command isn't found, install the NVIDIA driver for your distribution first.

Step 2. Install the NVIDIA Container Toolkit

Add NVIDIA's repository and signing key:

shell
1curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
2  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
3curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
4  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
5  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Install the package:

shell
1sudo apt-get update
2sudo apt-get install -y nvidia-container-toolkit

Don't install nvidia-docker2. It's deprecated, and the toolkit replaces it.

Step 3. Configure Docker to use the NVIDIA runtime

shell
1sudo nvidia-ctk runtime configure --runtime=docker
2sudo systemctl restart docker

The first command registers the nvidia runtime in /etc/docker/daemon.json. The restart makes Docker pick it up.

Step 4. Run nvidia-smi inside a container

shell
1docker run --rm --gpus all nvidia/cuda:13.4.3-base-ubuntu24.04 nvidia-smi

The output should match what nvidia-smi showed on the host. If it does, GPU access works.

Pick an image tag whose CUDA version is not newer than the "CUDA Version" from Step 1. The full list of tags is on Docker Hub: base has only the CUDA runtime, runtime adds the math libraries, devel adds the compiler and headers.

To give a container specific GPUs instead of all of them:

shell
1docker run --rm --gpus device=0 nvidia/cuda:13.4.3-base-ubuntu24.04 nvidia-smi
2docker run --rm --gpus '"device=0,2"' nvidia/cuda:13.4.3-base-ubuntu24.04 nvidia-smi

Step 5. Build your own CUDA image

A multi-stage build keeps the compiler out of the final image. Here is a small CUDA program, gpu-check.cu, that lists the GPUs it can see:

shell
1#include <cstdio>
2#include <cuda_runtime.h>
3
4int main() {
5    int count = 0;
6    cudaError_t err = cudaGetDeviceCount(&count);
7    if (err != cudaSuccess) {
8        printf("CUDA error: %s\n", cudaGetErrorString(err));
9        return 1;
10    }
11    for (int i = 0; i < count; i++) {
12        cudaDeviceProp p;
13        cudaGetDeviceProperties(&p, i);
14        printf("GPU %d: %s, %zu MiB\n", i, p.name, p.totalGlobalMem >> 20);
15    }
16    return 0;
17}

It's compiled in the devel image and copied into the much smaller base image. Put both files in one folder:

dockerfile
1FROM nvidia/cuda:13.4.3-devel-ubuntu24.04 AS build
2WORKDIR /src
3COPY gpu-check.cu .
4RUN nvcc -O2 -o gpu-check gpu-check.cu
5
6FROM nvidia/cuda:13.4.3-base-ubuntu24.04
7COPY --from=build /src/gpu-check /usr/local/bin/gpu-check
8ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
9CMD ["gpu-check"]

NVIDIA_DRIVER_CAPABILITIES controls which driver libraries the toolkit mounts. The default is compute,utility, which covers CUDA and nvidia-smi. Video workloads that decode or encode on the GPU, like our camera streams, also need video.

Build and run it:

shell
1docker build -t gpu-check .
2docker run --rm --gpus all gpu-check

On a GPU host it prints one line per GPU with its name and memory.

Step 6. Request a GPU in Docker Compose

yaml
1services:
2  analyzer:
3    build: .
4    restart: unless-stopped
5    deploy:
6      resources:
7        reservations:
8          devices:
9            - driver: nvidia
10              count: 1
11              capabilities: [gpu]

Use count: all for every GPU, or device_ids: ["0", "2"] for specific ones. count and device_ids can't be used together, and capabilities is required. Details are in the Compose GPU docs.

Common errors

• could not select device driver "nvidia" with capabilities: [[gpu]] or unknown or invalid runtime name: nvidia: the toolkit isn't installed, or Docker wasn't restarted after Step 3.
• The container exits right away with a CUDA version requirement error: the image's CUDA version is newer than the host driver supports. Use an older image tag or update the driver.
• Failed to initialize NVML: Unknown Error in a long-running container: the container lost GPU access after a systemd reload. Recreate the container, and see NVIDIA's troubleshooting guide for permanent fixes.
• Kubernetes: don't configure each node by hand. The NVIDIA GPU Operator installs the driver, the toolkit and the device plugin on GPU nodes. The toolkit also generates a CDI spec, the vendor-neutral way container runtimes describe GPUs.

Need help with GPU infrastructure?

GPU servers are expensive, and a misconfigured one sits idle. If you need help with GPU infrastructure like this, from Docker hosts to Kubernetes clusters, just ping us and let us take care of the infrastructure side.

Focus on your business and let the professionals do their job!