MIT · docker-vram

Open source runtime

Repo: github.com/santiagolertora/docker-vram. This is the plug. It does not ship a daemon. Without one the container still starts; vram.limit is recorded and not enforced.

What you get

vram-runtime wraps runc. You pass --runtime=vram and the labels Docker already knows how to attach. On Linux the runtime also bind-mounts the host NVIDIA nodes, so the guest sees the GPU without nvidia-container-toolkit.

Without a grant, tenant A fills the GPU and tenant B is CUDA OOM. — This clone, no daemon: both tenants can fight for the same card. That is the default NVIDIA story.

The grant (local OOM, neighbor stays) is the enterprise page.

Install (Linux)

Need: Rust (cargo), Docker, an NVIDIA GPU.

git clone https://github.com/santiagolertora/docker-vram.git
cd docker-vram
cargo build --release -p vram-cli -p vram-runtime
./scripts/install.sh

install.sh copies vram and vram-runtime only. Path is /usr/local/bin if you can write it, otherwise ~/.local/bin. Put that on PATH.

Once, add the runtime in /etc/docker/daemon.json and restart Docker:

{
  "runtimes": {
    "vram": { "path": "/usr/local/bin/vram-runtime" }
  }
}

sudo systemctl restart docker
docker run --runtime=vram -l vram.limit=4g ollama/ollama

The guest can see the GPU. The 4g is not a grant until something answers on http://127.0.0.1:8765.

Labels

Label	Meaning
`vram.limit`	Ceiling. `4g`, `512m`.
`vram.guarantee`	Reserved, if a daemon is there.
`vram.priority`	`low` / `normal` / `high` / `critical`

Same names as flags on vram run.

vram init

Writes ./docker-compose.yml in the current directory. One Ollama service, runtime: vram, labels already filled. If that file exists, it prints a warning and does not touch it.

vram init
docker compose up -d
docker compose logs -f llama

What it writes:

services:
  llama:
    image: ollama/ollama
    runtime: vram
    ports:
      - "11434:11434"
    labels:
      vram.limit: "8g"
      vram.guarantee: "4g"
      vram.priority: high

On Linux, after install.sh and daemon.json, that compose starts. The 8g is not a grant until a getvram daemon is on :8765. Edit the file if you want another image or another limit. vram init will not overwrite your edits.

CLI

vram --help lists every command. vram <command> --help is the rest.

Command	What it does
`vram init`	Write docker-compose.yml (Ollama + runtime: vram).
`vram run --limit=1g -- …`	Native process. Apple Silicon: this, not Docker Desktop.
`vram status`	GPU + grants. Needs a daemon.
`vram ps`	Registered containers. Needs a daemon.
`vram top`	Live TUI. Needs a daemon.
`vram inspect <id>`	One grant. Needs a daemon.
`vram set <id> --limit=4g`	Change a limit. Needs a daemon.

VRAM_DAEMON_URL overrides the daemon URL (default http://127.0.0.1:8765). If nothing is listening, status / ps / top / inspect / set print that and exit 0.

Two Ollama, one GPU

Script ships in the MIT repo: examples/linux-two-ollama.sh. B is 4g, A is 512m, both try tinyllama.

	This clone (no daemon)	With a getvram daemon
B 4g	Serves. Can die when A eats the card.	Serves. Still serves after A dies.
A 512m	Label is recorded. Load can take the card.	Local OOM. B keeps answering.

Run the script after install. That is the without. The with is enterprise.

Apple Silicon

M1 / M2 / M3 / M4. Unified memory, Metal, not CUDA. Docker Desktop is a Linux VM. --runtime=vram lives in that VM. Host Metal cannot see those containers.

cargo build --release -p vram-cli
./scripts/install.sh
vram run --limit=1g -- ollama serve

Without a daemon the process still starts. The Metal grant (libvram.dylib) is commercial, same as Linux libvram.so. There is no cascade OOM onto a neighbor the way a discrete NVIDIA card does. On a Mac the product is the quota.