Proxmox + Frigate, Part 2: GPU Object Detection (worksheet)

Sep 12

THE TUTORIAL IS MEANT TO BE USED WITH THIS VIDEO!

Proxmox VE 9.2, the Debian 13 Frigate VM from Part 1, Frigate 0.17 (tensorrt image), Ollama, and an Nvidia RTX 3060 12 GB. September 2026.

What you end up with: An Nvidia RTX video card passed through to VM 101, Frigate decoding and detecting on the card with a YOLOv9 model, semantic search turned on, and an Ollama container on the same card writing review summaries. Home Assistant needs no changes.

Before You Start

Part 1 finished and working: Proxmox, VM 100 (Home Assistant OS), VM 101 (Frigate in Docker), at least one camera showing in Frigate.
An Nvidia GeForce card with at least 8 GB of memory (the video uses an RTX 3060 12 GB). Detection and search fit in a couple of GB; the Ollama model in Steps 9 and 10 needs the rest. Most of this works the same for any Nvidia card from the GTX 10-series forward; Frigate wants driver 545 or newer and a card with compute capability 5.0 or better.
A power supply with a free 8-pin PCIe power connector. The 3060 draws up to about 170 W.
A free x16 slot. Use the top one, the slot wired to the CPU. On many boards the second x16 slot hangs off the chipset, and can't be used.
Room on VM 101's disk. The tensorrt image, the model build, and the Ollama model add roughly 15 GB. Check with df -h / in the VM; if you have less than 20 GB free, grow the disk first using the resize steps in Part 1's "If It Doesn't Work" section (64 GB is comfortable).
About 200 minutes of camera downtime while VM 101 is off and rebuilt.
Nothing plugged into the card's outputs once the install is done: no monitor, no HDMI capture box. The host never draws on the card after Step 2, and the VM does not either. A display sink on the card is one more thing that can keep it from settling.

Commands run in one of two places: the Proxmox shell (web UI, click the node, then Shell) or an SSH session to VM 101 (ssh netserv@YOUR-FRIGATE-VM-IP, from Part 1). Each step says which. Paste multi-line blocks in one go.

My Proxmox IP (from Part 1): ______________________________

My Frigate VM IP (from Part 1): ______________________________

GPU PCI address (Step 2.2, looks like 0b:00): ______________________________

GPU vendor:device IDs (Step 2.2, two of them, look like 10de:2504 and 10de:228e): ______________________________

CPU inference speed before (Frigate System metrics, from Part 1): ______________________________

Step 1 - Install the Card (if not already installed)

1.1 - Shut down both VMs from the Proxmox web UI (VM 100 and VM 101, Shutdown), then shut down the node (node, Shutdown). Unplug the power.

1.2 - Remove whatever graphics card was in the machine. Install the 3060 in the TOP x16 slot. Plug in the 8-pin power connector.

1.3 - Plug a monitor into the 3060 for this one boot and go into the BIOS. Confirm:

IOMMU (AMD-Vi, or SVM Mode plus IOMMU) is Enabled. You did this in Part 1.
Above 4G Decoding is Enabled. Usually under PCI or Advanced settings.
CSM is Disabled, if your board has it.
Resizable BAR can stay off. Nvidia cards do not need it here.

1.4 - Save and boot. Proxmox comes up normally, and the web UI at https://YOUR-PROXMOX-IP:8006 works. The monitor and keyboard come off again after this step, for good.

Step 2 - Reserve the Card on the Proxmox Host

Proxmox shell. Everything in this step is on the host, not in a VM.

2.1 - Confirm the IOMMU is on:

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

You want lines mentioning AMD-Vi (or DMAR on Intel). If there are none, the BIOS setting from 1.3 did not take.

2.2 - Find the card and write down its address and IDs:

lspci -nn | grep -i nvidia

You get two lines, something like:

0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060] [10de:2504] (rev a1)
0b:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)

The address is the part before the dot (0b:00 here). The IDs are the two bracketed vendor:device pairs (10de:2504 and 10de:228e here; yours may differ, there are several 3060 variants). Write both in the boxes at the top. The rest of this worksheet uses the example values; substitute yours.

2.3 - Check the IOMMU group. Both of the card's functions should be in a group by themselves. This lists everything that shares the card's group; use YOUR address from 2.2 and keep the 0000: prefix:

for d in /sys/bus/pci/devices/0000:0b:00.0/iommu_group/devices/*; do lspci -nns "${d##*/}"; done

The only two lines should be the card's VGA controller and its audio controller. If SATA controllers, USB, or anything else shows up, the card is in a slot that shares a group, and it has to move to the top x16 slot (see Before You Start, item 4). If the command says no such file or directory, the address is wrong or the IOMMU is off (recheck 2.1 and 2.2).

2.4 - Kernel command line. Add iommu=pt, which tells the host to skip address translation for devices it keeps for itself:

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"/' /etc/default/grub
grep CMDLINE_LINUX_DEFAULT /etc/default/grub
update-grub

The grep should print the line with iommu=pt in it. If you installed Proxmox on ZFS instead of ext4, it boots with systemd-boot: add iommu=pt to the single line in /etc/kernel/cmdline instead, then run proxmox-boot-tool refresh.

2.5 - Load vfio at boot:

cat > /etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
EOF

2.6 - Tell vfio which card to claim. Use YOUR two IDs from 2.2, comma separated, no spaces:

echo "options vfio-pci ids=10de:2504,10de:228e" > /etc/modprobe.d/vfio.conf

2.7 - Keep the host's own Nvidia drivers away from it:

cat > /etc/modprobe.d/blacklist-nvidia.conf <<'EOF'
blacklist nouveau
blacklist nvidiafb
blacklist nvidia
EOF

2.8 - Rebuild the boot image and reboot:

update-initramfs -u -k all
reboot

The host's screen will go blank or freeze at the boot logo from now on. That is correct: vfio owns the card and nothing draws on it. Manage the host from the browser, as always.

2.9 - Verify after the reboot, in the Proxmox shell:

lspci -nnk -s 0b:00

Both functions must say Kernel driver in use: vfio-pci. Ignore the "Kernel modules:" line under it that lists nvidiafb, nouveau, or snd_hda_intel; that is lspci naming drivers that could claim the device, not what did. If the "driver in use" line says nouveau or is missing, see "If It Doesn't Work."

Step 3 - Hand the Card to VM 101

Proxmox web UI.

3.1 - VM 101 is probably running again; Part 1 set it to start with the host, and the host just rebooted. VM 101 > Shutdown, and wait for the status to show stopped.

3.2 - Memory: VM 101 > Hardware > Memory > Edit. Memory: 16384. Passthrough pins the VM's memory for the card's DMA, so this RAM is fully reserved from the moment the VM boots, whatever the ballooning setting says. (The box has 32 GB; HAOS uses 4.)

3.3 - Processors (optional): 6 cores instead of 4 if you have them to spare. The 2700 has 8.

3.4 - Confirm the machine type: VM 101 > Options > Machine should say q35, and BIOS should say OVMF (UEFI). The Part 1 script set both. PCI Express passthrough needs q35.

3.5 - Add the card: VM 101 > Hardware > Add > PCI Device. Choose Raw Device and pick the line for the 3060's VGA function (0000:0b:00.0 in the example). Check PCI-Express. Leave All Functions UNCHECKED: you are handing the VM the graphics function only. Leave Primary GPU UNCHECKED; that is for VMs that need the card as a display, and this one has no display. Leave ROM-Bar checked. Add.

Why not All Functions: the card's HDMI audio function has no function-level reset of its own, so when QEMU opens it the kernel resets the whole slot, which yanks the GPU out from under QEMU mid-setup. On some boards (the video's X470 among them) the GPU never comes back from that and the VM fails with "invalid PCI interrupt pin 255." Frigate has no use for HDMI audio. The audio function stays parked on vfio-pci on the host, which satisfies the IOMMU group rule without giving it to the VM.

Confirm from the Proxmox shell; the line should end in .0:

qm config 101 | grep hostpci

3.6 - Start VM 101 (from the Proxmox UI; if it was running while you made these changes, Shutdown first, wait for stopped, then Start. A reboot from inside the VM does not apply hardware changes) and SSH in from your computer:

ssh netserv@YOUR-FRIGATE-VM-IP
lspci | grep -i nvidia

You should see one Nvidia device inside the VM, the GA104 VGA controller, usually at 01:00.0. The VM cannot use the card yet, but it can see it.

STOP HERE and take a breath. Passthrough is done. Everything from here on is ordinary Linux inside the VM.

Step 4 - Install the Nvidia Driver in the VM

SSH session to VM 101.

4.1 - Turn on Debian's non-free package sections. The Nvidia driver lives there:

sudo sed -i 's/^Components: .*/Components: main contrib non-free non-free-firmware/' /etc/apt/sources.list.d/debian.sources
sudo apt update

If that file does not exist on your image, edit /etc/apt/sources.list with nano and add contrib non-free non-free-firmware to the end of each deb line, then apt update.

4.2 - Kernel headers and the detector tool:

sudo apt install -y linux-headers-$(uname -r | sed 's/[^-]*-[^-]*-//') nvidia-detect
nvidia-detect

nvidia-detect should name the card and recommend the nvidia-driver package.

4.3 - Install the driver. This builds a kernel module and takes a few minutes:

sudo apt install -y nvidia-driver firmware-misc-nonfree
sudo reboot

4.4 - Verify after the reboot:

ssh netserv@YOUR-FRIGATE-VM-IP
nvidia-smi

You get a table: GeForce RTX 3060, a driver version, and 12288MiB of memory with almost nothing used. Frigate needs the driver version to be 545 or newer; Debian 13's is.

Driver version: ______________________________

Step 5 - Install the NVIDIA Container Toolkit

SSH session to VM 101. This is what lets Docker containers see the card.

5.1 - Add Nvidia's package repository:

sudo apt install -y curl gnupg2 ca-certificates
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

5.2 - Install it and register it with Docker:

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

5.3 - Test from inside a throwaway container:

docker run --rm --gpus all ubuntu nvidia-smi

If the same nvidia-smi table prints, containers can reach the card. That was the last piece of plumbing.

Step 6 - Build the YOLOv9 Model

SSH session to VM 101. Frigate does not ship this model; this one command downloads the weights and converts them to an ONNX file. It pulls a couple of GB and takes a few minutes. Do it in an empty folder so Docker does not upload your whole Frigate folder as build context.

6.1 - Build. Paste the whole block at the prompt in one go:

mkdir -p ~/yolo-export && cd ~/yolo-export
docker build . --build-arg MODEL_SIZE=s --build-arg IMG_SIZE=320 --output . -f- <<'EOF'
FROM python:3.11 AS build
RUN apt-get update && apt-get install --no-install-recommends -y cmake libgl1 && rm -rf /var/lib/apt/lists/*
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
WORKDIR /yolov9
ADD https://github.com/WongKinYiu/yolov9.git .
RUN uv pip install --system -r requirements.txt
RUN uv pip install --system onnx==1.18.0 onnxruntime onnx-simplifier==0.4.* onnxscript
ARG MODEL_SIZE
ARG IMG_SIZE
ADD https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-${MODEL_SIZE}-converted.pt yolov9-${MODEL_SIZE}.pt
RUN sed -i "s/ckpt = torch.load(attempt_download(w), map_location='cpu')/ckpt = torch.load(attempt_download(w), map_location='cpu', weights_only=False)/g" models/experimental.py
RUN python3 export.py --weights ./yolov9-${MODEL_SIZE}.pt --imgsz ${IMG_SIZE} --simplify --include onnx
FROM scratch
ARG MODEL_SIZE
ARG IMG_SIZE
COPY --from=build /yolov9/yolov9-${MODEL_SIZE}.onnx /yolov9-${MODEL_SIZE}-${IMG_SIZE}.onnx
EOF
ls -lh ~/yolo-export

You should see yolov9-s-320.onnx, a few tens of MB.

6.2 - Move it into Frigate's model cache and clean up the build. The model_cache folder already exists and belongs to root (Frigate made it on an earlier start), hence the sudo:

sudo mkdir -p ~/frigate/config/model_cache
sudo mv ~/yolo-export/yolov9-s-320.onnx ~/frigate/config/model_cache/
ls -l ~/frigate/config/model_cache/
docker builder prune -f

Sizes t, s, m, c, and e exist, and IMG_SIZE can be 640. A 3060 handles s or m at 320 with ease. Whatever you export, the width and height in Step 7.2 must match IMG_SIZE, and the path must match the file name.

Step 7 - Switch Frigate to the GPU

SSH session to VM 101.

7.1 - Replace the compose file. Two changes from Part 1: the image tag ends in -tensorrt, and a deploy block gives the container the GPU. Paste the whole block:

cat > ~/frigate/docker-compose.yml <<'EOF'
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    stop_grace_period: 30s
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    shm_size: "512mb"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - /media/frigate:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000"
      - "8554:8554"
      - "8555:8555/tcp"
      - "8555:8555/udp"
EOF
cd ~/frigate && docker compose config --quiet && echo OK

7.2 - Edit the config. Open it:

nano ~/frigate/config/config.yml

Delete the whole detectors block from Part 1 (the one with cpu1 in it) and put this in its place. Then add the model and ffmpeg blocks right below it, at the top level (no indentation on the first line of each block). Leave everything else, including your cameras, alone.

detectors:
  onnx:
    type: onnx

model:
  model_type: yolo-generic
  width: 320
  height: 320
  input_tensor: nchw
  input_dtype: float
  path: /config/model_cache/yolov9-s-320.onnx
  labelmap_path: /labelmap/coco-80.txt

ffmpeg:
  hwaccel_args: preset-nvidia

Save with Control-X, then Y, then Enter.

What these do: detectors says use the ONNX detector, which picks the Nvidia backend on its own in the tensorrt image. model tells Frigate which file, what input size, and which labelmap turns the model's numbers into names. ffmpeg moves video decoding for every camera onto the card's decoder.

7.3 - Pull and start. The tensorrt image is several GB; the first pull takes a while:

cd ~/frigate
docker compose pull
docker compose up -d
docker logs -f frigate

On the first start, Frigate compiles the ONNX model into a TensorRT engine for this exact GPU. The log will sit on that for anywhere from two to ten minutes and look stuck. It is not. It only happens once; the result is cached in config/model_cache. Wait for the line that says the detector started, then Control-C.

7.4 - Look at the number. Open https://YOUR-FRIGATE-VM-IP:8971, then System metrics. Inference speed should be single digits to low teens in milliseconds. A GPU section now appears on the same page with the card's load and memory. Compare the CPU graph to what it was.

GPU inference speed after: ______________________________

7.5 - Check the card from the VM while a camera has motion:

nvidia-smi

The frigate process shows up in the table with a couple of GB of memory in use.

STOP HERE if all you wanted was GPU detection. Everything below is optional, and it all rides on the same card.

Step 8 - Semantic Search

SSH session to VM 101. No new software; Frigate downloads the search model itself on the first start.

8.1 - Add this block to config.yml at the top level (nano, as in 7.2):

semantic_search:
  enabled: true
  reindex: false
  model: jinav1
  model_size: large

large is the version that runs on the GPU. small is a cut-down CPU version for people without a card. jinav1 is English-only and lighter; jinav2 handles other languages and costs a lot more memory.

8.2 - Restart and watch the download:

docker compose restart frigate
docker logs -f frigate

The first start downloads the Jina CLIP model (a few hundred MB). Control-C when the log settles down.

8.3 - Try it. In the Frigate UI, open Explore. Type something into the search box that describes an event you know happened: "person carrying a box," "red car," "dog in the yard." Every event from this point forward is indexed automatically.

Events from before you turned this on are not indexed. To index them once, set reindex: true, restart, wait for the log to finish (it works through every stored event), then set it back to false and restart again, or it will do it on every start.

Step 9 - Ollama on the Same Card

SSH session to VM 101. Ollama goes in the same compose file as Frigate so it shares the VM's driver and the two containers can find each other by name.

9.1 - Replace the compose file again, this time with both services:

cat > ~/frigate/docker-compose.yml <<'EOF'
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    stop_grace_period: 30s
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    shm_size: "512mb"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - /media/frigate:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000"
      - "8554:8554"
      - "8555:8555/tcp"
      - "8555:8555/udp"

  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    environment:
      OLLAMA_KEEP_ALIVE: "-1"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - ./ollama:/root/.ollama
    ports:
      - "11434:11434"
EOF
cd ~/frigate && docker compose config --quiet && echo OK
docker compose up -d

OLLAMA_KEEP_ALIVE=-1 keeps the model loaded on the card instead of unloading it after five idle minutes. Port 11434 is exposed on the VM so Home Assistant can use this same Ollama later.

9.2 - Pull the vision model. About 3.4 GB:

docker exec -it ollama ollama pull qwen3.5:4b

Qwen 3.5 is multimodal at every size (it takes images as well as text) and is the newer generation of the qwen3-vl model that Frigate's docs use as their example. Same size on the card.

9.3 - Test it. The --think=false part matters; without it the model reasons out loud for a while before answering:

docker exec -it ollama ollama run qwen3.5:4b --think=false "Describe what a home security camera is for, in one sentence."

An answer comes back in a few seconds. Then check the card:

nvidia-smi

Two processes now: frigate and ollama, and still several GB free. That is the reason for the 12 GB card.

Step 10 - Review Summaries

SSH session to VM 101 for the config, then the Frigate UI.

10.1 - Add these two blocks to config.yml at the top level (nano, as in 7.2). The genai block says which model to talk to and how; the review block turns the summaries on.

genai:
  provider: ollama
  base_url: http://ollama:11434
  model: qwen3.5:4b
  provider_options:
    think: false
    keep_alive: -1
    options:
      num_ctx: 8192

review:
  genai:
    enabled: true
    activity_context_prompt: |
      This is a single-family home. Family members and their dogs are often in the yard and driveway at any hour. Delivery drivers come to the front
      door during the day, leave a package, and go. Nobody should be near the shed,
      the vehicles, or the windows at night. Anyone trying doors, looking into
      windows, or lingering near vehicles is suspicious.
    additional_concerns:
      - animals other than dogs

base_url uses the container name ollama, which works because both services are in the same compose file. Under provider_options: think: false stops Qwen 3.5 from reasoning out loud before it answers (Frigate 0.17 does not do this for you; newer releases may). keep_alive: -1 repeats the setting from the compose file, so the model stays on the card either way. num_ctx: 8192 is the context window; Frigate reads this number to decide how many frames it can send per summary, and if it is missing it assumes 4096, which is a tight fit for twenty images. The activity_context_prompt is yours to rewrite; it is how the model decides what counts as normal at your house. additional_concerns is a list of extra things you want flagged. Those last two are optional; delete them to use Frigate's defaults.

10.2 - Restart:

docker compose restart frigate
docker logs -f frigate

Watch for any line mentioning genai or ollama that looks like an error. Control-C when it settles.

10.3 - Make an event. Walk in front of a camera, do something for fifteen seconds, and walk away. Wait about a minute after you are out of frame; the summary is generated when the review item ends. Then in the Frigate UI, open Review and click the item. The title, the short summary, and the full scene description are there. If you turned on Frigate's notifications in Settings, the title and short summary are what arrives on the phone.

STOP HERE. Everything is on the card. Go read what it said about you.

Daily Use

Frigate: https://FRIGATE-VM-IP:8971. System metrics shows inference speed and the GPU section. Explore is the search page. Review is where summaries live.
Watch the card from the VM: nvidia-smi, or watch -n 1 nvidia-smi to see it live.
Ollama from anywhere on the LAN: http://FRIGATE-VM-IP:11434 (Home Assistant's Ollama integration can point here).
While a summary is being written, the detector shares the card and inference speed ticks up for a couple of seconds. Normal.

If It Doesn't Work

The Proxmox host boots to a blank screen or freezes at the logo: expected after Step 2. Use the web UI. If the web UI is also gone, the boot image did not build; plug the monitor in, pick the previous kernel from the boot menu, and re-run 2.8.
2.9 still says nouveau (or shows no driver) instead of vfio-pci: the file names in 2.5 to 2.7 must end in .conf. Check them, re-run update-initramfs -u -k all, reboot.
VM 101 starts, but lspci | grep -i nvidia inside it prints nothing: the card is not attached to the VM. Usually the PCI device was added while the VM was running, which Proxmox holds as a pending change (shown in orange on the Hardware tab), and a reboot from inside the guest does not apply it. On the Proxmox shell, qm config 101 | grep hostpci should print a hostpci0 line, and qm pending 101 shows anything still waiting. Then VM 101 > Shutdown (not Reboot), wait for stopped, then Start. If qm config shows no hostpci line at all, redo 3.5.
VM 101 will not start with invalid PCI interrupt pin 255, or it starts but the Nvidia device is missing from lspci inside the VM: the card stopped answering on the PCI bus. On the host, dmesg | grep -i -e vfio -e "0000:0b:00" tells the story. A run of "resetting" lines on both functions followed by "timed out waiting for pending transaction," "restoring BARs" over and over, or "Invalid PCI ROM header signature" means the card got wedged by the reset that happens when the audio function is opened. Check qm config 101 | grep hostpci: if the address has no .0 on the end, All Functions was checked. Fix it with qm set 101 --hostpci0 0000:0b:00.0,pcie=1 (your address). Then the card needs a real power cycle to recover, not a reboot: qm set 101 --onboot 0, then poweroff, switch the power supply off for 30 seconds, boot, confirm lspci -nnk -s 0b:00 still shows the card's details, then qm start 101. Once it works, qm set 101 --onboot 1.
Same symptoms, but hostpci0 already ends in .0: last resort is to stop vfio from resetting the card at all when the VM opens it. After a cold boot, before starting the VM, run echo "" > /sys/bus/pci/devices/0000:0b:00.0/reset_method (your address), then start the VM. If that is what fixes it, the setting has to be applied at every boot before the VM starts; a small systemd unit or Proxmox hookscript does it, and it is beyond this worksheet.
power_state for the card reads D3cold, or dmesg says "Unable to change power state from D3cold to D0, device inaccessible": that is the kernel describing a card that has stopped answering, not the cause. Treat it as the wedged-card case above.
VM 101 will not start and the error mentions vfio, "device is in use," or "failed to mmap": most often the host grabbed the card's framebuffer at boot because it is the only GPU. Add initcall_blacklist=sysfb_init to the kernel line from 2.4 (inside the quotes, after iommu=pt), run update-grub, reboot the host.
VM 101 will not start and the error says the IOMMU group contains other devices: the card is in a slot that shares a group. Move it to the top slot. (There is a kernel option that forces groups apart, pcie_acs_override; it weakens the isolation the IOMMU provides, and it is not covered here.)
VM 101 will not start with a memory error: passthrough pins the VM's RAM, and the host does not have 16 GB free. Lower the number in 3.2, or check what else is running.
nvidia-smi in the VM says "No devices were found" or "couldn't communicate with the NVIDIA driver": the module is not loaded. Run dkms status; it should list an nvidia module as installed for your kernel. If it lists nothing, the headers in 4.2 did not match the running kernel: uname -r versus dpkg -l | grep linux-headers, install the matching one, then sudo apt install --reinstall nvidia-kernel-dkms. If dkms is fine, check mokutil --sb-state; if Secure Boot is enabled, the unsigned module is refused. Turn it off in the VM's firmware: reboot VM 101, press Esc on the boot logo in the Proxmox console, Device Manager, Secure Boot Configuration, uncheck Attempt Secure Boot, save, reboot.
5.3 fails with "could not select device driver nvidia": the nvidia-ctk runtime configure step or the Docker restart was skipped. Re-run 5.2.
6.1 fails with an unknown --output flag or a message about buildx: run docker buildx version. If it is missing, sudo apt install -y docker-buildx-plugin and re-run 6.1.
Frigate log says not enough values to unpack (expected 4, got 3): the ONNX file was not exported by the build in 6.1 (a model downloaded from somewhere else, or the wrong export flags). Redo Step 6.
Frigate log complains about input shape or dimensions: width and height in 7.2 do not match the IMG_SIZE the model was built with.
Frigate log stays on the TensorRT engine build for more than ten minutes: run nvidia-smi in another SSH window. If frigate is listed and memory is climbing, it is working; give it time. If frigate is not listed, the container cannot see the card; re-check 5.3 and the deploy block in 7.1.
Frigate log says the CUDA driver version is insufficient: the driver is older than 545. Check the version line in nvidia-smi.
docker compose pull or the model build fails with no space left on device: df -h /. Grow the disk (Part 1's resize steps), then docker system prune -f to clear leftover build layers.
Semantic search finds nothing: the events happened before it was turned on. Use the reindex procedure in 8.3. Also check docker logs frigate for the model download; it needs internet on the first start.
Review summaries never appear: docker logs frigate 2>&1 | grep -i -e genai -e ollama. Confirm the model is there with docker exec ollama ollama list, and that Frigate can reach Ollama by name: docker exec frigate curl -s http://ollama:11434/api/tags should print JSON. If it does not, the two services are not in the same compose file.
Ollama is slow and docker exec ollama ollama ps shows the model running on CPU: it did not fit on the card next to Frigate. Use the 4b model, not a bigger one, and make sure nothing else is holding memory in nvidia-smi. On an 8 GB card, the 4b model is the ceiling; a 12 GB card has room for the 9b.
Summaries take a minute or more each, or the Frigate log shows Ollama returning huge token counts: the model is thinking before it answers. Check that provider_options in 10.1 has think: false and that it is indented under genai, not at the top level.
Frigate log shows an Ollama error mentioning thinking or "does not support thinking": the model you configured is a plain instruct model (qwen3-vl, for example) and does not accept the think setting. Delete the think line for that model.
You would rather use the model from Frigate's docs: pull qwen3-vl:4b, set model: qwen3-vl:4b, and delete the think line. Same size, same config otherwise.
Summaries are wrong or silly: they sometimes are. A 4B model on a handful of frames is a caption, not a verdict. Rewrite the activity_context_prompt to describe your house better; it helps more than you would expect.

Maintenance

Frigate updates: same as Part 1, but keep the -tensorrt tag. After a major Frigate update, the TensorRT engine may rebuild on the first start; expect the slow first start again.

cd ~/frigate && docker compose pull && docker compose up -d

Ollama updates: the same command updates the container. To update the model itself: docker exec -it ollama ollama pull qwen3.5:4b.
Debian updates in the VM: sudo apt update && sudo apt full-upgrade -y. A new kernel triggers a DKMS rebuild of the Nvidia module; reboot afterward and confirm with nvidia-smi.
Proxmox host updates: the passthrough config survives kernel updates. After a host reboot, confirm 2.9 still shows vfio-pci before starting VM 101.
Backups: Proxmox backups of VM 101 include the PCI device line. If you ever restore that backup onto a machine without the card, remove the PCI device from Hardware before starting it.
Exclude the Ollama model folder from backups if you like; it is a download, not data: it lives at ~/frigate/ollama inside the VM.