Initial commit: Hermes Agent Skills collection
This commit is contained in:
@@ -0,0 +1,239 @@
|
||||
# Frigate NVR Troubleshooting Reference
|
||||
|
||||
## Infrastructure
|
||||
|
||||
| Component | Value |
|
||||
|-----------|-------|
|
||||
| CT ID | 120 |
|
||||
| Node | **n5pro (10.0.20.91)** — migrated from proxmox4 |
|
||||
| IP | 10.0.30.104 |
|
||||
| Web UI | http://10.0.30.104:5000 |
|
||||
| API | http://10.0.30.104:5000/api/ |
|
||||
| go2RTC | http://10.0.30.104:8554/ |
|
||||
| Config | /config/config.yml (inside CT) |
|
||||
| Service | frigate.service (systemd) |
|
||||
| Database | /config/frigate.db (SQLite) |
|
||||
| GPU | **AMD Radeon (gfx1150)** via /dev/dri/card0 + /dev/dri/renderD128 |
|
||||
| GPU Driver | Mesa Gallium radeonsi, VAAPI 1.22 (H264/HEVC decode+encode) |
|
||||
| onboot | yes |
|
||||
|
||||
## Version & Features
|
||||
|
||||
- Frigate 0.16.4 (also tested 0.17-0)
|
||||
- Detector: OpenVINO with yolov9s Plus model (320x320), **device: CPU** (OpenVINO is Intel-only, AMD GPU unsupported)
|
||||
- Tracked objects: person, face, license_plate, dog, cat, car, amazon, ups, package (+ extended postal labels)
|
||||
- Face recognition: enabled (large model) — **runs on CPU** (OpenVINO GPU fails, auto-fallback to ONNX Runtime CPU)
|
||||
- License plate recognition: **disabled** (was device: GPU, OpenVINO doesn't support AMD)
|
||||
- Semantic search: enabled (jinav1 large) — **runs on CPU** (same fallback)
|
||||
- Hardware accel: **none** (commented out — see GPU Migration section)
|
||||
|
||||
## Cameras
|
||||
|
||||
### Einfahrt (Driveway)
|
||||
- IP: 10.0.50.102
|
||||
- RTSP main: `rtsp://admin:<password>@10.0.50.102:554/h264Preview_01_main`
|
||||
- RTSP sub: `rtsp://admin:<password>@10.0.50.102:554/h264Preview_01_sub`
|
||||
- Detection: 640x360 @ 5fps
|
||||
- Motion threshold: 47, contour area: 19
|
||||
- LPR mask zones configured
|
||||
|
||||
### Terrasse (Patio)
|
||||
- IP: 10.0.50.103
|
||||
- Same RTSP pattern as Einfahrt
|
||||
- Motion threshold: 40, contour area: 27
|
||||
|
||||
## MQTT Configuration
|
||||
|
||||
```yaml
|
||||
mqtt:
|
||||
enabled: true
|
||||
host: 10.0.30.10
|
||||
port: 1883
|
||||
user: frigate
|
||||
password: <see /config/config.yml or 1Password>
|
||||
topic_prefix: frigate
|
||||
client_id: frigate
|
||||
stats_interval: 60
|
||||
qos: 0
|
||||
```
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Check CT status (n5pro, NOT proxmox4)
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 "pct status 120"
|
||||
|
||||
# Start if stopped
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 "pct start 120"
|
||||
|
||||
# Check Frigate API
|
||||
curl -s http://10.0.30.104:5000/api/version
|
||||
curl -s http://10.0.30.104:5000/api/config | python3 -m json.tool | head -50
|
||||
|
||||
# Check service logs
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 "pct exec 120 -- cat /dev/shm/logs/frigate/current | tail -30"
|
||||
|
||||
# Reload HA Frigate integration (does NOT require full HA restart)
|
||||
TOKEN=$(cat /tmp/ha_token.txt)
|
||||
curl -s -X POST "http://10.0.30.10:8123/api/config/config_entries/entry/01KKSHKS4HXBAW1JAKDNHMYHKZ/reload" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
## GPU Migration: Intel → AMD (2026-07-12)
|
||||
|
||||
Frigate CT 120 was migrated from proxmox4 (Intel iGPU) to n5pro (AMD Radeon gfx1150). Three issues arose:
|
||||
|
||||
### Issue 1: CT Device Mapping
|
||||
**Symptom:** `pct start 120` fails with "Device /dev/dri/card1 does not exist"
|
||||
**Cause:** CT config referenced `/dev/dri/card1` (Intel) and `renderD128` with gid=104 (Intel video group). n5pro has AMD GPU: `/dev/dri/card0` (gid=44=video) and `/dev/dri/renderD128` (gid=993=render).
|
||||
**Fix:** Update CT config device mappings:
|
||||
```
|
||||
dev0: /dev/dri/renderD128,gid=993
|
||||
dev1: /dev/dri/card0,gid=44
|
||||
```
|
||||
Check gids on the target node first: `stat -c '%g %G' /dev/dri/renderD128`
|
||||
|
||||
### Issue 2: OpenVINO EP Crash on AMD GPU
|
||||
**Symptom:** Frigate crashes during startup with:
|
||||
```
|
||||
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: RUNTIME_EXCEPTION:
|
||||
Exception during initialization: ... OpenVINO-EP ...
|
||||
Check '!m_device_map.empty()' failed ...
|
||||
[GPU] Can't get OPTIMIZATION_CAPABILITIES property as no supported devices found
|
||||
```
|
||||
**Cause:** OpenVINO Execution Provider is Intel-only. It calls `ov.compile_model(device_name="GPU")` which fails on AMD. Affects: detectors (if device: GPU), face_recognition embeddings, semantic_search embeddings, LPR.
|
||||
**Fix:** Set `device: CPU` for detectors and LPR in config.yml. Face recognition and semantic search auto-fallback to CPU — the runner code catches the OpenVINO exception and logs `"OpenVINO failed to build model, using CPU instead"` then uses ONNX Runtime CPUExecutionProvider.
|
||||
**Can face_recognition/semantic_search run on AMD GPU?** Not with stock Frigate. Would need ONNX Runtime built with ROCm/HIP Execution Provider, which is not shipped with Frigate. The OpenVINO code path is hardcoded for Intel GPU.
|
||||
|
||||
### Issue 3: FFmpeg Hardware Acceleration
|
||||
**Symptom:** `preset-intel-qsv-h264` causes ffmpeg crash. `preset-vaapi-h264` also fails with "Unable to choose an output format".
|
||||
**Cause:** Intel QSV is Intel-only. `preset-vaapi-h264` is NOT a valid Frigate FFmpeg preset name (Frigate interprets it as an output format, not a hwaccel arg).
|
||||
**Fix:** Comment out `hwaccel_args` entirely. CPU decode for 2 cameras is sufficient. AMD VAAPI could theoretically work with explicit ffmpeg args (e.g. `-hwaccel vaapi -hwaccel_device /dev/dri/renderD128`) but Frigate's preset system doesn't expose this cleanly.
|
||||
**Note:** `vainfo` on n5pro confirms AMD VAAPI works for H264/HEVC decode+encode. Installing `mesa-va-drivers` and `vainfo` packages on the Proxmox host enables it.
|
||||
|
||||
### Config Changes Applied
|
||||
|
||||
```yaml
|
||||
# BEFORE (Intel)
|
||||
ffmpeg:
|
||||
hwaccel_args: preset-intel-qsv-h264
|
||||
detectors:
|
||||
ov:
|
||||
type: openvino
|
||||
device: GPU
|
||||
lpr:
|
||||
enabled: true
|
||||
device: GPU
|
||||
|
||||
# AFTER (AMD)
|
||||
ffmpeg:
|
||||
#hwaccel_args: preset-vaapi-h264 # commented out — CPU decode
|
||||
detectors:
|
||||
ov:
|
||||
type: openvino
|
||||
device: CPU
|
||||
lpr:
|
||||
enabled: false # was device: GPU, OpenVINO doesn't support AMD
|
||||
# face_recognition and semantic_search remain enabled — auto-fallback to CPU
|
||||
```
|
||||
|
||||
## Conflicting HA Addon
|
||||
|
||||
**Critical:** There is a SECOND Frigate running as an HA addon (`ccab4aaf_frigate`, repo ccab4aaf). It was in `error` state and conflicts with the CT-based Frigate. Must be stopped:
|
||||
```bash
|
||||
# Via HA CLI through qm guest exec
|
||||
qm guest exec 106 -- sh -c 'ha addons stop ccab4aaf_frigate'
|
||||
```
|
||||
|
||||
Also stop `hm2mqtt` addon (`1376549a_hm2mqtt`) if HAME Energy entities were cleaned up — otherwise it re-discovers them via MQTT:
|
||||
```bash
|
||||
qm guest exec 106 -- sh -c 'ha addons stop 1376549a_hm2mqtt'
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### CT Stopped (Most Common)
|
||||
Symptom: All Frigate entities `unavailable` in HA, 10.0.30.104 unreachable.
|
||||
Fix: `pct start 120` on **n5pro (10.0.20.91)**, wait 60s, reload HA Frigate integration.
|
||||
|
||||
### OOM Killer (FIXED 2026-06-27)
|
||||
Symptom: Frigate process dies repeatedly, log shows "Out of memory: Killed process".
|
||||
Cause: 4GB RAM was insufficient for 2 cameras + face rec + LPR + embeddings + semantic search.
|
||||
Fix Applied: CT RAM increased 4GB → 8GB via `pct set 120 -memory 8192` (requires CT stop/start).
|
||||
Current state: 1.2GB used, 6.9GB free — no OOM risk.
|
||||
|
||||
### RTSP 461 Unsupported Transport
|
||||
Symptom: Log shows `method SETUP failed: 461 Unsupported transport`.
|
||||
Impact: Transient, doesn't prevent operation. go2rtc falls back to TCP transport.
|
||||
Action: Ignore — recordings and detection still work.
|
||||
|
||||
### Config File Confusion
|
||||
- `/opt/frigate/config/config.yml` — DEMO config (plays a test video). IGNORE.
|
||||
- `/config/config.yml` — REAL config with MQTT creds and camera RTSP URLs. THIS is loaded.
|
||||
- Frigate API `/api/config` shows the effective merged config (file + DB overrides + defaults).
|
||||
|
||||
### HA Integration Not Reconnecting
|
||||
Symptom: Frigate is running, MQTT is publishing, but HA entities stay `unavailable`.
|
||||
Cause: HA Frigate integration doesn't auto-reconnect when backend comes back.
|
||||
Fix: Reload via REST API `POST /api/config/config_entries/entry/{ENTRY_ID}/reload`. Full HA restart also works but is slower.
|
||||
|
||||
## HA Integration Details
|
||||
|
||||
- Config entry ID: `01KKSHKS4HXBAW1JAKDNHMYHKZ`
|
||||
- Domain: `frigate`
|
||||
- Title: `10.0.30.104:5000`
|
||||
- ~96 entities (cameras, switches, sensors, binary_sensors, images)
|
||||
- Entity naming: `<type>.einfahrt_<feature>_2`, `<type>.terrasse_<feature>_2`
|
||||
|
||||
## Frigate Update Path (0.16.4 → 0.17.2)
|
||||
|
||||
### Why Not In-Place Update
|
||||
|
||||
The CT 120 Frigate was installed via **community-scripts/ProxmoxVE** (`/bin/update` exists). The community script's `update` command redirects to the full CT creation script — it does **not** support in-place updates. It creates a new CT from scratch.
|
||||
|
||||
Frigate itself distributes **Docker images only** (no .deb, no pip wheel since 0.8+). The bare-metal source build at `/opt/frigate` cannot be upgraded by simply pulling new source — it requires a full rebuild with correct deps.
|
||||
|
||||
### Docker-in-LXC Dead End (Tested 2026-07-12)
|
||||
|
||||
Attempting to install Docker inside CT 120 (LXC) to use the official Docker image **fails systematically**:
|
||||
|
||||
| Failure | Cause |
|
||||
|---------|-------|
|
||||
| `dockerd: iptables not found` | iptables binary missing, symlinks broken |
|
||||
| `docker.socket: Failed to resolve group docker` | docker group not created during install |
|
||||
| `failed to create NAT chain DOCKER: iptables not found` | Even after fixing iptables, bridge driver fails |
|
||||
| dpkg database corruption | Interrupted apt-get leaves `/var/lib/dpkg/updates/` in bad state, `dpkg --configure -a` loops |
|
||||
| `modprobe overlay` fails (ExecStartPre) | LXC lacks kernel module loading capability |
|
||||
|
||||
**Conclusion:** Docker in Proxmox LXC is not viable. Frigate docs explicitly warn: *"If you choose to run Frigate via LXC the setup can be problematic."* Use a **QEMU VM** instead.
|
||||
|
||||
### Recommended Migration Path
|
||||
|
||||
1. Create a new **QEMU VM** on n5pro (not LXC CT)
|
||||
2. Install Debian 12 + Docker
|
||||
3. Use `ghcr.io/blakeblankshear/frigate:stable-rocm` (AMD GPU ROCm build)
|
||||
4. Copy `/config/config.yml` and `/config/frigate.db` from CT 120
|
||||
5. Map `/dev/dri/renderD128` for AMD GPU access
|
||||
6. Update HA Frigate integration URL to new VM IP
|
||||
7. Stop CT 120 Frigate once VM is verified
|
||||
|
||||
### Docker Image Tags Reference
|
||||
|
||||
| Tag | Use Case |
|
||||
|-----|----------|
|
||||
| `stable` | Standard amd64/arm64 (Intel iGPU, Coral) |
|
||||
| `stable-rocm` | **AMD GPU** (our n5pro case) |
|
||||
| `stable-tensorrt` | Nvidia GPU |
|
||||
| `stable-rk` | Rockchip SBCs |
|
||||
|
||||
### Current State (Post-Cleanup 2026-07-12)
|
||||
|
||||
- CT 120: Frigate 0.16.4 still active (restored after failed Docker attempt)
|
||||
- Docker packages purged from CT 120, cores reverted from 8 back to 4
|
||||
- CT 120 dpkg repaired (`dpkg --configure -a --force-all`)
|
||||
- HA addon `ccab4aaf_frigate` stopped (conflict)
|
||||
- HA addon `1376549a_hm2mqtt` stopped (HAME energy bridge)
|
||||
|
||||
## Double Take (Defunct)
|
||||
|
||||
Double Take (face recognition companion) was completely removed — no service, container, addon, or config remains. 13 orphaned entities linger in HA states but have no entity registry entries. They will disappear on HA restart. Do NOT attempt to restore Double Take unless reinstalling from scratch.
|
||||
Reference in New Issue
Block a user