Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,105 @@
+# HA Recorder Silent Connection Drop
+
+## Symptom
+
+HA shows no historical data (History/Logbook empty or stale), but HA itself
+runs normally — entities update, automations fire, no error logs.
+
+## Root Cause
+
+HA Recorder loses its MySQL/Galera DB connection silently. Unlike the
+exhausted-retries-at-boot case, no `ERROR (Recorder)` entries appear in
+the HA log. The recorder thread is alive but disconnected — it never
+reconnects on its own.
+
+Observed triggers:
+- MaxScale `FLUSH HOSTS` during Galera maintenance
+- Galera node restart (brief connection interruption)
+- MaxScale VM failover (VIP moves to standby)
+
+## Diagnosis Procedure
+
+### 1. Check HA logs for recorder errors
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10 \
+  "docker logs homeassistant 2>&1 | grep -i recorder | tail -20"
+```
+
+If EMPTY — this is the silent-drop pattern (not the exhausted-retries case,
+which floods the log with `ERROR (Recorder)` messages).
+
+### 2. Check MaxScale for active HA sessions
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.82 "maxctrl list sessions"
+```
+
+If `ha_recorder` is NOT listed among active sessions, the recorder
+connection is dead. Only `recipe_app` or other apps may be connected.
+
+### 3. Check `recorder_runs` table on Galera
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
+  "sudo mariadb homeassistant -e 'SELECT run_id, start, \`end\` FROM recorder_runs ORDER BY run_id DESC LIMIT 3\G'"
+```
+
+An open run (`end = NULL`) with an old `start` timestamp confirms the
+recorder started but never closed the run (connection dropped mid-run).
+
+### 4. Check latest state timestamp
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
+  "sudo mariadb homeassistant -e 'SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;'"
+```
+
+Compare with current time. If `latest_state` is hours/days behind, recorder
+stopped writing at that point.
+
+### 5. Check `states` table size (information_schema — fast, no COUNT(*))
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
+  "sudo mariadb homeassistant -e 'SELECT table_name, table_rows, ROUND(data_length/1024/1024) AS data_mb FROM information_schema.tables WHERE table_schema=\"homeassistant\";'"
+```
+
+**⚠️ Do NOT run `SELECT COUNT(*) FROM states`** — with 62M+ rows and 6.6 GB,
+this query hangs for 30+ seconds. Use `information_schema.table_rows`
+instead (instant, approximate).
+
+## Fix
+
+```bash
+# Restart HA core — recorder reconnects and resumes writing
+ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10 "ha core restart"
+```
+
+Wait 2-3 minutes for HA to fully start (large DB schema check takes time).
+
+## Verification
+
+```bash
+# Check latest state is now current
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
+  "sudo mariadb homeassistant -e 'SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;'"
+# Should show a timestamp within the last few seconds
+
+# Check MaxScale now has ha_recorder session
+ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.82 "maxctrl list sessions"
+```
+
+## Data Loss
+
+The gap between the silent disconnect and the restart is permanently lost.
+No state/event data was recorded during that period. InfluxDB (if configured
+for HA) may still have the data since it's a separate integration with its
+own connection.
+
+## Session Reference
+
+- **2026-07-08**: Recorder silently dropped after MaxScale `FLUSH HOSTS`
+  during Galera maintenance. Latest state was 2026-07-04 12:48 (4 days gap).
+  62.7M states, 6.6 GB on Galera. `ha core restart` fixed immediately.
+  `recorder_runs` run_id=106 open since 2026-07-01, `end=NULL`.