106 lines
3.5 KiB
Markdown
106 lines
3.5 KiB
Markdown
# HA Recorder Silent Connection Drop
|
|
|
|
## Symptom
|
|
|
|
HA shows no historical data (History/Logbook empty or stale), but HA itself
|
|
runs normally — entities update, automations fire, no error logs.
|
|
|
|
## Root Cause
|
|
|
|
HA Recorder loses its MySQL/Galera DB connection silently. Unlike the
|
|
exhausted-retries-at-boot case, no `ERROR (Recorder)` entries appear in
|
|
the HA log. The recorder thread is alive but disconnected — it never
|
|
reconnects on its own.
|
|
|
|
Observed triggers:
|
|
- MaxScale `FLUSH HOSTS` during Galera maintenance
|
|
- Galera node restart (brief connection interruption)
|
|
- MaxScale VM failover (VIP moves to standby)
|
|
|
|
## Diagnosis Procedure
|
|
|
|
### 1. Check HA logs for recorder errors
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10 \
|
|
"docker logs homeassistant 2>&1 | grep -i recorder | tail -20"
|
|
```
|
|
|
|
If EMPTY — this is the silent-drop pattern (not the exhausted-retries case,
|
|
which floods the log with `ERROR (Recorder)` messages).
|
|
|
|
### 2. Check MaxScale for active HA sessions
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.82 "maxctrl list sessions"
|
|
```
|
|
|
|
If `ha_recorder` is NOT listed among active sessions, the recorder
|
|
connection is dead. Only `recipe_app` or other apps may be connected.
|
|
|
|
### 3. Check `recorder_runs` table on Galera
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
|
|
"sudo mariadb homeassistant -e 'SELECT run_id, start, \`end\` FROM recorder_runs ORDER BY run_id DESC LIMIT 3\G'"
|
|
```
|
|
|
|
An open run (`end = NULL`) with an old `start` timestamp confirms the
|
|
recorder started but never closed the run (connection dropped mid-run).
|
|
|
|
### 4. Check latest state timestamp
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
|
|
"sudo mariadb homeassistant -e 'SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;'"
|
|
```
|
|
|
|
Compare with current time. If `latest_state` is hours/days behind, recorder
|
|
stopped writing at that point.
|
|
|
|
### 5. Check `states` table size (information_schema — fast, no COUNT(*))
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
|
|
"sudo mariadb homeassistant -e 'SELECT table_name, table_rows, ROUND(data_length/1024/1024) AS data_mb FROM information_schema.tables WHERE table_schema=\"homeassistant\";'"
|
|
```
|
|
|
|
**⚠️ Do NOT run `SELECT COUNT(*) FROM states`** — with 62M+ rows and 6.6 GB,
|
|
this query hangs for 30+ seconds. Use `information_schema.table_rows`
|
|
instead (instant, approximate).
|
|
|
|
## Fix
|
|
|
|
```bash
|
|
# Restart HA core — recorder reconnects and resumes writing
|
|
ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10 "ha core restart"
|
|
```
|
|
|
|
Wait 2-3 minutes for HA to fully start (large DB schema check takes time).
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
# Check latest state is now current
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 \
|
|
"sudo mariadb homeassistant -e 'SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;'"
|
|
# Should show a timestamp within the last few seconds
|
|
|
|
# Check MaxScale now has ha_recorder session
|
|
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.82 "maxctrl list sessions"
|
|
```
|
|
|
|
## Data Loss
|
|
|
|
The gap between the silent disconnect and the restart is permanently lost.
|
|
No state/event data was recorded during that period. InfluxDB (if configured
|
|
for HA) may still have the data since it's a separate integration with its
|
|
own connection.
|
|
|
|
## Session Reference
|
|
|
|
- **2026-07-08**: Recorder silently dropped after MaxScale `FLUSH HOSTS`
|
|
during Galera maintenance. Latest state was 2026-07-04 12:48 (4 days gap).
|
|
62.7M states, 6.6 GB on Galera. `ha core restart` fixed immediately.
|
|
`recorder_runs` run_id=106 open since 2026-07-01, `end=NULL`.
|