---
name: smart-home-automation
description: "Class-level skill for smart home troubleshooting and configuration, covering Home Assistant debugging (recorder/history, integration config editing, WebSocket API), NUT/UPS integration, and EVCC solar charging YAML management. Provides unified workflow, reference docs, and template files for common smart-home tasks."
version: 1.0.0
tags: [home-assistant, evcc, solar, automation, troubleshooting]
---

## Overview

This umbrella skill consolidates:
- `home-assistant-troubleshooting`
- `evcc-solar-charging`

It provides a unified troubleshooting guide, YAML templates for EVCC, and reference material for common Home Assistant issues.

## Connection Details

- **External URL**: `homeassistant.familie-schoen.com`
- **Internal IP**: `10.0.30.10:8123`
- **API token**: Stored in 1Password (Vault "Hermes", item "Home Assistant API Token") and `~/.hermes/.env` as `HA_TOKEN`/`HA_URL`. If missing, ask user.
- **HA VM**: VM 106 on proxmox1 (10.0.20.10). Access via Proxmox guest agent: `qm guest exec 106 -- bash -c '...'` (SSH as root to 10.0.20.10 with `~/.ssh/id_ed25519_proxmox`)
- **Config path** (inside VM): `/mnt/data/supervisor/homeassistant/configuration.yaml`
- **Database**: MariaDB Galera cluster via MaxScale VIP `10.0.30.70:3306` (NOT SQLite). See `references/galera-cluster-recovery.md`.
- **API auth header**: `Authorization: Bearer <token>`
- **Browser fallback (no token needed)**: Log in to the HA web UI via `browser_navigate` → extract token from `window.hassConnection` (Promise) → query REST API via `browser_console` `fetch()`. See `references/ha-browser-console-access.md` for details.
- **SSH addon access (alternative to Proxmox guest agent)**: When guest agent is down, use the Advanced SSH & Web Terminal HAOS add-on. Connect with `sshpass -p '<password>' ssh hassio@10.0.30.10`. Provides `sudo docker exec homeassistant python3 -c "..."` for file/API access inside the HA container. See "HAOS Advanced SSH Addon Access" section below for details.

## Pitfalls

- **Verify entity IDs before saving dashboard configs.** Scene names can differ from expectations (e.g. `scene.alles_aus_2` not `scene.alles_aus`), binary sensors may lack `device_class` attributes (filter by name keywords like `tur`/`fenster` instead), and some sensors may be `unavailable`. Before writing a dashboard config with entity references, query `ha.hass.states` for each entity ID to confirm it exists and has a valid state. A dashboard with dead entity references renders "Entity not available" cards that are hard to debug after the fact.
- **Load this skill BEFORE scanning the network.** The internal IP is known (10.0.30.10); don't waste time scanning 254 addresses per subnet. Go straight to the API or Proxmox guest agent.
- **Don't grep large cache files blindly.** `~/.hermes/models_dev_cache.json` is ~7MB. Always scope greps with `--include` filters and exclude `sessions/`, `node_modules/`.
- **Check if the `homeassistant` Hermes toolset is enabled.** Default config only has `hermes-cli`. If HA tools are needed, enable via `hermes tools enable homeassistant`.
- **QEMU guest agent on HA VM 106 is frequently down.** `qm guest exec 106` often returns "QEMU guest agent is not running". Don't rely on it for file access. Use HA REST/WebSocket APIs instead, or wait for agent to recover (sometimes doesn't until HA restart). For `.storage/` file access when guest agent is down, there is no reliable workaround — schedule a maintenance window.
- **WebSocket `config_entry/reload` is NOT available** — returns `unknown_command`. However, the **REST API equivalent DOES work**: `POST /api/config/config_entries/entry/{ENTRY_ID}/reload` returns `{"require_restart":false}` and reloads the integration without a full HA restart. Use this instead of `homeassistant/restart` whenever possible. For Frigate specifically, a full HA restart may still be needed if the integration needs to re-establish the MQTT/WebSocket connection from scratch.
- **Supervisor API `/addons/{slug}` often fails** with "Identifier values have to increase" or "unknown_error". The `/addons` list endpoint works but individual addon detail queries are unreliable via WebSocket. Work around by accessing addon data directly on the HA VM filesystem (when guest agent is up) or via the HA UI.
- **HA startup with 18GB DB takes 2-3 minutes.** After restart, `api/config` returns `NOT_RUNNING` for a while. Poll every 10s up to 3 min before assuming failure.
- **Context compaction redacts tokens in `.env` files.** After a context window compaction, the `HA_TOKEN` in `~/.hermes/.env` may be replaced with the redacted form (`eyJhbG...tfCE`, 13 chars instead of 183). Always verify token length before use. If corrupted, retrieve from 1Password: `op item get <ITEM_ID> --vault Hermes --field credential --reveal > /tmp/ha_token.txt` (note: use item ID, not title — duplicates may exist). Then restore: `sed -i "s|^HA_TOKEN=.*|HA_TOKEN=$(cat /tmp/ha_token.txt)|" ~/.hermes/.env`.
- **Supervisor API addon uninstall works** (even though addon restart/detail often fail). Via WebSocket: `{'type': 'supervisor/api', 'endpoint': '/addons/{slug}/uninstall', 'method': 'post'}`. Confirmed working for `3b9a05b8_hame_relay`. Find addon slugs with `docker ps --format '{{.Names}} {{.Status}}'` on the HA VM.
- **Recorder exhausts retries if DB is down at boot.** HA Recorder has `db_max_retries` (default 10, set to 20 here). If DB isn't reachable when HA starts, Recorder gives up and History/Logbook stay unloaded. Fix the DB FIRST, then restart HA.
- **Recorder silently drops DB connection without logging errors.** HA Recorder can lose its MySQL/Galera connection (e.g. after MaxScale `FLUSH HOSTS`, Galera restart, or network blip) WITHOUT logging any recorder errors. HA continues running normally — entities update, automations fire — but NO state data is written to the DB. History/Logbook show no recent data. **Diagnosis:** (1) Check `recorder_runs` table for an open run (`end IS NULL`) with old `start` timestamp. (2) Check `MAX(last_updated_ts)` in `states` table — if it's hours/days behind current time, recorder is silently disconnected. (3) No `ERROR (Recorder)` entries in HA logs — absence of errors does NOT mean recorder is working. **Fix:** `ha core restart` — recorder reconnects and resumes writing immediately. Data gap between disconnect and restart is permanently lost. See `references/ha-recorder-silent-drop.md` for the full diagnosis procedure.
- **Galera SST deadlock with parallel joins.** When recovering a 3-node Galera cluster, join nodes ONE AT A TIME. Starting 2+ joiners simultaneously causes "No donor candidates temporarily available in suitable state" deadlock.
- **MaxScale won't route to Donor/Desynced nodes by default.** During SST, the donor node is marked Donor/Desynced and MaxScale excludes it. Set `maxctrl alter monitor galera-monitor available_when_donor=true` to allow routing to the donor. Reset to `false` after all nodes are synced.
- **mDNS/Zeroconf discovery doesn't cross VLAN boundaries.** Integrations discovered via `zeroconf` (Shelly, LIFX, TRADFRI, Apple TV, etc.) will enter `setup_retry` if the device is on a different VLAN/subnet than HA. Check `ip route show` inside the HA VM early in diagnosis. See "VLAN/Network Segmentation Diagnosis" section above.
- **Not all unavailable entities indicate problems.** Some integrations (Dreame vacuum, possibly others) create entities for all possible device features and leave unsupported ones `unavailable`. If the main device entity works, unavailable `select`/`button`/`number`/`time` sub-entities are normal. See "Feature-Not-Supported" section above.
- **MQTT discovery entities have NO `mqtt.` prefix or `integration=mqtt` attribute.** Searching `/api/states` for entities starting with `mqtt.` or having `integration: mqtt` returns ZERO even when MQTT works perfectly. MQTT-discovered entities are named by their discovery topic (e.g. `sensor.sn_3006252807_*`, `sensor.opendtu_4676e0_*`, `binary_sensor.frigate_*`). To verify MQTT is working: (1) subscribe to `homeassistant/#` via `mosquitto_sub` inside the Mosquitto add-on container, (2) check entity `last_changed` timestamps for recency. See "MQTT Broker Diagnosis" section below.
- **HA REST API config entry metadata omits the `data` field.** `/api/config/config_entries/entry` returns title, domain, state — but NOT the actual config data (broker IP, credentials, host, port). The WebSocket `config_entries/get` command is equally limited. To see the real config: read `/config/.storage/core.config_entries` via `docker exec homeassistant python3 -c "..."` inside the HA VM. A stale config entry title (e.g. `192.168.100.27`) does NOT mean the actual broker IP is stale — check the `data.broker` field.
- **Setpoint-vs-measured feedback loop in control automations.** When a HA automation computes a new setpoint from a formula that includes the *current setpoint* as a variable, it creates positive feedback — the setpoint spirals to the hardware limit every cycle. Symptom: battery/storage device gets commanded to max power but internally throttles (e.g. via `anti_feed`), leaving a residual gap on the grid. Fix: swap `number.*_einstellen` (setpoint entities) for `sensor.*` (measured entities) in the variable definitions, keeping all formula structures unchanged. This is classic control theory — the feedback path must measure the actual plant output, not the controller's commanded output. See `references/marstek-cascade-automation.md` for a worked example.
- **Bubble Card pop-up migration (v3.2.0) breaks tab rendering when using bottom-navigation.** Moving content cards INTO the pop-up's `cards:` property (as the v3.2.0 migration notice instructs) causes all views to render BLANK — the pop-up is an overlay that hides content until triggered. When using `bottom-navigation` for persistent tab switching, do NOT use pop-up cards at all. Just put content cards directly in the view alongside the `bottom-navigation` card. The migration notice is misleading for this use case. See `references/bubble-card-popup-migration.md` for the full story and revert script.
- **User explicitly REVOKED Bubble Cards entirely (2026-07-10).** "nutze keine bubble cards" — do NOT install, use, or recommend Bubble Cards for any HA dashboard work. Use only Mushroom Cards + native HA cards. Bottom navigation via native HA view tabs, not Bubble bottom-navigation. This supersedes all earlier decisions about Bubble Card usage.
- **Headless browser may fail to render HA dashboards after `homeassistant.reload_all`.** After calling `homeassistant.reload_all` or clearing service workers/caches, the headless browser may show 0 `ha-card` elements across ALL dashboards — even the old ones. The config is verified correct via WebSocket. This is a headless-browser caching issue, not a config issue. Test on a real browser/mobile device for reliable verification.

## NUT/UPS Integration Troubleshooting

When HA entities from the NUT (Network UPS Tools) integration show `unavailable`:

1. **Check the NUT config entry** — read `.storage/core.config_entries` (see `references/ha-integration-config-editing.md`) and grep for `"domain":"nut"` to find the configured host/port.
2. **Check if NUT server is reachable** — `python3 -c "import socket; s=socket.socket(); s.settimeout(3); s.connect(('HOST', 3493)); print('OK'); s.close()"` or `upsc apc@HOST`.
3. **Check NUT server config** (SSH to the NUT host):
   - `/etc/nut/nut.conf` — `MODE` must be `netserver` (not `none`)
   - `/etc/nut/upsd.conf` — `LISTEN` must bind to accessible interface (`0.0.0.0 3493` or correct IP)
   - `/etc/nut/ups.conf` — driver section must match USB device (`usbhid-ups`, correct vendor/product IDs)
   - `/etc/nut/upsd.users` — must have a `[monitor]` section with `upsmon master`
4. **Start NUT services**: `systemctl restart nut-driver.target nut-server`
5. **Verify locally**: `upsc apc@localhost` should return UPS variables
6. **Update HA config entry** if the NUT server IP changed (see `references/ha-integration-config-editing.md`)
7. **Restart HA** — config entry changes in `.storage/` require a restart to take effect

### NUT host in this homelab
- **Host**: 10.0.30.100 (was 192.168.100.27 before network migration)
- **UPS**: APC Back-UPS ES 700G (USB, serial 5B1940T54935)
- **Driver**: `usbhid-ups`, port `auto`
- **NUT credentials**: Stored in 1Password (Vault "Hermes", item "NUT UPS Monitor")

## Editing HA Integration Config Without UI

When an integration's host/IP needs updating but the HA UI is inaccessible or the integration was auto-discovered:

1. **Read**: `/mnt/data/supervisor/homeassistant/.storage/core.config_entries` (via Proxmox guest agent)
2. **Backup first**: `cp core.config_entries core.config_entries.bak`
3. **Edit**: `sed -i 's/"host":"OLD_IP"/"host":"NEW_IP"/g' core.config_entries`
4. **Restart HA** — changes to `.storage/` files only take effect after restart
5. **Verify**: Check entity states via API after restart

See `references/ha-integration-config-editing.md` for detailed walkthrough.

## HA WebSocket API Patterns

- Use `websockets` Python library with `max_size=10*1024*1024` — entity/device registries can exceed 5MB with 3500+ entries (observed: 3570 entries). The default 1MB limit and even 5MB will fail on large installations.
- Available commands: `config/device_registry/list`, `config/entity_registry/list`, `config_entries/get` (no params — returns all entries, but WITHOUT the `data` field)
- **NOT available**: `config_entries/list` — returns `unknown_command`. Use `config_entries/get` instead.
- **`config_entries/get` does NOT accept `entry_id` param** — calling with `{"entry_id": "..."}` returns "extra keys not allowed". It only works with no params, returning all entries. The entries omit the `data` field (credentials, broker IP, host, port). For full config data including credentials, read `/config/.storage/core.config_entries` via `docker exec homeassistant python3 -c "..."`.
- **`config_entries/update`** works via WebSocket with `{"entry_id": "...", ...}` — but only updates top-level fields (title, pref_disable_new_entities, etc.), not the `data` block. For changing broker IP/host, edit `.storage/` directly.
- **`config_entries/disable`** works with `{"entry_id": "...", "disabled_by": "user"}` (set to `null` to re-enable).
- Auth flow: receive `auth_required` → send `{type: auth, access_token}` → receive `auth_ok` → send commands with incrementing `id`

### NUT Battery Calibration (Runtime Relearn)

When the UPS reports `ups.status: OL LB` (Online + Low Battery) despite `battery.charge: 100`, the firmware's runtime estimate is stale — typically after a battery replacement that wasn't registered.

**Fix: Update battery date via `upsrw`**

```bash
# SSH to NUT server (10.0.30.100)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.100 '
  NUT_PASS="<password from 1Password>"
  # Set battery manufacture date to the actual replacement date
  upsrw -s battery.mfr.date=YYYY/MM/DD -u upsadmin -p "$NUT_PASS" apc@localhost
  # Optionally lower the low-runtime threshold (seconds of runtime before LB triggers)
  upsrw -s battery.runtime.low=30 -u upsadmin -p "$NUT_PASS" apc@localhost
'

# Verify
upsc apc@10.0.30.100 | grep -E "battery.mfr.date|battery.runtime|ups.status"
# Expected: ups.status: OL (no LB), battery.runtime: ~285 (was ~60)
```

**Key points:**
- `upsrw -s` must come BEFORE `-u`/`-p` in the argument order, otherwise it silently fails.
- The battery date may not persist in `upsc` output immediately (firmware cache), but the runtime recalibration takes effect.
- Battery was replaced **December 2024** in this homelab. Previous date was `2019/10/01` (never updated).
- After fix: runtime jumped from 60s → 285s, LB flag cleared.
- For a true runtime-relearn (full discharge cycle), the NUT shutdown chain must be disabled first to avoid triggering FSD during calibration.

## Credential Management

**User instruction: Store ALL access tokens, passwords, certificates in 1Password (Vault "Hermes") immediately.**

Items stored this session:
- "Home Assistant API Token" — HA long-lived access token
- "NUT UPS Monitor (10.0.30.100)" — NUT upsd monitor password

## Systematic HA Health Audit (Proactive)

Beyond diagnosing outages, a proactive health audit identifies degradation before it becomes visible. Run this when asked "what can be improved" or as periodic maintenance.

### Step-by-Step

1. **Fetch HA config** — `GET /api/config` → version, state, component count. Confirm `state: RUNNING`.
2. **Fetch all entity states** — `GET /api/states` → save to file (JSON can contain control chars; parse with `strict=False` or `decode('utf-8','replace')`).
3. **Compute health metrics:**
   - Total entities, unavailable count + percentage, unknown count + percentage
   - Domain distribution (`Counter` on `entity_id.split('.')[0]`)
   - **Group unavailable by prefix** — split entity_id on `_`, take first 3 segments as key. This clusters "all entities from integration X" together (e.g. `sensor_spoolman_spool: 101` immediately flags a dead integration).
4. **Audit config entries** — `GET /api/config/config_entries/entry` → group by `state`:
   - `loaded` = healthy
   - `setup_retry` = discovered but can't connect (check device/network)
   - `setup_error` = config or auth failure
   - `not_loaded` = disabled/ignored (usually intentional, but audit for stale ignores)
5. **Audit automations** — filter `automation.*` from states → count on/off. List the OFF ones with friendly names — these are often forgotten after troubleshooting.
6. **Detect stale entities** — compare `last_changed` to now. Entities >24h stale (excluding `unavailable`/`unknown`) indicate sensors that stopped reporting.
7. **Cross-reference findings** — map unavailable groups to config entry states. A group of 77 unavailable `hame_energy` entities + a `not_loaded` or absent config entry = integration was replaced but not cleaned up.

### Output Format

Present as a ranked table: Problem # | Area | Dead Entities | Root Cause. Sort by dead-entity count descending. This directly feeds ideation for improvement priorities.

### Pitfalls

- **`/api/error_log` may return 404** on newer HA versions (2026.7.x observed). The endpoint path may have changed. Don't rely on it for the audit — use config entry states and entity grouping instead.
- **Cronjob reports show incremental runs, not totals.** A scraper cronjob reporting "323 recipes this run" does NOT mean the DB only has 323 — always check the DB directly for totals.
- **Stale `update.*` entities are normal** — `update.home_assistant_*_update` with 41h staleness just means no update is available. Exclude `update.` domain from stale analysis.

## Bulk Entity Audit (Diagnosing Widespread Outages)

When many HA entities are `unavailable` (e.g. after a DB outage or mass reconnection event), use the bulk audit pattern to quickly identify which devices/integrations are affected:

### Method

1. **Fetch all entity states** — save to file to avoid JSON parsing issues with control characters:
```bash
curl -s "$HA_URL/api/states" -H "Authorization: Bearer $HA_TOKEN" -o /tmp/ha_states.json
```

2. **Parse with `strict=False`** — HA state JSON sometimes contains unescaped control characters:
```python
import json
with open("/tmp/ha_states.json", "rb") as f:
    data = json.loads(f.read().decode('utf-8', errors='replace'))
```

3. **Filter and group** — categorize `unavailable` entities by device/integration using entity_id pattern matching:
```python
from collections import defaultdict
groups = defaultdict(list)
for e in data:
    if e['state'] != 'unavailable':
        continue
    eid = e['entity_id']
    name = e.get('attributes', {}).get('friendly_name', '')
    # Match by known prefixes: 'evcc_', 'hame_energy', 'shellyplus1', 'tradfri',
    # 'i3_mega', 'staubsauger_', 'double_take', 'badezimmer_', 'sauna_', etc.
    # Group into device categories and count
```

4. **Report by severity** — sort groups by entity count descending. Groups with 50+ entities (e.g. HAME Energy with 281, EVCC with 128) indicate entire integrations offline, not individual device failures.

### Common Post-Outage Cascade Pattern

After a Galera/DB outage and HA restart, several integrations may fail to reconnect:
- **EVCC** — add-on runs healthy but all loadpoint entities stay `unavailable` (needs add-on restart)
- **Alexa/Echo** — all devices show `konnektivitat: unavailable` (integration disconnected)
- **Frigate + Double Take** — Frigate may be on a stopped CT. See "Frigate Troubleshooting" below for location and recovery steps. Double Take (face recognition) depends on Frigate and goes offline when Frigate is down.
- **Robot vacuums** — cloud-connected devices drop and don't auto-reconnect
- **Shelly/TRADFRI** — local network devices recover on their own once network is stable

### Quick Health Metric

```python
total = len(data)
unavail = sum(1 for e in data if e['state'] == 'unavailable')
print(f"{unavail}/{total} ({unavail/total*100:.0f}%) unavailable")
# Normal: <5%. After DB outage: 30-40% until integrations reconnect.
```

## Dead Integration Cleanup (Bulk Removal)

When dead integrations inflate the unavailable entity count, perform structured cleanup. Full procedure: `references/ha-dead-integration-cleanup.md`.

### Decision Matrix

| Situation | Action |
|-----------|--------|
| Own config entry, no MQTT | `DELETE /api/config/config_entries/entry/{ENTRY_ID}` |
| MQTT-discovered (shares broker entry) | WebSocket `config/entity_registry/remove` per entity + clear MQTT discovery topics |
| `setup_error`, unrecoverable | Delete config entry via REST API |
| `setup_retry`, may recover | Disable, don't delete |
| Feature-Not-Supported (vacuum sub-entities) | Leave — normal behavior |

### Key Rules

- **MQTT-discovered entities share the broker's `config_entry_id`** — deleting the broker entry destroys ALL MQTT devices. Check `domain == "mqtt"` before deleting.
- **Clear MQTT discovery topics** with empty retained payloads (`POST /api/services/mqtt/publish`), or entities regenerate on restart.
- **`config/entity_registry/remove` IS supported via WebSocket** — returns `{"success": true}`.
- **Always measure unavailable-rate before and after.** Baseline script in the reference file.
- **`config/device_registry/remove` may silently fail** for MQTT-discovered devices — focus on entity removal + topic clearing; orphaned devices auto-clean over time.

### Worked Example (2026-07-12)

Removed HAME Energy (81 MQTT entities + 37 discovery topics), Spoolman (117 entities + config entry), Kia UVO (config entry, setup_error). Result: 2,282→2,088 entities, 28.5%→21.8% unavailable.

## Frigate Troubleshooting

When Frigate camera entities are `unavailable` in HA:

### Location in This Homelab

- **CT 120** "frigate" on **proxmox4** (10.0.20.40), IP **10.0.30.104**
- Web UI: `http://10.0.30.104:5000`
- Config: `/config/config.yml` inside CT (has real MQTT creds, camera RTSP URLs)
- Service: `frigate.service` (systemd, auto-restart)
- `onboot=1` — starts automatically on Proxmox boot

### Recovery Steps

1. **Check CT status** — `pct status 120` on proxmox4 (10.0.20.40). If stopped, start it: `pct start 120`.
2. **Wait 30-60s** for Frigate to initialize (OpenVINO detector + go2rtc + embeddings).
3. **Verify web UI**: `curl -s http://10.0.30.104/api/version` — should return version string.
4. **Check recordings**: `ls -lh /tmp/cache/Einfahrt* /tmp/cache/Terrasse*` — growing files mean cameras are streaming.
5. **Verify MQTT**: Subscribe to `frigate/#` on `10.0.30.10:1883` — should see `frigate/available → online` and per-camera state topics.
6. **Restart HA** — even if Frigate is publishing to MQTT, HA's Frigate integration needs a restart to reconnect. WebSocket `config_entry/reload` returns `unknown_command`; only `homeassistant/restart` works.
7. **Verify entities**: After HA restart (~90s), check `camera.einfahrt_2` and `camera.terrasse_2` states.

### Cameras

- **Einfahrt**: 10.0.50.102 (Hikvision, RTSP `h264Preview_01_main`/`_sub`)
- **Terrasse**: 10.0.50.103 (same model)
- Credentials in `/config/config.yml` (RTSP URLs with embedded `admin:password@`)

### Known Issues

- **OOM FIXED (2026-06-27)**: CT memory was increased from 4GB → 8GB. With 2 cameras + face recognition + LPR + embeddings + semantic search, 4GB caused repeated OOM kills. 8GB resolved it (1.2GB used, 6.9GB free). Command: `pct set 120 -memory 8192` (requires CT stop/start). Verified: Frigate stable, all 70/70 entities available after restart.
- **RTSP 461 "Unsupported transport"**: go2rtc initial probe may fail with `method SETUP failed: 461 Unsupported transport` — this is transient and doesn't prevent recording. Recordings still work via go2rtc restream.
- **Config file confusion**: `/opt/frigate/config/config.yml` is a demo/test config. The REAL config is at `/config/config.yml` (loaded by default). The Frigate API (`/api/config`) shows the effective merged config including DB-stored overrides.

### MQTT Credentials

- User: `frigate`, password in `/config/config.yml` inside CT 120.
- Also stored in 1Password (Vault "Hermes").

## EVCC Custom Charger via HA REST API

When EVCC cannot directly connect to a charger (TLS cert issues, API changes, device on unreachable VLAN), use EVCC's `type: custom` charger with `source: http` plugin to bridge through HA REST API entities. This is a powerful fallback when the native HA integration works but EVCC's direct connection doesn't.

### How It Works

- **Getters** (status, enabled, power, energy): `source: http` with `jq` to extract values from HA's `/api/states/{entity_id}` JSON response
- **Setters** (enable, maxcurrent): `source: http` with `method: POST` to HA's `/api/services/{domain}/{service}` endpoints, using Go templates (`{{.enable}}`, `{{.maxcurrent}}`) for value interpolation
- **Auth**: `Authorization: Bearer <HA_TOKEN>` header on every request

### Example: SMA EV Charger Bridge

The SMA EV Charger (10.0.50.205) has a self-signed cert with CN=`SMA3010510346.local` (no IP-SANs). Go 1.17+ ignores CN for TLS validation, so EVCC's direct HTTPS connection fails. The native HA SMA integration reaches the charger via a different network path (192.168.100.95). Solution: bridge EVCC through HA.

Entity mappings:
- **status** ← `sensor.ev_charger_schon_status_ladevorgang` (jq: `active_mode`→C, `sleep_mode`→B, `not_connected`→A)
- **enabled** ← `switch.ev_charger_schon_manuelle_ladefreigabe` (jq: `.state == "on"`)
- **enable** → `switch.turn_on`/`turn_off` service on `switch.ev_charger_schon_manuelle_ladefreigabe`
- **maxcurrent** → `number.set_value` service on `number.ev_charger_schon_ac_strom_begrenzung` (body: `{"entity_id":"...","value":{{.maxcurrent}}}`)
- **power** ← `sensor.ev_charger_schon_leistung_ladestation` (jq: `.state | tonumber`)
- **energy** ← `sensor.ev_charger_schon_zahlerstand_ladestation` (jq: `.state | tonumber`, scale: 0.001 for Wh→kWh)

See `templates/evcc-custom-charger-ha-bridge.yaml` for the full working config.

### EVCC HTTP Plugin Reference

- Registry name: `http` (use `source: http` in YAML)
- Supports: `uri`, `method`, `headers`, `body`, `jq`, `scale`, `insecure`, `auth`, `timeout`, `cache`
- Value templating: Go `text/template` with sprig functions. `${param}` regex syntax for backward compat. Template receives `{param: value}` map.
- Setter signature: `BoolSetter(param)` / `IntSetter(param)` / `FloatSetter(param)` — the `param` name becomes the template key (e.g. `{{.enable}}`, `{{.maxcurrent}}`)
- jq uses `github.com/itchyny/gojq` — standard jq syntax, supports `tonumber`, conditionals, etc.

### Extending the HA Bridge Pattern to Meters & Batteries

The `type: custom` + `source: http` pattern works for **meters and batteries** too, not just chargers. When a device's Modbus/API is unreachable from EVCC but HA has a working integration, define it as a `custom` meter in the `meters:` section using the same HA REST API bridge.

**Critical:** Do NOT add `usage: battery` — the `custom` type doesn't accept it. The `soc:` field's presence implicitly marks the meter as a battery.

#### Working Example: Marstek Venus Battery via HA REST API

Marstek Venus (10.0.50.113) Modbus port 502 is closed from EVCC's network, but HA's Marstek Modbus integration reads it fine. Bridge through HA:

```yaml
meters:
  - name: marstek
    type: custom
    capacity: 5                          # kWh, top-level (NOT under battery:)
    maxchargepower: 2500                 # W, optional power limit
    maxdischargepower: 2500              # W, optional power limit
    power:                               # required — AC power (positive=discharge)
      source: http
      uri: http://10.0.30.10:8123/api/states/sensor.marstek_venus_modbus_ac_leistung
      headers:
        Authorization: "Bearer __HA_TOKEN__"
      jq: '.state | tonumber'
    soc:                                 # presence of soc: makes this a battery
      source: http
      uri: http://10.0.30.10:8123/api/states/sensor.marstek_venus_modbus_batterie_ladezustand
      headers:
        Authorization: "Bearer __HA_TOKEN__"
      jq: '.state | tonumber'
    energy:                              # optional — total discharged energy
      source: http
      uri: http://10.0.30.10:8123/api/states/sensor.marstek_venus_modbus_gesamte_entladeenergie
      headers:
        Authorization: "Bearer __HA_TOKEN__"
      jq: '.state | tonumber'
```

Verified working (2026-06-27): EVCC reports `marstek: power=0W, soc=100%, energy=336.34kWh`. Note: `controllable=False` — for battery mode control, add `limitsoc` or `batterymode` setters targeting HA service calls (e.g. `number.marstek_venus_modbus_ladeleistung_einstellen`).

#### EVCC Custom Battery Field Reference (from source `meter/meter.go`)

| Field | Type | Required | Purpose |
|-------|------|----------|---------|
| `power` | plugin.Config | yes | Current power (W) |
| `soc` | plugin.Config | battery only | State of charge (%) — presence makes it a battery |
| `energy` | plugin.Config | optional | Total energy counter (kWh) |
| `capacity` | float | optional | Battery capacity in kWh |
| `maxchargepower` | float | optional | Max charge power (W) |
| `maxdischargepower` | float | optional | Max discharge power (W) |
| `limitsoc` | plugin.Config | optional | FloatSetter for SOC limit control |
| `batterymode` | plugin.Config | optional | IntSetter for battery mode control |

All `plugin.Config` fields accept `source: http` with `uri`, `headers`, `jq`, `method`, `body`, `scale`.

### EVCC API Inspection (Diagnostic Endpoint)

EVCC exposes a REST API on port 7070 (same port as the web UI):

```bash
# Full system state — grid, PV, battery (per-device), loadpoints, tariff, forecast
curl -s http://10.0.30.10:7070/api/state | python3 -m json.tool

# Key fields to check:
# - .grid.power, .grid.energy          — grid import/export
# - .pv[]                              — array of PV inverters with power/energy
# - .battery.power, .battery.soc       — aggregated battery
# - .battery.devices[]                  — per-battery breakdown (name, power, soc, controllable)
# - .loadpoints[]                       — charger status, mode, charge power
# - .tariffGrid                         — current grid price
# - .config                             — path to active config file
```

Only `/api/state` (GET) works reliably. No `/api/config`, `/api/restart`, or `/api/settings` endpoints exist. The WebSocket API (`/ws`) uses protobuf, not JSON.

### EVCC Battery Discharge Diagnosis (Why Didn't the Battery Cover the Gap?)

When batteries have sufficient SOC but grid import still occurs, the problem is on the EVCC/battery-control side, not the consumption side. Query `api/state` for the key fields:

| Field | Meaning | Impact |
|-------|---------|--------|
| `battery.devices[].controllable` | Whether EVCC can command this battery | `false` = EVCC reads SoC/power but cannot adjust discharge rate |
| `batteryDischargeControl` | Whether EVCC actively manages discharge | `false` = passive monitoring only |
| `residualPower` | Watts EVCC tolerates from grid before reacting | 100 = EVCC accepts 100W grid import as baseline |
| `batteryMode` | Current battery mode | `unknown` = no active mode set |
| `battery.devices[].power` | Current per-battery output (W) | Compare to home power to see if battery is maxed out |
| `battery.devices[].soc` | Per-battery SOC (%) | High SOC + low output = battery is limited, not empty |

**Diagnosis flow:**
1. Check `controllable` — if `false`, EVCC cannot increase discharge. The battery's own internal limit is the bottleneck.
2. Check `residualPower` — if >0, EVCC deliberately allows that much grid import.
3. Compare `battery.devices[].power` to home power — if battery output << home power and SOC is high, the battery's internal discharge limit (not EVCC) is the constraint.
4. Verify by checking if the battery ever exceeded its typical output during the session (e.g. Marstek spiked to 1,437W during heat pump cycle but settled back to ~650W steady-state → internal limiter, not capacity).

**⚠️ Before concluding "internal limiter" — check the HA automation!** In this homelab the Marstek is controlled by the HA automation "Marstek Kaskaden-Steuerung (Modbus)" (`automation.marstek_modbus_steuerung`), NOT by EVCC directly. EVCC sees `controllable: false` because it's only a passive observer. The automation had a **feedback-loop bug** in its `echter_bedarf` formula: it used `number.*_einstellen` (setpoints) instead of `sensor.*` (measured values), causing the setpoint to spiral to 2500W. The Marstek's `anti_feed` mode then throttled to ~650W, leaving a ~200W gap on the grid. **Fixed 2026-07-10**: swapped the two variable definitions from `number.*_einstellen` to `sensor.marstek_ladeleistung` / `sensor.marstek_entladeleistung` (measured values), keeping all formula structures unchanged per user request. See `references/marstek-cascade-automation.md` for the full analysis and fix.

**Fix options (depending on root cause):**
- If `controllable: false` AND a HA automation controls the battery: Fix the automation's control formula (common bug: using setpoint instead of measured values → positive feedback loop).
- If `residualPower > 0`: Lower it in EVCC config (but only effective if batteries are `controllable: true`).
- If battery internal limit: Increase `maxdischargepower` in EVCC config (if controllable) or change the battery's own discharge setting.

### EVCC Config File Locations (HAOS)

Three copies exist on the HA VM filesystem:
- **Main config**: `/mnt/data/supervisor/homeassistant/evcc.yaml` — the active config referenced by EVCC
- **Addon-managed copy**: `/mnt/data/supervisor/app_configs/<hash>_evcc/evcc.yaml` — addon's internal copy
- **Temp copy**: `/mnt/data/supervisor/tmp/tmpXXXXXX/data/evcc.yaml` — transient, ignore

Find with: `qm guest exec 106 -- find /mnt/data -name "evcc.yaml" -maxdepth 5`

### Pitfalls

- **`custom` meter type does NOT accept `usage:` field.** Only `type: template` meters use `usage: battery/pv/grid`. For `type: custom`, the presence of `soc:` field implicitly marks it as a battery. Adding `usage: battery` to a custom meter causes: `FATAL: has invalid keys: usage`. Remove the `usage` line entirely.
- **Custom battery fields (from EVCC source `meter/meter.go`):** `power` (required, plugin.Config), `soc` (optional, plugin.Config — makes it a battery), `energy` (optional), `capacity` (float, kWh), `maxchargepower` / `maxdischargepower` (float, W), `limitsoc` (optional setter for battery control), `batterymode` (optional IntSetter for mode control). These are top-level keys under the meter entry, NOT nested under a `battery:` block.
- **EVCC has a native `homeassistant` meter type** (`meter/homeassistant.go`) that supports `power`, `energy`, `soc`, `modeNormal/modeHold/modeCharge` entity IDs directly. However it requires OAuth2 flow (interactive browser auth), making it impractical for headless setups. Use `type: custom` + `source: http` with `jq` instead — simpler and works with a static bearer token.
- **File transfer to HAOS VM via `qm guest exec` + base64 can corrupt UTF-8.** Non-ASCII characters (e.g. "Schön" in YAML comments) get double-encoded through the base64→shell→guest-agent pipeline, producing control characters (U+0083) that YAML parsers reject with `"control characters are not allowed"`. **Fix: strip all non-ASCII from YAML files before transfer** — `python3 -c "open('out.yaml','wb').write(open('in.yaml','rb').read().encode('ascii','ignore'))"` — or simply use ASCII-only comments.
- **The Hermes `terminal` tool redacts bearer tokens in its output.** A 183-char HA token appears as `eyJhbG...tfCE` (13 chars) when echoed through Python via the terminal tool. This makes it impossible to pass tokens through Python intermediate variables. **Workaround:** write the token to a file (`/tmp/ha_token.txt`) using `op item get ... --reveal > /tmp/ha_token.txt`, then read it directly in shell or Python `open()` — never through the terminal tool's stdout.
- **1Password `op item get` with `--fields credential --format plain` may return empty** even when the credential exists. This happens when the field label doesn't match exactly (e.g. the field is labeled `password` internally but `credential` is requested). **Fallback**: use `--format json` and parse the full item JSON with Python to extract the field by label. See the pattern below.
- **1Password `--value` flag does NOT exist in service account mode.** `op item get <ID> --fields label=credential --reveal --value` fails with `unknown flag: --value`. Use `--reveal` only (without `--value`) — it outputs the value, possibly with surrounding quotes. Strip quotes with `tr -d '"'` in shell.
- **1Password may have duplicate items with the same title.** `op item get "Home Assistant API Token"` fails with "More than one item matches". Use the item ID directly: `op item get <ID> --vault Hermes --fields label=credential --reveal`. List IDs with `op item list --vault Hermes --format json`. The HA API token item ID is `mntt343ghnetkyeyeboxxksfde`.
- **HA devices can ONLY be created by integrations, not via WebSocket API.** `config/device_registry/create` returns `unknown_command`. To group entities under a device, create a `rest` integration config entry via REST API (`POST /api/config/config_entries/flow` with `handler: "rest"`). Template entities (platform=`template`, `config_entry_id: None`) cannot be linked to devices — they must be migrated to the integration first.
- **`device_info` is NOT supported in HA's `rest` integration YAML.** Neither the legacy `sensor: - platform: rest` syntax NOR the modern `rest: - resource: ... sensor: [...]` syntax accepts `device_info:` — HA rejects it with "Invalid config for 'rest': 'device_info' is an invalid option for 'rest'". This means you CANNOT attach a device to REST sensors via YAML alone. Workaround: manually create the device in `.storage/core.device_registry` (see "Manual Device Creation in .storage/" section below).
- **`device_info` is ALSO not supported in `template:` entities.** Adding `device_info:` to a template sensor/switch causes "Invalid config for 'template' at configuration.yaml: 'device_info' is an invalid option for 'template'". The `device_info` key only works for integrations that explicitly support it (some UI-based config entries do; YAML-based `rest` and `template` do not).
- **Manual `.storage/` device registry edits ARE persisted by HA on restart** — IF the device entry has all required fields matching HA's internal schema. Missing fields cause HA to silently drop the entry on startup. Required fields (verified 2026-07-10): `config_entries` (list), `configuration_url`, `connections` (list), `identifiers` (list of lists), `manufacturer`, `model`, `name`, `name_by_user` (null), `serial_number`, `sw_version`, `hw_version` (null), `via_device_id` (null), `area_id` (null), `disabled_by` (null), `entry_type` (null), `id` (uuid4), `config_entries_subentries` (dict), `created_at` (ISO datetime), `modified_at` (ISO datetime), `labels` (list), `model_id` (null), `primary_config_entry` (null). Do NOT include `name_by_user_set` — it is not a valid field and causes rejection.
- **Entity-to-device linking via `.storage/core.entity_registry` also survives restart** — set `device_id` on each entity in the registry file, restart HA, and the links persist. Unlike device entries, entity entries are more tolerant of manual edits (adding a `device_id` field to an existing entity works fine).
- **EVCC add-on restart via Docker (when Supervisor API is unreliable).** The HA Supervisor WebSocket API (`supervisor/api` → `/addons/{slug}/restart`) often returns `unknown_error`. Direct Docker restart works: `qm guest exec 106 -- docker restart addon_49686a9f_evcc`. Find the container name with `docker ps --format '{{.Names}} {{.Status}}' | grep evcc`.
- **EVCC add-on overwrites `evcc.yaml` on restart.** The add-on regenerates the config from its internal options/store. Manual edits to `evcc.yaml` on a running HAOS are lost on next add-on restart. To persist changes: stop the HA VM, mount the disk, write the file, start the VM. Or use the EVCC web UI config editor (EVCC 0.309+ has a built-in config UI, but its WebSocket API uses protobuf, not JSON — not scriptable).
- **RBD live-mount causes filesystem corruption.** Mapping and mounting an RBD disk while the VM is running WILL corrupt the ext4 filesystem. Symptoms: files disappear, `evcc.yaml` truncated to 0 bytes, files move to `lost+found`. ALWAYS stop the VM (`qm stop 106`) before mapping/mounting. If corruption occurs: `fsck.ext4 -y /dev/rbd3p8` then check `lost+found` for recovered files.
- **SMA EV Charger `/api/v1/token` returns 400** regardless of credentials or Content-Type. The API may require a specific firmware mode or the credentials may have changed. Don't waste time trying different encoding — use the HA bridge approach instead.
- **EVCC version 0.309.2** has limited REST API: only `/api/state` (GET) works reliably. No `/api/config`, `/api/restart`, or `/api/settings` endpoints.

## HAOS Filesystem Access via RBD Mapping (Last Resort)

When the QEMU guest agent is down and HA REST/WebSocket APIs can't reach the file you need:

```bash
# SSH to proxmox1 (10.0.20.10) as root
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10

# CRITICAL: Stop the VM first to avoid filesystem corruption!
qm stop 106
sleep 5

# Map the RBD disk
rbd map vm_disks/vm-106-disk-1
# Usually appears as /dev/rbd3

# The HAOS data partition is the largest partition (typically p8)
mount /dev/rbd3p8 /mnt/haos

# HA config: /mnt/haos/supervisor/homeassistant/
# Add-on data: /mnt/haos/supervisor/apps/data/

# When done:
umount /mnt/haos
rbd unmap /dev/rbd3
qm start 106
```

**Partitions on HAOS disk:**
- p1: boot/EFI
- p2: kernel
- p3: overlay (small)
- p8: data (largest — this is what you want)

## HA Config Flow via REST API (Adding Integrations Programmatically)

When you need to add an integration with a known device IP (e.g. after adding a second NIC for cross-VLAN access), use the REST API Config Flow. The WebSocket equivalents (`config/config_entries/flow/init`, `config/config_entries/flow/advance`) return `unknown_command` — **only the REST API works**.

### Two-Step Flow

```bash
TOKEN=$(cat /tmp/ha_token.txt)

# Step 1: Initiate flow for the integration handler
flow=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  "http://10.0.30.10:8123/api/config/config_entries/flow" \
  -d '{"handler":"shelly"}')
flow_id=$(echo "$flow" | python3 -c "import sys,json; print(json.loads(sys.stdin.read())['flow_id'])")

# Step 2: Submit the form with device details (schema varies per integration)
# For Shelly: {"host":"IP","port":80}
# For others: check the data_schema in the flow initiation response
result=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  "http://10.0.30.10:8123/api/config/config_entries/flow/$flow_id" \
  -d '{"host":"10.0.50.100","port":80}')

# Success: {"type":"create_entry","title":"shellyplus1-XXX","result":{"state":"loaded"}}
# Failure: {"type":"form","errors":{"base":"cannot_connect"}}
# Already exists: {"type":"abort","reason":"already_configured"}
```

### Batch-Adding Multiple Devices

```bash
for ip in 10.0.50.100 10.0.50.109 10.0.50.114 10.0.50.117; do
  flow=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    "http://10.0.30.10:8123/api/config/config_entries/flow" -d '{"handler":"shelly"}')
  fid=$(echo "$flow" | python3 -c "import sys,json; print(json.loads(sys.stdin.read())['flow_id'])")
  curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    "http://10.0.30.10:8123/api/config/config_entries/flow/$fid" \
    -d "{\"host\":\"$ip\",\"port\":80}" | python3 -c "
import sys,json; r=json.loads(sys.stdin.read())
print(f'$ip: {r.get(\"title\",\"?\")} - {r.get(\"result\",{}).get(\"state\",r.get(\"reason\",\"?\"))}')"
done
```

### Pitfalls

- **WebSocket config flow commands DON'T work** — `config/config_entries/flow/init` and `config/config_entries/flow/advance` both return `unknown_command`. Use REST API exclusively.
- **`already_configured` abort** — if a config entry already exists (even in `setup_retry` state), the flow aborts. Delete the old entry first: `DELETE /api/config/config_entries/entry/{entry_id}`.
- **`cannot_connect` error** — device may be sleeping (Shelly Plus 1 sleeps aggressively). Wake it with repeated `curl http://IP/rpc/Shelly.GetDeviceInfo` attempts, then retry the flow.
- **Flow forms are integration-specific** — check the `data_schema` in the initiation response to know which fields to submit. Shelly wants `host` + `port`; other integrations differ.

## Config Entry Reload via REST API

Unlike the WebSocket `config_entry/reload` (which returns `unknown_command`), the REST API CAN reload individual config entries:

```bash
curl -s -X POST "$HA_URL/api/config/config_entries/entry/{ENTRY_ID}/reload" \
  -H "Authorization: Bearer $HA_TOKEN"
# Success: {"require_restart":false}
```

This is much faster than a full HA restart when you need to reload a single integration (e.g. after fixing its backend service). Find the entry ID in `.storage/core.config_entries` or via `curl -s "$HA_URL/api/config/config_entries/entry" -H "Authorization: Bearer $HA_TOKEN"`.

## VLAN/Network Segmentation Diagnosis

When HA integrations are stuck in `setup_retry` but the physical devices are online, the root cause is often **network segmentation** — HA and IoT devices are on different VLANs/subnets that either lack routing between them or block mDNS/Zeroconf multicast.

### Diagnostic Workflow

1. **Check config entry states** via REST API:
```bash
curl -s "$HA_URL/api/config/config_entries/entry" -H "Authorization: Bearer $HA_TOKEN" | \
  python3 -c "import sys,json; [print(f'{e[\"title\"]}: {e[\"state\"]}') for e in json.load(sys.stdin) if e.get('domain')=='shelly']"
```
States: `loaded` = working, `setup_retry` = discovered but can't connect, `not_loaded` = never set up.

2. **Check HA's routing table** — does HA even have a route to the device subnet?
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 'qm guest exec 106 -- sh -c "ip route show"'
```
If only `10.0.30.0/24` is directly connected and the default gateway doesn't route to `10.0.50.0/24`, HA cannot reach those devices at all.

3. **Probe devices directly** to confirm they're alive (from a host that CAN reach them):
```bash
# Shelly devices expose a simple RPC API
curl -s --connect-timeout 2 "http://<IP>/rpc/Shelly.GetDeviceInfo" | python3 -m json.tool
# Returns: name, id, mac, model, gen, fw_id, ver, app
```

4. **Identify which devices are on which subnet** — scan the IoT subnet for Shelly API endpoints:
```bash
for ip in <list_of_alive_IPs>; do
  resp=$(curl -s --connect-timeout 2 -m 3 "http://$ip/rpc/Shelly.GetDeviceInfo" 2>/dev/null)
  echo "$resp" | grep -qi "shelly" && echo "$ip → $(echo $resp | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["id"])')"
done
```

5. **Shelly devices may sleep** — Shelly Plus 1 devices sleep aggressively to save power. A device that responds once may not respond again for 30+ seconds. Use multiple attempts with short timeouts. Some devices (e.g. D0EF76D9BEB8 at 10.0.50.114) may be completely offline (not pingable from any host) — this indicates a physical problem (power/WLAN), not a HA config issue.

### Root Cause: mDNS/Zeroconf Doesn't Cross VLAN Boundaries

Many HA integrations (Shelly, LIFX, TRADFRI, Apple TV, etc.) discover devices via mDNS/Zeroconf multicast. Multicast traffic (`224.0.0.251:5353`) does NOT traverse VLAN boundaries by default. Even if unicast routing between VLANs works, the discovery mechanism fails.

**Symptoms:**
- Config entries show `source: zeroconf` and `state: setup_retry`
- Config entry `data: {}` is empty (no cached IP/host)
- Devices respond to direct HTTP probes from other hosts
- HA's routing table lacks a route to the device subnet

**Solutions (in order of preference):**

1. **Add a second NIC to the HA VM on the IoT VLAN** (BEST — gives HA direct L2 access for both unicast AND mDNS). See "Adding a Second NIC to HA VM" below for exact commands.
2. **Add a firewall/routing rule** allowing HA (10.0.30.10) to reach the IoT subnet (10.0.50.0/24) — minimum: TCP to device ports (80/443 for Shelly). Note: this enables unicast but NOT mDNS multicast.
3. **Configure an mDNS repeater/reflector** on the router (e.g. avahi-reflector, or router-native mdns proxy) to forward mDNS between VLANs
4. **Move IoT devices to HA's subnet** (10.0.30.x) — simplest but may not fit network design
5. **Manually add devices via HA Config Flow REST API** — works even without mDNS if HA can reach the device IP. See "HA Config Flow via REST API" below.

### Adding a Second NIC to HA VM (Confirmed Working 2026-06-27)

When HA needs direct Layer-2 access to another VLAN (for mDNS discovery AND unicast), add a second virtual NIC tagged to that VLAN:

```bash
# 1. Add NIC to VM (on proxmox1, VLAN-aware bridge vmbr0, tag=50 for IoT VLAN)
#    Use a known MAC if recreating a previously-removed NIC (check VM description)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  'qm set 106 -net1 virtio=BC:24:11:63:C7:77,bridge=vmbr0,tag=50'

# 2. Reboot the HA VM so HAOS detects the new interface
qm reboot 106   # or qm stop 106 && qm start 106 if reboot hangs

# 3. Wait for HA to come back (poll API every 15s, up to 3 min)
for i in $(seq 1 12); do
  sleep 15
  curl -s -o /dev/null -w "%{http_code}" "http://10.0.30.10:8123/api/" && break
done

# 4. Enable DHCP on the new interface via HAOS CLI (interface name = enp0s<N>)
#    Method is "auto" (NOT "dhcp" — HAOS uses NetworkManager naming)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  'qm guest exec 106 -- sh -c "ha network update enp0s19 --ipv4-method auto"'

# 5. Verify the interface got an IP
qm guest exec 106 -- sh -c "ip addr show enp0s19"
# Expected: inet 10.0.50.XXX/24 scope global dynamic

# 6. Verify HA can now reach devices on the IoT VLAN
qm guest exec 106 -- sh -c "curl -s http://10.0.50.100/rpc/Shelly.GetDeviceInfo"
```

**Key points:**
- `vmbr0` must be `bridge-vlan-aware yes` (already set in this homelab) for `tag=50` to work
- HAOS uses NetworkManager — the method is called `auto` (DHCP), `static`, or `disabled` — NOT `dhcp`
- The new interface appears as `enp0s19` (incrementing from existing `enp0s18`)
- After adding the NIC, the old zeroconf-discovered config entries may still be in `setup_retry` — delete them and re-add via Config Flow (see below), OR restart HA and hope zeroconf rediscovers (may need multiple restarts)
- **Default route conflict**: HA will get two default routes (one per NIC). The original NIC's route has lower metric and wins. This is fine — the second NIC is for direct subnet access, not internet.

### This Homelab's Network Layout

- **HA**: 10.0.30.10 (VM 106 on proxmox1) — only routes to 10.0.30.0/24 directly
- **IoT/VLAN 50**: 10.0.50.x — Shelly Plus 1 devices, cameras, SMA devices, Marstek
- **Proxmox mgmt**: 10.0.20.x
- **Affected (2026-06-27)**: 8 Shelly Plus 1 devices on 10.0.50.x in `setup_retry`:
  - Licht Terrasse Gartenhaus (d48afc44115c, 10.0.50.100) → fixed via Config Flow
  - LED Streifen Windfang (a0a3b3c396e4, 10.0.50.109) → fixed via Config Flow
  - Bewässerungskreis hinten (10061c7a8220, 10.0.50.117) → fixed via Config Flow
  - Bewässerungspumpe (10061c79b388, 10.0.50.137) → fixed via Config Flow
  - Bewässerungskreis Hochbeete (a0dd6c499b5c, 10.0.50.120) → fixed via Config Flow
  - Bewässerungskreis Beete (d0ef76d9beb8, 10.0.50.114) → OFFLINE (device not responding on network)
  - Bewässerungskreis mitte (d0ef76d91930, 10.0.50.116) → recovered after HA restart
  - Bewässerungskreis vorne (d0ef76d9beb8, 10.0.50.114) → same device as Beete, OFFLINE
- **Working**: 3 Shellys on 10.0.30.x (Stromverbrauch Heizung, Strommessung Sauna, shellypro4pm on 10.0.50.136) — `loaded`
- **Fix applied**: Second NIC (net1, VLAN 50) added to HA VM 106 → HA now on 10.0.50.127. 7 broken config entries deleted, 6 re-added via Config Flow REST API. 9/9 loaded after fix.
- **Resolved (2026-06-27)**: All 10 Shellys now loaded. D0EF76D9BEB8 (10.0.50.114, "Bewässerungspumpe") was eventually reached — device sleeps aggressively, took multiple attempts. D48AFC449BF8 (10.0.50.129) was a new device not in the original list. Final count: 10/10 loaded, 0 setup_retry.

## "Feature-Not-Supported" Unavailable Entities (Normal Behavior)

Some integrations create entities for ALL possible features of a device family, then leave unsupported ones as `unavailable`. This is **not a defect** — the device is online and working.

### Pattern: Dreame Vacuum Integration

Two robot vacuums (staubsauger_oben, x40_master) showed 190 unavailable entities combined. Investigation revealed:
- Main `vacuum.*` entity is `docked`/`idle` (working perfectly)
- Battery sensors report 99-100%
- Unavailable entities are: `select` (room configs for unsupported models), `button` (reset functions), `switch` (pet detection, max suction, DND sub-options), `sensor` (mapping_time, drying_progress), `number` (unsupported settings), `time` (unconfigured schedules)

**Rule of thumb:** If the main device entity (`vacuum.*`, `light.*`, `switch.*`) is available and reporting state, and the unavailable sub-entities are `select`/`button`/`number`/`switch`/`time` types representing advanced features — it's normal. Don't attempt to "fix" these.

## Midea AC Integration Troubleshooting

When Midea Smart AC (`midea_ac` custom integration, `midea-ac-py` by mill1000) is stuck in `setup_retry`:

### Integration Details

- **Integration**: `midea-ac-py` v2026.4.0, requires `msmart-ng==2026.4.1`
- **Config flow**: `config_flow.py` in `/mnt/data/supervisor/homeassistant/custom_components/midea_ac/`
- **Discovery modes**: `discover` (needs cloud credentials + host) or `manual` (needs device ID, host, port, optionally token+k1)
- **Cloud credentials**: Hardcoded in `_CLOUD_CREDENTIALS` dict in `config_flow.py` — keyed by country code. Default DE credentials: `midea_eu@mailinator.com` / `das_ist_passwort1`. These shared credentials may expire → `cloud_connection_failed` abort.

### Device Discovery

Midea AC devices are **discovered via UDP broadcast on port 6445**, then communicated with via **TCP port 6444** (V3 protocol). TCP 6444 IS open but requires authentication (token+key handshake) — unauthenticated TCP connections time out with "No response from host".

```bash
# Local discovery via midealocal library (works for finding the device)
python3 -m midealocal.cli discover \
  --username "USER@email.com" --password 'PASS' \
  --cloud-name "SmartHome" --host "10.0.40.54" --get_sn
# Returns: device_id, type, ip, port, model, sn, protocol
```

### Cloud Credential Retrieval (Token + K1)

The integration needs a **Token** and **K1** key from the Midea cloud to establish local UDP communication. These are retrieved via the Midea Cloud API:

1. Login to Midea Cloud (MSmartHome app credentials, NOT NetHome Plus)
2. Call `/v1/iot/secure/getToken` with the device's UDP ID
3. Response contains `tokenlist` with `token` and `key` fields

**Known issue (2026-06-27)**: The `getToken` API returns `"code":"3004","msg":"value is illegal"` for ALL credential sets tested:
- Shared DE credentials (`midea_eu@mailinator.com`) — fails
- User's personal MSmartHome credentials (`dominik@familie-schoen.com`) — login succeeds, but getToken still returns 3004
- All udpid calculation methods (msmart-ng little/big endian, midealocal methods 0/1/2, btMac-derived, SN-derived, MAC-derived) — all fail

Root cause: The device has NO LAN token in the Midea cloud despite being connected via WLAN and remotely controllable through the MSmartHome app. Cloud API returns `btToken` + `btMac` but NO `mac` field. `connectType: "1"` indicates initial BLE pairing, but the device subsequently joined WiFi. The `getToken` endpoint consistently returns `"3004: value is illegal"` for ALL udpid calculation methods — the cloud simply has no LAN token provisioned for this device.

**Resolution**: Re-pair the device via **WLAN setup** in the MSmartHome app (not Bluetooth) to force LAN token generation. After re-pairing, re-run `cloud.get_cloud_keys(153931628805781)` — if it returns keys, use the manual config flow.

### ⚠️ CRITICAL: midealocal/msmart-ng Package Conflict

`pip install midealocal` installs an old `msmart==0.2.5` package that **OVERWRITES the `msmart` namespace** used by `msmart-ng==2026.4.1`. This breaks `msmart-ng`'s `security` module with `ImportError: cannot import name 'MSGTYPE_ENCRYPTED_REQUEST' from 'msmart.const'`.

**Never install both on the same Python environment.** If already broken:
```bash
pip uninstall -y msmart-ng midealocal msmart --break-system-packages
rm -rf ~/.local/lib/python3.*/site-packages/msmart
pip install --break-system-packages msmart-ng==2026.4.1
```

Use `midealocal` ONLY for cloud API debugging (listing devices, checking cloud info). Use `msmart-ng` for everything else (discovery, authentication, HA integration). They share the `msmart` package namespace but have incompatible internal structures.

### Patching Cloud Credentials in the Integration

When the shared credentials expire, patch `config_flow.py` with the user's own MSmartHome credentials:

```bash
# ⚠️ SEE "qm guest exec Shell Escaping" pitfall below — ! in passwords is HARD
# The working method uses awk with sprintf("%c", 33) to produce a literal !

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 'qm guest exec 106 -- sh -c "
  awk \"{gsub(/OLD_EMAIL\\\\|OLD_PASS/, \\\"NEW_EMAIL\\\\\\\"\\\\, \\\"NEW_PASS\\\"); print}\" \
  /mnt/data/supervisor/homeassistant/custom_components/midea_ac/config_flow.py > /tmp/cf_fixed && \
  mv /tmp/cf_fixed /mnt/data/supervisor/homeassistant/custom_components/midea_ac/config_flow.py
"'
# Then restart HA to load the patched integration
```

### This Homelab's Midea AC

- **Device**: "OGKlimaanlage" (ID 153931628805781, type 172/AC)
- **IP**: 10.0.40.54 (VLAN 40)
- **SN**: 000000P0000000Q1102C8D344AAE0000
- **Model**: 00000Q11, subtype 524
- **Protocol**: V3
- **Credentials**: dominik@familie-schoen.com (MSmartHome app) — stored in 1Password (Vault "Hermes", item "Midea MSmartHome", ID jzlkqibo44e6nz6gtvcqf7dg6q)
- **Status (2026-06-27)**: Device responds to UDP discovery but cloud token retrieval fails for ALL udpid methods (0=REVERSED_BIG, 1=BIG, 2=LITTLE, plus btMac-derived). Cloud API returns `"code":"3004","msg":"value is illegal"` consistently. Device IS connected via WLAN and remotely controllable through MSmartHome app — but cloud has no LAN token provisioned. Default keys from `midealocal.DEFAULT_KEYS` (method 99) also fail authentication ("Error packet received"). **Resolution**: re-pair device via WLAN setup in MSmartHome app (not BLE) to generate a LAN token, then use manual config flow.

## qm guest exec Shell Escaping Pitfalls (CRITICAL)

Editing files inside HAOS via `qm guest exec` is extremely painful due to shell escaping. Key issues:

### HAOS Has Minimal Tools

- **Available**: `sed`, `awk`, `sh`, `curl`, `grep`, `ha` (HAOS CLI)
- **NOT available**: `python3`, `perl`, `busybox`, `tcpdump`
- Cannot use Python heredocs or Perl one-liners for file manipulation

### The `!` Character Gets Mangled

The `!` in passwords (e.g. `qN4rF7ZtNL!Lex7R`) is interpreted by bash history expansion at EVERY layer of the SSH → qm → sh -c pipeline. Approaches that FAILED:
- `sed 's/x21/!/g'` → produces literal `x21` or gets eaten by shell
- `printf '%c' 33` → shell interprets `!` before printf sees it
- `perl -pi -e` → perl not available in HAOS
- `python3 -c` → python3 not available in HAOS
- Base64 round-trip → works for content but `!` still gets eaten during `echo | base64 -d`

**Working approach — `awk` with `sprintf`:**
```bash
# awk's sprintf("%c", 33) produces a literal ! without shell interpretation
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 'qm guest exec 106 -- sh -c "
  awk \"{gsub(/x21Lex7R/, sprintf(\\\"%c\\\",33) \\\"Lex7R\\\"); print}\" \
  /path/to/file > /tmp/fixed && mv /tmp/fixed /path/to/file && echo OK
"'
```

### HAOS Redacts Secrets in Output

When using `grep` or `cat` via `qm guest exec` on config files containing credentials, HAOS/the guest agent replaces credential-like strings with `***`. Example: `_CLOUD_CREDENTIALS=***` instead of showing the actual dict. This makes it **impossible to verify password patches via grep** — you must test functionality instead (e.g. trigger the config flow and check if it succeeds).

### General Strategy for File Edits in HAOS

1. **Prefer REST API** over file editing whenever possible
2. **Use `awk`** for substitutions involving special characters (!, @, $, etc.)
3. **Base64 for whole-file transfers** — read file via `qm guest exec -- cat`, modify locally, write back via base64. But beware: `!` in the base64-decoded content can still get eaten if passed through `echo`
4. **Test functionality** rather than verifying file contents via grep (redaction)
5. **SCP alternative**: if guest agent is working, `scp` directly to the VM may work (but HAOS doesn't run sshd by default)

## REST-Based Device Integration (Non-HA-Native Devices)

When a device has a local REST API but no native HA integration, it can be integrated via HA's `rest` sensor/switch platform. This was confirmed working with a Xenia espresso machine (ESP32-based controller).

### Case Study: Xenia Coffee Machine (10.0.50.128)

**Device**: Xenia Coffee (ESP32, firmware 3.9). Web UI at `http://10.0.50.128/` with Bootstrap frontend. No official HA integration.

**API Discovery**: The JS bundle (`/js/index_combined.js`) reveals REST endpoints. Use `curl -s --compressed http://10.0.50.128/js/index_combined.js | grep -oP '/api/v2/[a-z_]+'` to enumerate.

**Working API Endpoints**:
| Endpoint | Method | Returns |
|----------|--------|---------|
| `/api/v2/overview` | GET | Machine status: temps, pressure, power, extractions, energy |
| `/api/v2/machine` | GET | Machine config: type, firmware, serial, max ampere |
| `/api/v2/machine/control` | GET/POST | Machine control (power on/off) |
| `/api/v2/toggle_sb` | GET | Toggle steam boiler |
| `/api/v2/inc_dec` | GET | Increment/decrement parameter |
| `/get_all_params` | GET | All parameters |

**Key Fields** (from `/api/v2/overview`):
- `MA_STATUS`: 0=Off, 1=On
- `MA_CUR_PWR`: Current power in watts
- `BG_SENS_TEMP_A`: Brew group temperature (°C)
- `BB_SENS_TEMP_A`: Brew boiler temperature (°C)
- `MA_ENERGY_TOTAL_KWH`: Total energy counter
- `MA_EXTRACTIONS`: Shot counter

**HA Integration Status** (confirmed working 2026-06-29):
- `sensor.xenia_machine_data` — ✅ REST sensor polling `/api/v2/overview`
- `sensor.xenia_set_temperatures` — ✅ REST sensor for set temperatures
- `sensor.xenia_status` — ✅ Derived from `MA_STATUS` (On/Off)
- `sensor.brew_group_temperature` — ✅ Temperature sensor
- `sensor.brew_boiler_temperature` — ✅ Temperature sensor
- `switch.xenia_power_2` — ✅ REST switch (turn_on/turn_off both work, verified)
- `rest_command.xenia_machine_control` — ✅ REST command for machine control
- `sensor.xenia_current_power_2` — ❌ unavailable (broken REST sensor)
- `sensor.xenia_total_energy_2` — ❌ unavailable (broken REST sensor)
- `switch.xenia_power` — ❌ unavailable (old/orphaned entity, superseded by `_2` variant)

**Testing Power Control via HA API**:
```bash
# Turn on
curl -sk -X POST -H "Authorization: Bearer $HA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"entity_id":"switch.xenia_power_2"}' \
  "$HA_URL/api/services/switch/turn_on"
# Verify via device API
curl -s --compressed http://10.0.50.128/api/v2/overview | python3 -c "
import json,sys; d=json.load(sys.stdin)
print(f'MA_STATUS: {d[\"MA_STATUS\"]} (0=off, 1=on)')
"
```

## HAOS Advanced SSH Addon Access (Alternative to Proxmox Guest Agent)

When the QEMU guest agent is down or unavailable, the **Advanced SSH & Web Terminal** HAOS add-on provides direct shell access to the HA container and host. This is a more reliable access method than `qm guest exec`.

### Setup

1. **Install the add-on** via HA UI: Settings → Add-ons → Advanced SSH & Web Terminal
2. **Configure credentials** in the add-on options (set a password or SSH key)
3. **Start the add-on** — port 22 opens on the HA host (10.0.30.10:22)
4. **Connect**: `sshpass -p '<password>' ssh -o StrictHostKeyChecking=no hassio@10.0.30.10`

### Capabilities

- `sudo docker exec homeassistant python3 -c "..."` — run Python inside the HA container (has full Python3 with json/pathlib/etc.)
- `sudo docker exec homeassistant cat /config/configuration.yaml` — read HA config files
- `sudo cat /config/configuration.yaml` — read from host filesystem (HAOS mounts config at `/config/`)
- `sudo docker restart homeassistant` — restart HA
- `sudo docker exec homeassistant python3 /config/script.py` — run scripts placed in `/config/`

### Script Transfer Pattern (Local → HA Container)

Write a Python script locally, base64-encode it, transfer via SSH, and execute inside the container:

```bash
# 1. Write script locally
write_file /tmp/my_script.py "..."

# 2. Base64 encode and transfer
B64=$(base64 -w0 /tmp/my_script.py)
sshpass -p '<password>' ssh hassio@10.0.30.10 \
  "echo '$B64' | base64 -d | sudo tee /config/my_script.py > /dev/null && \
   sudo docker exec homeassistant python3 /config/my_script.py"
```

**Pitfall:** Base64 strings can get corrupted when embedded in SSH commands through the terminal tool. Always verify the decoded script content before execution: `echo "$B64" | base64 -d | grep "<expected_line>"`.

### This Homelab's SSH Addon Credentials

- **User**: `hassio`
- **Password**: Stored in 1Password (Vault "Hermes")
- **Port**: 22 on 10.0.30.10 (only open when the add-on is running)
- **Access level**: `sudo NOPASSWD` — full root access to the HAOS host and Docker containers

## Manual Device Creation in .storage/ (When YAML device_info Fails)

When `device_info` is not supported by the integration platform (confirmed for `rest` and `template`), you can manually create a device in the `.storage/core.device_registry` file and link entities to it via `.storage/core.entity_registry`. This requires SSH access to the HA container (see "HAOS Advanced SSH Addon Access" above).

### Step-by-Step Procedure

1. **Stop HA** (optional but safer) or proceed on a running instance
2. **Read the current device registry** to understand the schema:
   ```python
   import json, pathlib
   dr = json.loads(pathlib.Path('/config/.storage/core.device_registry').read_text())
   # Print an existing device to see all required fields
   print(json.dumps(dr['data']['devices'][0], indent=2))
   ```
3. **Create a new device entry** with ALL required fields (see pitfall above for the complete list). Use `uuid.uuid4()` for the `id`.
4. **Append to the devices list** and write back
5. **Link entities** by setting `device_id` on matching entities in `core.entity_registry`
6. **Restart HA** — `sudo docker restart homeassistant`
7. **Verify** via WebSocket API: `config/device_registry/list` should show the new device

### Working Script (Xenia Coffee Machine Example)

```python
import json, pathlib, uuid

# Create device
dr_path = pathlib.Path('/config/.storage/core.device_registry')
dr = json.loads(dr_path.read_text())
dev_id = str(uuid.uuid4())
new_device = {
    'config_entries': [],
    'configuration_url': 'http://10.0.50.128',
    'connections': [],
    'identifiers': [['xenia', 'coffee_machine']],
    'manufacturer': 'Xenia Coffee',
    'model': 'Xenia Espresso',
    'name': 'Xenia',
    'name_by_user': None,
    'serial_number': '313253795029',
    'sw_version': '3.9',
    'hw_version': None,
    'via_device_id': None,
    'area_id': None,
    'disabled_by': None,
    'entry_type': None,
    'id': dev_id,
    'config_entries_subentries': {},
    'created_at': '2026-07-10T15:30:00.000000+00:00',
    'modified_at': '2026-07-10T15:30:00.000000+00:00',
    'labels': [],
    'model_id': None,
    'primary_config_entry': None
}
dr['data']['devices'].append(new_device)
dr_path.write_text(json.dumps(dr, indent=2))

# Link entities
er_path = pathlib.Path('/config/.storage/core.entity_registry')
er = json.loads(er_path.read_text())
for e in er['data']['entities']:
    if 'xenia' in e.get('entity_id', '').lower():
        e['device_id'] = dev_id
er_path.write_text(json.dumps(er, indent=2))
```

### Modern rest: Integration Syntax (Legacy Conversion)

When migrating from legacy `sensor: - platform: rest` to modern `rest:` syntax (needed for some features), the structure changes:

**Legacy (does NOT support device_info):**
```yaml
sensor:
  - platform: rest
    name: xenia_machine_data
    resource: http://10.0.50.128/api/v2/overview
    method: GET
    value_template: >-
      {% if value_json is defined %}OK{% else %}Error{% endif %}
    json_attributes:
      - MA_STATUS
      - BG_SENS_TEMP_A
    scan_interval: 1
    timeout: 30
```

**Modern (also does NOT support device_info, but cleaner structure):**
```yaml
rest:
  - resource: http://10.0.50.128/api/v2/overview
    method: GET
    timeout: 30
    scan_interval: 1
    sensor:
      - name: "Xenia Machine Data"
        unique_id: xenia_machine_data
        value_template: >-
          {% if value_json is defined %}OK{% else %}Error{% endif %}
        json_attributes:
          - MA_STATUS
          - BG_SENS_TEMP_A
```

Note: `unique_id` is at the sensor level in modern syntax (not at the resource level). The `name` field uses a friendly name string. Template entities referencing these sensors (e.g. `{% set status = states('sensor.xenia_machine_data') %}`) continue to work unchanged as long as the `unique_id` stays the same.

## Alexa Exposure Troubleshooting (HA Cloud)

When a HA entity (switch, light, etc.) works via HA API/UI but cannot be controlled via Alexa voice commands:

### Root Causes

1. **Entity not exposed to Alexa**: HA Cloud (Nabu Casa) requires explicit entity exposure. REST-based switches with minimal attributes (only `friendly_name`, no `device_class`) are often NOT auto-exposed. Check: Settings → Cloud → Alexa → expose entities.

2. **Old/orphaned entity was previously exposed**: If an entity was renamed or recreated (e.g. `switch.xenia_power` → `switch.xenia_power_2`), Alexa may still reference the old entity ID. The old entity shows `unavailable` with `restored: true` — Alexa can't control it.

3. **Missing `device_class`**: REST switches created without `device_class: switch` may not be recognized by Alexa's smart home skill as a switch type.

### Diagnostic Workflow

1. **Check HA components**: `curl -sk -H "Authorization: Bearer $HA_TOKEN" "$HA_URL/api/config"` — look for `cloud` and `alexa` in components list.

2. **Check Alexa config entry**: Look for `alexa_devices` domain in config entries (this is the Alexa Media Player HACS integration, NOT the same as HA Cloud Alexa exposure).

3. **Check entity attributes**: An entity exposed to Alexa via HA Cloud typically has richer attributes. Bare-minimum attributes (only `friendly_name`) suggest the entity was NOT added to the Alexa exposure list.

4. **Check for orphaned entities**: Search `/api/states` for duplicate/near-duplicate entity names (e.g. `switch.xenia_power` and `switch.xenia_power_2`). The `restored: true` attribute on the old entity confirms it's orphaned.

### Fixes

1. **Expose entity via HA UI**: Settings → Cloud → Alexa → add `switch.xenia_power_2` to the exposure list. Then in Alexa app: "Devices → Discover" to re-detect.

2. **Use `switch_as_x` integration**: Convert the REST switch to a `switch_as_x` light entity (as done for `beleuchtung_beet_gabionen` in this homelab). Lights are more reliably exposed to Alexa than bare REST switches.

3. **Clean up orphaned entities**: Remove the old `switch.xenia_power` entity from HA entity registry to prevent Alexa confusion.

4. **Remove broken REST sensors**: `sensor.xenia_current_power_2` and `sensor.xenia_total_energy_2` show `unavailable` — their REST sensor configs need fixing or removal. Check the REST sensor configuration in `configuration.yaml` or `rest.sensor` config entries.

## MQTT Broker Diagnosis

When MQTT-dependent integrations (Frigate, Shelly, custom sensors) show no data or entities are missing, diagnose the MQTT broker and integration config:

### Workflow

1. **Check MQTT config entry via REST API**:
```python
import urllib.request, json
req = urllib.request.Request(
    f"{HA_URL}/api/config/config_entries/entry",
    headers={"Authorization": f"Bearer {token}"}
)
entries = json.loads(urllib.request.urlopen(req, timeout=10).read())
mqtt_entries = [e for e in entries if e.get('domain') == 'mqtt']
# Check: title (should be broker IP), state (should be 'loaded')
```

2. **Scan for active MQTT brokers** on known VLANs (1883 = plain, 8883 = TLS):
   - Use `execute_code` with `ThreadPoolExecutor(max_workers=128)` and `socket.connect_ex` with 0.3s timeout — scans a /24 in ~2s. Faster and more reliable than `nmap` (often not installed in CTs).
   - HAOS has 3 NICs (VLAN30/40/50) — the same broker appears on all 3 IPs. Don't be fooled into thinking there are multiple brokers.

3. **Test broker auth with raw MQTT CONNECT packet** (no mosquitto_pub/clients needed):
   - Build a minimal MQTT v3.1.1 CONNECT, send via socket, read 4-byte CONNACK.
   - Return codes: 0=Accepted, 1=Bad proto version, 2=Client ID rejected, 3=Server unavailable, 4=Bad user/pass, 5=Not authorized.
   - See `scripts/mqtt-broker-scan.py` for the reusable implementation.

4. **Count MQTT entities** via `/api/states` — if 0, the integration isn't connected even if the config entry says `loaded`.

### Critical Pitfall: Config Entry TITLE ≠ Actual Broker IP

The MQTT config entry's `title` field (shown in HA UI and REST API metadata) can be stale/misleading. In this homelab, the title says `"192.168.100.27"` (a dead pre-migration IP), but the actual `data.broker` field is `"127.0.0.1"`. **Always read the full `.storage/core.config_entries` data, not just the REST API metadata**, which omits the `data` field.

```bash
# Read the ACTUAL broker config (includes credentials)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  "qm guest exec 106 -- /bin/sh -c 'docker exec homeassistant python3 -c \"
import json
with open(\\\"/config/.storage/core.config_entries\\\") as f:
    data = json.load(f)
for e in data[\\\"data\\\"][\\\"entries\\\"]:
    if e.get(\\\"domain\\\") == \\\"mqtt\\\":
        print(e[\\\"data\\\"])
\"'"
# Output: {'broker': '127.0.0.1', 'password': '...', 'port': 1883, 'protocol': '5', 'username': 'opendtu'}
```

### Critical Pitfall: MQTT Discovery Entities Have NO `mqtt.` Prefix

Searching for entities with `mqtt.` prefix or `integration=mqtt` attribute returns ZERO even when MQTT works perfectly. MQTT discovery creates entities named by the discovery topic — e.g. `sensor.sn_3006252807_pv_current_a`, `sensor.opendtu_4676e0_ac_power`, `binary_sensor.frigate_einfahrt_person_occupancy`. These have NO integration attribute in HA 2026.6.4.

**Correct way to verify MQTT is working:**
1. Subscribe to `homeassistant/#` discovery topics via `mosquitto_sub` inside the Mosquitto add-on container
2. Subscribe to data topics (e.g. `hoymiles/#`, `frigate/#`) to see live data flowing
3. Check entity `last_changed` timestamps — if recent, MQTT is delivering data

```bash
# Verify MQTT discovery + data flow (run inside Mosquitto add-on container)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  "qm guest exec 106 -- /bin/sh -c 'docker exec addon_core_mosquitto timeout 5 \
   mosquitto_sub -h 127.0.0.1 -p 1883 -u USER -P PASS -t \"homeassistant/#\" -C 5 -W 3 -v'"
```

### Common Misconfiguration: Dead IP from Network Migration

`192.168.100.27` is a pre-migration IP that keeps appearing in HA config entry titles. Both NUT and MQTT entries had this stale title. **However, the MQTT entry's actual `data.broker` was already `127.0.0.1` — only the title was stale.** After any network migration, audit config entries by reading the `.storage/` data, not just the REST API metadata.

### This Homelab's MQTT Setup (verified working 2026-06-28)

- **Broker**: Mosquitto add-on (`addon_core_mosquitto`, v7.1.0) on HAOS, listening on `127.0.0.1:1883` (also externally on `10.0.30.10:1883`, `10.0.40.61:1883`, `10.0.50.127:1883` — same VM, 3 NICs)
- **Authentication**: Required. MQTT user `opendtu` with password `28acaneltO!#` (stored in HA `.storage/core.config_entries`)
- **MQTT integration config entry**: Title `192.168.100.27` (STALE — cosmetic only), actual `data.broker` = `127.0.0.1:1883`, protocol v5. State `loaded`, working correctly.
- **116+ MQTT-discovered entities**: 103 `sn_*` (Hoymiles/SMA inverters) + 13 `opendtu_*` (OpenDTU diagnostics) — all updating live
- **Frigate MQTT**: Uses `frigate` user with credentials in `/config/config.yml` on CT 120

### OpenDTU / Hoymiles Microinverter MQTT Topology

OpenDTU (ESP32, IP `10.0.40.55`, firmware v25.5.10) publishes Hoymiles microinverter data via MQTT:

- **Discovery topics**: `homeassistant/sensor/OpenDTU-4676E0_*/config`
- **Data topics**: `hoymiles/<serial>/#` (name, status, AC/DC power, voltage, current, yield, temperature, radio stats)
- **3 Hoymiles inverters**:
  - HM-1500 (serial 116183069389) — 1500W microinverter
  - Schuppen 1 (serial 138290602874)
  - Schuppen 2 (serial 138290602981)
- **SN_ entities** in HA (`sensor.sn_3006252807_*`, `sensor.sn_3006252817_*`, `sensor.sn_3007105790_*`) are created via MQTT discovery from OpenDTU. Serials 3006252807/3006252817 correspond to the Hoymiles inverters; 3007105790 is the SMA SunnyBoyStorage with battery (also exposed via OpenDTU or SMA Speedwire integration).

### Separate Non-MQTT PV Integrations

These are NOT via MQTT — separate HA integrations:
- **SMA SunnyBoy L1** (10.0.50.201) — `sma` integration (Speedwire)
- **SMA SunnyBoy L2** (10.0.50.202) — `sma` integration
- **SMA SunnyBoyStorage** (10.0.50.203) — `sma` integration
- **SMA EV Charger** (192.168.100.95) — `smaev` integration

## Galera Database Credential Discovery (Recipes DB)

The `maxscale` monitoring user can `SHOW DATABASES` but has **no data access** (`GRANT SHOW DATABASES, SUPER, REPLICATION SLAVE, BINLOG MONITOR ON *.*`). When you need to query the recipes DB directly (count rows, verify data, debug the MCP server):

### Credential Chain

1. **Try `maxscale` user** → can list databases but CANNOT `USE recipes` (Access denied 1044).
2. **`root` user** → Access denied 1045 (MaxScale doesn't route root connections).
3. **Actual app user** → Check the MCP server config in `~/.hermes/config.yaml` under `mcp_servers.recipes.env`:
   - `RECIPES_DB_HOST` = `10.0.30.70`
   - `RECIPES_DB_USER` = `recipe_app`
   - `RECIPES_DB_PASS` = stored in plaintext in config.yaml env section
   - `RECIPES_DB_NAME` = `recipes`

### Working Connection

```python
import pymysql
conn = pymysql.connect(
    host='10.0.30.70', port=3306,
    user='recipe_app', password='<from config.yaml>',
    database='recipes', charset='utf8mb4',
)
```

### Pitfalls

- **1Password item `mariadb-galera-vm` stores infrastructure passwords** (root, maxscale, mariabackup, ha-recorder) but NOT the `recipe_app` user password. That lives only in the MCP server's env config.
- **The `recipe_app` user has full CRUD on `recipes` DB** — use read-only queries unless explicitly modifying data.
- **Recipes DB tables**: `recipes`, `ingredients`, `recipe_ingredient_links`, `recipe_semantics` (embeddings), `meal_decisions`, `preference_rules`, `ingredient_substitutes`.
- **No `language` column** in recipes table — translations are stored inline in title/description, not as a separate field.
- **`scraped_at` column** tracks ingestion date (range observed: 2026-06-18 → 2026-07-09).

## HA Browser Console Access (No Local Token)

When no HA API token is available locally (`.env` missing, token redacted after compaction, 1Password inaccessible), the HA web UI's browser session can be used to query the REST API directly. Log in via `browser_navigate`, then extract the bearer token from `window.hassConnection` (a Promise — must `await` it; token at `conn.auth.data.access_token`). Use `browser_console` with `fetch('/api/states', {headers: {'Authorization': 'Bearer ' + token}})` or `/api/history/period/...` for time-range queries.

### Token Extraction — Two Methods

**Method 1: `window.hassConnection` (fresh page load)**
```javascript
(async () => {
  const conn = await window.hassConnection;  // It's a Promise!
  const token = conn.auth.data.access_token;  // snake_case key!
  return 'Token len: ' + token.length;
})()
```

**Method 2: DOM element (after re-login / stale session)**
```javascript
(async () => {
  const ha = document.querySelector("home-assistant");
  // Wait for hass to be available (page may still be loading)
  for (let i = 0; i < 30; i++) {
    if (ha?.hass?.connection) break;
    await new Promise(r => setTimeout(r, 1000));
  }
  const token = ha.hass.auth.accessToken;  // Direct property, camelCase getter
  return 'Token len: ' + token.length;
})()
```

**When to use which:**
| Scenario | Method |
|----------|--------|
| Fresh page load, quick query | `window.hassConnection` (Method 1) |
| After re-login, stale session | DOM element (Method 2) |
| `hassConnection` returns empty `{}` | DOM element (Method 2) |

**Key details:**
- `window.hassConnection` is a **Promise** — must `await` it.
- Token key in `hassConnection` path: `auth.data.access_token` (snake_case).
- Token key in DOM path: `hass.auth.accessToken` (camelCase getter).
- `hass.auth` has keys: `["data", "_saveTokens"]`.
- `hass.connection.auth` is NOT populated — use `hass.auth` instead.
- After re-login, the page may take 5-10s to fully load `home-assistant` element. Poll for `ha.hass.connection` availability before accessing.

See `references/ha-browser-console-access.md` for full technique, history query patterns, and pitfalls.

## Energy/Power Analysis (Nighttime Grid Import Diagnosis)

To diagnose unexpected power consumption (e.g. nighttime grid draw):

1. **Confirm anomaly** — query `sensor.evcc_grid_power` history in 30-min buckets.
2. **Decompose** — `grid = home - pv - battery_discharge + battery_charge`. Query all four.
3. **Battery SOC trajectory** — both batteries (SMA `sn_3007105790_battery_soc_total`, Marstek `marstek_venus_modbus_batterie_ladezustand`). If drained overnight → grid fills the gap.
4. **Identify consumer** — query large consumers individually (heat pump `shellypro3em_08f9e0e98b9c_total_active_power`, sauna, EV charger, dryer).
5. **Heat pump correlation** — `cu401b_s_kompressorphase` (off/preparing/heating/pause), `cu401b_s_verdichtermodulation` (%), `cu401b_s_ww_speichertemperatur` (WW tank temp). A WW heating cycle can draw 5kW+ for 10-30 min.

Common root cause: **heat pump WW heating at night** drains batteries → remaining overnight demand exceeds depleted battery output → grid import. Prevention: shift WW schedule to PV hours (10–15h), increase hysteresis, ensure batteries start night at 100%.

See `references/ha-energy-analysis.md` for the complete sensor inventory, query templates, and prevention strategies.

## Deleting Dashboards

To delete a dashboard (removes it from the sidebar and deletes its config):

```javascript
await conn.sendMessagePromise({
  type: 'lovelace/dashboards/delete',
  dashboard_id: 'dashboard_test'  // ← uses underscore ID, NOT url_path
});
// Returns: null on success
```

**Key:** `dashboard_id` uses underscores (`dashboard_test`), while `lovelace/config` uses `url_path` with hyphens (`dashboard-test`). The `lovelace/dashboards/list` response provides both `id` and `url_path` fields — use `id` for deletion.

## Listing Lovelace Resources (Custom Cards)

To check which custom card resources are registered (after HACS install or for auditing):

```javascript
const resources = await conn.sendMessagePromise({type: 'lovelace/resources'});
// Returns: [{id, url, type}]
// Example URLs: /hacsfiles/lovelace-mushroom/mushroom.js?hacstag=444350375511
```

HACS auto-registers installed cards as resources. If a custom card type doesn't render after install, check this list — the resource URL may be missing or malformed.

## HA Entity Inventory via Browser Console

For dashboard design or automation planning, quickly inventory all entities grouped by domain:

```javascript
(async () => {
  const ha = document.querySelector("home-assistant");
  const states = ha.hass.states;
  const domains = {};
  for (const [eid, s] of Object.entries(states)) {
    const domain = eid.split('.')[0];
    if (!domains[domain]) domains[domain] = [];
    domains[domain].push(eid);
  }
  // Summary counts + entity lists for key domains
  const summary = {};
  for (const d of ['light', 'cover', 'climate', 'switch', 'sensor', 'binary_sensor',
                   'camera', 'lock', 'media_player', 'scene', 'automation']) {
    summary[d] = domains[d]?.length || 0;
  }
  return JSON.stringify({summary, covers: domains.cover, lights: domains.light,
                        climates: domains.climate, cameras: domains.camera}, null, 2);
})()
```

This gives instant visibility into what's available for dashboard building without scrolling the HA UI.

## HACS Repository Management via WebSocket

HACS exposes a **partial WebSocket API** for repository listing, release browsing, and installation. The UI (iframe-based, heavy shadow DOM) is extremely hard to automate via browser tools — prefer the WebSocket API.

### Working Commands

```javascript
const conn = ha.hass.connection;

// 1. List ALL repositories (installed + available)
const repos = await conn.sendMessagePromise({type: 'hacs/repositories/list'});
// Returns: [{id, name, full_name, category, installed, can_download, installed_version, available_version, ...}]
// Filter: repos.filter(r => r.installed)  — installed repos only
// Filter: repos.filter(r => r.name.toLowerCase().includes('mushroom'))  — search by name

// 2. List releases for a specific repository
const releases = await conn.sendMessagePromise({
  type: 'hacs/repository/releases',
  repository: '444350375'  // ← repository ID (string), NOT repository_id
});
// Returns: [{name, tag, published_at, prerelease}]

// 3. Install/download a repository
await conn.sendMessagePromise({
  type: 'hacs/repository/download',
  repository: '444350375'  // ← repository ID (string), NOT repository_id
});
// Returns: {} on success. Installation is async — resources appear within seconds.
```

### Parameter Naming (CRITICAL)

| Command | Correct key | Wrong key | Error if wrong |
|---------|------------|-----------|----------------|
| `hacs/repository/releases` | `repository` | `repository_id` | `"extra keys not allowed @ data['repository_id']"` |
| `hacs/repository/download` | `repository` | `repository_id` | `"extra keys not allowed @ data['repository_id']"` |

### Verifying Installation

After `hacs/repository/download`, verify via Lovelace resources:

```javascript
const resources = await conn.sendMessagePromise({type: 'lovelace/resources'});
// Returns: [{id, url, type}]
// HACS auto-registers installed cards as resources:
// /hacsfiles/lovelace-mushroom/mushroom.js?hacstag=444350375511
// /hacsfiles/Bubble-Card/bubble-card.js?hacstag=680112919324
```

### Finding Repository IDs

Use `hacs/repositories/list` and filter by name or `full_name`:
```javascript
const repos = await conn.sendMessagePromise({type: 'hacs/repositories/list'});
const mushroom = repos.find(r => r.full_name === 'piitaya/lovelace-mushroom');
// mushroom.id = '444350375'
```

### Commands That DON'T Work

- `hacs/repos` → `unknown_command` (wrong command name — use `hacs/repositories/list`)
- `hacs/status` → `unknown_command`
- `hacs/repository/install` → `unknown_command` (use `hacs/repository/download` instead)
- `hacs/repository` → `unknown_command`

### HACS UI Automation Pitfall

The HACS dashboard (`/hacs/entry`) renders inside an iframe with heavy shadow DOM. `browser_click` on search results and download buttons is unreliable — elements appear in the accessibility tree but clicks don't reach them. The WebSocket API is the reliable path for programmatic installation.

## Bubble Card Pop-up Migration (v3.2.0+ Standalone Format)

Since Bubble Card v3.2.0, pop-ups must be migrated from the legacy format (cards as siblings in a vertical-stack) to the standalone format (cards nested inside the pop-up's `cards:` property). The migration notice appears in the HA UI when legacy pop-ups are detected.

### Legacy vs Standalone Structure

**Legacy (pre-v3.2.0):** Pop-up card is a sibling of content cards in the view:
```yaml
views:
  - cards:
      - type: custom:bubble-card        # pop-up header only
        card_type: pop-up
        hash: "#home"
      - type: custom:mushroom-template-card  # content card (sibling)
        ...
      - type: grid                            # more content (sibling)
        ...
      - type: custom:bubble-card             # navigation (sibling)
        card_type: bottom-navigation
```

**Standalone (v3.2.0+):** Content cards are nested INSIDE the pop-up:
```yaml
views:
  - cards:
      - type: custom:bubble-card
        card_type: pop-up
        hash: "#home"
        cards:                               # ← content moved HERE
          - type: custom:mushroom-template-card
            ...
          - type: grid
            ...
      - type: custom:bubble-card             # navigation stays sibling
        card_type: bottom-navigation
```

### Migration Script (WebSocket)

```javascript
(async () => {
  const ha = document.querySelector("home-assistant");
  const conn = ha.hass.connection;

  const cfg = await conn.sendMessagePromise({type: 'lovelace/config', url_path: 'dashboard-haus'});

  for (const view of cfg.views) {
    const cards = view.cards || [];
    const popupIdx = cards.findIndex(c => c.card_type === 'pop-up');
    if (popupIdx === -1) continue;

    const popup = cards[popupIdx];
    const navIdx = cards.findIndex(c => c.card_type === 'bottom-navigation');

    // Content cards = everything EXCEPT pop-up and bottom-nav
    const contentCards = cards.filter((c, i) => i !== popupIdx && i !== navIdx);

    // Move content INTO the pop-up
    popup.cards = contentCards;

    // Rebuild: [pop-up (with content), bottom-nav]
    const newCards = [popup];
    if (navIdx !== -1) newCards.push(cards[navIdx]);

    view.cards = newCards;
  }

  await conn.sendMessagePromise({
    type: 'lovelace/config/save',
    url_path: 'dashboard-haus',
    config: cfg
  });
})();
```

### Key Points

- **Bottom-navigation stays OUTSIDE** the pop-up — it's global nav, not pop-up content.
- **All other content cards move INSIDE** the pop-up's `cards:` property.
- **Verify after migration:** Check `cfg.views[].cards[].cards` exists and has the expected card count. Zero render errors confirms success.
- **The migration is reversible** — move cards back out of the pop-up to revert.
- Migration can be done via the HA UI editor ("Migrate to standalone" button) or via YAML/WebSocket as above.

## HA Session Expiry Handling (Browser)

HA browser sessions expire frequently (observed: every ~15-20 min of active WebSocket use). Pattern for long-running browser sessions:

1. **Detect:** `browser_console` returns `TypeError: Cannot read properties of null (reading 'hass')` or `ha.hass` is undefined.
2. **Re-login:** Navigate to `http://10.0.30.10:8123`, fill username (`@e8`) + password (`@e9`), click Log in (`@e6`).
3. **Wait for ready:** Poll `document.querySelector("home-assistant")?.hass?.connection` for up to 30s.
4. **Resume** the interrupted WebSocket operation.

This happened 3+ times during the dashboard redesign session. Build scripts to be resilient: wrap WebSocket calls in a function that detects null `hass` and re-logs in automatically.

## HA Lovelace Dashboard Enumeration

Dashboards are enumerated and managed via the **WebSocket API only**. REST endpoints (`/api/lovelace/config`, `/api/config/dashboard/config`) return 404.

```javascript
// List all dashboards
const dashboards = await conn.sendMessagePromise({type: 'lovelace/dashboards/list'});
// Returns: [{id, title, url_path, icon, mode, show_in_sidebar}]

// Read dashboard config (use url_path, NOT id)
const cfg = await conn.sendMessagePromise({type: 'lovelace/config', url_path: 'dashboard-haus'});
// Returns: {title, views: [{title, path, icon, cards: [...]}]}
```

See `references/ha-lovelace-dashboard-api.md` for full enumeration, card auditing, and write-back patterns.

- `references/ha-browser-console-access.md` — Access HA REST API via browser console when no local token is available (extract token from `window.hassConnection`, use `fetch()` in `browser_console`)
- `references/ha-lovelace-dashboard-api.md` — Enumerate and modify HA Lovelace dashboards via WebSocket API (REST endpoints return 404). Dashboard listing, config reading, card auditing, and write-back patterns.
- `references/marstek-cascade-automation.md` — Marstek Kaskaden-Steuerung (Modbus) HA automation: full YAML, control entities, feedback-loop bug analysis (setpoint vs measured in `echter_bedarf`), fix, and how to retrieve automation YAML via REST API
- `references/ha-energy-analysis.md` — Energy/power analysis methodology: sensor inventory, nighttime grid import diagnosis, heat pump WW scheduling, battery SOC management
- `references/ha-health-audit.md` — Proactive HA health audit: 6-step methodology (config metadata, entity stats, unavailable grouping, config-entry states, automation status, staleness) producing a prioritized improvement list. Use when the user asks "what can I improve?" rather than diagnosing a specific outage.
- `references/ha-dashboard-redesign.md` — HA dashboard redesign brainstorming session: entity inventory, mobile-first Mushroom+Bubble design, dashboard cleanup plan, approved view structure
- `references/bubble-card-popup-migration.md` — Bubble Card v3.2.0 pop-up migration: legacy→standalone format, WebSocket migration script, verification, session expiry pitfall

- `references/ha-health-audit-2026-07-12.md` — Proactive HA health audit snapshot: entity health, dead integrations, setup_retry entries, automations off, cleanup priorities. Baseline for comparing future audits.
- `references/ha-dead-integration-cleanup.md` — Dead integration cleanup: decision matrix, WebSocket bulk entity removal, MQTT discovery topic clearing, before/after verification, worked example (2026-07-12)
- `references/ha-debugging.md` — HA history/recorder troubleshooting checklist, API endpoints, common fixes (SQLite + MariaDB)
- `references/ha-recorder-silent-drop.md` — HA Recorder silently dropping DB connection without errors: diagnosis via recorder_runs/states timestamps, fix with ha core restart
- `references/galera-cluster-recovery.md` — Galera cluster crash recovery: bootstrap, sequential SST, MaxScale routing, HA recorder reconnection
- `references/ha-integration-config-editing.md` — Editing HA integration config entries directly in `.storage/` (NUT/UPS IP migration walkthrough)
- `references/frigate-troubleshooting.md` — Frigate NVR: CT 120 on n5pro (10.0.20.91), AMD GPU migration (OpenVINO Intel-only, CPU fallback), camera/MQTT verification, conflicting HA addon, recovery steps
- `references/midea-ac-debugging.md` — Midea AC integration: cloud credential flow, token/k1 retrieval, library comparison, config flow patching
- `templates/evcc-custom-charger-ha-bridge.yaml` — Minimal EVCC config with only the custom charger bridged through HA REST API (SMA EV Charger example)
- `templates/evcc-full-config.yaml` — Full EVCC config including grid meter, 3 PV inverters, 2 batteries, custom charger via HA, site config, and Tibber tariff
- `references/1password-credential-extraction.md` — 1Password `op item get` fallback patterns when `--fields credential --format plain` returns empty
- `references/xenia-rest-api.md` — Xenia coffee machine REST API: endpoints, HA integration status, Alexa exposure issue, broken entity inventory
- `references/evcc-device-inventory.md` — Complete device inventory for this homelab's EVCC setup: IPs, templates, Modbus registers, diagnostic commands, config file locations
- `scripts/verify-ha-recorder.sh` — Verify HA recorder/history/logbook status via REST API
- `scripts/mqtt-broker-scan.py` — Scan VLANs for MQTT brokers and test auth via raw MQTT CONNECT/CONNACK packets (no mosquitto clients needed)
