Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,134 @@
+# Galera Performance Tuning Session (June 29, 2026)
+
+## Problem
+
+HA state history for "Stromverbrauch Haus netto" took very long to load. User suspected MySQL was too slow.
+
+## Diagnosis
+
+### Root Cause: NOT Galera itself — HA `states` table explosion
+
+| Finding | Value |
+|---------|-------|
+| Registered HA entities | 3,416 (1,860 sensors) |
+| `states` table size | 16,338 MB (16.3 GB) |
+| `states` table rows | 61,378,779 |
+| Total HA DB size | 18.31 GB |
+| States per entity (30d) | ~2,049,022 for one high-frequency sensor |
+| `wsrep_local_recv_queue_avg` | 7.36 (should be < 1) |
+| `innodb_buffer_pool_size` | 4 GB (too small for 16 GB states table) |
+| `innodb_io_capacity` | 200 (HDD-tier, should be 2000+) |
+| `wsrep_slave_threads` | 4 (insufficient for write load) |
+| `gcache.size` | 128 MB (too small, risks expensive SST on rejoin) |
+
+### Query Performance BEFORE Tuning
+
+| Time Range | Rows | Duration |
+|------------|------|----------|
+| 24h | 79,062 | 0.18s |
+| 7 days | 220,340 | **10.1s** |
+| 30 days | 2,049,022 | **60.0s** |
+
+Index `ix_states_metadata_id_last_updated_ts` was used correctly but the sheer row count made even indexed scans slow.
+
+### How to Connect for Diagnosis
+
+1. **MySQL access via MaxScale VIP** using `maxscale` user (has `SUPER` privilege):
+```python
+import pymysql
+conn = pymysql.connect(host="10.0.30.70", port=3306,
+    user="maxscale", password="<maxscale-password>", charset="utf8mb4")
+```
+
+2. **HA database access** using `ha_recorder` user:
+```python
+conn = pymysql.connect(host="10.0.30.70", port=3306,
+    user="ha_recorder", password="<ha-recorder-password>",
+    database="homeassistant", charset="utf8mb4")
+```
+
+3. **Credentials source**: 1Password item `mariadb-galera-vm`, vault `Hermes`:
+```bash
+op item get "mariadb-galera-vm" --vault "Hermes" --reveal \
+  --fields maxscale-password,ha-recorder-password,root-password
+```
+
+4. **HAOS SSH** (port 22222) for HA-side investigation:
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
+# Inside: docker exec homeassistant <command>
+```
+
+## Changes Applied (SET GLOBAL, instant effect)
+
+All 9 parameters changed successfully via `maxscale` MySQL user through MaxScale VIP — no SSH required:
+
+| Parameter | Before | After | Effect |
+|-----------|--------|-------|--------|
+| `wsrep_slave_threads` | 4 | 16 | More applier threads for replication backlog |
+| `innodb_io_capacity` | 200 | 2000 | SSD-appropriate write throughput |
+| `innodb_io_capacity_max` | 2000 | 4000 | Burst capacity |
+| `innodb_read_io_threads` | 4 | 8 | Double read parallelism |
+| `innodb_write_io_threads` | 4 | 8 | Double write parallelism |
+| `innodb_flush_neighbors` | 1 | 0 | Remove HDD-era coalescing on SSD |
+| `innodb_purge_threads` | 4 | 8 | Faster cleanup of deleted rows |
+| `innodb_adaptive_hash_index` | OFF | ON | Speed up point lookups on large tables |
+| `long_query_time` | 10 | 2 | Log slow queries for diagnosis |
+
+## Query Performance AFTER Tuning
+
+| Time Range | Rows | Before | After | Improvement |
+|------------|------|--------|-------|-------------|
+| 24h | 79,130 | 0.18s | 0.02s | **9×** |
+| 7 days | 220,408 | 10.1s | 0.05s | **200×** |
+| 30 days | 2,049,090 | 60.0s | 0.39s | **154×** |
+| SELECT + JOIN LIMIT 1000 | 1000 | — | 0.01s | — |
+| SELECT + JOIN LIMIT 500 | 500 | — | 0.008s | — |
+
+## Parameters Requiring Restart (Not Yet Changed)
+
+These cannot be changed via `SET GLOBAL` and require editing the server config + restarting each node:
+
+| Parameter | Current | Target | Notes |
+|-----------|---------|--------|-------|
+| `innodb_buffer_pool_size` | 4 GB | 8-16 GB | Must fit working set (states table = 16 GB) |
+| `innodb_log_file_size` | 96 MB | 512 MB | Small redo log throttles write throughput |
+| `gcache.size` | 128 MB | 1 GB | Prevents expensive SST on node rejoin |
+| `innodb_buffer_pool_instances` | unknown | 4-8 | Should match buffer pool / 1GB |
+
+## Making Changes Permanent
+
+`SET GLOBAL` changes are lost on MariaDB restart. To persist, SSH to each Galera node and append to server config:
+
+```bash
+ssh debian@10.0.30.71
+sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
+[mariadb]
+wsrep_slave_threads=16
+innodb_io_capacity=2000
+innodb_io_capacity_max=4000
+innodb_read_io_threads=8
+innodb_write_io_threads=8
+innodb_flush_neighbors=0
+innodb_purge_threads=8
+innodb_adaptive_hash_index=ON
+long_query_time=2
+innodb_buffer_pool_size=8G
+innodb_log_file_size=512M
+wsrep_provider_options="gcache.size=1G"
+EOF
+sudo systemctl restart mariadb
+```
+Repeat on all 3 nodes sequentially (verify `wsrep_local_state_comment=Synced` before next node).
+
+## HA Recorder Recommendations (Not Applied — User Declined)
+
+User was offered but declined recorder config changes:
+- `recorder.exclude` for noisy domains (automation, button, select, update)
+- `recorder.commit_interval: 5` (reduce write frequency from 1s to 5s)
+- Move long-term history to InfluxDB (already configured at 10.0.30.109:8086, bucket `ha_hot_90d`)
+- `OPTIMIZE TABLE states` to reclaim 619 MB fragmented space
+
+## Key Insight
+
+The `maxscale` MySQL user has `SUPER` privilege and can run `SET GLOBAL` through the MaxScale VIP. This enables rapid Galera tuning without SSH access to the individual nodes — critical when SSH keys aren't set up for the DB VMs but 1Password credentials are available.