dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/mariadb-galera-cluster-administration/references/galera-cluster-topology.md
+92
View File
@@ -0,0 +1,92 @@
# Homelab Galera/MaxScale Topology
## MariaDB Galera Cluster (3 nodes)
| Node | VM ID | Proxmox Host | IP | Hostname |
|------|-------|---------------|----|----------|
| db1 | 300 | proxmox2 (10.0.20.20) | 10.0.30.71 | mariadb-01 |
| db2 | 301 | proxmox6 (10.0.20.60) | 10.0.30.72 | mariadb-02 |
| db3 | 302 | proxmox4 (10.0.20.40) | 10.0.30.73 | mariadb-03 |
- MariaDB version: 11.4.10-MariaDB-deb12
- Cluster name: `mariadb-galera`
- Galera address: `gcomm://10.0.30.71,10.0.30.72,10.0.30.73`
- DB size: ~18GB (homeassistant schema)
- **Note**: PVE HA may migrate VMs. Always verify current placement
with `qm list` on each node before operating.
## MaxScale Proxy (1 active + VIP)
| Node | VM ID | Proxmox Host | IP | Hostname | Status |
|------|-------|--------------|----|----------|--------|
| maxscale-01 | 310 | proxmox4 (10.0.20.40) | 10.0.30.81 | maxscale-01 | Active |
| VIP | — | — | 10.0.30.70 | keepalived VIP | — |
- MaxScale version: 24.02.9
- Services: `rw-router` (readwritesplit), `ro-router` (readconnroute)
- Monitor: `galera-monitor` (galeramon)
- Listener: `rw-listener` on port 3306
- **Note**: A second MaxScale VM (VM501 "MaxScale1") exists on
proxmox4 but is **stopped**. If a standby MaxScale is needed, check
VM501's config. The VIP `10.0.30.70` is the authoritative entry point.
## SSH Access
```bash
# MariaDB VMs (user: debian)
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.72
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73
# MaxScale VMs (user: debian)
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81
# Proxmox hosts (user: root)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 # proxmox1 (HA VM host)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 # n5pro
```
## Credentials
All credentials stored in 1Password Vault "Hermes", item `mariadb-galera-vm`:
- `root-password` — MySQL root (localhost only on each node, not accessible via MaxScale VIP)
- `maxscale-password` — MaxScale MySQL user with `SUPER` privilege — **can run `SET GLOBAL` via VIP** (see Section 6)
- `mariabackup-password` — Galera SST transfer auth
- `ha-recorder-password` — HA Recorder MySQL user, has SELECT/INSERT/UPDATE on `homeassistant` DB
- `keepalived-auth-pass` — keepalived VIP failover auth
Retrieve all fields:
```bash
op item get "mariadb-galera-vm" --vault "Hermes" --reveal --fields password,root-password,mariabackup-password,maxscale-password,ha-recorder-password
```
Note: The generic `password` field is empty. Use the named fields instead.
## Firewall (UFW on MariaDB VMs)
- Port 3306: ALLOW from 10.0.30.81, 10.0.30.82 (MaxScale only)
- Ports 4567, 4568, 4444: ALLOW between Galera nodes (10.0.30.71-73)
## Home Assistant
- VM 106 on proxmox1 (10.0.20.10)
- Internal: http://10.0.30.10:8123
- External: https://homeassistant.familie-schoen.com
- Config: `/mnt/data/supervisor/homeassistant/configuration.yaml`
- Recorder DB URL: `mysql://ha_recorder:***@10.0.30.70:3306/homeassistant`
- InfluxDB: 10.0.30.109:8086, bucket `ha_hot_90d` (long-term history)
- HA masks passwords with `***` in configuration.yaml — retrieve real passwords from 1Password, not the config file
### HAOS SSH Access (Port 22222)
HAOS exposes SSH on port 22222 (requires SSH addon enabled):
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
```
Inside HAOS, HA runs as Docker container `homeassistant`:
```bash
docker exec homeassistant cat /config/configuration.yaml
docker exec homeassistant python3 -c "..."
docker logs homeassistant 2>&1 | grep -i recorder | tail -20
```
Note: `pip` modules like `pymysql` are NOT available inside the HA container. Use `sqlalchemy` (bundled with HA) or query the DB from outside.
devops/mariadb-galera-cluster-administration/references/galera-performance-tuning.md
+134
View File
@@ -0,0 +1,134 @@
# Galera Performance Tuning Session (June 29, 2026)
## Problem
HA state history for "Stromverbrauch Haus netto" took very long to load. User suspected MySQL was too slow.
## Diagnosis
### Root Cause: NOT Galera itself — HA `states` table explosion
| Finding | Value |
|---------|-------|
| Registered HA entities | 3,416 (1,860 sensors) |
| `states` table size | 16,338 MB (16.3 GB) |
| `states` table rows | 61,378,779 |
| Total HA DB size | 18.31 GB |
| States per entity (30d) | ~2,049,022 for one high-frequency sensor |
| `wsrep_local_recv_queue_avg` | 7.36 (should be < 1) |
| `innodb_buffer_pool_size` | 4 GB (too small for 16 GB states table) |
| `innodb_io_capacity` | 200 (HDD-tier, should be 2000+) |
| `wsrep_slave_threads` | 4 (insufficient for write load) |
| `gcache.size` | 128 MB (too small, risks expensive SST on rejoin) |
### Query Performance BEFORE Tuning
| Time Range | Rows | Duration |
|------------|------|----------|
| 24h | 79,062 | 0.18s |
| 7 days | 220,340 | **10.1s** |
| 30 days | 2,049,022 | **60.0s** |
Index `ix_states_metadata_id_last_updated_ts` was used correctly but the sheer row count made even indexed scans slow.
### How to Connect for Diagnosis
1. **MySQL access via MaxScale VIP** using `maxscale` user (has `SUPER` privilege):
```python
import pymysql
conn = pymysql.connect(host="10.0.30.70", port=3306,
user="maxscale", password="<maxscale-password>", charset="utf8mb4")
```
2. **HA database access** using `ha_recorder` user:
```python
conn = pymysql.connect(host="10.0.30.70", port=3306,
user="ha_recorder", password="<ha-recorder-password>",
database="homeassistant", charset="utf8mb4")
```
3. **Credentials source**: 1Password item `mariadb-galera-vm`, vault `Hermes`:
```bash
op item get "mariadb-galera-vm" --vault "Hermes" --reveal \
--fields maxscale-password,ha-recorder-password,root-password
```
4. **HAOS SSH** (port 22222) for HA-side investigation:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
# Inside: docker exec homeassistant <command>
```
## Changes Applied (SET GLOBAL, instant effect)
All 9 parameters changed successfully via `maxscale` MySQL user through MaxScale VIP — no SSH required:
| Parameter | Before | After | Effect |
|-----------|--------|-------|--------|
| `wsrep_slave_threads` | 4 | 16 | More applier threads for replication backlog |
| `innodb_io_capacity` | 200 | 2000 | SSD-appropriate write throughput |
| `innodb_io_capacity_max` | 2000 | 4000 | Burst capacity |
| `innodb_read_io_threads` | 4 | 8 | Double read parallelism |
| `innodb_write_io_threads` | 4 | 8 | Double write parallelism |
| `innodb_flush_neighbors` | 1 | 0 | Remove HDD-era coalescing on SSD |
| `innodb_purge_threads` | 4 | 8 | Faster cleanup of deleted rows |
| `innodb_adaptive_hash_index` | OFF | ON | Speed up point lookups on large tables |
| `long_query_time` | 10 | 2 | Log slow queries for diagnosis |
## Query Performance AFTER Tuning
| Time Range | Rows | Before | After | Improvement |
|------------|------|--------|-------|-------------|
| 24h | 79,130 | 0.18s | 0.02s | **9×** |
| 7 days | 220,408 | 10.1s | 0.05s | **200×** |
| 30 days | 2,049,090 | 60.0s | 0.39s | **154×** |
| SELECT + JOIN LIMIT 1000 | 1000 | — | 0.01s | — |
| SELECT + JOIN LIMIT 500 | 500 | — | 0.008s | — |
## Parameters Requiring Restart (Not Yet Changed)
These cannot be changed via `SET GLOBAL` and require editing the server config + restarting each node:
| Parameter | Current | Target | Notes |
|-----------|---------|--------|-------|
| `innodb_buffer_pool_size` | 4 GB | 8-16 GB | Must fit working set (states table = 16 GB) |
| `innodb_log_file_size` | 96 MB | 512 MB | Small redo log throttles write throughput |
| `gcache.size` | 128 MB | 1 GB | Prevents expensive SST on node rejoin |
| `innodb_buffer_pool_instances` | unknown | 4-8 | Should match buffer pool / 1GB |
## Making Changes Permanent
`SET GLOBAL` changes are lost on MariaDB restart. To persist, SSH to each Galera node and append to server config:
```bash
ssh debian@10.0.30.71
sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
[mariadb]
wsrep_slave_threads=16
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_neighbors=0
innodb_purge_threads=8
innodb_adaptive_hash_index=ON
long_query_time=2
innodb_buffer_pool_size=8G
innodb_log_file_size=512M
wsrep_provider_options="gcache.size=1G"
EOF
sudo systemctl restart mariadb
```
Repeat on all 3 nodes sequentially (verify `wsrep_local_state_comment=Synced` before next node).
## HA Recorder Recommendations (Not Applied — User Declined)
User was offered but declined recorder config changes:
- `recorder.exclude` for noisy domains (automation, button, select, update)
- `recorder.commit_interval: 5` (reduce write frequency from 1s to 5s)
- Move long-term history to InfluxDB (already configured at 10.0.30.109:8086, bucket `ha_hot_90d`)
- `OPTIMIZE TABLE states` to reclaim 619 MB fragmented space
## Key Insight
The `maxscale` MySQL user has `SUPER` privilege and can run `SET GLOBAL` through the MaxScale VIP. This enables rapid Galera tuning without SSH access to the individual nodes — critical when SSH keys aren't set up for the DB VMs but 1Password credentials are available.
devops/mariadb-galera-cluster-administration/references/galera-recovery-runbook.md
+66
View File
@@ -0,0 +1,66 @@
# Galera Cluster Recovery Runbook
Based on June 27, 2026 recovery of total Galera cluster failure (down since June 24 ~14:46 CEST).
## Timeline
1. **June 24 ~14:46** — All 3 Galera nodes crashed simultaneously. Root cause: likely simultaneous network issue preventing inter-node communication → "Failed to reach primary view" → all nodes aborted.
2. **June 27 13:00** — Recovery began. db1 bootstrapped, db2/db3 stopped to resolve SST deadlock.
3. **June 27 13:07** — db2 SST started (mariabackup, ~18GB)
4. **June 27 13:17** — db2 SST completed (~40 min), state=Synced, cluster_size=2
5. **June 27 13:17** — db3 SST started
6. **June 27 ~13:50** — db3 SST completed, cluster_size=3
## Key Decisions Made
### 1. Bootstrap from Node 1 (no grastate.ini)
No `grastate.ini` existed on any node, so no `safe_to_bootstrap` flag. Any node can be chosen. Picked db1 (10.0.30.71).
### 2. Stop db2/db3 to Resolve SST Deadlock
Initially tried starting both db2 and db3 after db1 bootstrap. Both entered SST but deadlocked: `"No donor candidates temporarily available in suitable state"`. Solution: stop both, then start them one at a time.
### 3. MaxScale `available_when_donor=true`
Without this, MaxScale refused to route to db1 while it was Donor/Desynced serving SST to db2. HA Recorder got `Lost connection to server during query` errors. Setting this flag fixed routing immediately.
### 4. Two HA Restarts Needed
First restart failed because MaxScale wasn't routing yet (before `available_when_donor=true`). Second restart after the fix succeeded — Recorder connected, history/logbook loaded.
## Pitfalls Encountered
### PITFALL: `galera_new_cluster` Appears to Timeout
`galera_new_cluster` may appear to hang/timeout in the terminal but actually launches mariadbd in the background successfully. Always verify separately with `systemctl is-active` and `mariadb -e "SHOW STATUS LIKE 'wsrep_%'"`.
### PITFALL: Parallel SST Causes Deadlock
Starting 2+ nodes simultaneously when only 1 donor exists → all joiners compete for the same donor → donor can't serve multiple SST streams → deadlock → all abort.
### PITFALL: HA Recorder Retries Exhaust Before DB Recovers
With `db_max_retries: 20` and 5-second intervals, Recorder gives up after ~100 seconds. If MaxScale isn't routing by then, Recorder shuts down. Requires HA restart after DB is confirmed accessible.
### PITFALL: MaxScale `set server db1 master` Doesn't Work
Monitored servers can only have `maintenance`/`drain` set manually. Role assignment is automatic via the galeramon monitor. Don't waste time trying to force roles.
### PITFALL: HA Takes Long to Reach RUNNING State
With ~18GB DB, HA's schema migration check (`pre_migrate_schema`) takes significant time. State stays `NOT_RUNNING` for 2-3 minutes. Don't assume failure — check docker logs for actual errors.
## Post-Recovery Cleanup
```bash
# Revert available_when_donor
maxctrl alter monitor galera-monitor available_when_donor=false
# Verify all nodes
maxctrl list servers
# All should show: "Synced, Running" with no Donor/Desynced
# Test HA
curl -s "$HA_URL/api/services" -H "Authorization: Bearer $HA_TOKEN" | python3 -c "
import sys,json
data=json.load(sys.stdin)
services=set(s['domain'] for s in data)
for d in ['recorder','history','logbook']:
print(f'{d}: {\"✓\" if d in services else \"✗\"}')
"
# Test history data
curl -s "$HA_URL/api/history/period?filter_entity_id=sun.sun" -H "Authorization: Bearer $HA_TOKEN"
```