dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add home-assistant-dashboard-conventions skill + update multiple skills

Browse Source 
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards)
- Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills
- New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/mariadb-galera-cluster-administration/SKILL.md
+218 -12
View File
@@ -1,6 +1,6 @@
---
name: mariadb-galera-cluster-administration
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, and HA Recorder dependency troubleshooting."
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, HA Recorder dependency troubleshooting, OOM-killer diagnosis (PVE host kills KVM), VM placement strategy, and balloon:0 rationale for DB VMs."
version: 1.0.0
tags: [mariadb, galera, maxscale, database, cluster, sst, mysql, ha, homeassistant]
---
@@ -205,25 +205,105 @@ resolved it immediately — `maxctrl list servers` showed all 3 nodes
with correct grants and password, but the host was blocked at the
MariaDB connection-error-counter level (error 1129).
### 2.6 MaxScale VIP Architecture (Homelab)
### 2.6 MaxScale Source-IP ACL Blocking (CRITICAL)
**Symptoms:** `Access denied for user 'root'@'10.0.30.XXX' (using password: YES)` from any host outside the MaxScale VM itself, even with correct credentials. Affects root, maxscale, app users — ALL users. The error comes from MaxScale, not Galera.
**Root Cause:** MaxScale's listener configuration restricts which source IPs may connect. Unlike `max_connection_errors` (§2.5, which blocks the MaxScale VM itself), this is an intentional ACL in the MaxScale listener/service definition. Even localhost (127.0.0.1) on the MaxScale VM may be blocked for certain users.
**Diagnosis:**
```bash
# Test from your host:
python3 -c "
import pymysql
try:
conn = pymysql.connect(host='10.0.30.70', port=3306, user='root', password='<pw>', charset='utf8mb4')
print('OK')
except Exception as e:
print(f'Denied: {e}')
"
# Error: (1045, "Access denied for user 'root'@'10.0.30.230' (using password: YES)")
```
The source IP in the error message is YOUR host's IP, not the Galera node. MaxScale rejects before forwarding to any backend.
**Affected source IPs observed:**
- `.230` (Hermes host) — blocked
- `.62` (K8s worker) — blocked
- `127.0.0.1` (MaxScale VM localhost) — blocked for root, may work for other users
- `.82` (MaxScale VM external IP) — blocked for root
**⚠️ Galera VM SSH and Guest Agent:**
- Galera VMs (.71/.72/.73) have SSH **closed** (port 22 connection refused)
- QEMU Guest Agent is **not running** on Galera VMs — `qm guest exec` fails
- No `qm guest cmd <vmid> ping` works — agent not installed/running
- Direct network access to .71/.72/.73 is blocked even from PVE nodes and MaxScale VM
- The **only** access path is MaxScale VIP .70:3306, and only from allowlisted IPs
**⚠️ 1Password Item Names (corrected):**
- `mariadb-root` — root password (NOT `mariadb-galera-vm`)
- `mariadb-maxscale` — MaxScale admin password
- `mariadb-app-user` — username=dominik, password for app access
- All in vault "Kubernetes ESO" (ID: `334ykdtj5kar3jlpcrztjvx2fu`)
- ESO token retrievable: `kubectl get secret onepassword-token -n external-secrets -o jsonpath="{.data.token}" | base64 -d`
**⚠️ MaxScale VM Access:**
- Hostname: `maxscale-02`
- SSH: `ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.70` (works!)
- Has `mariadb` client installed at `/usr/bin/mysql`
- But even from MaxScale VM localhost, root SQL is denied — MaxScale ACL blocks before backend
**Workaround — SSH to MaxScale VM and use `maxctrl` or local `mariadb` client:**
```bash
# MaxScale VM accepts SSH as debian user
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.70
# Once on MaxScale VM, use maxctrl to manage the cluster
maxctrl list servers
maxctrl list services
# For SQL operations, try connecting through MaxScale with a user that has
# the correct grant for the MaxScale VM's source IP:
mariadb --defaults-extra-file=/tmp/creds.cnf -h 127.0.0.1 -e "CREATE DATABASE ..."
```
**If ALL users are blocked from ALL sources:**
1. Check `maxctrl list listeners` for IP binding restrictions
2. Check MaxScale service `user`/`password` matching the MySQL user grants
3. The `maxscale` MySQL user (with SUPER) may work through `maxctrl` commands even when direct SQL is blocked
4. As last resort: SSH to a Galera node directly (if SSH is open) and run `sudo mariadb` locally
**Prevention:** When setting up MaxScale, document which source IPs are allowed and ensure the application's expected source IP is in the allowlist. For K8s migrations, the K8s worker IPs must be allowed if applications connect through the MaxScale VIP.
### 2.7 MaxScale VIP Architecture (Homelab — Updated 2026-07-13)
| Component | Address | Role |
|-----------|---------|------|
| MaxScale VIP | `10.0.30.70:3306` | rw (readwritesplit) |
| MaxScale VIP | `10.0.30.70:3307` | ro (readconnroute) |
| MaxScale VM 01 | `10.0.30.81` (VM 310, proxmox4) | Active MaxScale instance |
| db1 | `10.0.30.71` (VM 300, proxmox2) | Galera node |
| db2 | `10.0.30.72` (VM 301, proxmox6) | Galera node |
| db3 | `10.0.30.73` (VM 302, proxmox4) | Galera node |
| MaxScale VM 01 | `10.0.30.81` (VM 310, n5pro) | Active MaxScale instance |
| MaxScale VM 02 | `10.0.30.82` (VM 311, proxmox3) | Standby MaxScale instance |
| db1 | `10.0.30.71` (VM 300, n5pro) | Galera node |
| db2 | `10.0.30.72` (VM 301, proxmox3) | Galera node |
| db3 | `10.0.30.73` (VM 302, proxmox6) | Galera node |
Applications connect through `10.0.30.70:3306` for automatic failover and
load balancing. The `maxscale` MySQL user (with SUPER privilege) can be used
for `SET GLOBAL` tuning commands through the VIP without SSH access to nodes.
**Note**: A second MaxScale VM (VM501 "MaxScale1") exists on proxmox4
but is currently **stopped**. The VIP `10.0.30.70` is served by
Keepalived on the active MaxScale VM. PVE HA may migrate VMs — always
verify current placement with `qm list` before operating.
**HA Rules**: All 5 VMs (300, 301, 302, 310, 311) restricted to n5pro +
proxmox3 + proxmox6 (3 hardware-diverse hosts for quorum safety) via
strict `db-nodes` node-affinity rule. Anti-affinity keeps MaxScale
instances apart (310≠311) and Galera nodes apart (301≠302).
See `references/galera-cluster-topology.md` for full rules.cfg content.
**⚠️ Quorum rule**: Each Galera node MUST be on a different physical host.
2 nodes on the same host = quorum loss if that host fails.
**⚠️ Anti-affinity blocks migrations**: `qm migrate` fails if the target
host already has a VM with negative affinity. Check `rules.cfg` first.
PVE HA may migrate VMs — always verify current placement with `qm list`
before operating.
---
@@ -392,9 +472,135 @@ FULLTEXT indexes on MariaDB require `ENGINE=InnoDB` and `CHARSET=utf8mb4`. Creat
---
---
### 5.3b OOM-Killer: PVE Host Kills Galera KVM Process (2026-07-13)
## Section 6: Dynamic Performance Tuning via SET GLOBAL
**Different from §1.1** — this is NOT a MariaDB crash. The PVE host's Linux
kernel OOM-killer terminates the entire KVM process when host RAM is exhausted.
**Symptoms:**
- VM shows `running` in `qm status` but Guest Agent is down (`qm guest cmd <vmid> ping` fails)
- SSH to the VM is refused/closed (all guest services died with the KVM process)
- QEMU restarts the VM (onboot=1) but the guest OS boots incompletely (VmRSS far below allocated RAM)
- Galera cluster drops from 3→2 nodes; surviving nodes show `forgetting <UUID>` in journal
**Diagnosis (on the PVE HOST, not the VM):**
```bash
# SSH to the PVE node hosting the affected VM
ssh root@<pve-node-ip> "dmesg -T | grep -iE 'oom|killed|out of memory' | tail -20"
# Look for: "Out of memory: Killed process <PID> (kvm)" with task_memcg=/qemu.slice/<VMID>.scope
```
**Accessing a partially-booted VM via `qm guest exec`:**
When SSH is down but QEMU restarted the VM (onboot=1), the guest agent may
respond even though regular `qm agent <vmid> ping` fails. Use `qm guest exec`
to run commands directly:
```bash
# From the PVE host where the VM runs:
qm guest exec <vmid> -- /bin/bash -c "hostname; uptime; systemctl is-active mariadb; free -m | head -2"
# Returns JSON with exitcode, out-data, out-truncated fields
```
This is the fastest way to check if MariaDB is still in SST/activating state
when SSH is not yet available. Parse output with:
```bash
qm guest exec <vmid> -- /bin/bash -c "systemctl is-active mariadb" | python3 -c "import json,sys; print(json.load(sys.stdin).get('out-data','').strip())"
```
**Root Cause:** RAM overcommit on the PVE host:
```
overcommit = sum(running VM maxmem) / node physical RAM
```
Example: proxmox2 had 15 GB RAM but 22 GB VMs allocated (VM 129=12GB + VM 300=8GB + VM 311=2GB = 147%).
**Why `balloon: 0` is correct for Galera VMs:**
MySQL/Galera's InnoDB buffer pool holds hot pages under constant access.
If the PVE balloon driver reclaims memory, the guest kernel swaps those hot
pages to swap → massive I/O latency spikes → Galera flow control kicks in
and throttles writes across the ENTIRE cluster. One ballooned Galera node
can slow down the whole cluster. `balloon: 0` (disabled) ensures the VM
always holds its full allocated RAM.
**Fix sequence (user preference: RCA → restart → mitigation, in that order):**
1. **RCA first** — diagnose root cause BEFORE restarting. Check `dmesg -T | grep oom` on the PVE host, correlate with Galera journal timestamps on surviving nodes. Understand WHY the VM died before bringing it back.
2. **Restart the killed VM** — `qm stop <vmid>; qm start <vmid>` (or via HA manager). Wait for SST to complete (Galera rejoins via State Transfer from a surviving node). Expect 30-60 minutes SST time depending on DB size. During SST, the node shows `activating` in systemctl and `Joiner` in wsrep state.
3. **Mitigation** — address the root cause to prevent recurrence:
- Migrate the VM to a node with RAM headroom
- Set strict HA node-affinity rules to prevent failover to small nodes
- Deploy RAM-based rebalancer (see PVE skill `scripts/pve-ram-rebalancer.sh`)
4. **Verify** — confirm Galera cluster size = 3, all nodes Synced, MaxScale routing restored
### 5.3c Galera/MaxScale VM Placement on PVE (Updated 2026-07-13)
**⚠️ CRITICAL: Quorum-Aware Placement Rule**
Each Galera node MUST be on a different physical PVE host. With 3 Galera
nodes, losing any single host must leave ≥2 nodes alive (= quorum). If 2
nodes share a host and that host fails, only 1 survives → **no quorum,
entire cluster goes down** (non-primary state, all writes blocked).
**Placement checklist:**
1. Count Galera nodes per PVE host — max 1 per host
2. Verify: for EACH host, if it fails, ≥2 Galera nodes remain on other hosts
3. Include MaxScale VMs in the analysis — they should also be distributed
4. Document the 3 allowed hosts in the HA `db-nodes` rule (strict=1)
**Previous placement (problematic — caused OOM + quorum risk):**
| VM | Name | Node | RAM | Issue |
|----|------|------|-----|-------|
| 300 | mariadb-01 | proxmox2 (15GB) | 8 GB | OOM killed — co-located with 12GB K8s CP |
| 301 | mariadb-02 | proxmox6 (15GB) | 8 GB | Overcommit risk |
| 302 | mariadb-03 | proxmox4 (15GB) | 8 GB | 200% overcommit with K8s worker |
| 310 | maxscale-01 | proxmox4 (15GB) | 2 GB | Same node as mariadb-03 |
| 311 | maxscale-02 | proxmox2 (15GB) | 2 GB | Same node as mariadb-01 |
**Intermediate placement (quorum risk — 2 nodes on proxmox3):**
| VM | Name | Node | Issue |
|----|------|------|-------|
| 300 | mariadb-01 | proxmox3 | 2 of 3 Galera on same host — quorum risk |
| 301 | mariadb-02 | proxmox3 | If proxmox3 fails, only 1 node survives |
| 302 | mariadb-03 | n5pro | OK |
**Corrected placement (3 different hosts, quorum-safe):**
| VM | Name | Node | RAM | Safe? |
|----|------|------|-----|-------|
| 300 | mariadb-01 | n5pro (91GB) | 8 GB | ✅ Massive headroom |
| 301 | mariadb-02 | proxmox3 (31GB) | 8 GB | ✅ Good headroom |
| 302 | mariadb-03 | proxmox6 (15GB) | 8 GB | ⚠️ 113% overcommit but effective use is only 5.5 GB |
| 310 | maxscale-01 | n5pro (91GB) | 2 GB | ✅ |
| 311 | maxscale-02 | proxmox3 (31GB) | 2 GB | ✅ |
HA rule `db-nodes` (strict=1) restricts all 5 VMs to n5pro + proxmox3 +
proxmox6 (3 hardware-diverse hosts). Anti-affinity rules keep MaxScale
instances apart (310≠311) and Galera nodes apart (301≠302).
**⚠️ PITFALL: Live Migration Can Kill Recently-Recovering Galera Nodes**
Migrating a Galera VM that is still in `activating` state (SST in progress)
or that was recently OOM-killed and restarted can cause it to drop from the
cluster and become unreachable (guest agent down, SSH refused). Sequence:
1. OOM-kill destroys KVM process
2. QEMU restarts VM (onboot=1) but guest OS boots incompletely
3. Live migration triggers → VM freezes during migration
4. After migration: guest agent down, SSH "No route to host", MariaDB stuck
**Safe migration sequence for recovering Galera VMs:**
1. Wait until `systemctl is-active mariadb` = `active` AND `wsrep_local_state_comment` = `Synced`
2. THEN migrate — never migrate during SST/activating state
3. After migration, verify guest agent responds: `qm agent <vmid> ping`
4. If guest agent is down post-migration: `qm stop <vmid>; qm start <vmid>` (full restart, not live migrate)
**⚠️ PITFALL: Anti-Affinity Rules Block Migrations**
Negative resource-affinity rules (e.g., `vm:310` ↔ `vm:311` must not
co-locate) will cause `qm migrate` to fail with:
```
cannot migrate resource 'vm:311' to node 'n5pro':
- resource 'vm:310' on target node 'n5pro' in negative affinity with resource 'vm:311'
```
Always check `/etc/pve/ha/rules.cfg` for conflicting rules before migrating.
The target node must not contain any VM that has negative affinity with
the VM being migrated.
---
### 6.1 The `maxscale` MySQL User Trick
devops/mariadb-galera-cluster-administration/references/galera-cluster-topology.md
+60 -38
View File
@@ -1,67 +1,98 @@
# Homelab Galera/MaxScale Topology
# Homelab Galera/MaxScale Topology (Updated 2026-07-13)
## MariaDB Galera Cluster (3 nodes)
## MariaDB Galera Cluster (3 nodes — quorum-safe: 3 different PVE hosts)
| Node | VM ID | Proxmox Host | IP | Hostname |
|------|-------|---------------|----|----------|
| db1 | 300 | proxmox2 (10.0.20.20) | 10.0.30.71 | mariadb-01 |
| db2 | 301 | proxmox6 (10.0.20.60) | 10.0.30.72 | mariadb-02 |
| db3 | 302 | proxmox4 (10.0.20.40) | 10.0.30.73 | mariadb-03 |
| Node | VM ID | Proxmox Host | IP | Hostname | RAM |
|------|-------|---------------|----|----------|-----|
| db1 | 300 | n5pro (10.0.20.91) | 10.0.30.71 | mariadb-01 | 8 GB |
| db2 | 301 | proxmox3 (10.0.20.30) | 10.0.30.72 | mariadb-02 | 8 GB |
| db3 | 302 | proxmox6 (10.0.20.60) | 10.0.30.73 | mariadb-03 | 8 GB |
- MariaDB version: 11.4.10-MariaDB-deb12
- Cluster name: `mariadb-galera`
- Galera address: `gcomm://10.0.30.71,10.0.30.72,10.0.30.73`
- DB size: ~18GB (homeassistant schema)
- **Effective RAM usage**: ~5.5 GB per node (4 GB innodb_buffer_pool + 1.5 GB OS/Galera)
- `balloon: 0` on all Galera VMs — see SKILL.md §5.3b for rationale
- **Quorum guarantee**: Any single PVE host failure leaves ≥2 Galera nodes alive
- **Note**: PVE HA may migrate VMs. Always verify current placement
with `qm list` on each node before operating.
## MaxScale Proxy (1 active + VIP)
## MaxScale Proxy (2 instances + VIP)
| Node | VM ID | Proxmox Host | IP | Hostname | Status |
|------|-------|--------------|----|----------|--------|
| maxscale-01 | 310 | proxmox4 (10.0.20.40) | 10.0.30.81 | maxscale-01 | Active |
| VIP | — | — | 10.0.30.70 | keepalived VIP | — |
| Node | VM ID | Proxmox Host | IP | Hostname | RAM | Status |
|------|-------|---------------|----|----------|-----|--------|
| maxscale-01 | 310 | n5pro (10.0.20.91) | 10.0.30.81 | maxscale-01 | 2 GB | Active |
| maxscale-02 | 311 | proxmox3 (10.0.20.30) | 10.0.30.82 | maxscale-02 | 2 GB | Standby |
| VIP | — | — | 10.0.30.70 | keepalived VIP | — | — |
- MaxScale version: 24.02.9
- Services: `rw-router` (readwritesplit), `ro-router` (readconnroute)
- Monitor: `galera-monitor` (galeramon)
- Listener: `rw-listener` on port 3306
- **Note**: A second MaxScale VM (VM501 "MaxScale1") exists on
proxmox4 but is **stopped**. If a standby MaxScale is needed, check
VM501's config. The VIP `10.0.30.70` is the authoritative entry point.
## HA Rules (PVE 8 rules.cfg)
```
node-affinity: db-nodes
nodes n5pro,proxmox3,proxmox6
resources vm:300,vm:301,vm:302,vm:310,vm:311
strict 1
resource-affinity: ha-rule-ea9e852a-8d82
affinity negative
resources vm:310,vm:311
resource-affinity: ha-rule-d61699eb-2baf
affinity negative
resources vm:301,vm:302
```
- `db-nodes` (strict=1): Galera + MaxScale VMs restricted to n5pro + proxmox3 + proxmox6 (3 hardware-diverse hosts)
- Negative resource-affinity: MaxScale 310↔311 kept apart, Galera 301↔302 kept apart
- At node failure, HA manager restarts VM on one of the other 2 allowed nodes (deterministic)
- **⚠️ Anti-affinity rules can block `qm migrate`** — check rules.cfg before migrating
## PVE Host RAM Capacity
| Host | IP | Physical RAM | Notes |
|------|-----|-------------|-------|
| n5pro | 10.0.20.91 | 91 GB | Massive headroom, primary DB host |
| proxmox3 | 10.0.20.30 | 31 GB | Good headroom, secondary DB host |
| proxmox6 | 10.0.20.60 | 15 GB | Tight — 113% overcommit with Galera VM, but effective use is only 5.5 GB |
| proxmox1-2,4-5,7 | 10.0.20.{10,20,40,50,??} | 15 GB each | NOT in db-nodes rule — DB VMs cannot failover here |
## SSH Access
```bash
# MariaDB VMs (user: debian)
# MariaDB VMs (user: debian) — may vary if HA migrated the VM
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.72
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73
# MaxScale VMs (user: debian)
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.70
# Proxmox hosts (user: root)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 # proxmox1 (HA VM host)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 # proxmox1
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.30 # proxmox3
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.60 # proxmox6
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 # n5pro
```
## Credentials
All credentials stored in 1Password Vault "Hermes", item `mariadb-galera-vm`:
- `root-password` — MySQL root (localhost only on each node, not accessible via MaxScale VIP)
- `maxscale-password` — MaxScale MySQL user with `SUPER` privilege — **can run `SET GLOBAL` via VIP** (see Section 6)
Credentials in 1Password Vault "Kubernetes ESO" (ID: `334ykdtj5kar3jlpcrztjvx2fu`):
- `mariadb-root` — root password (localhost only, `sudo mariadb` on node)
- `mariadb-app-user` — username=dominik, app access
- `gitea-db` — Gitea database user/password
Legacy item `mariadb-galera-vm` in vault "Hermes" may still contain:
- `maxscale-password` — MaxScale MySQL user with SUPER privilege
- `mariabackup-password` — Galera SST transfer auth
- `ha-recorder-password` — HA Recorder MySQL user, has SELECT/INSERT/UPDATE on `homeassistant` DB
- `ha-recorder-password` — HA Recorder MySQL user
- `keepalived-auth-pass` — keepalived VIP failover auth
Retrieve all fields:
```bash
op item get "mariadb-galera-vm" --vault "Hermes" --reveal --fields password,root-password,mariabackup-password,maxscale-password,ha-recorder-password
```
Note: The generic `password` field is empty. Use the named fields instead.
## Firewall (UFW on MariaDB VMs)
- Port 3306: ALLOW from 10.0.30.81, 10.0.30.82 (MaxScale only)
@@ -75,18 +106,9 @@ Note: The generic `password` field is empty. Use the named fields instead.
- Config: `/mnt/data/supervisor/homeassistant/configuration.yaml`
- Recorder DB URL: `mysql://ha_recorder:***@10.0.30.70:3306/homeassistant`
- InfluxDB: 10.0.30.109:8086, bucket `ha_hot_90d` (long-term history)
- HA masks passwords with `***` in configuration.yaml — retrieve real passwords from 1Password, not the config file
### HAOS SSH Access (Port 22222)
HAOS exposes SSH on port 22222 (requires SSH addon enabled):
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
```
Inside HAOS, HA runs as Docker container `homeassistant`:
```bash
docker exec homeassistant cat /config/configuration.yaml
docker exec homeassistant python3 -c "..."
docker logs homeassistant 2>&1 | grep -i recorder | tail -20
```
Note: `pip` modules like `pymysql` are NOT available inside the HA container. Use `sqlalchemy` (bundled with HA) or query the DB from outside.