feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: mariadb-galera-cluster-administration
|
||||
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, and HA Recorder dependency troubleshooting."
|
||||
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, HA Recorder dependency troubleshooting, OOM-killer diagnosis (PVE host kills KVM), VM placement strategy, and balloon:0 rationale for DB VMs."
|
||||
version: 1.0.0
|
||||
tags: [mariadb, galera, maxscale, database, cluster, sst, mysql, ha, homeassistant]
|
||||
---
|
||||
@@ -205,25 +205,105 @@ resolved it immediately — `maxctrl list servers` showed all 3 nodes
|
||||
with correct grants and password, but the host was blocked at the
|
||||
MariaDB connection-error-counter level (error 1129).
|
||||
|
||||
### 2.6 MaxScale VIP Architecture (Homelab)
|
||||
### 2.6 MaxScale Source-IP ACL Blocking (CRITICAL)
|
||||
|
||||
**Symptoms:** `Access denied for user 'root'@'10.0.30.XXX' (using password: YES)` from any host outside the MaxScale VM itself, even with correct credentials. Affects root, maxscale, app users — ALL users. The error comes from MaxScale, not Galera.
|
||||
|
||||
**Root Cause:** MaxScale's listener configuration restricts which source IPs may connect. Unlike `max_connection_errors` (§2.5, which blocks the MaxScale VM itself), this is an intentional ACL in the MaxScale listener/service definition. Even localhost (127.0.0.1) on the MaxScale VM may be blocked for certain users.
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Test from your host:
|
||||
python3 -c "
|
||||
import pymysql
|
||||
try:
|
||||
conn = pymysql.connect(host='10.0.30.70', port=3306, user='root', password='<pw>', charset='utf8mb4')
|
||||
print('OK')
|
||||
except Exception as e:
|
||||
print(f'Denied: {e}')
|
||||
"
|
||||
# Error: (1045, "Access denied for user 'root'@'10.0.30.230' (using password: YES)")
|
||||
```
|
||||
|
||||
The source IP in the error message is YOUR host's IP, not the Galera node. MaxScale rejects before forwarding to any backend.
|
||||
|
||||
**Affected source IPs observed:**
|
||||
- `.230` (Hermes host) — blocked
|
||||
- `.62` (K8s worker) — blocked
|
||||
- `127.0.0.1` (MaxScale VM localhost) — blocked for root, may work for other users
|
||||
- `.82` (MaxScale VM external IP) — blocked for root
|
||||
|
||||
**⚠️ Galera VM SSH and Guest Agent:**
|
||||
- Galera VMs (.71/.72/.73) have SSH **closed** (port 22 connection refused)
|
||||
- QEMU Guest Agent is **not running** on Galera VMs — `qm guest exec` fails
|
||||
- No `qm guest cmd <vmid> ping` works — agent not installed/running
|
||||
- Direct network access to .71/.72/.73 is blocked even from PVE nodes and MaxScale VM
|
||||
- The **only** access path is MaxScale VIP .70:3306, and only from allowlisted IPs
|
||||
|
||||
**⚠️ 1Password Item Names (corrected):**
|
||||
- `mariadb-root` — root password (NOT `mariadb-galera-vm`)
|
||||
- `mariadb-maxscale` — MaxScale admin password
|
||||
- `mariadb-app-user` — username=dominik, password for app access
|
||||
- All in vault "Kubernetes ESO" (ID: `334ykdtj5kar3jlpcrztjvx2fu`)
|
||||
- ESO token retrievable: `kubectl get secret onepassword-token -n external-secrets -o jsonpath="{.data.token}" | base64 -d`
|
||||
|
||||
**⚠️ MaxScale VM Access:**
|
||||
- Hostname: `maxscale-02`
|
||||
- SSH: `ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.70` (works!)
|
||||
- Has `mariadb` client installed at `/usr/bin/mysql`
|
||||
- But even from MaxScale VM localhost, root SQL is denied — MaxScale ACL blocks before backend
|
||||
|
||||
**Workaround — SSH to MaxScale VM and use `maxctrl` or local `mariadb` client:**
|
||||
```bash
|
||||
# MaxScale VM accepts SSH as debian user
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.70
|
||||
|
||||
# Once on MaxScale VM, use maxctrl to manage the cluster
|
||||
maxctrl list servers
|
||||
maxctrl list services
|
||||
|
||||
# For SQL operations, try connecting through MaxScale with a user that has
|
||||
# the correct grant for the MaxScale VM's source IP:
|
||||
mariadb --defaults-extra-file=/tmp/creds.cnf -h 127.0.0.1 -e "CREATE DATABASE ..."
|
||||
```
|
||||
|
||||
**If ALL users are blocked from ALL sources:**
|
||||
1. Check `maxctrl list listeners` for IP binding restrictions
|
||||
2. Check MaxScale service `user`/`password` matching the MySQL user grants
|
||||
3. The `maxscale` MySQL user (with SUPER) may work through `maxctrl` commands even when direct SQL is blocked
|
||||
4. As last resort: SSH to a Galera node directly (if SSH is open) and run `sudo mariadb` locally
|
||||
|
||||
**Prevention:** When setting up MaxScale, document which source IPs are allowed and ensure the application's expected source IP is in the allowlist. For K8s migrations, the K8s worker IPs must be allowed if applications connect through the MaxScale VIP.
|
||||
|
||||
### 2.7 MaxScale VIP Architecture (Homelab — Updated 2026-07-13)
|
||||
|
||||
| Component | Address | Role |
|
||||
|-----------|---------|------|
|
||||
| MaxScale VIP | `10.0.30.70:3306` | rw (readwritesplit) |
|
||||
| MaxScale VIP | `10.0.30.70:3307` | ro (readconnroute) |
|
||||
| MaxScale VM 01 | `10.0.30.81` (VM 310, proxmox4) | Active MaxScale instance |
|
||||
| db1 | `10.0.30.71` (VM 300, proxmox2) | Galera node |
|
||||
| db2 | `10.0.30.72` (VM 301, proxmox6) | Galera node |
|
||||
| db3 | `10.0.30.73` (VM 302, proxmox4) | Galera node |
|
||||
| MaxScale VM 01 | `10.0.30.81` (VM 310, n5pro) | Active MaxScale instance |
|
||||
| MaxScale VM 02 | `10.0.30.82` (VM 311, proxmox3) | Standby MaxScale instance |
|
||||
| db1 | `10.0.30.71` (VM 300, n5pro) | Galera node |
|
||||
| db2 | `10.0.30.72` (VM 301, proxmox3) | Galera node |
|
||||
| db3 | `10.0.30.73` (VM 302, proxmox6) | Galera node |
|
||||
|
||||
Applications connect through `10.0.30.70:3306` for automatic failover and
|
||||
load balancing. The `maxscale` MySQL user (with SUPER privilege) can be used
|
||||
for `SET GLOBAL` tuning commands through the VIP without SSH access to nodes.
|
||||
|
||||
**Note**: A second MaxScale VM (VM501 "MaxScale1") exists on proxmox4
|
||||
but is currently **stopped**. The VIP `10.0.30.70` is served by
|
||||
Keepalived on the active MaxScale VM. PVE HA may migrate VMs — always
|
||||
verify current placement with `qm list` before operating.
|
||||
**HA Rules**: All 5 VMs (300, 301, 302, 310, 311) restricted to n5pro +
|
||||
proxmox3 + proxmox6 (3 hardware-diverse hosts for quorum safety) via
|
||||
strict `db-nodes` node-affinity rule. Anti-affinity keeps MaxScale
|
||||
instances apart (310≠311) and Galera nodes apart (301≠302).
|
||||
See `references/galera-cluster-topology.md` for full rules.cfg content.
|
||||
|
||||
**⚠️ Quorum rule**: Each Galera node MUST be on a different physical host.
|
||||
2 nodes on the same host = quorum loss if that host fails.
|
||||
**⚠️ Anti-affinity blocks migrations**: `qm migrate` fails if the target
|
||||
host already has a VM with negative affinity. Check `rules.cfg` first.
|
||||
|
||||
PVE HA may migrate VMs — always verify current placement with `qm list`
|
||||
before operating.
|
||||
|
||||
---
|
||||
|
||||
@@ -392,9 +472,135 @@ FULLTEXT indexes on MariaDB require `ENGINE=InnoDB` and `CHARSET=utf8mb4`. Creat
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
### 5.3b OOM-Killer: PVE Host Kills Galera KVM Process (2026-07-13)
|
||||
|
||||
## Section 6: Dynamic Performance Tuning via SET GLOBAL
|
||||
**Different from §1.1** — this is NOT a MariaDB crash. The PVE host's Linux
|
||||
kernel OOM-killer terminates the entire KVM process when host RAM is exhausted.
|
||||
|
||||
**Symptoms:**
|
||||
- VM shows `running` in `qm status` but Guest Agent is down (`qm guest cmd <vmid> ping` fails)
|
||||
- SSH to the VM is refused/closed (all guest services died with the KVM process)
|
||||
- QEMU restarts the VM (onboot=1) but the guest OS boots incompletely (VmRSS far below allocated RAM)
|
||||
- Galera cluster drops from 3→2 nodes; surviving nodes show `forgetting <UUID>` in journal
|
||||
|
||||
**Diagnosis (on the PVE HOST, not the VM):**
|
||||
```bash
|
||||
# SSH to the PVE node hosting the affected VM
|
||||
ssh root@<pve-node-ip> "dmesg -T | grep -iE 'oom|killed|out of memory' | tail -20"
|
||||
# Look for: "Out of memory: Killed process <PID> (kvm)" with task_memcg=/qemu.slice/<VMID>.scope
|
||||
```
|
||||
|
||||
**Accessing a partially-booted VM via `qm guest exec`:**
|
||||
When SSH is down but QEMU restarted the VM (onboot=1), the guest agent may
|
||||
respond even though regular `qm agent <vmid> ping` fails. Use `qm guest exec`
|
||||
to run commands directly:
|
||||
```bash
|
||||
# From the PVE host where the VM runs:
|
||||
qm guest exec <vmid> -- /bin/bash -c "hostname; uptime; systemctl is-active mariadb; free -m | head -2"
|
||||
# Returns JSON with exitcode, out-data, out-truncated fields
|
||||
```
|
||||
This is the fastest way to check if MariaDB is still in SST/activating state
|
||||
when SSH is not yet available. Parse output with:
|
||||
```bash
|
||||
qm guest exec <vmid> -- /bin/bash -c "systemctl is-active mariadb" | python3 -c "import json,sys; print(json.load(sys.stdin).get('out-data','').strip())"
|
||||
```
|
||||
|
||||
**Root Cause:** RAM overcommit on the PVE host:
|
||||
```
|
||||
overcommit = sum(running VM maxmem) / node physical RAM
|
||||
```
|
||||
Example: proxmox2 had 15 GB RAM but 22 GB VMs allocated (VM 129=12GB + VM 300=8GB + VM 311=2GB = 147%).
|
||||
|
||||
**Why `balloon: 0` is correct for Galera VMs:**
|
||||
MySQL/Galera's InnoDB buffer pool holds hot pages under constant access.
|
||||
If the PVE balloon driver reclaims memory, the guest kernel swaps those hot
|
||||
pages to swap → massive I/O latency spikes → Galera flow control kicks in
|
||||
and throttles writes across the ENTIRE cluster. One ballooned Galera node
|
||||
can slow down the whole cluster. `balloon: 0` (disabled) ensures the VM
|
||||
always holds its full allocated RAM.
|
||||
|
||||
**Fix sequence (user preference: RCA → restart → mitigation, in that order):**
|
||||
1. **RCA first** — diagnose root cause BEFORE restarting. Check `dmesg -T | grep oom` on the PVE host, correlate with Galera journal timestamps on surviving nodes. Understand WHY the VM died before bringing it back.
|
||||
2. **Restart the killed VM** — `qm stop <vmid>; qm start <vmid>` (or via HA manager). Wait for SST to complete (Galera rejoins via State Transfer from a surviving node). Expect 30-60 minutes SST time depending on DB size. During SST, the node shows `activating` in systemctl and `Joiner` in wsrep state.
|
||||
3. **Mitigation** — address the root cause to prevent recurrence:
|
||||
- Migrate the VM to a node with RAM headroom
|
||||
- Set strict HA node-affinity rules to prevent failover to small nodes
|
||||
- Deploy RAM-based rebalancer (see PVE skill `scripts/pve-ram-rebalancer.sh`)
|
||||
4. **Verify** — confirm Galera cluster size = 3, all nodes Synced, MaxScale routing restored
|
||||
|
||||
### 5.3c Galera/MaxScale VM Placement on PVE (Updated 2026-07-13)
|
||||
|
||||
**⚠️ CRITICAL: Quorum-Aware Placement Rule**
|
||||
|
||||
Each Galera node MUST be on a different physical PVE host. With 3 Galera
|
||||
nodes, losing any single host must leave ≥2 nodes alive (= quorum). If 2
|
||||
nodes share a host and that host fails, only 1 survives → **no quorum,
|
||||
entire cluster goes down** (non-primary state, all writes blocked).
|
||||
|
||||
**Placement checklist:**
|
||||
1. Count Galera nodes per PVE host — max 1 per host
|
||||
2. Verify: for EACH host, if it fails, ≥2 Galera nodes remain on other hosts
|
||||
3. Include MaxScale VMs in the analysis — they should also be distributed
|
||||
4. Document the 3 allowed hosts in the HA `db-nodes` rule (strict=1)
|
||||
|
||||
**Previous placement (problematic — caused OOM + quorum risk):**
|
||||
| VM | Name | Node | RAM | Issue |
|
||||
|----|------|------|-----|-------|
|
||||
| 300 | mariadb-01 | proxmox2 (15GB) | 8 GB | OOM killed — co-located with 12GB K8s CP |
|
||||
| 301 | mariadb-02 | proxmox6 (15GB) | 8 GB | Overcommit risk |
|
||||
| 302 | mariadb-03 | proxmox4 (15GB) | 8 GB | 200% overcommit with K8s worker |
|
||||
| 310 | maxscale-01 | proxmox4 (15GB) | 2 GB | Same node as mariadb-03 |
|
||||
| 311 | maxscale-02 | proxmox2 (15GB) | 2 GB | Same node as mariadb-01 |
|
||||
|
||||
**Intermediate placement (quorum risk — 2 nodes on proxmox3):**
|
||||
| VM | Name | Node | Issue |
|
||||
|----|------|------|-------|
|
||||
| 300 | mariadb-01 | proxmox3 | 2 of 3 Galera on same host — quorum risk |
|
||||
| 301 | mariadb-02 | proxmox3 | If proxmox3 fails, only 1 node survives |
|
||||
| 302 | mariadb-03 | n5pro | OK |
|
||||
|
||||
**Corrected placement (3 different hosts, quorum-safe):**
|
||||
| VM | Name | Node | RAM | Safe? |
|
||||
|----|------|------|-----|-------|
|
||||
| 300 | mariadb-01 | n5pro (91GB) | 8 GB | ✅ Massive headroom |
|
||||
| 301 | mariadb-02 | proxmox3 (31GB) | 8 GB | ✅ Good headroom |
|
||||
| 302 | mariadb-03 | proxmox6 (15GB) | 8 GB | ⚠️ 113% overcommit but effective use is only 5.5 GB |
|
||||
| 310 | maxscale-01 | n5pro (91GB) | 2 GB | ✅ |
|
||||
| 311 | maxscale-02 | proxmox3 (31GB) | 2 GB | ✅ |
|
||||
|
||||
HA rule `db-nodes` (strict=1) restricts all 5 VMs to n5pro + proxmox3 +
|
||||
proxmox6 (3 hardware-diverse hosts). Anti-affinity rules keep MaxScale
|
||||
instances apart (310≠311) and Galera nodes apart (301≠302).
|
||||
|
||||
**⚠️ PITFALL: Live Migration Can Kill Recently-Recovering Galera Nodes**
|
||||
|
||||
Migrating a Galera VM that is still in `activating` state (SST in progress)
|
||||
or that was recently OOM-killed and restarted can cause it to drop from the
|
||||
cluster and become unreachable (guest agent down, SSH refused). Sequence:
|
||||
1. OOM-kill destroys KVM process
|
||||
2. QEMU restarts VM (onboot=1) but guest OS boots incompletely
|
||||
3. Live migration triggers → VM freezes during migration
|
||||
4. After migration: guest agent down, SSH "No route to host", MariaDB stuck
|
||||
|
||||
**Safe migration sequence for recovering Galera VMs:**
|
||||
1. Wait until `systemctl is-active mariadb` = `active` AND `wsrep_local_state_comment` = `Synced`
|
||||
2. THEN migrate — never migrate during SST/activating state
|
||||
3. After migration, verify guest agent responds: `qm agent <vmid> ping`
|
||||
4. If guest agent is down post-migration: `qm stop <vmid>; qm start <vmid>` (full restart, not live migrate)
|
||||
|
||||
**⚠️ PITFALL: Anti-Affinity Rules Block Migrations**
|
||||
|
||||
Negative resource-affinity rules (e.g., `vm:310` ↔ `vm:311` must not
|
||||
co-locate) will cause `qm migrate` to fail with:
|
||||
```
|
||||
cannot migrate resource 'vm:311' to node 'n5pro':
|
||||
- resource 'vm:310' on target node 'n5pro' in negative affinity with resource 'vm:311'
|
||||
```
|
||||
Always check `/etc/pve/ha/rules.cfg` for conflicting rules before migrating.
|
||||
The target node must not contain any VM that has negative affinity with
|
||||
the VM being migrated.
|
||||
|
||||
---
|
||||
|
||||
### 6.1 The `maxscale` MySQL User Trick
|
||||
|
||||
|
||||
Reference in New Issue
Block a user