dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/mariadb-galera-cluster-administration/SKILL.md
+504
View File
@@ -0,0 +1,504 @@
---
name: mariadb-galera-cluster-administration
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, and HA Recorder dependency troubleshooting."
version: 1.0.0
tags: [mariadb, galera, maxscale, database, cluster, sst, mysql, ha, homeassistant]
---
## Overview
This umbrella skill covers MariaDB Galera cluster administration with MaxScale as the SQL proxy layer:
- **Total Cluster Failure Recovery** — bootstrap, sequential node rejoin, SST deadlock avoidance
- **MaxScale Routing During Recovery** — `available_when_donor`, readwritesplit tuning, maintenance mode
- **Health Verification** — wsrep status, cluster size, SST progress monitoring
- **Application Dependency Recovery** — HA Recorder reconnect after DB outage
Load this skill for any Galera cluster operation: startup, shutdown, node addition, SST troubleshooting, split-brain recovery, or MaxScale routing issues.
---
## Section 1: Total Cluster Failure Recovery
### 1.1 Symptoms of Total Failure
- All nodes show `systemctl status mariadb` → `Active: failed`
- Journal shows `Failed to reach primary view` and `Failed to open channel 'mariadb-galera' at 'gcomm://...'`: `-110 (Connection timed out)`
- No `grastate.ini` present (or `safe_to_bootstrap: 0` on all nodes — pick any node if absent)
- Application errors: `MySQLdb.OperationalError: (2013, 'Lost connection to server during query')`
### 1.2 Recovery Procedure (CRITICAL — sequential, not parallel!)
**Step 1: Bootstrap ONE node**
```bash
# On the chosen bootstrap node (e.g. db1 at 10.0.30.71):
sudo galera_new_cluster
# This runs mariadbd with --wsrep-new-cluster
# May appear to timeout but continues in background — verify separately
```
Verify bootstrap:
```bash
sudo systemctl is-active mariadb # → active
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" # → Synced
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';" # → 1
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';" # → Primary
```
**Step 2: Start remaining nodes SEQUENTIALLY (one at a time)**
```bash
# On db2:
sudo systemctl start --no-block mariadb
# POLL until wsrep_local_state_comment = "Synced" BEFORE starting db3
```
**⚠️ PITFALL: Starting multiple nodes simultaneously causes SST DEADLOCK.**
All joining nodes request SST from `*any*`, but the only available donor becomes overloaded serving multiple concurrent SST streams. Result: `"No donor candidates temporarily available in suitable state"` — all nodes abort.
**Step 3: Repeat for each subsequent node** only after the previous one reaches `Synced`.
### 1.3 SST Monitoring
```bash
# On the joining node (joiner):
sudo systemctl is-active mariadb # → "activating" during SST
sudo du -sh /var/lib/mysql/ # grows toward source size (~18-20GB)
sudo journalctl -u mariadb -n 5 # "Waiting for SST streaming to complete!"
ps aux | grep -E 'socat|mariabackup|mbstream' # SST processes running
sudo ss -tnp | grep 4444 # active SST connection from donor
# On the source node (donor):
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" # → "Donor/Desynced"
ps aux | grep mariabackup # backup process streaming to joiner
```
SST duration scales with DB size: ~18GB ≈ 30-40 minutes per node on typical homelab hardware.
### 1.4 Background Polling Pattern
SST takes too long for foreground terminal timeout. Use background polling:
```bash
ssh debian@JOINER_IP "
for i in \$(seq 1 120); do
STATUS=\$(sudo systemctl is-active mariadb 2>/dev/null)
if [ \"\$STATUS\" = 'active' ]; then
WSREP=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'\" 2>/dev/null | awk '/wsrep_local_state_comment/{print \$2}')
SIZE=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'\" 2>/dev/null | awk '/wsrep_cluster_size/{print \$2}')
echo \"Poll \$i: ACTIVE, state=\$WSREP, cluster_size=\$SIZE\"
if [ \"\$WSREP\" = 'Synced' ]; then
echo 'NODE FULLY SYNCED!'
break
fi
else
SIZE_DIR=\$(sudo du -sh /var/lib/mysql/ 2>/dev/null | awk '{print \$1}')
echo \"Poll \$i: \$STATUS, datadir=\$SIZE_DIR\"
fi
sleep 10
done
"
```
Use `background=true, notify_on_complete=true` with `timeout=600` (or higher).
---
## Section 2: MaxScale Routing Configuration
### 2.1 The Donor Routing Problem (CRITICAL)
During SST, the donor node has state `Donor/Desynced`. By default, MaxScale's galeramon monitor marks donor nodes as unavailable, so `readwritesplit` won't route traffic to them. With only one synced node acting as donor, **MaxScale has zero routable backends** → applications get connection errors.
**Fix: Enable `available_when_donor`**
```bash
maxctrl alter monitor galera-monitor available_when_donor=true
```
This makes the donor node accept traffic while serving SST. Verify:
```bash
maxctrl list servers
# Donor node should show: "Master, Synced, Running, Donor/Desynced" with Connections > 0
```
**After all nodes are synced, revert:**
```bash
maxctrl alter monitor galera-monitor available_when_donor=false
```
### 2.2 Other Useful MaxScale Commands
```bash
# Server states
maxctrl list servers
# Set/clear maintenance (prevent traffic to a node)
maxctrl set server db2 maintenance
maxctrl clear server db2 maintenance
# Allow reads on master when no slaves available
maxctrl alter service rw-router master_accept_reads=true
# View monitor/service config
maxctrl show monitor galera-monitor
maxctrl show service rw-router
# MaxScale listener check
maxctrl list listeners
```
### 2.3 Testing Connectivity Through MaxScale
```bash
# From MaxScale host (localhost):
mariadb -h 127.0.0.1 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 'OK' AS r, @@hostname;"
# From application host (via VIP):
mariadb -h 10.0.30.70 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 1;"
```
### 2.4 Common MaxScale Issues
| Symptom | Cause | Fix |
|---------|-------|-----|
| No routable backends | Donor excluded by default | `available_when_donor=true` |
| "No valid servers from which to query MariaDB user accounts" | All backends down | Recover Galera first |
| readwritesplit finds no slave | Only master available | `master_accept_reads=true` |
| `set server db1 master` fails | Monitored server, can't set role manually | Use `maintenance`/`drain` only |
| All servers show "Down" but MariaDB is running | MaxScale VM IP blocked by `max_connection_errors` on Galera nodes | `FLUSH HOSTS;` on all 3 nodes (see §2.5) |
| `Could not find valid server for target type TARGET_ALL` in logs | All backends Down — same root cause as above or Galera cluster down | Fix Galera first or FLUSH HOSTS |
### 2.5 MaxScale IP Blocked by max_connection_errors (CRITICAL)
**Symptoms:** `maxctrl list servers` shows all 3 Galera nodes as `Down`, but
`mariadb` on each node is running and `wsrep_cluster_size=3`. MaxScale logs show
`Could not find valid server for target type TARGET_ALL` every ~30 seconds.
**Root Cause:** The MaxScale VM IP (e.g., 10.0.30.82) accumulated too many
connection errors on the Galera nodes (typically after a cluster outage or
restart). MariaDB's `max_connect_errors` counter blocks the host after reaching
the threshold (default 100), returning error 1129:
`Host '10.0.30.82' is blocked because of many connection errors`.
**Diagnosis:**
```bash
# Test MaxScale monitor connectivity from MaxScale VM to a Galera node:
ssh debian@10.0.30.70 "mariadb -h 10.0.30.71 -P 3306 -u maxscale -p'<maxscale-password>' -e 'SELECT 1;'"
# ERROR 2002 (HY000): ... 1129 - Host '10.0.30.82' is blocked because of many connection errors
```
**Fix:** Flush hosts on ALL 3 Galera nodes:
```bash
for node in 10.0.30.71 10.0.30.72 10.0.30.73; do
ssh -i ~/.ssh/id_ed25519_proxmox debian@$node "sudo mariadb -e 'FLUSH HOSTS;'"
done
```
Wait 5 seconds for the monitor to re-check, then verify:
```bash
ssh debian@10.0.30.70 "maxctrl list servers"
# All 3 should show: Slave/Master, Synced, Running
```
**Prevention:** Increase `max_connect_errors` on Galera nodes or add
`max_connect_errors=1000000` to the server config to prevent blocking during
cluster instability.
**Note:** This scenario recurs frequently after any cluster instability
(restart, network blip, SST). Always check `maxctrl list servers` for "Down"
nodes first — if Galera itself is healthy (`wsrep_cluster_size=3`,
`wsrep_ready=ON`), `FLUSH HOSTS` is almost certainly the fix.
**Confirmed again 2026-07-06**: MaxScale VM 10.0.30.82 blocked on all 3
Galera nodes after extended downtime. `FLUSH HOSTS` on all 3 nodes
resolved it immediately — `maxctrl list servers` showed all 3 nodes
"Synced, Running" within seconds. The `maxscale@%` MySQL user existed
with correct grants and password, but the host was blocked at the
MariaDB connection-error-counter level (error 1129).
### 2.6 MaxScale VIP Architecture (Homelab)
| Component | Address | Role |
|-----------|---------|------|
| MaxScale VIP | `10.0.30.70:3306` | rw (readwritesplit) |
| MaxScale VIP | `10.0.30.70:3307` | ro (readconnroute) |
| MaxScale VM 01 | `10.0.30.81` (VM 310, proxmox4) | Active MaxScale instance |
| db1 | `10.0.30.71` (VM 300, proxmox2) | Galera node |
| db2 | `10.0.30.72` (VM 301, proxmox6) | Galera node |
| db3 | `10.0.30.73` (VM 302, proxmox4) | Galera node |
Applications connect through `10.0.30.70:3306` for automatic failover and
load balancing. The `maxscale` MySQL user (with SUPER privilege) can be used
for `SET GLOBAL` tuning commands through the VIP without SSH access to nodes.
**Note**: A second MaxScale VM (VM501 "MaxScale1") exists on proxmox4
but is currently **stopped**. The VIP `10.0.30.70` is served by
Keepalived on the active MaxScale VM. PVE HA may migrate VMs — always
verify current placement with `qm list` before operating.
---
## Section 3: Home Assistant Recorder Dependency
### 3.1 Architecture
```
Home Assistant (HAOS VM) → MaxScale VIP (10.0.30.70:3306) → Galera Cluster (3 nodes)
```
HA `configuration.yaml` recorder config:
```yaml
recorder:
db_url: mysql://ha_recorder:PASSWORD@10.0.30.70:3306/homeassistant?charset=utf8mb4
purge_keep_days: 30
db_max_retries: 20
```
### 3.2 When Recorder Fails
- HA logs: `Error during connection setup: (retrying in 5 seconds)` / `MySQLdb.OperationalError: (2013, 'Lost connection to server during query')`
- After retries exhausted: `Recorder setup failed, recorder shutting down`
- Downstream: `history`, `logbook` integrations fail (depend on recorder)
- `recorder` may show as loaded in API but `history`/`logbook` domains missing
### 3.3 Recovery
1. Ensure Galera cluster + MaxScale routing is functional FIRST
2. Restart HA: `POST /api/services/homeassistant/restart` with bearer token
3. Wait for `state: RUNNING` (may take 2-3 min with large DB schema migration check)
4. Verify: `GET /api/services` includes `recorder`, `history`, `logbook`
5. Test: `GET /api/history/period?filter_entity_id=sun.sun` → HTTP 200 with data
### 3.3b Silent Recorder Disconnect (No Errors Logged)
**Different from §3.2** — Recorder can lose its DB connection WITHOUT
logging any errors. HA runs normally (entities update, automations fire),
but no state data is written to the DB. History/Logbook show no recent data.
**Triggers:** MaxScale `FLUSH HOSTS`, Galera node restart, MaxScale VIP
failover — any brief connection interruption that the recorder thread
doesn't recover from.
**Diagnosis (does NOT require HA API — all checks on Galera directly):**
1. Check `recorder_runs` for an open run:
```sql
SELECT run_id, start, `end` FROM recorder_runs ORDER BY run_id DESC LIMIT 3\G
```
Open run (`end = NULL`) with old `start` = recorder disconnected mid-run.
2. Check latest state timestamp:
```sql
SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;
```
If hours/days behind current time → recorder stopped writing.
3. **⚠️ Do NOT run `SELECT COUNT(*) FROM states`** — with 62M+ rows / 6.6 GB,
this hangs for 30+ seconds. Use `information_schema.table_rows` instead:
```sql
SELECT table_name, table_rows, ROUND(data_length/1024/1024) AS data_mb
FROM information_schema.tables WHERE table_schema='homeassistant';
```
4. Check MaxScale sessions for `ha_recorder`:
```bash
maxctrl list sessions
```
If `ha_recorder` absent → connection is dead.
**Fix:** `ha core restart` (via SSH to HAOS on port 22222):
```bash
ssh -p 22222 root@10.0.30.10 "ha core restart"
```
Recorder reconnects and resumes writing immediately. Data gap between
disconnect and restart is permanently lost. InfluxDB (if configured) may
still have the data — it's a separate integration with its own connection.
### 3.4 HA Access Methods
- HA API: `curl -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/...`
- HA VM logs via Proxmox guest agent: `qm guest exec 106 -- bash -c 'docker logs homeassistant 2>&1 | tail -30'`
- HA config: `/mnt/data/supervisor/homeassistant/configuration.yaml` (accessed via Proxmox host SSH)
---
## Section 4: Verification Checklist
After full cluster recovery:
1. ✅ All 3 nodes: `wsrep_local_state_comment = Synced`, `wsrep_cluster_size = 3`
2. ✅ MaxScale: all servers `Running, Synced`, no `Donor/Desynced`
3. ✅ `available_when_donor` reverted to `false`
4. ✅ MaxScale connection test succeeds via VIP
5. ✅ HA services: `recorder`, `history`, `logbook` all loaded
6. ✅ HA History API returns HTTP 200 with data
7. ✅ No `ERROR (Recorder)` entries in recent HA logs
## Section 5: Application-Level Performance on Galera
### 5.1 Bulk-Load IDs for Dedup (Avoid Per-Row Roundtrips)
Every query through MaxScale → Galera incurs network latency + replication overhead. Patterns that do per-row existence checks (`SELECT 1 FROM tbl WHERE id=?` inside a loop) are catastrophically slow — a scraper checking 30k URLs against 28k existing rows took minutes appearing "hung" with zero output.
**Wrong:**
```python
for url in urls:
if db_has_id(likely_id): # one roundtrip per URL
continue
fetch_and_insert(url)
```
**Right:**
```python
existing_ids = set() # load once
cur.execute("SELECT id FROM recipes")
existing_ids = {r[0] for r in cur.fetchall()}
for url in urls:
if likely_id in existing_ids: # in-memory, nanoseconds
continue
fetch_and_insert(url)
existing_ids.add(new_id) # keep set fresh
```
Loading 28k IDs into a Python `set()` takes ~0.3s. Every subsequent check is O(1) in-memory.
### 5.2 PyMySQL `executemany` Dict Field Pitfall
`pymysql.cursors.Cursor.executemany()` raises `TypeError: dict can not be used as parameter` if any value in the batch tuple is a Python `dict`. This occurs when source data has inconsistent typing — e.g., Schema.org JSON-LD `recipeCategory` is usually a string but sometimes a dict (`{"@type": "Thing", "name": "Dessert"}`) or a list.
**Fix:** Normalize every field to a scalar before building the row tuple:
```python
cat = r.get("category", "") or ""
if isinstance(cat, dict):
cat = cat.get("name", "") or str(cat)
elif isinstance(cat, list):
cat = ", ".join(str(c) for c in cat)
```
Apply the same pattern to `calories`, `rating_value`, and any field sourced from semi-structured JSON-LD. Validate with a dry-run scan (`isinstance(field, dict)` check across all records) before the first `executemany` batch.
### 5.3 Creating Application Databases on Galera
To add a new database + user for an application (e.g., recipe storage):
```bash
# SSH to any Galera node (e.g., 10.0.30.71)
ssh debian@10.0.30.71
sudo mariadb -e "
CREATE DATABASE IF NOT EXISTS <dbname> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS '<user>'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON <dbname>.* TO '<user>'@'%';
FLUSH PRIVILEGES;
"
```
The database propagates to all nodes automatically via Galera replication. Applications connect through the MaxScale VIP (`10.0.30.70:3306`), never directly to individual nodes.
**PyMySQL connection** (when `mysql` CLI is not installed on the application host):
```python
import pymysql
conn = pymysql.connect(
host="10.0.30.70", port=3306,
user="<user>", password="<password>",
charset="utf8mb4", database="<dbname>",
)
```
FULLTEXT indexes on MariaDB require `ENGINE=InnoDB` and `CHARSET=utf8mb4`. Create them in the schema DDL, not as a post-migration `ALTER TABLE` (which locks the table on large datasets).
---
---
## Section 6: Dynamic Performance Tuning via SET GLOBAL
### 6.1 The `maxscale` MySQL User Trick
The `maxscale` MySQL user (stored in 1Password item `mariadb-galera-vm`, field `maxscale-password`) has `SUPER` privilege. This means you can run `SET GLOBAL` commands **through the MaxScale VIP** without SSH access to the Galera nodes:
```python
import pymysql
conn = pymysql.connect(
host="10.0.30.70", port=3306,
user="maxscale", password="<maxscale-password>",
charset="utf8mb4"
)
cur = conn.cursor()
cur.execute("SET GLOBAL wsrep_slave_threads=16")
```
This is the fastest way to tune Galera — no SSH, no config file edits, instant effect. Changes persist until the next MariaDB restart. To make them permanent, the same parameters must be added to the server config (`/etc/mysql/mariadb.conf.d/50-server.cnf` or equivalent) via SSH.
### 6.2 Key Tunable Parameters (Ordered by Impact)
| Parameter | Default | Recommended | Why |
|-----------|---------|-------------|-----|
| `wsrep_slave_threads` | 4 | 16 | Applier threads — if `wsrep_local_recv_queue_avg` > 1, nodes can't keep up |
| `innodb_io_capacity` | 200 | 2000 | 200 is HDD-tier; SSDs need 2000+ |
| `innodb_io_capacity_max` | 2000 | 4000 | Burst capacity for flush storms |
| `innodb_read_io_threads` | 4 | 8 | Parallel read threads |
| `innodb_write_io_threads` | 4 | 8 | Parallel write threads |
| `innodb_flush_neighbors` | 1 | 0 | 1 coalesces neighboring pages (HDD optimization); unnecessary on SSD |
| `innodb_purge_threads` | 4 | 8 | Cleanup threads for massive deletes (e.g., HA purges) |
| `innodb_adaptive_hash_index` | OFF | ON | Speeds up point lookups on large tables (states, recipes) |
| `long_query_time` | 10 | 2 | Catch slow queries in the slow query log |
Parameters that **cannot** be changed at runtime (require restart + config file):
- `innodb_buffer_pool_size` (4 GB → 8-16 GB recommended for HA workloads)
- `innodb_log_file_size` (96 MB → 512 MB)
- `gcache.size` (128 MB → 1 GB, wsrep_provider_options, prevents expensive SST on rejoin)
### 6.3 Diagnosing HA Recorder Slowness
When HA history/state charts load slowly, the bottleneck is almost always the `states` table in the `homeassistant` database, not Galera itself.
**Diagnosis procedure:**
1. Connect as `ha_recorder` (password in 1Password `mariadb-galera-vm`, field `ha-recorder-password`):
```python
conn = pymysql.connect(host="10.0.30.70", port=3306, user="ha_recorder",
password="<ha-recorder-password>", database="homeassistant", charset="utf8mb4")
```
2. Check `states` table size:
```sql
SELECT table_name, ROUND(SUM(data_length+index_length)/1024/1024,1) AS size_mb, table_rows
FROM information_schema.tables WHERE table_schema='homeassistant'
GROUP BY table_name ORDER BY size_mb DESC LIMIT 10;
```
3. Find the entity's `metadata_id`:
```sql
SELECT metadata_id, entity_id FROM states_meta
WHERE entity_id LIKE '%stromverbrauch%';
```
4. Time queries at different ranges (24h, 7d, 30d):
```sql
SELECT COUNT(*) FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>');
```
5. Check the index is being used:
```sql
EXPLAIN SELECT * FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>')
ORDER BY last_updated_ts;
```
**Red flags:**
- `states` table > 10 GB or > 50M rows → needs purging or recorder excludes
- A single entity with > 1M state changes in 30 days → high-frequency sensor flooding the recorder
- Query time for 7-day range > 5s → Galera applier backlog or insufficient buffer pool
- `wsrep_local_recv_queue_avg` > 1 → applier threads can't keep up (increase `wsrep_slave_threads`)
### 6.4 Making Changes Permanent
`SET GLOBAL` changes are lost on restart. To persist, SSH to each Galera node and edit the server config:
```bash
ssh debian@10.0.30.71
sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
[mariadb]
wsrep_slave_threads=16
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_neighbors=0
innodb_purge_threads=8
innodb_adaptive_hash_index=ON
long_query_time=2
EOF
sudo systemctl restart mariadb
```
Repeat on all 3 nodes. Restart sequentially (one at a time, verify `Synced` before next).
---
- `references/galera-cluster-topology.md` — Homelab Galera/MaxScale topology, IPs, VM mapping, credentials locations
- `references/galera-recovery-runbook.md` — Step-by-step recovery transcript from June 2026 outage
- `references/galera-performance-tuning.md` — Performance tuning session: before/after metrics, HA states table diagnosis