dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/mariadb-galera-cluster-administration/SKILL.md
+504
View File
@@ -0,0 +1,504 @@
---
name: mariadb-galera-cluster-administration
description: "Class-level skill for MariaDB Galera cluster + MaxScale proxy administration. Covers total-cluster-failure recovery, sequential SST, MaxScale routing during donor/desynced states, health verification, and HA Recorder dependency troubleshooting."
version: 1.0.0
tags: [mariadb, galera, maxscale, database, cluster, sst, mysql, ha, homeassistant]
---
## Overview
This umbrella skill covers MariaDB Galera cluster administration with MaxScale as the SQL proxy layer:
- **Total Cluster Failure Recovery** — bootstrap, sequential node rejoin, SST deadlock avoidance
- **MaxScale Routing During Recovery** — `available_when_donor`, readwritesplit tuning, maintenance mode
- **Health Verification** — wsrep status, cluster size, SST progress monitoring
- **Application Dependency Recovery** — HA Recorder reconnect after DB outage
Load this skill for any Galera cluster operation: startup, shutdown, node addition, SST troubleshooting, split-brain recovery, or MaxScale routing issues.
---
## Section 1: Total Cluster Failure Recovery
### 1.1 Symptoms of Total Failure
- All nodes show `systemctl status mariadb` → `Active: failed`
- Journal shows `Failed to reach primary view` and `Failed to open channel 'mariadb-galera' at 'gcomm://...'`: `-110 (Connection timed out)`
- No `grastate.ini` present (or `safe_to_bootstrap: 0` on all nodes — pick any node if absent)
- Application errors: `MySQLdb.OperationalError: (2013, 'Lost connection to server during query')`
### 1.2 Recovery Procedure (CRITICAL — sequential, not parallel!)
**Step 1: Bootstrap ONE node**
```bash
# On the chosen bootstrap node (e.g. db1 at 10.0.30.71):
sudo galera_new_cluster
# This runs mariadbd with --wsrep-new-cluster
# May appear to timeout but continues in background — verify separately
```
Verify bootstrap:
```bash
sudo systemctl is-active mariadb # → active
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" # → Synced
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';" # → 1
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';" # → Primary
```
**Step 2: Start remaining nodes SEQUENTIALLY (one at a time)**
```bash
# On db2:
sudo systemctl start --no-block mariadb
# POLL until wsrep_local_state_comment = "Synced" BEFORE starting db3
```
**⚠️ PITFALL: Starting multiple nodes simultaneously causes SST DEADLOCK.**
All joining nodes request SST from `*any*`, but the only available donor becomes overloaded serving multiple concurrent SST streams. Result: `"No donor candidates temporarily available in suitable state"` — all nodes abort.
**Step 3: Repeat for each subsequent node** only after the previous one reaches `Synced`.
### 1.3 SST Monitoring
```bash
# On the joining node (joiner):
sudo systemctl is-active mariadb # → "activating" during SST
sudo du -sh /var/lib/mysql/ # grows toward source size (~18-20GB)
sudo journalctl -u mariadb -n 5 # "Waiting for SST streaming to complete!"
ps aux | grep -E 'socat|mariabackup|mbstream' # SST processes running
sudo ss -tnp | grep 4444 # active SST connection from donor
# On the source node (donor):
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" # → "Donor/Desynced"
ps aux | grep mariabackup # backup process streaming to joiner
```
SST duration scales with DB size: ~18GB ≈ 30-40 minutes per node on typical homelab hardware.
### 1.4 Background Polling Pattern
SST takes too long for foreground terminal timeout. Use background polling:
```bash
ssh debian@JOINER_IP "
for i in \$(seq 1 120); do
STATUS=\$(sudo systemctl is-active mariadb 2>/dev/null)
if [ \"\$STATUS\" = 'active' ]; then
WSREP=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'\" 2>/dev/null | awk '/wsrep_local_state_comment/{print \$2}')
SIZE=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'\" 2>/dev/null | awk '/wsrep_cluster_size/{print \$2}')
echo \"Poll \$i: ACTIVE, state=\$WSREP, cluster_size=\$SIZE\"
if [ \"\$WSREP\" = 'Synced' ]; then
echo 'NODE FULLY SYNCED!'
break
fi
else
SIZE_DIR=\$(sudo du -sh /var/lib/mysql/ 2>/dev/null | awk '{print \$1}')
echo \"Poll \$i: \$STATUS, datadir=\$SIZE_DIR\"
fi
sleep 10
done
"
```
Use `background=true, notify_on_complete=true` with `timeout=600` (or higher).
---
## Section 2: MaxScale Routing Configuration
### 2.1 The Donor Routing Problem (CRITICAL)
During SST, the donor node has state `Donor/Desynced`. By default, MaxScale's galeramon monitor marks donor nodes as unavailable, so `readwritesplit` won't route traffic to them. With only one synced node acting as donor, **MaxScale has zero routable backends** → applications get connection errors.
**Fix: Enable `available_when_donor`**
```bash
maxctrl alter monitor galera-monitor available_when_donor=true
```
This makes the donor node accept traffic while serving SST. Verify:
```bash
maxctrl list servers
# Donor node should show: "Master, Synced, Running, Donor/Desynced" with Connections > 0
```
**After all nodes are synced, revert:**
```bash
maxctrl alter monitor galera-monitor available_when_donor=false
```
### 2.2 Other Useful MaxScale Commands
```bash
# Server states
maxctrl list servers
# Set/clear maintenance (prevent traffic to a node)
maxctrl set server db2 maintenance
maxctrl clear server db2 maintenance
# Allow reads on master when no slaves available
maxctrl alter service rw-router master_accept_reads=true
# View monitor/service config
maxctrl show monitor galera-monitor
maxctrl show service rw-router
# MaxScale listener check
maxctrl list listeners
```
### 2.3 Testing Connectivity Through MaxScale
```bash
# From MaxScale host (localhost):
mariadb -h 127.0.0.1 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 'OK' AS r, @@hostname;"
# From application host (via VIP):
mariadb -h 10.0.30.70 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 1;"
```
### 2.4 Common MaxScale Issues
| Symptom | Cause | Fix |
|---------|-------|-----|
| No routable backends | Donor excluded by default | `available_when_donor=true` |
| "No valid servers from which to query MariaDB user accounts" | All backends down | Recover Galera first |
| readwritesplit finds no slave | Only master available | `master_accept_reads=true` |
| `set server db1 master` fails | Monitored server, can't set role manually | Use `maintenance`/`drain` only |
| All servers show "Down" but MariaDB is running | MaxScale VM IP blocked by `max_connection_errors` on Galera nodes | `FLUSH HOSTS;` on all 3 nodes (see §2.5) |
| `Could not find valid server for target type TARGET_ALL` in logs | All backends Down — same root cause as above or Galera cluster down | Fix Galera first or FLUSH HOSTS |
### 2.5 MaxScale IP Blocked by max_connection_errors (CRITICAL)
**Symptoms:** `maxctrl list servers` shows all 3 Galera nodes as `Down`, but
`mariadb` on each node is running and `wsrep_cluster_size=3`. MaxScale logs show
`Could not find valid server for target type TARGET_ALL` every ~30 seconds.
**Root Cause:** The MaxScale VM IP (e.g., 10.0.30.82) accumulated too many
connection errors on the Galera nodes (typically after a cluster outage or
restart). MariaDB's `max_connect_errors` counter blocks the host after reaching
the threshold (default 100), returning error 1129:
`Host '10.0.30.82' is blocked because of many connection errors`.
**Diagnosis:**
```bash
# Test MaxScale monitor connectivity from MaxScale VM to a Galera node:
ssh debian@10.0.30.70 "mariadb -h 10.0.30.71 -P 3306 -u maxscale -p'<maxscale-password>' -e 'SELECT 1;'"
# ERROR 2002 (HY000): ... 1129 - Host '10.0.30.82' is blocked because of many connection errors
```
**Fix:** Flush hosts on ALL 3 Galera nodes:
```bash
for node in 10.0.30.71 10.0.30.72 10.0.30.73; do
ssh -i ~/.ssh/id_ed25519_proxmox debian@$node "sudo mariadb -e 'FLUSH HOSTS;'"
done
```
Wait 5 seconds for the monitor to re-check, then verify:
```bash
ssh debian@10.0.30.70 "maxctrl list servers"
# All 3 should show: Slave/Master, Synced, Running
```
**Prevention:** Increase `max_connect_errors` on Galera nodes or add
`max_connect_errors=1000000` to the server config to prevent blocking during
cluster instability.
**Note:** This scenario recurs frequently after any cluster instability
(restart, network blip, SST). Always check `maxctrl list servers` for "Down"
nodes first — if Galera itself is healthy (`wsrep_cluster_size=3`,
`wsrep_ready=ON`), `FLUSH HOSTS` is almost certainly the fix.
**Confirmed again 2026-07-06**: MaxScale VM 10.0.30.82 blocked on all 3
Galera nodes after extended downtime. `FLUSH HOSTS` on all 3 nodes
resolved it immediately — `maxctrl list servers` showed all 3 nodes
"Synced, Running" within seconds. The `maxscale@%` MySQL user existed
with correct grants and password, but the host was blocked at the
MariaDB connection-error-counter level (error 1129).
### 2.6 MaxScale VIP Architecture (Homelab)
| Component | Address | Role |
|-----------|---------|------|
| MaxScale VIP | `10.0.30.70:3306` | rw (readwritesplit) |
| MaxScale VIP | `10.0.30.70:3307` | ro (readconnroute) |
| MaxScale VM 01 | `10.0.30.81` (VM 310, proxmox4) | Active MaxScale instance |
| db1 | `10.0.30.71` (VM 300, proxmox2) | Galera node |
| db2 | `10.0.30.72` (VM 301, proxmox6) | Galera node |
| db3 | `10.0.30.73` (VM 302, proxmox4) | Galera node |
Applications connect through `10.0.30.70:3306` for automatic failover and
load balancing. The `maxscale` MySQL user (with SUPER privilege) can be used
for `SET GLOBAL` tuning commands through the VIP without SSH access to nodes.
**Note**: A second MaxScale VM (VM501 "MaxScale1") exists on proxmox4
but is currently **stopped**. The VIP `10.0.30.70` is served by
Keepalived on the active MaxScale VM. PVE HA may migrate VMs — always
verify current placement with `qm list` before operating.
---
## Section 3: Home Assistant Recorder Dependency
### 3.1 Architecture
```
Home Assistant (HAOS VM) → MaxScale VIP (10.0.30.70:3306) → Galera Cluster (3 nodes)
```
HA `configuration.yaml` recorder config:
```yaml
recorder:
db_url: mysql://ha_recorder:PASSWORD@10.0.30.70:3306/homeassistant?charset=utf8mb4
purge_keep_days: 30
db_max_retries: 20
```
### 3.2 When Recorder Fails
- HA logs: `Error during connection setup: (retrying in 5 seconds)` / `MySQLdb.OperationalError: (2013, 'Lost connection to server during query')`
- After retries exhausted: `Recorder setup failed, recorder shutting down`
- Downstream: `history`, `logbook` integrations fail (depend on recorder)
- `recorder` may show as loaded in API but `history`/`logbook` domains missing
### 3.3 Recovery
1. Ensure Galera cluster + MaxScale routing is functional FIRST
2. Restart HA: `POST /api/services/homeassistant/restart` with bearer token
3. Wait for `state: RUNNING` (may take 2-3 min with large DB schema migration check)
4. Verify: `GET /api/services` includes `recorder`, `history`, `logbook`
5. Test: `GET /api/history/period?filter_entity_id=sun.sun` → HTTP 200 with data
### 3.3b Silent Recorder Disconnect (No Errors Logged)
**Different from §3.2** — Recorder can lose its DB connection WITHOUT
logging any errors. HA runs normally (entities update, automations fire),
but no state data is written to the DB. History/Logbook show no recent data.
**Triggers:** MaxScale `FLUSH HOSTS`, Galera node restart, MaxScale VIP
failover — any brief connection interruption that the recorder thread
doesn't recover from.
**Diagnosis (does NOT require HA API — all checks on Galera directly):**
1. Check `recorder_runs` for an open run:
```sql
SELECT run_id, start, `end` FROM recorder_runs ORDER BY run_id DESC LIMIT 3\G
```
Open run (`end = NULL`) with old `start` = recorder disconnected mid-run.
2. Check latest state timestamp:
```sql
SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;
```
If hours/days behind current time → recorder stopped writing.
3. **⚠️ Do NOT run `SELECT COUNT(*) FROM states`** — with 62M+ rows / 6.6 GB,
this hangs for 30+ seconds. Use `information_schema.table_rows` instead:
```sql
SELECT table_name, table_rows, ROUND(data_length/1024/1024) AS data_mb
FROM information_schema.tables WHERE table_schema='homeassistant';
```
4. Check MaxScale sessions for `ha_recorder`:
```bash
maxctrl list sessions
```
If `ha_recorder` absent → connection is dead.
**Fix:** `ha core restart` (via SSH to HAOS on port 22222):
```bash
ssh -p 22222 root@10.0.30.10 "ha core restart"
```
Recorder reconnects and resumes writing immediately. Data gap between
disconnect and restart is permanently lost. InfluxDB (if configured) may
still have the data — it's a separate integration with its own connection.
### 3.4 HA Access Methods
- HA API: `curl -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/...`
- HA VM logs via Proxmox guest agent: `qm guest exec 106 -- bash -c 'docker logs homeassistant 2>&1 | tail -30'`
- HA config: `/mnt/data/supervisor/homeassistant/configuration.yaml` (accessed via Proxmox host SSH)
---
## Section 4: Verification Checklist
After full cluster recovery:
1. ✅ All 3 nodes: `wsrep_local_state_comment = Synced`, `wsrep_cluster_size = 3`
2. ✅ MaxScale: all servers `Running, Synced`, no `Donor/Desynced`
3. ✅ `available_when_donor` reverted to `false`
4. ✅ MaxScale connection test succeeds via VIP
5. ✅ HA services: `recorder`, `history`, `logbook` all loaded
6. ✅ HA History API returns HTTP 200 with data
7. ✅ No `ERROR (Recorder)` entries in recent HA logs
## Section 5: Application-Level Performance on Galera
### 5.1 Bulk-Load IDs for Dedup (Avoid Per-Row Roundtrips)
Every query through MaxScale → Galera incurs network latency + replication overhead. Patterns that do per-row existence checks (`SELECT 1 FROM tbl WHERE id=?` inside a loop) are catastrophically slow — a scraper checking 30k URLs against 28k existing rows took minutes appearing "hung" with zero output.
**Wrong:**
```python
for url in urls:
if db_has_id(likely_id): # one roundtrip per URL
continue
fetch_and_insert(url)
```
**Right:**
```python
existing_ids = set() # load once
cur.execute("SELECT id FROM recipes")
existing_ids = {r[0] for r in cur.fetchall()}
for url in urls:
if likely_id in existing_ids: # in-memory, nanoseconds
continue
fetch_and_insert(url)
existing_ids.add(new_id) # keep set fresh
```
Loading 28k IDs into a Python `set()` takes ~0.3s. Every subsequent check is O(1) in-memory.
### 5.2 PyMySQL `executemany` Dict Field Pitfall
`pymysql.cursors.Cursor.executemany()` raises `TypeError: dict can not be used as parameter` if any value in the batch tuple is a Python `dict`. This occurs when source data has inconsistent typing — e.g., Schema.org JSON-LD `recipeCategory` is usually a string but sometimes a dict (`{"@type": "Thing", "name": "Dessert"}`) or a list.
**Fix:** Normalize every field to a scalar before building the row tuple:
```python
cat = r.get("category", "") or ""
if isinstance(cat, dict):
cat = cat.get("name", "") or str(cat)
elif isinstance(cat, list):
cat = ", ".join(str(c) for c in cat)
```
Apply the same pattern to `calories`, `rating_value`, and any field sourced from semi-structured JSON-LD. Validate with a dry-run scan (`isinstance(field, dict)` check across all records) before the first `executemany` batch.
### 5.3 Creating Application Databases on Galera
To add a new database + user for an application (e.g., recipe storage):
```bash
# SSH to any Galera node (e.g., 10.0.30.71)
ssh debian@10.0.30.71
sudo mariadb -e "
CREATE DATABASE IF NOT EXISTS <dbname> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS '<user>'@'%' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON <dbname>.* TO '<user>'@'%';
FLUSH PRIVILEGES;
"
```
The database propagates to all nodes automatically via Galera replication. Applications connect through the MaxScale VIP (`10.0.30.70:3306`), never directly to individual nodes.
**PyMySQL connection** (when `mysql` CLI is not installed on the application host):
```python
import pymysql
conn = pymysql.connect(
host="10.0.30.70", port=3306,
user="<user>", password="<password>",
charset="utf8mb4", database="<dbname>",
)
```
FULLTEXT indexes on MariaDB require `ENGINE=InnoDB` and `CHARSET=utf8mb4`. Create them in the schema DDL, not as a post-migration `ALTER TABLE` (which locks the table on large datasets).
---
---
## Section 6: Dynamic Performance Tuning via SET GLOBAL
### 6.1 The `maxscale` MySQL User Trick
The `maxscale` MySQL user (stored in 1Password item `mariadb-galera-vm`, field `maxscale-password`) has `SUPER` privilege. This means you can run `SET GLOBAL` commands **through the MaxScale VIP** without SSH access to the Galera nodes:
```python
import pymysql
conn = pymysql.connect(
host="10.0.30.70", port=3306,
user="maxscale", password="<maxscale-password>",
charset="utf8mb4"
)
cur = conn.cursor()
cur.execute("SET GLOBAL wsrep_slave_threads=16")
```
This is the fastest way to tune Galera — no SSH, no config file edits, instant effect. Changes persist until the next MariaDB restart. To make them permanent, the same parameters must be added to the server config (`/etc/mysql/mariadb.conf.d/50-server.cnf` or equivalent) via SSH.
### 6.2 Key Tunable Parameters (Ordered by Impact)
| Parameter | Default | Recommended | Why |
|-----------|---------|-------------|-----|
| `wsrep_slave_threads` | 4 | 16 | Applier threads — if `wsrep_local_recv_queue_avg` > 1, nodes can't keep up |
| `innodb_io_capacity` | 200 | 2000 | 200 is HDD-tier; SSDs need 2000+ |
| `innodb_io_capacity_max` | 2000 | 4000 | Burst capacity for flush storms |
| `innodb_read_io_threads` | 4 | 8 | Parallel read threads |
| `innodb_write_io_threads` | 4 | 8 | Parallel write threads |
| `innodb_flush_neighbors` | 1 | 0 | 1 coalesces neighboring pages (HDD optimization); unnecessary on SSD |
| `innodb_purge_threads` | 4 | 8 | Cleanup threads for massive deletes (e.g., HA purges) |
| `innodb_adaptive_hash_index` | OFF | ON | Speeds up point lookups on large tables (states, recipes) |
| `long_query_time` | 10 | 2 | Catch slow queries in the slow query log |
Parameters that **cannot** be changed at runtime (require restart + config file):
- `innodb_buffer_pool_size` (4 GB → 8-16 GB recommended for HA workloads)
- `innodb_log_file_size` (96 MB → 512 MB)
- `gcache.size` (128 MB → 1 GB, wsrep_provider_options, prevents expensive SST on rejoin)
### 6.3 Diagnosing HA Recorder Slowness
When HA history/state charts load slowly, the bottleneck is almost always the `states` table in the `homeassistant` database, not Galera itself.
**Diagnosis procedure:**
1. Connect as `ha_recorder` (password in 1Password `mariadb-galera-vm`, field `ha-recorder-password`):
```python
conn = pymysql.connect(host="10.0.30.70", port=3306, user="ha_recorder",
password="<ha-recorder-password>", database="homeassistant", charset="utf8mb4")
```
2. Check `states` table size:
```sql
SELECT table_name, ROUND(SUM(data_length+index_length)/1024/1024,1) AS size_mb, table_rows
FROM information_schema.tables WHERE table_schema='homeassistant'
GROUP BY table_name ORDER BY size_mb DESC LIMIT 10;
```
3. Find the entity's `metadata_id`:
```sql
SELECT metadata_id, entity_id FROM states_meta
WHERE entity_id LIKE '%stromverbrauch%';
```
4. Time queries at different ranges (24h, 7d, 30d):
```sql
SELECT COUNT(*) FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>');
```
5. Check the index is being used:
```sql
EXPLAIN SELECT * FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>')
ORDER BY last_updated_ts;
```
**Red flags:**
- `states` table > 10 GB or > 50M rows → needs purging or recorder excludes
- A single entity with > 1M state changes in 30 days → high-frequency sensor flooding the recorder
- Query time for 7-day range > 5s → Galera applier backlog or insufficient buffer pool
- `wsrep_local_recv_queue_avg` > 1 → applier threads can't keep up (increase `wsrep_slave_threads`)
### 6.4 Making Changes Permanent
`SET GLOBAL` changes are lost on restart. To persist, SSH to each Galera node and edit the server config:
```bash
ssh debian@10.0.30.71
sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
[mariadb]
wsrep_slave_threads=16
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_neighbors=0
innodb_purge_threads=8
innodb_adaptive_hash_index=ON
long_query_time=2
EOF
sudo systemctl restart mariadb
```
Repeat on all 3 nodes. Restart sequentially (one at a time, verify `Synced` before next).
---
- `references/galera-cluster-topology.md` — Homelab Galera/MaxScale topology, IPs, VM mapping, credentials locations
- `references/galera-recovery-runbook.md` — Step-by-step recovery transcript from June 2026 outage
- `references/galera-performance-tuning.md` — Performance tuning session: before/after metrics, HA states table diagnosis
devops/mariadb-galera-cluster-administration/references/galera-cluster-topology.md
+92
View File
@@ -0,0 +1,92 @@
# Homelab Galera/MaxScale Topology
## MariaDB Galera Cluster (3 nodes)
| Node | VM ID | Proxmox Host | IP | Hostname |
|------|-------|---------------|----|----------|
| db1 | 300 | proxmox2 (10.0.20.20) | 10.0.30.71 | mariadb-01 |
| db2 | 301 | proxmox6 (10.0.20.60) | 10.0.30.72 | mariadb-02 |
| db3 | 302 | proxmox4 (10.0.20.40) | 10.0.30.73 | mariadb-03 |
- MariaDB version: 11.4.10-MariaDB-deb12
- Cluster name: `mariadb-galera`
- Galera address: `gcomm://10.0.30.71,10.0.30.72,10.0.30.73`
- DB size: ~18GB (homeassistant schema)
- **Note**: PVE HA may migrate VMs. Always verify current placement
with `qm list` on each node before operating.
## MaxScale Proxy (1 active + VIP)
| Node | VM ID | Proxmox Host | IP | Hostname | Status |
|------|-------|--------------|----|----------|--------|
| maxscale-01 | 310 | proxmox4 (10.0.20.40) | 10.0.30.81 | maxscale-01 | Active |
| VIP | — | — | 10.0.30.70 | keepalived VIP | — |
- MaxScale version: 24.02.9
- Services: `rw-router` (readwritesplit), `ro-router` (readconnroute)
- Monitor: `galera-monitor` (galeramon)
- Listener: `rw-listener` on port 3306
- **Note**: A second MaxScale VM (VM501 "MaxScale1") exists on
proxmox4 but is **stopped**. If a standby MaxScale is needed, check
VM501's config. The VIP `10.0.30.70` is the authoritative entry point.
## SSH Access
```bash
# MariaDB VMs (user: debian)
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.72
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73
# MaxScale VMs (user: debian)
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81
# Proxmox hosts (user: root)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 # proxmox1 (HA VM host)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.91 # n5pro
```
## Credentials
All credentials stored in 1Password Vault "Hermes", item `mariadb-galera-vm`:
- `root-password` — MySQL root (localhost only on each node, not accessible via MaxScale VIP)
- `maxscale-password` — MaxScale MySQL user with `SUPER` privilege — **can run `SET GLOBAL` via VIP** (see Section 6)
- `mariabackup-password` — Galera SST transfer auth
- `ha-recorder-password` — HA Recorder MySQL user, has SELECT/INSERT/UPDATE on `homeassistant` DB
- `keepalived-auth-pass` — keepalived VIP failover auth
Retrieve all fields:
```bash
op item get "mariadb-galera-vm" --vault "Hermes" --reveal --fields password,root-password,mariabackup-password,maxscale-password,ha-recorder-password
```
Note: The generic `password` field is empty. Use the named fields instead.
## Firewall (UFW on MariaDB VMs)
- Port 3306: ALLOW from 10.0.30.81, 10.0.30.82 (MaxScale only)
- Ports 4567, 4568, 4444: ALLOW between Galera nodes (10.0.30.71-73)
## Home Assistant
- VM 106 on proxmox1 (10.0.20.10)
- Internal: http://10.0.30.10:8123
- External: https://homeassistant.familie-schoen.com
- Config: `/mnt/data/supervisor/homeassistant/configuration.yaml`
- Recorder DB URL: `mysql://ha_recorder:***@10.0.30.70:3306/homeassistant`
- InfluxDB: 10.0.30.109:8086, bucket `ha_hot_90d` (long-term history)
- HA masks passwords with `***` in configuration.yaml — retrieve real passwords from 1Password, not the config file
### HAOS SSH Access (Port 22222)
HAOS exposes SSH on port 22222 (requires SSH addon enabled):
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
```
Inside HAOS, HA runs as Docker container `homeassistant`:
```bash
docker exec homeassistant cat /config/configuration.yaml
docker exec homeassistant python3 -c "..."
docker logs homeassistant 2>&1 | grep -i recorder | tail -20
```
Note: `pip` modules like `pymysql` are NOT available inside the HA container. Use `sqlalchemy` (bundled with HA) or query the DB from outside.
devops/mariadb-galera-cluster-administration/references/galera-performance-tuning.md
+134
View File
@@ -0,0 +1,134 @@
# Galera Performance Tuning Session (June 29, 2026)
## Problem
HA state history for "Stromverbrauch Haus netto" took very long to load. User suspected MySQL was too slow.
## Diagnosis
### Root Cause: NOT Galera itself — HA `states` table explosion
| Finding | Value |
|---------|-------|
| Registered HA entities | 3,416 (1,860 sensors) |
| `states` table size | 16,338 MB (16.3 GB) |
| `states` table rows | 61,378,779 |
| Total HA DB size | 18.31 GB |
| States per entity (30d) | ~2,049,022 for one high-frequency sensor |
| `wsrep_local_recv_queue_avg` | 7.36 (should be < 1) |
| `innodb_buffer_pool_size` | 4 GB (too small for 16 GB states table) |
| `innodb_io_capacity` | 200 (HDD-tier, should be 2000+) |
| `wsrep_slave_threads` | 4 (insufficient for write load) |
| `gcache.size` | 128 MB (too small, risks expensive SST on rejoin) |
### Query Performance BEFORE Tuning
| Time Range | Rows | Duration |
|------------|------|----------|
| 24h | 79,062 | 0.18s |
| 7 days | 220,340 | **10.1s** |
| 30 days | 2,049,022 | **60.0s** |
Index `ix_states_metadata_id_last_updated_ts` was used correctly but the sheer row count made even indexed scans slow.
### How to Connect for Diagnosis
1. **MySQL access via MaxScale VIP** using `maxscale` user (has `SUPER` privilege):
```python
import pymysql
conn = pymysql.connect(host="10.0.30.70", port=3306,
user="maxscale", password="<maxscale-password>", charset="utf8mb4")
```
2. **HA database access** using `ha_recorder` user:
```python
conn = pymysql.connect(host="10.0.30.70", port=3306,
user="ha_recorder", password="<ha-recorder-password>",
database="homeassistant", charset="utf8mb4")
```
3. **Credentials source**: 1Password item `mariadb-galera-vm`, vault `Hermes`:
```bash
op item get "mariadb-galera-vm" --vault "Hermes" --reveal \
--fields maxscale-password,ha-recorder-password,root-password
```
4. **HAOS SSH** (port 22222) for HA-side investigation:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 -p 22222
# Inside: docker exec homeassistant <command>
```
## Changes Applied (SET GLOBAL, instant effect)
All 9 parameters changed successfully via `maxscale` MySQL user through MaxScale VIP — no SSH required:
| Parameter | Before | After | Effect |
|-----------|--------|-------|--------|
| `wsrep_slave_threads` | 4 | 16 | More applier threads for replication backlog |
| `innodb_io_capacity` | 200 | 2000 | SSD-appropriate write throughput |
| `innodb_io_capacity_max` | 2000 | 4000 | Burst capacity |
| `innodb_read_io_threads` | 4 | 8 | Double read parallelism |
| `innodb_write_io_threads` | 4 | 8 | Double write parallelism |
| `innodb_flush_neighbors` | 1 | 0 | Remove HDD-era coalescing on SSD |
| `innodb_purge_threads` | 4 | 8 | Faster cleanup of deleted rows |
| `innodb_adaptive_hash_index` | OFF | ON | Speed up point lookups on large tables |
| `long_query_time` | 10 | 2 | Log slow queries for diagnosis |
## Query Performance AFTER Tuning
| Time Range | Rows | Before | After | Improvement |
|------------|------|--------|-------|-------------|
| 24h | 79,130 | 0.18s | 0.02s | **9×** |
| 7 days | 220,408 | 10.1s | 0.05s | **200×** |
| 30 days | 2,049,090 | 60.0s | 0.39s | **154×** |
| SELECT + JOIN LIMIT 1000 | 1000 | — | 0.01s | — |
| SELECT + JOIN LIMIT 500 | 500 | — | 0.008s | — |
## Parameters Requiring Restart (Not Yet Changed)
These cannot be changed via `SET GLOBAL` and require editing the server config + restarting each node:
| Parameter | Current | Target | Notes |
|-----------|---------|--------|-------|
| `innodb_buffer_pool_size` | 4 GB | 8-16 GB | Must fit working set (states table = 16 GB) |
| `innodb_log_file_size` | 96 MB | 512 MB | Small redo log throttles write throughput |
| `gcache.size` | 128 MB | 1 GB | Prevents expensive SST on node rejoin |
| `innodb_buffer_pool_instances` | unknown | 4-8 | Should match buffer pool / 1GB |
## Making Changes Permanent
`SET GLOBAL` changes are lost on MariaDB restart. To persist, SSH to each Galera node and append to server config:
```bash
ssh debian@10.0.30.71
sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
[mariadb]
wsrep_slave_threads=16
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_neighbors=0
innodb_purge_threads=8
innodb_adaptive_hash_index=ON
long_query_time=2
innodb_buffer_pool_size=8G
innodb_log_file_size=512M
wsrep_provider_options="gcache.size=1G"
EOF
sudo systemctl restart mariadb
```
Repeat on all 3 nodes sequentially (verify `wsrep_local_state_comment=Synced` before next node).
## HA Recorder Recommendations (Not Applied — User Declined)
User was offered but declined recorder config changes:
- `recorder.exclude` for noisy domains (automation, button, select, update)
- `recorder.commit_interval: 5` (reduce write frequency from 1s to 5s)
- Move long-term history to InfluxDB (already configured at 10.0.30.109:8086, bucket `ha_hot_90d`)
- `OPTIMIZE TABLE states` to reclaim 619 MB fragmented space
## Key Insight
The `maxscale` MySQL user has `SUPER` privilege and can run `SET GLOBAL` through the MaxScale VIP. This enables rapid Galera tuning without SSH access to the individual nodes — critical when SSH keys aren't set up for the DB VMs but 1Password credentials are available.
devops/mariadb-galera-cluster-administration/references/galera-recovery-runbook.md
+66
View File
@@ -0,0 +1,66 @@
# Galera Cluster Recovery Runbook
Based on June 27, 2026 recovery of total Galera cluster failure (down since June 24 ~14:46 CEST).
## Timeline
1. **June 24 ~14:46** — All 3 Galera nodes crashed simultaneously. Root cause: likely simultaneous network issue preventing inter-node communication → "Failed to reach primary view" → all nodes aborted.
2. **June 27 13:00** — Recovery began. db1 bootstrapped, db2/db3 stopped to resolve SST deadlock.
3. **June 27 13:07** — db2 SST started (mariabackup, ~18GB)
4. **June 27 13:17** — db2 SST completed (~40 min), state=Synced, cluster_size=2
5. **June 27 13:17** — db3 SST started
6. **June 27 ~13:50** — db3 SST completed, cluster_size=3
## Key Decisions Made
### 1. Bootstrap from Node 1 (no grastate.ini)
No `grastate.ini` existed on any node, so no `safe_to_bootstrap` flag. Any node can be chosen. Picked db1 (10.0.30.71).
### 2. Stop db2/db3 to Resolve SST Deadlock
Initially tried starting both db2 and db3 after db1 bootstrap. Both entered SST but deadlocked: `"No donor candidates temporarily available in suitable state"`. Solution: stop both, then start them one at a time.
### 3. MaxScale `available_when_donor=true`
Without this, MaxScale refused to route to db1 while it was Donor/Desynced serving SST to db2. HA Recorder got `Lost connection to server during query` errors. Setting this flag fixed routing immediately.
### 4. Two HA Restarts Needed
First restart failed because MaxScale wasn't routing yet (before `available_when_donor=true`). Second restart after the fix succeeded — Recorder connected, history/logbook loaded.
## Pitfalls Encountered
### PITFALL: `galera_new_cluster` Appears to Timeout
`galera_new_cluster` may appear to hang/timeout in the terminal but actually launches mariadbd in the background successfully. Always verify separately with `systemctl is-active` and `mariadb -e "SHOW STATUS LIKE 'wsrep_%'"`.
### PITFALL: Parallel SST Causes Deadlock
Starting 2+ nodes simultaneously when only 1 donor exists → all joiners compete for the same donor → donor can't serve multiple SST streams → deadlock → all abort.
### PITFALL: HA Recorder Retries Exhaust Before DB Recovers
With `db_max_retries: 20` and 5-second intervals, Recorder gives up after ~100 seconds. If MaxScale isn't routing by then, Recorder shuts down. Requires HA restart after DB is confirmed accessible.
### PITFALL: MaxScale `set server db1 master` Doesn't Work
Monitored servers can only have `maintenance`/`drain` set manually. Role assignment is automatic via the galeramon monitor. Don't waste time trying to force roles.
### PITFALL: HA Takes Long to Reach RUNNING State
With ~18GB DB, HA's schema migration check (`pre_migrate_schema`) takes significant time. State stays `NOT_RUNNING` for 2-3 minutes. Don't assume failure — check docker logs for actual errors.
## Post-Recovery Cleanup
```bash
# Revert available_when_donor
maxctrl alter monitor galera-monitor available_when_donor=false
# Verify all nodes
maxctrl list servers
# All should show: "Synced, Running" with no Donor/Desynced
# Test HA
curl -s "$HA_URL/api/services" -H "Authorization: Bearer $HA_TOKEN" | python3 -c "
import sys,json
data=json.load(sys.stdin)
services=set(s['domain'] for s in data)
for d in ['recorder','history','logbook']:
print(f'{d}: {\"✓\" if d in services else \"✗\"}')
"
# Test history data
curl -s "$HA_URL/api/history/period?filter_entity_id=sun.sun" -H "Authorization: Bearer $HA_TOKEN"
```