hermes-skills/devops/mariadb-galera-cluster-administration/SKILL.md at e9cc10662516e4b1d2ba8d8803ed80060e4bda62

dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

21 KiB

Raw Blame History

name, description, version, tags

name

description

version

Overview

This umbrella skill covers MariaDB Galera cluster administration with MaxScale as the SQL proxy layer:

Total Cluster Failure Recovery — bootstrap, sequential node rejoin, SST deadlock avoidance
MaxScale Routing During Recovery — available_when_donor, readwritesplit tuning, maintenance mode
Health Verification — wsrep status, cluster size, SST progress monitoring
Application Dependency Recovery — HA Recorder reconnect after DB outage

Load this skill for any Galera cluster operation: startup, shutdown, node addition, SST troubleshooting, split-brain recovery, or MaxScale routing issues.

Section 1: Total Cluster Failure Recovery

1.1 Symptoms of Total Failure

All nodes show systemctl status mariadb → Active: failed
Journal shows Failed to reach primary view and Failed to open channel 'mariadb-galera' at 'gcomm://...': -110 (Connection timed out)
No grastate.ini present (or safe_to_bootstrap: 0 on all nodes — pick any node if absent)
Application errors: MySQLdb.OperationalError: (2013, 'Lost connection to server during query')

1.2 Recovery Procedure (CRITICAL — sequential, not parallel!)

Step 1: Bootstrap ONE node

# On the chosen bootstrap node (e.g. db1 at 10.0.30.71):
sudo galera_new_cluster
# This runs mariadbd with --wsrep-new-cluster
# May appear to timeout but continues in background — verify separately

Verify bootstrap:

sudo systemctl is-active mariadb          # → active
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"  # → Synced
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';"          # → 1
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"       # → Primary

Step 2: Start remaining nodes SEQUENTIALLY (one at a time)

# On db2:
sudo systemctl start --no-block mariadb
# POLL until wsrep_local_state_comment = "Synced" BEFORE starting db3

⚠️ PITFALL: Starting multiple nodes simultaneously causes SST DEADLOCK. All joining nodes request SST from *any*, but the only available donor becomes overloaded serving multiple concurrent SST streams. Result: "No donor candidates temporarily available in suitable state" — all nodes abort.

Step 3: Repeat for each subsequent node only after the previous one reaches Synced.

1.3 SST Monitoring

# On the joining node (joiner):
sudo systemctl is-active mariadb    # → "activating" during SST
sudo du -sh /var/lib/mysql/         # grows toward source size (~18-20GB)
sudo journalctl -u mariadb -n 5     # "Waiting for SST streaming to complete!"
ps aux | grep -E 'socat|mariabackup|mbstream'  # SST processes running
sudo ss -tnp | grep 4444            # active SST connection from donor

# On the source node (donor):
sudo mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"  # → "Donor/Desynced"
ps aux | grep mariabackup           # backup process streaming to joiner

SST duration scales with DB size: ~18GB ≈ 30-40 minutes per node on typical homelab hardware.

1.4 Background Polling Pattern

SST takes too long for foreground terminal timeout. Use background polling:

ssh debian@JOINER_IP "
for i in \$(seq 1 120); do
  STATUS=\$(sudo systemctl is-active mariadb 2>/dev/null)
  if [ \"\$STATUS\" = 'active' ]; then
    WSREP=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'\" 2>/dev/null | awk '/wsrep_local_state_comment/{print \$2}')
    SIZE=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'\" 2>/dev/null | awk '/wsrep_cluster_size/{print \$2}')
    echo \"Poll \$i: ACTIVE, state=\$WSREP, cluster_size=\$SIZE\"
    if [ \"\$WSREP\" = 'Synced' ]; then
      echo 'NODE FULLY SYNCED!'
      break
    fi
  else
    SIZE_DIR=\$(sudo du -sh /var/lib/mysql/ 2>/dev/null | awk '{print \$1}')
    echo \"Poll \$i: \$STATUS, datadir=\$SIZE_DIR\"
  fi
  sleep 10
done
"

Use background=true, notify_on_complete=true with timeout=600 (or higher).

Section 2: MaxScale Routing Configuration

2.1 The Donor Routing Problem (CRITICAL)

During SST, the donor node has state Donor/Desynced. By default, MaxScale's galeramon monitor marks donor nodes as unavailable, so readwritesplit won't route traffic to them. With only one synced node acting as donor, MaxScale has zero routable backends → applications get connection errors.

Fix: Enable available_when_donor

maxctrl alter monitor galera-monitor available_when_donor=true

This makes the donor node accept traffic while serving SST. Verify:

maxctrl list servers
# Donor node should show: "Master, Synced, Running, Donor/Desynced" with Connections > 0

After all nodes are synced, revert:

maxctrl alter monitor galera-monitor available_when_donor=false

2.2 Other Useful MaxScale Commands

# Server states
maxctrl list servers

# Set/clear maintenance (prevent traffic to a node)
maxctrl set server db2 maintenance
maxctrl clear server db2 maintenance

# Allow reads on master when no slaves available
maxctrl alter service rw-router master_accept_reads=true

# View monitor/service config
maxctrl show monitor galera-monitor
maxctrl show service rw-router

# MaxScale listener check
maxctrl list listeners

2.3 Testing Connectivity Through MaxScale

# From MaxScale host (localhost):
mariadb -h 127.0.0.1 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 'OK' AS r, @@hostname;"

# From application host (via VIP):
mariadb -h 10.0.30.70 -P 3306 -u ha_recorder -p'PASSWORD' --skip-ssl -e "SELECT 1;"

2.4 Common MaxScale Issues

Symptom	Cause	Fix
No routable backends	Donor excluded by default	`available_when_donor=true`
"No valid servers from which to query MariaDB user accounts"	All backends down	Recover Galera first
readwritesplit finds no slave	Only master available	`master_accept_reads=true`
`set server db1 master` fails	Monitored server, can't set role manually	Use `maintenance`/`drain` only
All servers show "Down" but MariaDB is running	MaxScale VM IP blocked by `max_connection_errors` on Galera nodes	`FLUSH HOSTS;` on all 3 nodes (see §2.5)
`Could not find valid server for target type TARGET_ALL` in logs	All backends Down — same root cause as above or Galera cluster down	Fix Galera first or FLUSH HOSTS

2.5 MaxScale IP Blocked by max_connection_errors (CRITICAL)

Symptoms: maxctrl list servers shows all 3 Galera nodes as Down, but mariadb on each node is running and wsrep_cluster_size=3. MaxScale logs show Could not find valid server for target type TARGET_ALL every ~30 seconds.

Root Cause: The MaxScale VM IP (e.g., 10.0.30.82) accumulated too many connection errors on the Galera nodes (typically after a cluster outage or restart). MariaDB's max_connect_errors counter blocks the host after reaching the threshold (default 100), returning error 1129: Host '10.0.30.82' is blocked because of many connection errors.

Diagnosis:

# Test MaxScale monitor connectivity from MaxScale VM to a Galera node:
ssh debian@10.0.30.70 "mariadb -h 10.0.30.71 -P 3306 -u maxscale -p'<maxscale-password>' -e 'SELECT 1;'"
# ERROR 2002 (HY000): ... 1129 - Host '10.0.30.82' is blocked because of many connection errors

Fix: Flush hosts on ALL 3 Galera nodes:

for node in 10.0.30.71 10.0.30.72 10.0.30.73; do
  ssh -i ~/.ssh/id_ed25519_proxmox debian@$node "sudo mariadb -e 'FLUSH HOSTS;'"
done

Wait 5 seconds for the monitor to re-check, then verify:

ssh debian@10.0.30.70 "maxctrl list servers"
# All 3 should show: Slave/Master, Synced, Running

Prevention: Increase max_connect_errors on Galera nodes or add max_connect_errors=1000000 to the server config to prevent blocking during cluster instability.

Note: This scenario recurs frequently after any cluster instability (restart, network blip, SST). Always check maxctrl list servers for "Down" nodes first — if Galera itself is healthy (wsrep_cluster_size=3, wsrep_ready=ON), FLUSH HOSTS is almost certainly the fix.

Confirmed again 2026-07-06: MaxScale VM 10.0.30.82 blocked on all 3 Galera nodes after extended downtime. FLUSH HOSTS on all 3 nodes resolved it immediately — maxctrl list servers showed all 3 nodes "Synced, Running" within seconds. The maxscale@% MySQL user existed with correct grants and password, but the host was blocked at the MariaDB connection-error-counter level (error 1129).

2.6 MaxScale VIP Architecture (Homelab)

Component	Address	Role
MaxScale VIP	`10.0.30.70:3306`	rw (readwritesplit)
MaxScale VIP	`10.0.30.70:3307`	ro (readconnroute)
MaxScale VM 01	`10.0.30.81` (VM 310, proxmox4)	Active MaxScale instance
db1	`10.0.30.71` (VM 300, proxmox2)	Galera node
db2	`10.0.30.72` (VM 301, proxmox6)	Galera node
db3	`10.0.30.73` (VM 302, proxmox4)	Galera node

Applications connect through 10.0.30.70:3306 for automatic failover and load balancing. The maxscale MySQL user (with SUPER privilege) can be used for SET GLOBAL tuning commands through the VIP without SSH access to nodes.

Note: A second MaxScale VM (VM501 "MaxScale1") exists on proxmox4 but is currently stopped. The VIP 10.0.30.70 is served by Keepalived on the active MaxScale VM. PVE HA may migrate VMs — always verify current placement with qm list before operating.

Section 3: Home Assistant Recorder Dependency

3.1 Architecture

Home Assistant (HAOS VM) → MaxScale VIP (10.0.30.70:3306) → Galera Cluster (3 nodes)

HA configuration.yaml recorder config:

recorder:
  db_url: mysql://ha_recorder:PASSWORD@10.0.30.70:3306/homeassistant?charset=utf8mb4
  purge_keep_days: 30
  db_max_retries: 20

3.2 When Recorder Fails

HA logs: Error during connection setup: (retrying in 5 seconds) / MySQLdb.OperationalError: (2013, 'Lost connection to server during query')
After retries exhausted: Recorder setup failed, recorder shutting down
Downstream: history, logbook integrations fail (depend on recorder)
recorder may show as loaded in API but history/logbook domains missing

3.3 Recovery

Ensure Galera cluster + MaxScale routing is functional FIRST
Restart HA: POST /api/services/homeassistant/restart with bearer token
Wait for state: RUNNING (may take 2-3 min with large DB schema migration check)
Verify: GET /api/services includes recorder, history, logbook
Test: GET /api/history/period?filter_entity_id=sun.sun → HTTP 200 with data

3.3b Silent Recorder Disconnect (No Errors Logged)

Different from §3.2 — Recorder can lose its DB connection WITHOUT logging any errors. HA runs normally (entities update, automations fire), but no state data is written to the DB. History/Logbook show no recent data.

Triggers: MaxScale FLUSH HOSTS, Galera node restart, MaxScale VIP failover — any brief connection interruption that the recorder thread doesn't recover from.

Diagnosis (does NOT require HA API — all checks on Galera directly):

Check recorder_runs for an open run:
```
SELECT run_id, start, `end` FROM recorder_runs ORDER BY run_id DESC LIMIT 3\G
```
Open run (end = NULL) with old start = recorder disconnected mid-run.
Check latest state timestamp:
```
SELECT FROM_UNIXTIME(MAX(last_updated_ts)) AS latest_state FROM states;
```
If hours/days behind current time → recorder stopped writing.

⚠️ Do NOT run SELECT COUNT(*) FROM states — with 62M+ rows / 6.6 GB, this hangs for 30+ seconds. Use information_schema.table_rows instead:

SELECT table_name, table_rows, ROUND(data_length/1024/1024) AS data_mb
FROM information_schema.tables WHERE table_schema='homeassistant';

Check MaxScale sessions for ha_recorder:
```
maxctrl list sessions
```
If ha_recorder absent → connection is dead.

Fix: ha core restart (via SSH to HAOS on port 22222):

ssh -p 22222 root@10.0.30.10 "ha core restart"

Recorder reconnects and resumes writing immediately. Data gap between disconnect and restart is permanently lost. InfluxDB (if configured) may still have the data — it's a separate integration with its own connection.

3.4 HA Access Methods

HA API: curl -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/...
HA VM logs via Proxmox guest agent: qm guest exec 106 -- bash -c 'docker logs homeassistant 2>&1 | tail -30'
HA config: /mnt/data/supervisor/homeassistant/configuration.yaml (accessed via Proxmox host SSH)

Section 4: Verification Checklist

After full cluster recovery:

✅ All 3 nodes: wsrep_local_state_comment = Synced, wsrep_cluster_size = 3
✅ MaxScale: all servers Running, Synced, no Donor/Desynced
✅ available_when_donor reverted to false
✅ MaxScale connection test succeeds via VIP
✅ HA services: recorder, history, logbook all loaded
✅ HA History API returns HTTP 200 with data
✅ No ERROR (Recorder) entries in recent HA logs

Section 5: Application-Level Performance on Galera

5.1 Bulk-Load IDs for Dedup (Avoid Per-Row Roundtrips)

Every query through MaxScale → Galera incurs network latency + replication overhead. Patterns that do per-row existence checks (SELECT 1 FROM tbl WHERE id=? inside a loop) are catastrophically slow — a scraper checking 30k URLs against 28k existing rows took minutes appearing "hung" with zero output.

Wrong:

for url in urls:
    if db_has_id(likely_id):  # one roundtrip per URL
        continue
    fetch_and_insert(url)

Right:

existing_ids = set()  # load once
cur.execute("SELECT id FROM recipes")
existing_ids = {r[0] for r in cur.fetchall()}
for url in urls:
    if likely_id in existing_ids:  # in-memory, nanoseconds
        continue
    fetch_and_insert(url)
    existing_ids.add(new_id)  # keep set fresh

Loading 28k IDs into a Python set() takes ~0.3s. Every subsequent check is O(1) in-memory.

5.2 PyMySQL `executemany` Dict Field Pitfall

pymysql.cursors.Cursor.executemany() raises TypeError: dict can not be used as parameter if any value in the batch tuple is a Python dict. This occurs when source data has inconsistent typing — e.g., Schema.org JSON-LD recipeCategory is usually a string but sometimes a dict ({"@type": "Thing", "name": "Dessert"}) or a list.

Fix: Normalize every field to a scalar before building the row tuple:

cat = r.get("category", "") or ""
if isinstance(cat, dict):
    cat = cat.get("name", "") or str(cat)
elif isinstance(cat, list):
    cat = ", ".join(str(c) for c in cat)

Apply the same pattern to calories, rating_value, and any field sourced from semi-structured JSON-LD. Validate with a dry-run scan (isinstance(field, dict) check across all records) before the first executemany batch.

5.3 Creating Application Databases on Galera

To add a new database + user for an application (e.g., recipe storage):

# SSH to any Galera node (e.g., 10.0.30.71)
ssh debian@10.0.30.71
sudo mariadb -e "
  CREATE DATABASE IF NOT EXISTS <dbname> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
  CREATE USER IF NOT EXISTS '<user>'@'%' IDENTIFIED BY '<password>';
  GRANT ALL PRIVILEGES ON <dbname>.* TO '<user>'@'%';
  FLUSH PRIVILEGES;
"

The database propagates to all nodes automatically via Galera replication. Applications connect through the MaxScale VIP (10.0.30.70:3306), never directly to individual nodes.

PyMySQL connection (when mysql CLI is not installed on the application host):

import pymysql
conn = pymysql.connect(
    host="10.0.30.70", port=3306,
    user="<user>", password="<password>",
    charset="utf8mb4", database="<dbname>",
)

FULLTEXT indexes on MariaDB require ENGINE=InnoDB and CHARSET=utf8mb4. Create them in the schema DDL, not as a post-migration ALTER TABLE (which locks the table on large datasets).

Section 6: Dynamic Performance Tuning via SET GLOBAL

6.1 The `maxscale` MySQL User Trick

The maxscale MySQL user (stored in 1Password item mariadb-galera-vm, field maxscale-password) has SUPER privilege. This means you can run SET GLOBAL commands through the MaxScale VIP without SSH access to the Galera nodes:

import pymysql
conn = pymysql.connect(
    host="10.0.30.70", port=3306,
    user="maxscale", password="<maxscale-password>",
    charset="utf8mb4"
)
cur = conn.cursor()
cur.execute("SET GLOBAL wsrep_slave_threads=16")

This is the fastest way to tune Galera — no SSH, no config file edits, instant effect. Changes persist until the next MariaDB restart. To make them permanent, the same parameters must be added to the server config (/etc/mysql/mariadb.conf.d/50-server.cnf or equivalent) via SSH.

6.2 Key Tunable Parameters (Ordered by Impact)

Parameter	Default	Recommended	Why
`wsrep_slave_threads`	4	16	Applier threads — if `wsrep_local_recv_queue_avg` > 1, nodes can't keep up
`innodb_io_capacity`	200	2000	200 is HDD-tier; SSDs need 2000+
`innodb_io_capacity_max`	2000	4000	Burst capacity for flush storms
`innodb_read_io_threads`	4	8	Parallel read threads
`innodb_write_io_threads`	4	8	Parallel write threads
`innodb_flush_neighbors`	1	0	1 coalesces neighboring pages (HDD optimization); unnecessary on SSD
`innodb_purge_threads`	4	8	Cleanup threads for massive deletes (e.g., HA purges)
`innodb_adaptive_hash_index`	OFF	ON	Speeds up point lookups on large tables (states, recipes)
`long_query_time`	10	2	Catch slow queries in the slow query log

Parameters that cannot be changed at runtime (require restart + config file):

innodb_buffer_pool_size (4 GB → 8-16 GB recommended for HA workloads)
innodb_log_file_size (96 MB → 512 MB)
gcache.size (128 MB → 1 GB, wsrep_provider_options, prevents expensive SST on rejoin)

6.3 Diagnosing HA Recorder Slowness

When HA history/state charts load slowly, the bottleneck is almost always the states table in the homeassistant database, not Galera itself.

Diagnosis procedure:

Connect as ha_recorder (password in 1Password mariadb-galera-vm, field ha-recorder-password):

conn = pymysql.connect(host="10.0.30.70", port=3306, user="ha_recorder",
    password="<ha-recorder-password>", database="homeassistant", charset="utf8mb4")

Check states table size:

SELECT table_name, ROUND(SUM(data_length+index_length)/1024/1024,1) AS size_mb, table_rows
FROM information_schema.tables WHERE table_schema='homeassistant'
GROUP BY table_name ORDER BY size_mb DESC LIMIT 10;

Find the entity's metadata_id:

SELECT metadata_id, entity_id FROM states_meta
WHERE entity_id LIKE '%stromverbrauch%';

Time queries at different ranges (24h, 7d, 30d):

SELECT COUNT(*) FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>');

Check the index is being used:

EXPLAIN SELECT * FROM states
WHERE metadata_id=<ID> AND last_updated_ts >= UNIX_TIMESTAMP('<datetime>')
ORDER BY last_updated_ts;

Red flags:

states table > 10 GB or > 50M rows → needs purging or recorder excludes
A single entity with > 1M state changes in 30 days → high-frequency sensor flooding the recorder
Query time for 7-day range > 5s → Galera applier backlog or insufficient buffer pool
wsrep_local_recv_queue_avg > 1 → applier threads can't keep up (increase wsrep_slave_threads)

6.4 Making Changes Permanent

SET GLOBAL changes are lost on restart. To persist, SSH to each Galera node and edit the server config:

ssh debian@10.0.30.71
sudo tee -a /etc/mysql/mariadb.conf.d/50-server.cnf << 'EOF'
[mariadb]
wsrep_slave_threads=16
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_flush_neighbors=0
innodb_purge_threads=8
innodb_adaptive_hash_index=ON
long_query_time=2
EOF
sudo systemctl restart mariadb

Repeat on all 3 nodes. Restart sequentially (one at a time, verify Synced before next).

references/galera-cluster-topology.md — Homelab Galera/MaxScale topology, IPs, VM mapping, credentials locations
references/galera-recovery-runbook.md — Step-by-step recovery transcript from June 2026 outage
references/galera-performance-tuning.md — Performance tuning session: before/after metrics, HA states table diagnosis

21 KiB Raw Blame History