# Galera Cluster Recovery for Home Assistant Database

## Architecture

```
HA (VM 106, 10.0.30.10)
  └─ recorder: mysql://ha_recorder@10.0.30.70:3306/homeassistant
       └─ MaxScale VIP (10.0.30.70, keepalived)
            ├─ maxscale-01 (10.0.30.81, VM 310, n5pro)
            └─ maxscale-02 (10.0.30.82, VM 311, proxmox2)
                 └─ readwritesplit → Galera cluster
                      ├─ db1 (10.0.30.71, VM 300, proxmox2)
                      ├─ db2 (10.0.30.72, VM 301, proxmox4)
                      └─ db3 (10.0.30.73, VM 302, proxmox5)
```

- **MariaDB**: 11.4.10-MariaDB-deb12
- **MaxScale**: 24.02.9
- **Galera cluster name**: `mariadb-galera`
- **Cluster address**: `gcomm://10.0.30.71,10.0.30.72,10.0.30.73`
- **DB size**: ~18GB (homeassistant schema)
- **SSH to DB nodes**: `ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.{71,72,73}`
- **SSH to MaxScale**: `ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81`

## Symptoms of Cluster Failure

- HA logs: `MySQLdb.OperationalError: (1045, "Access denied for user 'ha_recorder'@'10.0.30.10'")` or `(2013, "Lost connection to server during query")`
- HA: Recorder service loaded but History/Logbook domains missing
- All 3 MariaDB nodes: `systemctl status mariadb` shows `Active: failed`
- Port 3306 closed on all DB nodes
- MaxScale: all servers show "Down"

## Recovery Procedure

### Step 1: Verify All Nodes Are Down

```bash
for ip in 10.0.30.{71,72,73}; do
  echo "=== $ip ==="
  ssh -i ~/.ssh/id_ed25519_proxmox debian@$ip "
    sudo systemctl is-active mariadb
    sudo journalctl -u mariadb --no-pager -n 5
  "
done
```

Look for: `Failed to reach primary view`, `gcomm://...: -110 (Connection timed out)`, `Aborting`

### Step 2: Bootstrap From Node 1

```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71 "
  sudo systemctl stop mariadb 2>/dev/null
  sudo galera_new_cluster
"
```

Wait ~30s, then verify:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71 "
  sudo mariadb -e \"
    SHOW GLOBAL STATUS LIKE 'wsrep_ready';
    SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
    SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';
    SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
  \"
"
```

Expected: `wsrep_ready=ON`, `wsrep_cluster_size=1`, `wsrep_cluster_status=Primary`, `wsrep_local_state_comment=Synced`

### Step 3: Enable MaxScale Routing to Donor

During SST, the bootstrapped node becomes Donor/Desynced. MaxScale excludes Donor nodes by default.

```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81 "
  maxctrl alter monitor galera-monitor available_when_donor=true
  maxctrl list servers
"
```

Verify db1 shows "Master, Synced, Running, Donor/Desynced" and connections work:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81 "
  mariadb -h 127.0.0.1 -P 3306 -u ha_recorder -p'<password>' --skip-ssl -e \"SELECT 'OK' AS r, @@hostname;\"
"
```

### Step 4: Restart HA (NOW the DB is reachable)

```bash
source ~/.hermes/.env
curl -s -X POST "$HA_URL/api/services/homeassistant/restart" \
  -H "Authorization: Bearer $HA_TOKEN" \
  -H "Content-Type: application/json" -d '{}'
```

Poll for HA startup (takes 2-3 min with 18GB DB):
```bash
for i in $(seq 1 18); do
  sleep 10
  STATE=$(curl -s "$HA_URL/api/config" -H "Authorization: Bearer $HA_TOKEN" | python3 -c "import sys,json; print(json.load(sys.stdin).get('state','?'))" 2>/dev/null)
  echo "Check $i: $STATE"
  [ "$STATE" = "RUNNING" ] && break
done
```

Verify recorder/history/logbook:
```bash
curl -s "$HA_URL/api/services" -H "Authorization: Bearer $HA_TOKEN" | python3 -c "
import sys,json
data=json.load(sys.stdin)
services=set(s['domain'] for s in data)
for d in ['recorder','history','logbook']:
    print(f'{d}: {\"✓\" if d in services else \"✗\"}')
"
```

Test history API:
```bash
curl -s -w " HTTP:%{http_code}" "$HA_URL/api/history/period?filter_entity_id=sun.sun" \
  -H "Authorization: Bearer $HA_TOKEN" | head -1 | cut -c1-100
```

### Step 5: Join db2 (SEQUENTIAL — one at a time!)

```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.72 "sudo systemctl start --no-block mariadb"
```

Monitor SST progress (18GB DB takes ~30 min):
```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.72 "
  for i in \$(seq 1 120); do
    STATUS=\$(sudo systemctl is-active mariadb 2>/dev/null)
    if [ \"\$STATUS\" = 'active' ]; then
      WSREP=\$(sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'\" 2>/dev/null | awk '/wsrep_local_state_comment/{print \$2}')
      echo \"Poll \$i: ACTIVE, state=\$WSREP\"
      [ \"\$WSREP\" = 'Synced' ] && echo 'DB2 SYNCED!' && break
    else
      SIZE=\$(sudo du -sh /var/lib/mysql/ 2>/dev/null | awk '{print \$1}')
      echo \"Poll \$i: \$STATUS, datadir=\$SIZE\"
    fi
    sleep 10
  done
"
```

Also check from donor side:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.71 "
  sudo mariadb -e \"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';\"
  ps aux | grep mariabackup | grep -v grep
"
```

### Step 6: Join db3 (ONLY after db2 is Synced)

```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.73 "sudo systemctl start --no-block mariadb"
```

Same monitoring loop as Step 5.

### Step 7: Restore MaxScale Settings

Once all 3 nodes are Synced:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.81 "
  maxctrl alter monitor galera-monitor available_when_donor=false
  maxctrl clear server db2 maintenance
  maxctrl clear server db3 maintenance
  maxctrl list servers
"
```

All 3 should show "Synced, Running".

## Key Pitfalls

1. **Parallel SST deadlock**: Starting 2+ joiners simultaneously → "No donor candidates temporarily available in suitable state". Both stall forever. FIX: start one, wait for Synced, then start next.

2. **MaxScale `available_when_donor`**: Default `false` means MaxScale won't route ANY traffic while the only available node is a donor. HA Recorder gets "Lost connection" / "Access denied". FIX: `maxctrl alter monitor galera-monitor available_when_donor=true` before joining additional nodes.

3. **HA Recorder retry exhaustion**: `db_max_retries: 20` in configuration.yaml. If DB unreachable at boot, Recorder gives up after ~20 retries × 5s = 100s. History/Logbook never load. FIX: ensure DB is reachable via MaxScale BEFORE restarting HA.

4. **`maxctrl set server db1 master` fails on monitored servers**: "monitored server, only maintenance/drain can be altered". Don't try to manually set master role — the galera-monitor does this automatically based on wsrep state.

5. **HA startup time with large DB**: With 18GB DB, HA takes 2-3 minutes to start (schema migration/validation). Don't assume failure if `api/config` returns `NOT_RUNNING` for the first few minutes.

6. **grastate.dat may not exist**: On clean shutdown or fresh install, `grastate.ini`/`grastate.dat` may not be present. This doesn't prevent bootstrap — just run `galera_new_cluster` directly.

## Credentials

All DB credentials should be stored in 1Password (Vault "Hermes"):
- `ha_recorder` MySQL password (used in HA configuration.yaml db_url)
- MaxScale admin user/password
- Mariabackup SST auth credentials

Currently in HA configuration.yaml:
```yaml
recorder:
  db_url: mysql://ha_recorder:***@10.0.30.70:3306/homeassistant?charset=utf8mb4
  purge_keep_days: 30
  db_max_retries: 20
```