142 lines
8.4 KiB
Markdown
142 lines
8.4 KiB
Markdown
# Cluster Audit — 2026-07-03
|
|
|
|
## Session Context
|
|
|
|
Full cluster health audit performed at user request after HA management changes (added vm:106, ct:108, vm:230 to HA; normalized max_relocate/max_restart; cleaned ghost configs from removed "pve" node; repaired LRM on proxmox1).
|
|
|
|
## Cluster Overview
|
|
|
|
- **Cluster name:** Homelab
|
|
- **Nodes:** 8 (proxmox1-7, n5pro)
|
|
- **PVE version:** 9.2.3 on all nodes
|
|
- **Kernel variance:** proxmox1=6.17.4-2, proxmox2-3=7.0.2-6, proxmox4-7+n5pro=7.0.12-1
|
|
- **Corosync:** Single ring (knet, UDP, 10.0.20.0/24), link_mode=passive, quorum=5/8
|
|
- **HA-Manager:** 16 managed guests, CRM master on proxmox4, fencing armed
|
|
|
|
## Hardware Summary
|
|
|
|
| Node | CPU | RAM | Storage | OSDs | Mon | Role |
|
|
|------|-----|-----|---------|------|-----|------|
|
|
| proxmox1 | i7-8550U 8c | 15GB | 954G NVMe + 3.6T HDD | 1 (HDD) | — | HA VMs |
|
|
| proxmox2 | i3-9100T 4c | 15GB | 239G NVMe + 250G SSD | 2 (SSD) | — | MariaDB |
|
|
| proxmox3 | i3-9100T 4c | 31GB | 239G NVMe + 233G SSD | 1 (SSD) | — | — |
|
|
| proxmox4 | i3-9100T 4c | 15GB | 239G NVMe + 239G SSD | 1 (SSD) | YES | MariaDB, MaxScale |
|
|
| proxmox5 | i3-9100T 4c | 15GB | 239G NVMe + 239G SSD | 1 (SSD) | YES (leader) | — |
|
|
| proxmox6 | i3-9100T 4c | 15GB | 239G NVMe + 932G HDD | 1 (HDD) | — | Many CTs |
|
|
| proxmox7 | i3-9100T 4c | 15GB | 239G NVMe + 932G HDD | 1 (HDD) | YES | Hermes, LiteLLM |
|
|
| n5pro | Ryzen AI 9 HX PRO 370 24c | 92GB | 119G NVMe | 0 | — | GPU compute target |
|
|
|
|
## Issues Found & Actions Taken
|
|
|
|
### Actions Completed This Session
|
|
|
|
1. **HA status detection fix:** `/cluster/resources` API `ha` field is unreliable — all 55 guests reported as non-HA when 12 were HA-managed. Corrected method: use `ha-manager status` + `pvesh get /cluster/ha/resources`.
|
|
2. **LRM proxmox1 repair:** `pve-ha-lrm.service` was disabled+inactive since rolling update. Fixed: `systemctl enable && start pve-ha-lrm`.
|
|
3. **HA added:** vm:106 (homeassistant), ct:108 (gitea), vm:230 (hermes-agent) — all with max_relocate=3, max_restart=2.
|
|
4. **HA params normalized:** ct:104, ct:99999 reduced from 5/5 to 3/2.
|
|
5. **Ghost cleanup:** Deleted vm:101, vm:102, ct:902 from `/etc/pve/nodes/pve/` (removed node, not in corosync).
|
|
6. **Ceph Mons reduced:** 7→3 (removed ubuntu, proxmox1, proxmox2, proxmox3). MON_DISK_LOW warning eliminated.
|
|
7. **osd.6 purged:** Destroyed OSD removed from CRUSH tree. PG rebalancing initiated.
|
|
8. **proxmox2 root disk:** Removed unused pve-data thin pool (10.5G), extended pve-root 24.5G→37G (85%→50% usage).
|
|
|
|
### Remaining Recommendations (Prioritized)
|
|
|
|
#### High Priority
|
|
|
|
| # | Issue | Recommendation | Risk | Effort |
|
|
|---|-------|---------------|------|--------|
|
|
| 1 | tm_disks pool replica=2 | Increase to size=3 or decommission pool | Medium (data loss risk) | 10min + rebalance |
|
|
| 2 | OSD imbalance (VAR 0.64-1.96) | Reweight osd.7/10 down, osd.8/9 up | Low | 5min + time |
|
|
| 3 | BlueStore slow ops (osd.0, osd.1) | Related to OSD imbalance + old SSD (Samsung 860 EVO, 13k hrs) | — | Monitor/replace |
|
|
|
|
#### Medium Priority
|
|
|
|
| # | Issue | Recommendation | Risk | Effort |
|
|
|---|-------|---------------|------|--------|
|
|
| 4 | Single Corosync ring | Add second ring on separate VLAN | Low | Config change |
|
|
| 5 | Inconsistent bond modes | proxmox2/3/5: change round-robin→active-backup (single NIC) | Low | 5min/node |
|
|
| 6 | osd_memory_target=4GB on 15GB nodes | Reduce to 2-3GB | Low | 1 command |
|
|
| 7 | target_size_ratio=0.9 on 3 pools | Set realistic values (0.01-0.2) | Low | 5min |
|
|
| 8 | CephFS unused (0 clients) | Shut down MDS if not needed | Low | 5min |
|
|
| 9 | Kernel version inconsistency | Reboot proxmox1-3 to get 7.0.12 | Medium | Wartungsfenster |
|
|
|
|
#### Low Priority
|
|
|
|
| # | Issue | Recommendation |
|
|
|---|-------|---------------|
|
|
| 10 | osd_recovery_sleep=0.1 | Throttles recovery — set to 0 for urgent rebalances |
|
|
| 11 | OSD reweight 0.9 on osd.0/2/5 | Reset to 1.0 if not intentional |
|
|
| 12 | proxmox1 phantom disks (sda/sdb 0B) | Investigate and remove |
|
|
| 13 | n5pro ISO partition (sda 14.3G) | Remove leftover installation media |
|
|
| 14 | proxmox1 /boot/efi 85% full | Clean old kernel EFI entries |
|
|
|
|
## Node Root Disk Survey (2026-07-03)
|
|
|
|
| Node | Root-LV | Used | % | pve-data (thin) | VG Free | Notes |
|
|
|------|---------|------|---|-----------------|---------|-------|
|
|
| proxmox1 | 96G | 26G | 29% | — | 849G | Huge NVMe, mostly unallocated |
|
|
| proxmox2 | 37G | 18G | 50% | — (removed) | 0 | Extended this session |
|
|
| proxmox3 | 85G | 17G | 21% | 142G (0%) | 0 | Could reclaim pve-data |
|
|
| proxmox4 | 101G | 21G | 22% | 128G (0%) | 0 | Could reclaim pve-data |
|
|
| proxmox5 | 68G | 20G | 31% | 141G (0%) | 16G | Could reclaim pve-data |
|
|
| proxmox6 | 68G | 20G | 31% | — | 110G | Has 50G NVMe WAL (pve-ceph--db) |
|
|
| proxmox7 | 68G | 22G | 34% | — | 110G | Has 50G NVMe WAL (pve-ceph--db) |
|
|
| n5pro | 39G | 15G | 39% | 54G (0%) | 15G | Could reclaim pve-data |
|
|
|
|
## Ceph Pool Summary
|
|
|
|
| Pool | ID | PGs | Size | min_size | Crush Rule | Used | % | Application |
|
|
|------|----|-----|------|----------|------------|------|---|-------------|
|
|
| cephfs_data | 1 | 128 | 3 | 2 | replicated_rule | 0B | 0% | cephfs |
|
|
| cephfs_metadata | 2 | 128 | 3 | 2 | replicated_ssd | 332K | 0% | cephfs |
|
|
| vm_disks | 3 | 128 | 3 | 2 | replicated_ssd | 688G | 69% | rbd |
|
|
| .mgr | 4 | 1 | 3 | 2 | replicated_ssd | 303M | 0% | mgr |
|
|
| rbd | 5 | 32 | 3 | 2 | replicated_hdd | 36K | 0% | rbd |
|
|
| hdd_disk | 6 | 128 | 3 | 2 | replicated_hdd | 3.7T | 71% | rbd |
|
|
| tm_disks | 7 | 128 | **2** | **2** | replicated_hdd | 564G | 26% | rbd |
|
|
|
|
**Critical:** tm_disks has replica=2 with min_size=2 — any single OSD failure blocks all writes. No redundancy margin.
|
|
|
|
## CRUSH Rules
|
|
|
|
- **replicated_rule** (rule 0): default, all devices, host-level isolation
|
|
- **replicated_hdd** (rule 1): HDD class only, host-level isolation
|
|
- **replicated_ssd** (rule 2): SSD class only, host-level isolation
|
|
|
|
## Network Bond Modes Per Node
|
|
|
|
| Node | Bond Mode | NICs in bond | Notes |
|
|
|------|-----------|--------------|-------|
|
|
| proxmox1 | active-backup | 1 (USB NIC) | Single NIC, correct mode |
|
|
| proxmox2 | round-robin | 1 | Wrong mode for single NIC |
|
|
| proxmox3 | round-robin | 1 | Wrong mode for single NIC |
|
|
| proxmox4 | active-backup | 1 | Correct |
|
|
| proxmox5 | round-robin | 1 | Wrong mode for single NIC |
|
|
| proxmox6 | active-backup | 1 | Correct |
|
|
| proxmox7 | active-backup | 1 | Correct |
|
|
| n5pro | active-backup | 1 (nic0) | Correct |
|
|
|
|
Round-robin with a single slave provides no benefit and adds overhead. Should be active-backup.
|
|
|
|
## Updates 2026-07-04
|
|
|
|
### osd.9 Destroy + Rebuild with DB on Real Partition (Confirmed)
|
|
|
|
osd.9 on 10.0.30.100 had irreparably corrupt RocksDB (MANIFEST corruption from loopback DB device). Destroyed with `--force`, purged from CRUSH, disk wiped, and rebuilt as new osd.6 with DB on `osd-9-db` LV (30GB, sda4 PV). The `--block.db` flag (not `--db-dev`) was used successfully. OSD ID was recycled (9→6) — normal Ceph behavior. Backfill ran automatically from peer replicas (size=3, no data loss). Procedure documented in SKILL.md Section 6.10.
|
|
|
|
### Post-Shrink Cleanup Confirmed
|
|
|
|
The initramfs premount script (`/etc/initramfs-tools/scripts/init-premount/auto-shrink-root`) and GRUB rescue entry (`/etc/grub.d/40_custom_rescue`) were removed after the automated shrink completed successfully. GRUB + initramfs regenerated cleanly. Verified no `shrink_root` references remain in `grub.cfg` or initramfs scripts. The partition changes (sda2=160G, sda4=64G LVM) are persistent.
|
|
|
|
### CT 114 (smb-tm-noris) MMP Recovery + HA Addition
|
|
|
|
CT 114 on proxmox5 (Time Machine SMB target, 10.0.30.20) was stopped and couldn't start due to Ext4 MMP deadlock on its RBD root disk. Recovery followed Section 3.4 procedure: Ceph client blacklist → D-state release → `tune2fs -f -E clear_mmp` → container start. After recovery, `smbd` required manual start (`systemctl start smbd`). CT 114 subsequently added to HA-manager with `max_relocate=3, max_restart=2`.
|
|
|
|
### Time Machine Target Inventory
|
|
|
|
Two Time Machine storage locations exist in the homelab:
|
|
- **CT 114 (10.0.30.20)** — active SMB share `[TimeMachine]` with `fruit:time machine = yes`, Samba + Avahi. Last backup: 2026-05-20 (D112299.sparsebundle, 280GB). CT was stopped for ~6 weeks (MMP issue), preventing backups.
|
|
- **ZFS pool on 10.0.30.100** (`/pool01_n2_redundant/TimeMachine`) — legacy target, last backup 2026-03-15 (D112299.sparsebundle, 79GB). Same Mac migrated from ZFS to CT 114. Three older bundles from Sarah's MBP and MacBook Air (2025-2022).
|
|
|
|
CT 114 is the current active target. The ZFS share appears deprecated.
|