8.4 KiB
Cluster Audit — 2026-07-03
Session Context
Full cluster health audit performed at user request after HA management changes (added vm:106, ct:108, vm:230 to HA; normalized max_relocate/max_restart; cleaned ghost configs from removed "pve" node; repaired LRM on proxmox1).
Cluster Overview
- Cluster name: Homelab
- Nodes: 8 (proxmox1-7, n5pro)
- PVE version: 9.2.3 on all nodes
- Kernel variance: proxmox1=6.17.4-2, proxmox2-3=7.0.2-6, proxmox4-7+n5pro=7.0.12-1
- Corosync: Single ring (knet, UDP, 10.0.20.0/24), link_mode=passive, quorum=5/8
- HA-Manager: 16 managed guests, CRM master on proxmox4, fencing armed
Hardware Summary
| Node | CPU | RAM | Storage | OSDs | Mon | Role |
|---|---|---|---|---|---|---|
| proxmox1 | i7-8550U 8c | 15GB | 954G NVMe + 3.6T HDD | 1 (HDD) | — | HA VMs |
| proxmox2 | i3-9100T 4c | 15GB | 239G NVMe + 250G SSD | 2 (SSD) | — | MariaDB |
| proxmox3 | i3-9100T 4c | 31GB | 239G NVMe + 233G SSD | 1 (SSD) | — | — |
| proxmox4 | i3-9100T 4c | 15GB | 239G NVMe + 239G SSD | 1 (SSD) | YES | MariaDB, MaxScale |
| proxmox5 | i3-9100T 4c | 15GB | 239G NVMe + 239G SSD | 1 (SSD) | YES (leader) | — |
| proxmox6 | i3-9100T 4c | 15GB | 239G NVMe + 932G HDD | 1 (HDD) | — | Many CTs |
| proxmox7 | i3-9100T 4c | 15GB | 239G NVMe + 932G HDD | 1 (HDD) | YES | Hermes, LiteLLM |
| n5pro | Ryzen AI 9 HX PRO 370 24c | 92GB | 119G NVMe | 0 | — | GPU compute target |
Issues Found & Actions Taken
Actions Completed This Session
- HA status detection fix:
/cluster/resourcesAPIhafield is unreliable — all 55 guests reported as non-HA when 12 were HA-managed. Corrected method: useha-manager status+pvesh get /cluster/ha/resources. - LRM proxmox1 repair:
pve-ha-lrm.servicewas disabled+inactive since rolling update. Fixed:systemctl enable && start pve-ha-lrm. - HA added: vm:106 (homeassistant), ct:108 (gitea), vm:230 (hermes-agent) — all with max_relocate=3, max_restart=2.
- HA params normalized: ct:104, ct:99999 reduced from 5/5 to 3/2.
- Ghost cleanup: Deleted vm:101, vm:102, ct:902 from
/etc/pve/nodes/pve/(removed node, not in corosync). - Ceph Mons reduced: 7→3 (removed ubuntu, proxmox1, proxmox2, proxmox3). MON_DISK_LOW warning eliminated.
- osd.6 purged: Destroyed OSD removed from CRUSH tree. PG rebalancing initiated.
- proxmox2 root disk: Removed unused pve-data thin pool (10.5G), extended pve-root 24.5G→37G (85%→50% usage).
Remaining Recommendations (Prioritized)
High Priority
| # | Issue | Recommendation | Risk | Effort |
|---|---|---|---|---|
| 1 | tm_disks pool replica=2 | Increase to size=3 or decommission pool | Medium (data loss risk) | 10min + rebalance |
| 2 | OSD imbalance (VAR 0.64-1.96) | Reweight osd.7/10 down, osd.8/9 up | Low | 5min + time |
| 3 | BlueStore slow ops (osd.0, osd.1) | Related to OSD imbalance + old SSD (Samsung 860 EVO, 13k hrs) | — | Monitor/replace |
Medium Priority
| # | Issue | Recommendation | Risk | Effort |
|---|---|---|---|---|
| 4 | Single Corosync ring | Add second ring on separate VLAN | Low | Config change |
| 5 | Inconsistent bond modes | proxmox2/3/5: change round-robin→active-backup (single NIC) | Low | 5min/node |
| 6 | osd_memory_target=4GB on 15GB nodes | Reduce to 2-3GB | Low | 1 command |
| 7 | target_size_ratio=0.9 on 3 pools | Set realistic values (0.01-0.2) | Low | 5min |
| 8 | CephFS unused (0 clients) | Shut down MDS if not needed | Low | 5min |
| 9 | Kernel version inconsistency | Reboot proxmox1-3 to get 7.0.12 | Medium | Wartungsfenster |
Low Priority
| # | Issue | Recommendation |
|---|---|---|
| 10 | osd_recovery_sleep=0.1 | Throttles recovery — set to 0 for urgent rebalances |
| 11 | OSD reweight 0.9 on osd.0/2/5 | Reset to 1.0 if not intentional |
| 12 | proxmox1 phantom disks (sda/sdb 0B) | Investigate and remove |
| 13 | n5pro ISO partition (sda 14.3G) | Remove leftover installation media |
| 14 | proxmox1 /boot/efi 85% full | Clean old kernel EFI entries |
Node Root Disk Survey (2026-07-03)
| Node | Root-LV | Used | % | pve-data (thin) | VG Free | Notes |
|---|---|---|---|---|---|---|
| proxmox1 | 96G | 26G | 29% | — | 849G | Huge NVMe, mostly unallocated |
| proxmox2 | 37G | 18G | 50% | — (removed) | 0 | Extended this session |
| proxmox3 | 85G | 17G | 21% | 142G (0%) | 0 | Could reclaim pve-data |
| proxmox4 | 101G | 21G | 22% | 128G (0%) | 0 | Could reclaim pve-data |
| proxmox5 | 68G | 20G | 31% | 141G (0%) | 16G | Could reclaim pve-data |
| proxmox6 | 68G | 20G | 31% | — | 110G | Has 50G NVMe WAL (pve-ceph--db) |
| proxmox7 | 68G | 22G | 34% | — | 110G | Has 50G NVMe WAL (pve-ceph--db) |
| n5pro | 39G | 15G | 39% | 54G (0%) | 15G | Could reclaim pve-data |
Ceph Pool Summary
| Pool | ID | PGs | Size | min_size | Crush Rule | Used | % | Application |
|---|---|---|---|---|---|---|---|---|
| cephfs_data | 1 | 128 | 3 | 2 | replicated_rule | 0B | 0% | cephfs |
| cephfs_metadata | 2 | 128 | 3 | 2 | replicated_ssd | 332K | 0% | cephfs |
| vm_disks | 3 | 128 | 3 | 2 | replicated_ssd | 688G | 69% | rbd |
| .mgr | 4 | 1 | 3 | 2 | replicated_ssd | 303M | 0% | mgr |
| rbd | 5 | 32 | 3 | 2 | replicated_hdd | 36K | 0% | rbd |
| hdd_disk | 6 | 128 | 3 | 2 | replicated_hdd | 3.7T | 71% | rbd |
| tm_disks | 7 | 128 | 2 | 2 | replicated_hdd | 564G | 26% | rbd |
Critical: tm_disks has replica=2 with min_size=2 — any single OSD failure blocks all writes. No redundancy margin.
CRUSH Rules
- replicated_rule (rule 0): default, all devices, host-level isolation
- replicated_hdd (rule 1): HDD class only, host-level isolation
- replicated_ssd (rule 2): SSD class only, host-level isolation
Network Bond Modes Per Node
| Node | Bond Mode | NICs in bond | Notes |
|---|---|---|---|
| proxmox1 | active-backup | 1 (USB NIC) | Single NIC, correct mode |
| proxmox2 | round-robin | 1 | Wrong mode for single NIC |
| proxmox3 | round-robin | 1 | Wrong mode for single NIC |
| proxmox4 | active-backup | 1 | Correct |
| proxmox5 | round-robin | 1 | Wrong mode for single NIC |
| proxmox6 | active-backup | 1 | Correct |
| proxmox7 | active-backup | 1 | Correct |
| n5pro | active-backup | 1 (nic0) | Correct |
Round-robin with a single slave provides no benefit and adds overhead. Should be active-backup.
Updates 2026-07-04
osd.9 Destroy + Rebuild with DB on Real Partition (Confirmed)
osd.9 on 10.0.30.100 had irreparably corrupt RocksDB (MANIFEST corruption from loopback DB device). Destroyed with --force, purged from CRUSH, disk wiped, and rebuilt as new osd.6 with DB on osd-9-db LV (30GB, sda4 PV). The --block.db flag (not --db-dev) was used successfully. OSD ID was recycled (9→6) — normal Ceph behavior. Backfill ran automatically from peer replicas (size=3, no data loss). Procedure documented in SKILL.md Section 6.10.
Post-Shrink Cleanup Confirmed
The initramfs premount script (/etc/initramfs-tools/scripts/init-premount/auto-shrink-root) and GRUB rescue entry (/etc/grub.d/40_custom_rescue) were removed after the automated shrink completed successfully. GRUB + initramfs regenerated cleanly. Verified no shrink_root references remain in grub.cfg or initramfs scripts. The partition changes (sda2=160G, sda4=64G LVM) are persistent.
CT 114 (smb-tm-noris) MMP Recovery + HA Addition
CT 114 on proxmox5 (Time Machine SMB target, 10.0.30.20) was stopped and couldn't start due to Ext4 MMP deadlock on its RBD root disk. Recovery followed Section 3.4 procedure: Ceph client blacklist → D-state release → tune2fs -f -E clear_mmp → container start. After recovery, smbd required manual start (systemctl start smbd). CT 114 subsequently added to HA-manager with max_relocate=3, max_restart=2.
Time Machine Target Inventory
Two Time Machine storage locations exist in the homelab:
- CT 114 (10.0.30.20) — active SMB share
[TimeMachine]withfruit:time machine = yes, Samba + Avahi. Last backup: 2026-05-20 (D112299.sparsebundle, 280GB). CT was stopped for ~6 weeks (MMP issue), preventing backups. - ZFS pool on 10.0.30.100 (
/pool01_n2_redundant/TimeMachine) — legacy target, last backup 2026-03-15 (D112299.sparsebundle, 79GB). Same Mac migrated from ZFS to CT 114. Three older bundles from Sarah's MBP and MacBook Air (2025-2022).
CT 114 is the current active target. The ZFS share appears deprecated.