hermes-skills/devops/proxmox-ve-administration/references/cluster-analysis-2026-07-12.md

Cluster Analysis — 2026-07-12

Session Context

Comprehensive Proxmox + K8s cluster analysis requested by user after completing RKE2 v1.35.6 upgrade and Helm chart upgrades. Goal: identify performance optimizations, evaluate n5pro for GPU workloads, assess overall health.

Proxmox Cluster State (8 nodes)

Node	CPU	Cores	RAM	Disks	OSDs	VMs/CTs
proxmox1	i7-8550U	8T	15GB	3.6TB HDD + 954GB NVMe	1 (HDD)	VM106 (HA)
proxmox2	i3-9100T	4C	15GB	233GB SSD×2 + 239GB NVMe	2 (SSD)	CP-03, Worker-02
proxmox3	i3-9100T	4C	31GB	233GB SSD + 239GB NVMe	1 (SSD)	CP-02, Worker-03
proxmox4	i3-9100T	4C	15GB	238GB SSD + 239GB NVMe	1 (SSD)	VM302, VM310
proxmox5	i3-9100T	4C	15GB	238GB SSD + 239GB NVMe	1 (SSD)	CP-01, Worker-01
proxmox6	i3-9100T	4C	15GB	931GB HDD + 239GB NVMe	1 (HDD)	Many CTs
proxmox7	i3-9100T	4C	15GB	931GB HDD + 239GB NVMe	1 (HDD)	CTs, VM230
n5pro	Ryzen AI 9 HX 370	24C	91GB	128GB NVMe only	0	CT104, CT116

n5pro Hardware Details

• CPU: AMD Ryzen AI 9 HX PRO 370 (24C/24T) — most powerful node by far
• GPU: AMD Radeon 890M (integrated APU graphics — NOT discrete PCIe)
  • No PCIe slots available (Mini PC form factor)
  • VFIO/GPU passthrough NOT possible
  • SR-IOV not supported on integrated Radeon
  • MIG is NVIDIA-only — not applicable
• SATA: JMB58x AHCI controller present in lspci BUT no SATA disks connected
  • Only a 14.3GB SanDisk USB stick (sda) for installation media
• NVMe: 119.2GB (nearly full: OS + 2×30GB WAL/DB LVs)
  • pve-wal--db--osd-a (30GB) and pve-wal--db--osd-b (30GB) — created but UNUSED
  • No OSDs deployed on n5pro despite WAL/DB preparation
• RAM: 91GB — more than all other nodes combined

CRUSH Map Anomalies

Two ghost hosts in CRUSH tree:

• px-tmp20 (weight 0) — empty placeholder, no OSDs
• ubuntu (weight 5.49) — hosts osd.6 + osd.8 (2×2.8TB HDD)
  • This is likely an external Ceph contributor node, NOT one of the 8 PVE nodes
  • Hostname doesn't match any PVE node — verify provenance

K8s Cluster State

VM Configurations

All 6 K8s VMs have identical config:

• 4 vCPU, 12GB RAM, balloon: 8192, cpu: host, numa: 0
• aio=io_uring, cache=none, discard=on, iothread=1
• CPs on vm_disks pool (SSD tier), Workers on hdd_disk pool (HDD tier)
• All have HA configured (no HA groups defined)

Resource Utilization (very low)

Node	CPU	CPU%	Memory	Mem%
CP-01	194m	4%	3387Mi	43%
CP-02	250m	6%	4398Mi	48%
CP-03	210m	5%	3643Mi	46%
Worker-01	58m	1%	2652Mi	33%
Worker-02	78m	1%	2830Mi	35%
Worker-03	70m	1%	2470Mi	26%

Top CPU consumers: kube-apiserver (84m × 3), etcd (36-58m × 3), Cilium (27-31m × 6).

Workloads (light)

• ArgoCD (7 apps, 6 Healthy/Synced, backups OutOfSync, hindsight OutOfSync)
• CloudNativePG (postgres-main, 3 instances)
• External Secrets Operator
• Velero (backups + node-agent)
• Ceph CSI RBD (provisioner + nodeplugin)
• Hindsight (API + postgres)
• Cilium, CoreDNS, Traefik (RKE2-bundled)

Ceph Critical Issues (July 12)

Recurring osd.10 Near-Full Problem

Same issue as July 4 (see references/ceph-pool-full-recovery-2026-07.md) but now worse:

Metric	July 4	July 12
osd.10 %USE	95% (full_ratio)	96.72%
hdd_disk pool %USED	71%	99.09%
Pools backfillfull	6	7
PGs backfill_toofull	0	4
BlueFS spillover	0 OSDs	1 (osd.8)
Slow ops	4 OSDs	5 OSDs
Ceph version mix	uniform	5 OSDs on 19.2.3, rest on 19.2.4

Root cause unchanged: osd.10 (982GB HDD on proxmox6) is nearly full, blocking backfill for 4 PGs. The hdd_disk pool (where Worker VM disks reside) is at 99.09% — essentially no growth capacity.

Worker VM Disk Placement Problem

Worker-01/02/03 disks are on hdd_disk pool (HDD tier, 99% full). CP-01/02/03 disks are on vm_disks pool (SSD tier, 74.5% full). Workers should be on SSD tier for I/O performance, but SSD pool only has 87GB MAX AVAIL remaining.

OSD Imbalance (VAR 0.65-2.33, STDDEV 27.69%)

OSD	Size	%USE	VAR	PGs	Issue
osd.10	982GB	96.72%	2.33	283	Nearly full, blocking backfill
osd.4	233GB	73.90%	1.78	119	SSD, slow ops
osd.3	238GB	71.53%	1.73	114	SSD, slow ops
osd.5	238GB	71.33%	1.72	113	SSD, slow ops
osd.2	233GB	63.73%	1.54	97	SSD, slow ops
osd.7	982GB	63.82%	1.54	215	HDD
osd.0	188GB	57.61%	1.39	71	SSD, slow ops
osd.1	3.6TB	34.53%	0.83	437	HDD, slow ops
osd.6	2.8TB	27.10%	0.65	304	HDD (ubuntu host)
osd.8	2.8TB	27.25%	0.66	301	HDD, BlueFS spillover

Performance Optimization Recommendations

Priority 1: Acute Fixes

1. Drain osd.10 — ceph osd reweight 10 0.5 or lower. Move data to osd.6/8 (ubuntu host, 2×2.8TB, only 27% full).
2. Migrate Worker VM disks from hdd_disk (99% full, HDD) to vm_disks (SSD) via PVE storage live migration. Need to free SSD pool space first or add SSD capacity.
3. Upgrade 5 OSDs from Ceph 19.2.3 → 19.2.4 (eliminate version mix).
4. Clean CRUSH map — remove px-tmp20, investigate ubuntu host.

Priority 2: K8s Optimizations

5. Disable memory ballooning on K8s VMs — set balloon: 0. Ballooning causes non-deterministic latency spikes.
6. Node-Local DNS Cache — deploy node-local-dns DaemonSet.
7. Kubelet reserved resources — verify systemReserved/kubeReserved.
8. Pod topology spread — for multi-replica deployments.

Priority 3: Infrastructure

9. Jumbo frames (MTU 9000) — reduce Ceph network overhead.
10. CPU pinning — i3 nodes have 4 cores, 2 VMs each = 100% overcommit.
11. Utilize n5pro — 24 cores + 91GB RAM idle. Place worker VM there.
12. HA group with n5pro — define HA group so VMs can failover to n5pro.

n5pro GPU Evaluation

Conclusion: GPU passthrough NOT possible.

• Radeon 890M is integrated APU graphics (shares system RAM)
• No discrete PCIe GPU, no PCIe slots for expansion
• SR-IOV not supported on consumer integrated GPUs
• MIG is NVIDIA-only

Alternative uses for n5pro:

• CPU-pinned worker VM (excellent: 24C, 91GB RAM)
• Ceph OSD node (JMB58x SATA + 5 disks → new OSDs, WAL/DB LVs ready)
• vLLM CPU inference (possible but slow vs GPU)
• General compute workloads (best CPU/RAM in cluster)

Failover Analysis

• All 6 K8s VMs have HA configured ✅
• No HA groups defined ⚠️
• i3 nodes: 4 cores each, 2 K8s VMs per node = 8 vCPUs on 4 physical cores
• If proxmox5 fails (CP-01 + Worker-01), no other i3 node can absorb 8 vCPUs
• n5pro (24C, 91GB) could host all 6 VMs simultaneously — ideal HA fallback
• Recommendation: Define HA group [proxmox2, proxmox3, proxmox5, n5pro] with n5pro as last-resort fallback

Data Collection Methodology

Reliable VM Inventory (avoids SSH quoting issues)

# From PVE coordinator (10.0.20.10):
pvesh get /cluster/resources --type vm --output-format json
# Returns JSON array with vmid, name, node, status, maxmem, maxcpu, maxdisk
# Parse with python3 -c "import sys,json; ..."

SSH Chain Quoting Pitfall

Nested SSH commands with grep and awk break due to quote escaping through multiple SSH layers. Solutions:

1. Use pvesh get /cluster/resources --type vm --output-format json (API)
2. Use heredoc: ssh root@host 'bash -s' << 'SCRIPT' ... SCRIPT
3. Use pvesh API calls instead of parsing CLI output

Direct SSH to n5pro

n5pro is NOT reachable by hostname from proxmox1 — use IP:

ssh -o StrictHostKeyChecking=no 10.0.20.91 'commands'

pvesm list Fails During Ceph Backfillfull (2026-07-12)

When Ceph pool hdd_disk is in backfillfull state (osd.10 at 96.72%), pvesm list hdd_disk returns empty with error:

rbd error: rbd: listing images failed: (2) No such file or directory

Similarly pvesh get /nodes/{node}/storage/hdd_disk/content fails. Fallback: Use rbd ls hdd_disk directly (works even when PVE storage layer cannot enumerate). Cross-reference with pvesh get /cluster/resources --type vm for orphan detection.

HA Rule Created: k8s-prefer-n5pro (2026-07-12)

Successfully created node-affinity rule for K8s VMs:

• Rule name: k8s-prefer-n5pro
• Type: node-affinity
• Resources: vm:118,vm:128,vm:129,vm:130,vm:131,vm:132
• Nodes: n5pro,proxmox5,proxmox3,proxmox2 (ordered preference)
• All 6 VMs briefly went through "deleting" → "starting" → "started" transition
• Existing rules untouched (ct:104/ct:99999 → n5pro, vm:310/311 + vm:301/302 negative affinity)

See references/ha-manager-and-custom-dashboards-2026-07.md for full syntax.

K8s VM Anti-Affinity Problem (2026-07-12)

Current Distribution (BAD)

Node	CP	Worker	Risk
proxmox2	cp-03 (129)	worker-02 (132)	2 VMs lost on node failure
proxmox3	cp-02 (130)	worker-03 (131)	2 VMs lost on node failure
proxmox5	cp-01 (118)	worker-01 (128)	2 VMs lost on node failure

Each of the 3 K8s nodes hosts BOTH a CP and a Worker. A single node failure drops 1 CP + 1 Worker simultaneously. Quorum survives (2/3 CPs) but workload capacity drops 33%.

K8s VM Anti-Colocation: EXECUTED (2026-07-12) ✅

The proposed distribution was successfully implemented via live migration. All 3 worker VMs migrated to separate nodes (no node hosts >1 K8s VM):

Node	CP	Worker
proxmox1	—	worker-02 (132) ← migrated FROM proxmox2
proxmox2	cp-03 (129)	—
proxmox3	cp-02 (130)	—
proxmox4	—	worker-03 (131) ← migrated FROM proxmox3
proxmox5	cp-01 (118)	—
n5pro	—	worker-01 (128) ← migrated FROM proxmox5

Migration commands (ran from source node, not coordinator):

ssh root@10.0.20.20 'qm migrate 132 proxmox1 --online'  # proxmox2 → proxmox1
ssh root@10.0.20.30 'qm migrate 131 proxmox4 --online'  # proxmox3 → proxmox4
ssh root@10.0.20.50 'qm migrate 128 n5pro --online'     # proxmox5 → n5pro

HA rule k8s-prefer-n5pro was expanded to include ALL 8 nodes before migration (was missing proxmox1, proxmox4 — caused migration failure). See references/ha-manager-and-custom-dashboards-2026-07.md for details.

Result: Any single node failure now loses at most 1 K8s VM (was 2). Quorum survives any single CP node failure (2/3 CPs remain).