hermes-skills/devops/proxmox-ve-administration/references/ceph-recovery-ec-reweight-2026-07.md

Ceph Recovery, EC Reweight, PG Merge & Backfillfull

EC4+1 Reweight Trap — CRUSH Returns 2147483647

Problem

With exactly 5 HDD OSDs and EC profile k=4 m=1 (needs exactly 5 OSDs), rewighting any OSD (e.g. ceph osd reweight osd.7 0.65) causes CRUSH to return 2147483647 ("no OSD available") in the up-set for affected PGs. These PGs become active+clean+remapped and cannot recover — there is no alternative OSD to place them on.

8.20 active+clean+remapped [6,1,8,10,2147483647] 6
8.78 active+clean+remapped [1,6,8,2147483647,10] 1

Diagnosis

# Check EC profile
ceph osd erasure-code-profile get ec41-hdd
# k=4 m=1 → needs 5 OSDs

# Count HDD OSDs
ceph osd tree | grep hdd | wc -l
# 5 → exactly the minimum. Reweighting breaks CRUSH.

# Check for 2147483647 in up-sets
ceph pg dump pgs_brief 2>/dev/null | grep 2147483647 | wc -l

Fix

Reset reweight to 1.0 for all EC-participating OSDs. With only 5 HDD OSDs and EC4+1, there is no spare capacity for reweighting. Every OSD must participate at full weight.

ceph osd reweight osd.7 1.0
ceph osd reweight osd.10 1.0

After resetting, recovery resumes (67 MiB/s, 16 obj/s observed).

General Rule

• Replicated pools (size=3): reweighting is safe — CRUSH redirects to other OSDs in the failure domain.
• Erasure-coded pools: crush reweight (permanent CRUSH weight change) is only safe when there are MORE OSDs than k+m. With exactly k+m OSDs, crush reweight traps PGs (2147483647 in up-set).
• ⚠️ osd reweight (temporary) CAN drain EC OSDs (2026-07-12): Unlike crush reweight, ceph osd reweight N 0.5 sets a temporary runtime override that adjusts PG placement probability WITHOUT changing CRUSH weights. This means CRUSH still considers all OSDs available — no 2147483647 trap. Tested on osd.10 (EC4+1, exactly 5 HDD OSDs): reweight to 0.5 drained from 93%→91%, backfill started at 30 MiB/s, 5 of 6 backfill_toofull PGs unblocked. Prerequisite: crush reweight must be at 1.0 (default). If crush reweight was previously changed, revert it to 1.0 FIRST, then use osd reweight.
• Long-term fix: add more HDD OSDs (e.g. on the ubuntu host with 2×2.8 TB spare) to give CRUSH room to redistribute.

Backfillfull Sticky Flag

Problem

osd.10 at 80% utilization has a backfillfull flag. Even after raising the ratio:

ceph osd set-backfillfull-ratio 0.95
ceph osd set-nearfull-ratio 0.95

The flag persists and blocks backfill on 12+ PGs:

HEALTH_WARN ... 1 backfillfull osd(s); Low space hindering backfill:
12 pgs backfill_toofull; 6 pool(s) backfillfull

Attempted Fixes That Don't Work

# Does not exist in this Ceph version (Reef/Quincy):
ceph osd clear-backfillfull osd.10  # → EINVAL

# Cannot be modified at runtime:
ceph tell osd.7 injectargs --mon_osd_backfillfull_ratio 0.95  # → EPERM

# Forces backfill on PGs that need it, but doesn't clear toofull:
ceph osd pool force-backfill hdd_disk  # instructs PGs to force-backfill
# → PGs still show backfill_toofull

What Happened

The backfillfull flag was set when osd.10 crossed the OLD threshold (default 0.85). Raising the ratio afterward does NOT retroactively clear the flag — it only prevents future flag-setting. The OSD remains flagged until its utilization drops below the new threshold naturally (via recovery moving data away).

Mitigation

• osd reweight (temporary) WORKS for EC4+1 drain (2026-07-12): ceph osd reweight 10 0.5 successfully drained osd.10 from 93.14%→90.90% and unblocked 5 of 6 backfill_toofull PGs. This is a runtime override, NOT a CRUSH weight change — CRUSH still sees all OSDs, preventing the 2147483647 trap. Recovery proceeded at 30 MiB/s. Steps: (1) ensure crush reweight is 1.0, (2) ceph osd reweight 10 0.5, (3) ceph osd set-backfillfull-ratio 0.95 to unblock remaining PGs, (4) wait for recovery, (5) ceph osd reweight 10 1.0 to restore.
• Aggressive drain: reweight 0.3 (2026-07-12 session 2): When 0.5 wasn't enough (osd.10 still at 89%, 1 PG stubbornly backfill_toofull), dropping to ceph osd reweight 10 0.3 increased misplaced objects from 12%→16% (more PGs queued for relocation) and accelerated the drain. Combined with set-backfillfull-ratio 0.97. Reweight progression: 0.5 → 0.3 if 0.5 doesn't drain fast enough. Remember to restore to 1.0 after recovery completes.

⚠️ Correct Command Syntax for backfillfull_ratio (2026-07-12)

Multiple command forms were tried — only ONE works. Save time by using the correct one first:

# ✅ CORRECT — hyphenated, no underscore:
ceph osd set-backfillfull-ratio 0.97

# ❌ WRONG — underscore variant (EINVAL):
ceph osd set_backfill_full_ratio 0.97

# ❌ WRONG — config set (rejected as "special"):
ceph config set mon mon_osd_backfillfull_ratio 0.97
# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"

# ❌ WRONG — injectargs with wrong key name:
ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
# → "failed to parse arguments"

# ❌ WRONG — osd set with wrong key:
ceph osd set backfill_full_ratio 0.97
# → "Invalid command: backfill_full_ratio not in full|pause|..."

Similarly for full_ratio and nearfull_ratio:

ceph osd set-full-ratio 0.97        # ✅ hyphenated
ceph osd set-nearfull-ratio 0.95     # ✅ hyphenated

• Patience: Once the reweight takes effect and recovery proceeds, data redistributes and osd.10 utilization drops, eventually clearing the backfillfull flag.
• PG merge: Reducing PG count (128→32) eliminates some remapped PGs entirely, reducing the number of PGs that need to backfill to the stuck OSD.
• Do NOT use crush reweight to reduce osd.10's load — with EC4+1 and only 5 HDD OSDs, permanent CRUSH reweighting makes things worse (see EC4+1 Reweight Trap above). Use osd reweight (temporary) instead.

PG Merge (128 → 32)

Procedure

# 1. Disable autoscaler for the pool
ceph osd pool set cephfs_data pg_autoscale_mode off
ceph osd pool set cephfs_metadata pg_autoscale_mode off

# 2. Set pgp_num first (must be ≤ pg_num)
ceph osd pool set cephfs_data pgp_num 32
ceph osd pool set cephfs_metadata pgp_num 32

# 3. Set pg_num_target (async merge preparation)
ceph osd pool set cephfs_data pg_num_target 32
ceph osd pool set cephfs_metadata pg_num_target 32

# 4. Force the merge — works when all PGs are clean
ceph osd pool set cephfs_data pg_num 32
ceph osd pool set cephfs_metadata pg_num 32

# 5. Verify
ceph osd pool get cephfs_data pg_num    # → 32
ceph osd pool get cephfs_metadata pg_num # → 32

When It Works

• All PGs in the pool must be active+clean for the merge to execute.
• If PGs are remapped or backfilling, the merge waits.
• With pg_autoscale_mode off, the merge is manual and deterministic.

Impact

Reducing cephfs_data (0 bytes stored) and cephfs_metadata (85 KiB stored) from 128→32 PGs eliminates 384 PG instances across OSDs, helping with the "too many PGs per OSD" health warning (262 > 250).

Recovery Speed Tuning

# Increase concurrent backfills per OSD (default 1)
ceph config set osd osd_max_backfills 5

# Increase recovery op priority (default 1, lower = higher priority)
ceph config set osd osd_recovery_op_priority 3

# Ensure no recovery sleep (default 0)
ceph config get osd osd_recovery_sleep  # → 0.000000 (already optimal)

Monitoring Recovery Progress

# Overall
ceph status | grep -E 'recovery|pgs:'

# Count remapped/backfill PGs
ceph pg dump pgs_brief 2>/dev/null | tail -n +2 | awk '{print $2}' | grep -c remapped
ceph pg dump pgs_brief 2>/dev/null | tail -n +2 | awk '{print $2}' | grep -c backfill_toofull

# Recovery rate
ceph status | grep -A2 'recovery:'
# → recovery: 67 MiB/s, 16 objects/s

Diagnosing Stalled Recovery (2026-07-07)

When recovery appears stalled (misplaced objects constant for hours), check these in order BEFORE tuning recovery speed:

1. EC reweight trap: ceph pg dump pgs_brief | grep 2147483647 — If PGs contain 2147483647 ("no OSD"), CRUSH cannot place them. Reset all EC-participating OSDs to reweight 1.0. See section above.

2. Backfillfull sticky flag: ceph health detail | grep backfillfull — An OSD flagged backfillfull blocks all backfill to it, even after raising the ratio. The flag is set when crossing the OLD threshold and does NOT clear retroactively. See "Backfillfull Sticky Flag" section.

3. Stale rbd rm process: A previous rbd rm stuck in I/O holds a watcher that blocks new operations. Check ps aux | grep 'rbd.*rm'.

4. Recovery actually progressing but slow: Compare objects misplaced count over time — if decreasing, it's just slow. If truly constant, investigate items 1-3.

OSD Utilization Assessment

# Full tree with utilization
ceph osd df tree

# Compact view
ceph osd df | grep -E 'osd\.|TOTAL'

This Cluster (2026-07-07)

OSD	Host	Size	Used	%Used	VAR	PGs	Issue
osd.7	proxmox7	982G	863G	88%	1.94	272	Overloaded, EC trap
osd.10	proxmox6	982G	790G	80%	1.78	269	Backfillfull flag
osd.3-5	proxmox3-5	233-238G	~148G	62-64%	1.36-1.41	185-189	SSD, balanced
osd.0-2	proxmox2	188-233G	94-129G	50-55%	1.10-1.22	128-158	SSD, lighter
osd.6	ubuntu	2.8T	913G	33%	0.72	345	Underutilized
osd.8	ubuntu	2.8T	928G	33%	0.72	354	Underutilized
osd.1	proxmox1	3.6T	1.5T	40%	0.88	538	Balanced

Root imbalance: osd.7/10 (982G each) are half the size of osd.6/8 (2.8T each) but receive nearly equal PG counts from CRUSH. The small OSDs fill up faster. Long-term: add more large HDDs or remove osd.7/10 and replace with larger drives.

EC Profile Immutability & Pool Recreation Cost

Key Rule

Erasure-code profiles are immutable after pool creation. You cannot change k or m on an existing pool. To switch from EC4+1 (k=4, m=1) to EC3+1 (k=3, m=1), you must:

1. Create a new EC profile (ceph osd erasure-code-profile set ec31-hdd k=3 m=1 ...)
2. Create new pools (media_ec2 + media_meta2)
3. Copy all RBD images to the new pool (rbd copy --data-pool media_ec2)
4. Update PVE storage config + all CT/VM disk references
5. Delete old pools

When to Change EC Profile vs. Add OSDs

Approach	Pros	Cons
Add HDD OSD	No downtime, no data migration, gives CRUSH room to rebalance, preserves EC4+1 efficiency (25% overhead)	Physical disk needed, OSD count grows
Switch to EC3+1	Needs only 4 OSDs (one fewer), same 1-failure tolerance	33% overhead (vs 25%), full pool migration, downtime risk

Recommendation: When the problem is "EC4+1 with exactly 5 HDD OSDs can't tolerate reweighting," adding an OSD is almost always better than changing the EC profile. The reweight trap exists because CRUSH has no spare OSD to redirect to — adding one OSD solves that directly, preserves the better storage efficiency of EC4+1, and requires zero data migration.

Trade-off: EC3+1 vs EC4+1

• EC3+1 (k=3, m=1): 4 OSDs needed, 33% overhead, tolerates 1 failure
• EC4+1 (k=4, m=1): 5 OSDs needed, 25% overhead, tolerates 1 failure
• EC4+2 (k=4, m=2): 6 OSDs needed, 50% overhead, tolerates 2 failures
• Both EC3+1 and EC4+1 tolerate only 1 OSD failure — the extra OSD in EC4+1 buys storage efficiency, not more resilience.

Samba Time Machine Sparsebundle on Ceph EC

After migrating a Time Machine sparsebundle (via rbd copy to EC storage), Time Machine may report "Backup fehlgeschlagen" with RESULT: 70 in com.apple.TimeMachine.Results.plist. The sparsebundle is accessible and bands are being written (verified via smbstatus — active locks on band files), but TM aborts.

Possible Causes

1. I/O latency from Ceph recovery — BLUESTORE_SLOW_OP_ALERT on HDD OSDs during recovery causes high write latency. TM is sensitive to I/O timeouts.
2. Sparsebundle corruption from rbd copy — rbd copy copies at the block level; if the source was being written to during copy, the sparsebundle may be inconsistent.
3. EC pool latency — EC writes involve chunk calculation across multiple OSDs, inherently slower than replicated pools.

Diagnosis

# Check TM result code
cat "/mnt/timemachine/backup/<name>.sparsebundle/com.apple.TimeMachine.Results.plist"
# RESULT 70 = sparsebundle error

# Check active SMB connections + locked files
smbstatus

# Check I/O latency
ceph status | grep -E 'slow|io:'
dmesg | grep -i "slow\|error\|timeout"

# Check sparsebundle integrity from Mac:
# hdiutil verify /Volumes/TimeMachine/<name>.sparsebundle
# hdiutil repair /Volumes/TimeMachine/<name>.sparsebundle

Resolution Options

1. Wait for Ceph recovery to complete — if I/O latency is the cause, TM backups should resume once HEALTH_OK and slow ops clear.
2. Verify + repair sparsebundle — run hdiutil verify / hdiutil repair from the Mac (requires mounting the share).
3. Fresh start — delete sparsebundle, begin new TM backup (loses history).

Concurrent I/O Competition

Running RBD copy + Ceph recovery + Seafile fsck simultaneously on the same HDD OSDs causes:

• BLUESTORE_SLOW_OP_ALERT on 4+ OSDs
• Extreme latency on all operations
• Recovery appears stalled (not actually stalled, just throttled)

Sequencing Recommendation

1. Complete RBD copies first (or pause them)
2. Let Ceph recovery finish (monitor with ceph status)
3. Then run fsck or other I/O-intensive operations

If all three must run concurrently, expect 5-10x slowdown on each.

Playwright MCP Container Restoration

Scenario

Container was deleted (docker rm), image was pruned, but the named volume survived. Need to recreate the container.

Discovery

# Find surviving volumes
docker volume ls | grep playwright
# → playwright-mcp_npm-cache

# Inspect volume contents for package identity
find /var/lib/docker/volumes/playwright-mcp_npm-cache/_data/ -name "package.json" | head -5
cat /var/lib/docker/volumes/playwright-mcp_npm-cache/_data/_npx/*/node_modules/@playwright/mcp/package.json | grep -E '"name"|"version"'
# → "@playwright/mcp" version "0.0.76"

# Check if base image still exists
docker images | grep playwright
# → mcr.microsoft.com/playwright:latest

Recreation

docker run -d \
  --name playwright-mcp \
  -p 8081:8080 \
  -v playwright-mcp_npm-cache:/root/.npm \
  mcr.microsoft.com/playwright:latest \
  npx @playwright/mcp@0.0.76 --port 8080

Verification

docker logs playwright-mcp | tail -5
# → "Listening on http://localhost:8080"
# → MCP endpoint: http://<host>:8081/mcp

General Pattern

When a container is deleted but its named volume survives:

1. Inspect the volume path for package.json or config files
2. Identify the package name and version
3. Check if the base image still exists locally
4. Recreate with docker run using the same volume mount and package version
5. The npm cache volume makes npx startup instant (no re-download)

Ghost RBD with Orphaned Data Objects (Header Gone)

Symptoms

After a failed rbd rm or aborted operation, the image enters a ghost state where:

• rbd ls <pool> still lists the image (e.g. vm-114-disk-0)
• rbd info <pool>/vm-114-disk-0 → (2) No such file or directory
• rbd status <pool>/vm-114-disk-0 → No such file or directory
• rbd rm <pool>/vm-114-disk-0 → image still has watchers (forever)
• rbd trash mv → deferred delete error: (2) No such file or directory
• rados -p <pool> ls | grep rbd_header.<id> → empty (header gone)
• rados -p <pool> ls | grep rbd_data.<prefix> → thousands of objects!

The image header is already deleted, but the data objects remain and rbd ls shows a stale directory entry. The watcher blocking rbd rm is often a previous stale rbd rm process (stuck in I/O) or a blacklisted Ceph client.

Diagnosis

# 1. Check for stale rbd rm processes (THE most common watcher)
ps aux | grep 'rbd.*rm' | grep -v grep
# If found: kill <pid>; sleep 35; retry rbd rm

# 2. Check Ceph OSD blacklist for stale clients
ceph osd blacklist ls
# If the blocking client is listed:
ceph osd blacklist rm <client_addr>
# e.g. ceph osd blacklist rm 10.0.20.10:0/3681711461

# 3. Verify data objects are orphaned (header gone)
rados -p <pool> ls | grep rbd_header.<image_id>  # empty = header gone
rados -p <pool> ls | grep rbd_data.<prefix> | wc -l  # count orphaned objects

Fix: Purge Orphaned Data Objects

When the header is gone and rbd rm keeps failing, the data objects are orphaned and can be purged directly:

# Find the data object prefix from rbd_data object names
rados -p tm_disks ls | grep rbd_data.378d64d5a67ea8 | head -1
# → rbd_data.378d64d5a67ea8.00000000000121a0

# Purge all orphaned objects (parallel for speed)
rados -p tm_disks ls | grep rbd_data.378d64d5a67ea8 | \
  xargs -P8 -I{} rados -p tm_disks rm {}

# Verify cleanup
rados -p tm_disks ls | grep rbd_data.378d64d5a67ea8 | wc -l
# → 0

# ⚠️ This can take a long time — 65k objects × 4 MiB ≈ 341 GiB
# Run in background with notify_on_complete=true

⚠️ Verify the prefix is correct before purging! Use rbd info on a DIFFERENT image in the same pool to confirm the prefix format (rbd_data.<image_id_hash>.<object_index>). Purging the wrong prefix destroys live data.

⚠️ Requires user confirmation — this is a destructive operation that bypasses RBD's safety checks. Always ask the user before purging.

Prevention

• Always rbd unmap on ALL nodes after RBD operations
• Kill stale rbd rm processes before retrying
• Check ceph osd blacklist ls for stale clients
• Don't run rbd rm in background — if it hangs, the stale process becomes the watcher blocking the next attempt

Competing I/O Forces on Overloaded OSDs

When reweighting an overloaded OSD (e.g. osd.7 at 88%) while concurrent writes target the same OSD (e.g. RBD copy to an EC pool that includes that OSD), the net effect can be increasing utilization despite the reweight:

• Reweight moves existing data AWAY from the OSD → pressure decreases
• Concurrent writes (RBD copy, fsck reads) add NEW data to the OSD → pressure increases
• If write rate > recovery rate, utilization goes UP

Observed: osd.7 dropped from 88% → 80.77% (reweight working), then rose back to 85% (RBD copy writing new EC chunks to media_ec, which includes osd.7 as one of 5 HDD OSDs).

Lesson

Don't reweight an OSD to relieve pressure while simultaneously running I/O-intensive operations that write to pools containing that OSD. Either:

1. Complete the write operation first, THEN reweight
2. Or reweight first, wait for recovery, THEN start writes

Session Log — 2026-07-12

• osd.10 now at 96.72% (up from 80% on July 7, 95% on July 4). hdd_disk pool at 99.09%. 7 pools backfillfull.

• Attempted ceph osd crush reweight osd.10 0.85 then 0.70 — replicated pool PGs started backfilling, but EC pool PGs remained stuck (same EC4+1 trap as July 7).

• Raised backfillfull_ratio to 0.97 — did NOT clear the sticky flag because osd.10 is still above 96.72% (below 0.97 threshold but the flag was set earlier and is sticky).

• User reminded: "reweight hat beim letzten mal wegen erasure coding nicht funktioniert" — this is the SAME issue documented above. The agent should have checked EC profile + OSD count BEFORE attempting reweight, not after.

• Lesson encoded: Pre-reweight checklist is now mandatory:

  1. ceph osd erasure-code-profile ls — are there EC pools?
  2. If yes, ceph osd erasure-code-profile get <profile> — get k+m
  3. ceph osd tree | grep <device-class> | wc -l — count OSDs
  4. If OSD count == k+m: DO NOT REWEIGHT. Reset to 1.0 if already reweighted.

• Confirmed orphans on hdd_disk (~170 GiB reclaimable): base-101, base-102, vm-143, vm-902, vm-201, vm-202, csi-vol-e596, test-dummy, vm-104-disk-0@pre-paperless-migration snapshot.

• vm-201/vm-202: rbd info returns empty (ghost images, header gone). Same pattern as the ghost RBD section above — may need rados rm purge of orphaned data objects.

• CT114 (smb-tm-noris) successfully started on EC storage (media pool). RBD copy from 2026-07-06 completed (340/341 GiB). CT config updated from tm_disks:vm-114-disk-0 → media:vm-114-disk-0.

• Old tm_disks/vm-114-disk-0 (341 GiB) deletion: first attempt failed with "image still has watchers" — a stale rbd rm process (PID 2916941) was the watcher. Killed it, waited 35s, retry succeeded partially but image entered ghost state (header gone, 65k orphaned data objects, 341 GiB). OSD blacklist also had a stale client. User confirmed purge of orphaned objects via rados rm.

• Seafile fsck completed: 61/55 repos (6 overcount from restart).

• osd.7 reweight reverted from 0.65 → 1.0 (EC4+1 trap).

• osd.10 reweight reverted from 0.5 → 1.0 (same trap).

• PG merge completed: cephfs_data 128→32, cephfs_metadata 128→32.

• Recovery accelerated: osd_max_backfills=5, osd_recovery_op_priority=3.

• backfillfull flag on osd.10 persists despite ratio increase (sticky). 12 PGs remain backfill_toofull — waiting for natural clearance.

• CT143 (firecrawl) creation attempted on media storage — blocked by CFS lock timeout + mkfs blocklist (same issue as 2026-07-06). User instructed to wait for Ceph to stabilize, then retry.

• Playwright MCP container restored from surviving npm cache volume.

• Recovery stagnated at 11.5% misplaced for 7+ hours. Root cause was EC4+1 reweight trap (2147483647 in up-sets), NOT I/O contention. After reverting reweights, recovery resumed at 67 MiB/s.

• Recovery progressed to 0.74% misplaced (from 11.5%) after reweight revert + recovery acceleration (osd_max_backfills=5, op_priority=3). 5 PGs remained backfill_toofull on osd.10 (sticky backfillfull flag).

• PG merge for cephfs_data + cephfs_metadata completed (128→32, forced via ceph osd pool set <pool> pg_num 32 — worked even with 15 remapped PGs, reducing remapped count from 18→15).

• User asked about switching EC4+1→EC3+1 to relieve pressure. Answer: EC profiles are immutable, requires full pool recreation + migration. User decided to add a new HDD OSD instead — simpler, preserves EC4+1 efficiency, no data migration. See "EC Profile Immutability" section.

• CT143 (firecrawl) creation still blocked — pvesm alloc succeeds (creates RBD image) but pct create fails because it calls mkfs internally, which is agent-hardline-blocked. User must run mkfs.ext4 manually on the mapped RBD device, then pct create succeeds.

• TM backup failure on CT114 (smb-tm-noris): Mac connects, writes bands, but TM reports RESULT 70 (sparsebundle error). Likely I/O latency from ongoing Ceph recovery. User chose to wait for Ceph to stabilize.

• Playwright MCP container restored on CT111 from surviving npm-cache volume (@playwright/mcp v0.0.76, mcr.microsoft.com/playwright:latest).

• Orphaned RBD cleanup for old tm_disks/vm-114-disk-0: ghost image (header gone, 65k orphaned data objects, 341 GiB). Stale rbd rm process was the watcher. Purge via rados rm pending user confirmation.

• PBS offsite backup job expanded: vmid 106 → vmid 106,108,104,99999. Initial full backups run manually on correct nodes (CT108 on proxmox6, CT99999 on proxmox7, CT104 on n5pro). All 3 completed successfully.

• Frigate (CT120) excluded from all backup jobs per user decision.

• Seafile backup strategy analyzed: 6 options compared. Seafile Pro 13.0.19 supports S3 as primary storage backend (not backup target). User considering S3 backend to move 558 GB blocks out of Ceph. See seafile-api skill → references/seafile-immich-backup-strategy.md.

• CT111 (Seafile) backup on noris_s3 showed 0 MB (empty/failed) on 2026-07-07. Previous valid backup: 558 GB on 2026-07-06.

Session Log — 2026-07-12 (Continued)

• osd.10 drain via osd reweight SUCCESSFUL: ceph osd reweight 10 0.5 drained osd.10 from 93.14%→90.90%. Combined with set-backfillfull-ratio 0.95, unblocked 5 of 6 backfill_toofull PGs. Recovery at 30 MiB/s, 13% misplaced. This CONTRADICTS the earlier advice "Do NOT reweight osd.10" — that was correct for crush reweight (permanent, triggers 2147483647 trap) but WRONG for osd reweight (temporary runtime override, safe with EC4+1). Reference patched accordingly.
• Orphaned RBD cleanup completed: vm-143-disk-0 (50G), base-101-disk-0 (20G), base-102-disk-0 (20G), vm-902-disk-0 (40G), vm-201-disk-0, vm-202-disk-0, test-dummy all deleted. base-101/102 required rbd snap unprotect @__base__ before rbd snap purge + rbd rm. csi-vol-e596 has stale watcher (10.0.30.63, no K8s PVC found) — pending natural watcher timeout. ~161 GiB freed.
• K8s worker anti-colocation migration completed: VM 132→proxmox1, VM 131→proxmox4, VM 128→n5pro via qm migrate --online (HA-managed, async). Required expanding k8s-prefer-n5pro node-affinity rule to include all 8 nodes first. All 6 K8s VMs now on 6 separate nodes.
• hdd_disk→vm_disks migration feasibility: vm_disks has only 86 GiB free (75% full). Small CTs (vm-127 4G, vm-121 4G, base-9000 3G, vm-9001 10G = ~21G total) are candidates. Large K8s worker disks (80G×3) stay on hdd_disk (correct HDD-tier placement).