hermes-skills/devops/proxmox-ve-administration/references/ceph-ec-pool-and-service-migration-2026-07.md

Ceph EC Pool Creation & LXC Service Migration — 2026-07-04

EC Pool Creation (media_ec 4+1)

Created erasure-coded pool media_ec (k=4, m=1, HDD-only) with replicated metadata pool media_meta for RBD use.

Commands Executed

CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# EC profile
$CEPH osd erasure-code-profile set ec41-hdd \
  plugin=jerasure technique=reed_sol_van k=4 m=1 \
  crush-failure-domain=osd crush-device-class=hdd

# Pools
$CEPH osd pool create media_ec 128 128 erasure ec41-hdd
$CEPH osd pool create media_meta 32 32 replicated
$CEPH osd pool application enable media_ec rbd
$CEPH osd pool application enable media_meta rbd
$CEPH osd pool set media_ec allow_ec_overwrites true

# PVE storage registration
pvesm add rbd media --pool media_meta --data-pool media_ec \
  --username admin \
  --monhost 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
pvesm set media --content rootdir

Services Migrated to Ceph EC

CT 136 (smb-media) — 10.0.30.30

Item	Detail
Purpose	Samba media share (Filme, Serien, XXX)
Root disk	8 GB on vm_disks (replicated RBD)
Mount point	mp0: media:500,mp=/mnt/media (EC-backed)
Data migrated	370 GB total (Filme 3.5G, Serien 36G, XXX 330G)
Rsync source	10.0.30.100:/pool01_n2_redundant/Download/
HA	ct:136, max_relocate=3, max_restart=2

CT 137 (imap) — 10.0.30.40

Item	Detail
Purpose	Dovecot IMAP server
Root disk	8 GB on vm_disks (replicated RBD)
Mount point	mp0: media:20,mp=/var/mail (EC-backed)
Data migrated	15 GB mailboxes (99,690 files)
Rsync source	10.0.30.100:/var/lib/docker/volumes/mailserver/mail/vhosts/
Dovecot config	maildir:/var/mail/vhosts/%d/%n/mail
Dovecot version	2.3.19.1 (Debian 12)
Ports	IMAP 143, IMAPS 993, POP3 110, POP3S 995
Password backend	passwd-file with SHA512-CRYPT hashes (extracted from old PostfixAdmin MySQL)
HA	ct:137, max_relocate=3, max_restart=2

Mailbox Inventory

Domain	Mailbox	Files	Size
familie-schoen.com	dominik	65,493	3,975 MB
famille-schoen.com	sarah	9,685	3,791 MB
famille-schoen.com	sarah-coc	9,976	4,585 MB
famille-schoen.com	newsletter	14,200	421 MB
famille-schoen.com	dokumente	173	716 MB
famille-schoen.com	honig	159	7 MB
dominikschoen.de	(empty)	0	—
famschoen.eu	(empty)	0	—
schoen.sytes.net	(empty)	0	—
grafiniert.de	(empty)	0	—
Total		99,690	~15 GB

Password Extraction Technique

Passwords were extracted from the dead Docker host's PostfixAdmin MySQL database by spinning up a temporary MariaDB container with --skip-grant-tables:

docker run -d --rm --name tmp-mariadb \
  -v /var/lib/docker/volumes/mailserver/mysql/db:/var/lib/mysql \
  mariadb:latest --skip-grant-tables --skip-networking
docker exec tmp-mariadb mysql -uroot \
  -e "SELECT username,password,maildir FROM postfix.mailbox;"
docker stop tmp-mariadb

Hashes were then written to the CT via base64 round-trip to preserve $ characters in SHA512-CRYPT hashes (shell eats $ through nested bash/SSH).

Migration Pattern

Both CTs followed the same pattern:

1. Create CT with rootfs on replicated RBD (vm_disks)
2. Add EC-backed mount point via pct set -mp0 media:<size>,mp=<path>
3. Install SSH server + rsync in CT
4. Set root password + enable PermitRootLogin
5. Rsync from source host using sshpass -e transport
6. Install + configure service (Samba / Dovecot)
7. Add to HA manager

Pitfalls Encountered

• Template storage local disabled — used hdd_templates:vztmpl/ instead
• Wrong bridge name vmbr1 — Schön homelab uses vmbr0 with tag=30 for VLAN 30
• pvesm alloc creates unformatted RBD — pct set -mp0 creates AND formats in one step
• Fresh CT has no SSH server — must apt install openssh-server + set root password before rsync
• rsync over SSH needs sshpass in -e flag — rsync -e "sshpass -p PASS ssh"
• ZFS du hangs under rsync I/O load — use find -printf or wait for rsync to complete
• pct push writes null bytes when source file is missing — verify with cat after push
• Shell $ in SHA512-CRYPT hashes eaten through nested bash — use base64 round-trip
• pct create with nonexistent template name — check pveam list hdd_templates first

Password Reset Session (2026-07-04)

All 6 mailbox accounts were reset to a uniform password using doveadm pw:

# Generate hash inside CT
HASH=$(pct exec 137 -- doveadm pw -s SHA512-CRYPT -p "NEW_PASSWORD")
# Output: {SHA512-CRYPT}$6$xMdJTDRO/SDIbrdP$qONDQi...

# Build users file locally (same hash for all accounts)
for user in dominik sarah newsletter dokumente sarah-coc honig; do
  echo "${user}@familie-schoen.com:${HASH}"
done > /tmp/dovecot-users

# Base64 round-trip to CT (preserves $ characters)
B64=$(base64 /tmp/dovecot-users | tr -d '\n')
pct exec 137 -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"
pct exec 137 -- systemctl restart dovecot

# Verify login
pct exec 137 -- bash -c \
  "echo -e '1 LOGIN dominik@familie-schoen.com NEW_PASSWORD\n2 LIST \"\" \"INBOX\"\n3 LOGOUT' | timeout 5 nc localhost 143"

Seafile Version Identified

The Seafile installation on 10.0.30.100 was seafileltd/seafile-mc:11.0-latest (Docker image, 20 months old). Found via docker images | grep seafile after daemon recovery.

Post-Migration Cleanup (2026-07-04)

Source Data Deletion on 10.0.30.100

After successful migration to CT 136 (media) and CT 137 (IMAP), source data was deleted from 10.0.30.100:

Path Deleted	Size	Method	Duration
/var/lib/docker/volumes/mailserver/mail/vhosts/	11 GB	rm -rf	~30s (99k small files)
/pool01_n2_redundant/Download/Filme/	3.5 GB	rm -rf	~5s
/pool01_n2_redundant/Download/Serien/	33 GB	rm -rf	~10s
/pool01_n2_redundant/Download/XXX/	329 GB	rm -rf (background)	~15 min
/pool01_n2_redundant/Download/complete/	746 MB	rm -rf (background)	~5s

ZFS async free lag: After rm -rf completed, zfs list still showed 290 GB USED on the Download dataset, but du -sh showed only 3.1 GB live. ZFS frees blocks asynchronously via TXG commits — the gap is expected and resolves over time.

ZFS Dataset Destruction

The entire pool01_n2_redundant/Download dataset was destroyed to reclaim all space:

# 1. Find blocking processes (cwd in dataset)
lsof +D /pool01_n2_redundant/Download/
fuser -mv /pool01_n2_redundant/Download/

# 2. Kill ALL blockers (including background monitor processes!)
kill -9 <PID1> <PID2> ...

# 3. Destroy dataset with all snapshots
zfs destroy -r pool01_n2_redundant/Download
# ⚠️ Enters D-state (uninterruptible I/O) — cannot be killed
# ⚠️ On degraded RAIDZ2 with defective disks: 10+ minutes

Pitfalls encountered:

• Background monitoring processes (polling zfs list in a loop) had cwd in the dataset, blocking zfs destroy. Must kill ALL processes before destroy.
• zfs destroy entered D-state for 10+ minutes on the degraded RAIDZ2 pool (2 defective disks). This is normal I/O wait, not a hang.
• zpool sync also hangs during heavy async free operations — do not rely on it for progress monitoring.

Ceph Pool-Full Incident After Migration

Adding CT 137's 20 GB RBD to media_ec pushed osd.10 to exactly 95% (full_ratio), triggering a cluster-wide pool-full condition. See Section 6.12 of the SKILL.md for the full diagnosis and fix.

Download Dataset Contents (Final Inventory Before Deletion)

Remaining items in /pool01_n2_redundant/Download/ after media migration (all <3 GB, kept until dataset destroyed):

Path	Size	Description
VirtualBox VMs/	2.7 GB	Old VB-VM from 2017
Windows 8.1.vmwarevm/	270 MB	Old VMware VM (1 VMDK)
Circuit.dmg	83 MB	macOS app
NZBget dirs (incomplete/nzb/output/queue/tmp/vm)	0	Empty
complete/	746 MB	NZBget output (old macOS software, 1 film)