# Ceph EC Pool Creation & LXC Service Migration — 2026-07-04

## EC Pool Creation (media_ec 4+1)

Created erasure-coded pool `media_ec` (k=4, m=1, HDD-only) with replicated metadata pool `media_meta` for RBD use.

### Commands Executed

```bash
CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# EC profile
$CEPH osd erasure-code-profile set ec41-hdd \
  plugin=jerasure technique=reed_sol_van k=4 m=1 \
  crush-failure-domain=osd crush-device-class=hdd

# Pools
$CEPH osd pool create media_ec 128 128 erasure ec41-hdd
$CEPH osd pool create media_meta 32 32 replicated
$CEPH osd pool application enable media_ec rbd
$CEPH osd pool application enable media_meta rbd
$CEPH osd pool set media_ec allow_ec_overwrites true

# PVE storage registration
pvesm add rbd media --pool media_meta --data-pool media_ec \
  --username admin \
  --monhost 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
pvesm set media --content rootdir
```

## Services Migrated to Ceph EC

### CT 136 (smb-media) — 10.0.30.30

| Item | Detail |
|------|--------|
| Purpose | Samba media share (Filme, Serien, XXX) |
| Root disk | 8 GB on vm_disks (replicated RBD) |
| Mount point | mp0: media:500,mp=/mnt/media (EC-backed) |
| Data migrated | 370 GB total (Filme 3.5G, Serien 36G, XXX 330G) |
| Rsync source | 10.0.30.100:/pool01_n2_redundant/Download/ |
| HA | ct:136, max_relocate=3, max_restart=2 |

### CT 137 (imap) — 10.0.30.40

| Item | Detail |
|------|--------|
| Purpose | Dovecot IMAP server |
| Root disk | 8 GB on vm_disks (replicated RBD) |
| Mount point | mp0: media:20,mp=/var/mail (EC-backed) |
| Data migrated | 15 GB mailboxes (99,690 files) |
| Rsync source | 10.0.30.100:/var/lib/docker/volumes/mailserver/mail/vhosts/ |
| Dovecot config | maildir:/var/mail/vhosts/%d/%n/mail |
| Dovecot version | 2.3.19.1 (Debian 12) |
| Ports | IMAP 143, IMAPS 993, POP3 110, POP3S 995 |
| Password backend | passwd-file with SHA512-CRYPT hashes (extracted from old PostfixAdmin MySQL) |
| HA | ct:137, max_relocate=3, max_restart=2 |

#### Mailbox Inventory

| Domain | Mailbox | Files | Size |
|--------|---------|-------|------|
| familie-schoen.com | dominik | 65,493 | 3,975 MB |
| famille-schoen.com | sarah | 9,685 | 3,791 MB |
| famille-schoen.com | sarah-coc | 9,976 | 4,585 MB |
| famille-schoen.com | newsletter | 14,200 | 421 MB |
| famille-schoen.com | dokumente | 173 | 716 MB |
| famille-schoen.com | honig | 159 | 7 MB |
| dominikschoen.de | (empty) | 0 | — |
| famschoen.eu | (empty) | 0 | — |
| schoen.sytes.net | (empty) | 0 | — |
| grafiniert.de | (empty) | 0 | — |
| **Total** | | **99,690** | **~15 GB** |

#### Password Extraction Technique

Passwords were extracted from the dead Docker host's PostfixAdmin MySQL database by spinning up a temporary MariaDB container with `--skip-grant-tables`:

```bash
docker run -d --rm --name tmp-mariadb \
  -v /var/lib/docker/volumes/mailserver/mysql/db:/var/lib/mysql \
  mariadb:latest --skip-grant-tables --skip-networking
docker exec tmp-mariadb mysql -uroot \
  -e "SELECT username,password,maildir FROM postfix.mailbox;"
docker stop tmp-mariadb
```

Hashes were then written to the CT via base64 round-trip to preserve `$` characters in SHA512-CRYPT hashes (shell eats `$` through nested bash/SSH).

## Migration Pattern

Both CTs followed the same pattern:
1. Create CT with rootfs on replicated RBD (`vm_disks`)
2. Add EC-backed mount point via `pct set -mp0 media:<size>,mp=<path>`
3. Install SSH server + rsync in CT
4. Set root password + enable PermitRootLogin
5. Rsync from source host using `sshpass -e` transport
6. Install + configure service (Samba / Dovecot)
7. Add to HA manager

## Pitfalls Encountered

- **Template storage `local` disabled** — used `hdd_templates:vztmpl/` instead
- **Wrong bridge name `vmbr1`** — Schön homelab uses `vmbr0` with `tag=30` for VLAN 30
- **`pvesm alloc` creates unformatted RBD** — `pct set -mp0` creates AND formats in one step
- **Fresh CT has no SSH server** — must `apt install openssh-server` + set root password before rsync
- **rsync over SSH needs sshpass in `-e` flag** — `rsync -e "sshpass -p PASS ssh"`
- **ZFS `du` hangs under rsync I/O load** — use `find -printf` or wait for rsync to complete
- **`pct push` writes null bytes when source file is missing** — verify with `cat` after push
- **Shell `$` in SHA512-CRYPT hashes eaten through nested bash** — use base64 round-trip
- **`pct create` with nonexistent template name** — check `pveam list hdd_templates` first

## Password Reset Session (2026-07-04)

All 6 mailbox accounts were reset to a uniform password using `doveadm pw`:

```bash
# Generate hash inside CT
HASH=$(pct exec 137 -- doveadm pw -s SHA512-CRYPT -p "NEW_PASSWORD")
# Output: {SHA512-CRYPT}$6$xMdJTDRO/SDIbrdP$qONDQi...

# Build users file locally (same hash for all accounts)
for user in dominik sarah newsletter dokumente sarah-coc honig; do
  echo "${user}@familie-schoen.com:${HASH}"
done > /tmp/dovecot-users

# Base64 round-trip to CT (preserves $ characters)
B64=$(base64 /tmp/dovecot-users | tr -d '\n')
pct exec 137 -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"
pct exec 137 -- systemctl restart dovecot

# Verify login
pct exec 137 -- bash -c \
  "echo -e '1 LOGIN dominik@familie-schoen.com NEW_PASSWORD\n2 LIST \"\" \"INBOX\"\n3 LOGOUT' | timeout 5 nc localhost 143"
```

### Seafile Version Identified

The Seafile installation on 10.0.30.100 was `seafileltd/seafile-mc:11.0-latest` (Docker image, 20 months old). Found via `docker images | grep seafile` after daemon recovery.

## Post-Migration Cleanup (2026-07-04)

### Source Data Deletion on 10.0.30.100

After successful migration to CT 136 (media) and CT 137 (IMAP), source data was deleted from 10.0.30.100:

| Path Deleted | Size | Method | Duration |
|-------------|------|--------|----------|
| `/var/lib/docker/volumes/mailserver/mail/vhosts/` | 11 GB | `rm -rf` | ~30s (99k small files) |
| `/pool01_n2_redundant/Download/Filme/` | 3.5 GB | `rm -rf` | ~5s |
| `/pool01_n2_redundant/Download/Serien/` | 33 GB | `rm -rf` | ~10s |
| `/pool01_n2_redundant/Download/XXX/` | 329 GB | `rm -rf` (background) | ~15 min |
| `/pool01_n2_redundant/Download/complete/` | 746 MB | `rm -rf` (background) | ~5s |

**ZFS async free lag:** After `rm -rf` completed, `zfs list` still showed 290 GB USED on the Download dataset, but `du -sh` showed only 3.1 GB live. ZFS frees blocks asynchronously via TXG commits — the gap is expected and resolves over time.

### ZFS Dataset Destruction

The entire `pool01_n2_redundant/Download` dataset was destroyed to reclaim all space:

```bash
# 1. Find blocking processes (cwd in dataset)
lsof +D /pool01_n2_redundant/Download/
fuser -mv /pool01_n2_redundant/Download/

# 2. Kill ALL blockers (including background monitor processes!)
kill -9 <PID1> <PID2> ...

# 3. Destroy dataset with all snapshots
zfs destroy -r pool01_n2_redundant/Download
# ⚠️ Enters D-state (uninterruptible I/O) — cannot be killed
# ⚠️ On degraded RAIDZ2 with defective disks: 10+ minutes
```

**Pitfalls encountered:**
- Background monitoring processes (polling `zfs list` in a loop) had cwd in the dataset, blocking `zfs destroy`. Must kill ALL processes before destroy.
- `zfs destroy` entered D-state for 10+ minutes on the degraded RAIDZ2 pool (2 defective disks). This is normal I/O wait, not a hang.
- `zpool sync` also hangs during heavy async free operations — do not rely on it for progress monitoring.

### Ceph Pool-Full Incident After Migration

Adding CT 137's 20 GB RBD to `media_ec` pushed osd.10 to exactly 95% (`full_ratio`), triggering a cluster-wide pool-full condition. See Section 6.12 of the SKILL.md for the full diagnosis and fix.

### Download Dataset Contents (Final Inventory Before Deletion)

Remaining items in `/pool01_n2_redundant/Download/` after media migration (all <3 GB, kept until dataset destroyed):

| Path | Size | Description |
|------|------|-------------|
| VirtualBox VMs/ | 2.7 GB | Old VB-VM from 2017 |
| Windows 8.1.vmwarevm/ | 270 MB | Old VMware VM (1 VMDK) |
| Circuit.dmg | 83 MB | macOS app |
| NZBget dirs (incomplete/nzb/output/queue/tmp/vm) | 0 | Empty |
| complete/ | 746 MB | NZBget output (old macOS software, 1 film) |