195 lines
8.1 KiB
Markdown
195 lines
8.1 KiB
Markdown
# Ceph EC Pool Creation & LXC Service Migration — 2026-07-04
|
|
|
|
## EC Pool Creation (media_ec 4+1)
|
|
|
|
Created erasure-coded pool `media_ec` (k=4, m=1, HDD-only) with replicated metadata pool `media_meta` for RBD use.
|
|
|
|
### Commands Executed
|
|
|
|
```bash
|
|
CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"
|
|
|
|
# EC profile
|
|
$CEPH osd erasure-code-profile set ec41-hdd \
|
|
plugin=jerasure technique=reed_sol_van k=4 m=1 \
|
|
crush-failure-domain=osd crush-device-class=hdd
|
|
|
|
# Pools
|
|
$CEPH osd pool create media_ec 128 128 erasure ec41-hdd
|
|
$CEPH osd pool create media_meta 32 32 replicated
|
|
$CEPH osd pool application enable media_ec rbd
|
|
$CEPH osd pool application enable media_meta rbd
|
|
$CEPH osd pool set media_ec allow_ec_overwrites true
|
|
|
|
# PVE storage registration
|
|
pvesm add rbd media --pool media_meta --data-pool media_ec \
|
|
--username admin \
|
|
--monhost 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
|
|
pvesm set media --content rootdir
|
|
```
|
|
|
|
## Services Migrated to Ceph EC
|
|
|
|
### CT 136 (smb-media) — 10.0.30.30
|
|
|
|
| Item | Detail |
|
|
|------|--------|
|
|
| Purpose | Samba media share (Filme, Serien, XXX) |
|
|
| Root disk | 8 GB on vm_disks (replicated RBD) |
|
|
| Mount point | mp0: media:500,mp=/mnt/media (EC-backed) |
|
|
| Data migrated | 370 GB total (Filme 3.5G, Serien 36G, XXX 330G) |
|
|
| Rsync source | 10.0.30.100:/pool01_n2_redundant/Download/ |
|
|
| HA | ct:136, max_relocate=3, max_restart=2 |
|
|
|
|
### CT 137 (imap) — 10.0.30.40
|
|
|
|
| Item | Detail |
|
|
|------|--------|
|
|
| Purpose | Dovecot IMAP server |
|
|
| Root disk | 8 GB on vm_disks (replicated RBD) |
|
|
| Mount point | mp0: media:20,mp=/var/mail (EC-backed) |
|
|
| Data migrated | 15 GB mailboxes (99,690 files) |
|
|
| Rsync source | 10.0.30.100:/var/lib/docker/volumes/mailserver/mail/vhosts/ |
|
|
| Dovecot config | maildir:/var/mail/vhosts/%d/%n/mail |
|
|
| Dovecot version | 2.3.19.1 (Debian 12) |
|
|
| Ports | IMAP 143, IMAPS 993, POP3 110, POP3S 995 |
|
|
| Password backend | passwd-file with SHA512-CRYPT hashes (extracted from old PostfixAdmin MySQL) |
|
|
| HA | ct:137, max_relocate=3, max_restart=2 |
|
|
|
|
#### Mailbox Inventory
|
|
|
|
| Domain | Mailbox | Files | Size |
|
|
|--------|---------|-------|------|
|
|
| familie-schoen.com | dominik | 65,493 | 3,975 MB |
|
|
| famille-schoen.com | sarah | 9,685 | 3,791 MB |
|
|
| famille-schoen.com | sarah-coc | 9,976 | 4,585 MB |
|
|
| famille-schoen.com | newsletter | 14,200 | 421 MB |
|
|
| famille-schoen.com | dokumente | 173 | 716 MB |
|
|
| famille-schoen.com | honig | 159 | 7 MB |
|
|
| dominikschoen.de | (empty) | 0 | — |
|
|
| famschoen.eu | (empty) | 0 | — |
|
|
| schoen.sytes.net | (empty) | 0 | — |
|
|
| grafiniert.de | (empty) | 0 | — |
|
|
| **Total** | | **99,690** | **~15 GB** |
|
|
|
|
#### Password Extraction Technique
|
|
|
|
Passwords were extracted from the dead Docker host's PostfixAdmin MySQL database by spinning up a temporary MariaDB container with `--skip-grant-tables`:
|
|
|
|
```bash
|
|
docker run -d --rm --name tmp-mariadb \
|
|
-v /var/lib/docker/volumes/mailserver/mysql/db:/var/lib/mysql \
|
|
mariadb:latest --skip-grant-tables --skip-networking
|
|
docker exec tmp-mariadb mysql -uroot \
|
|
-e "SELECT username,password,maildir FROM postfix.mailbox;"
|
|
docker stop tmp-mariadb
|
|
```
|
|
|
|
Hashes were then written to the CT via base64 round-trip to preserve `$` characters in SHA512-CRYPT hashes (shell eats `$` through nested bash/SSH).
|
|
|
|
## Migration Pattern
|
|
|
|
Both CTs followed the same pattern:
|
|
1. Create CT with rootfs on replicated RBD (`vm_disks`)
|
|
2. Add EC-backed mount point via `pct set -mp0 media:<size>,mp=<path>`
|
|
3. Install SSH server + rsync in CT
|
|
4. Set root password + enable PermitRootLogin
|
|
5. Rsync from source host using `sshpass -e` transport
|
|
6. Install + configure service (Samba / Dovecot)
|
|
7. Add to HA manager
|
|
|
|
## Pitfalls Encountered
|
|
|
|
- **Template storage `local` disabled** — used `hdd_templates:vztmpl/` instead
|
|
- **Wrong bridge name `vmbr1`** — Schön homelab uses `vmbr0` with `tag=30` for VLAN 30
|
|
- **`pvesm alloc` creates unformatted RBD** — `pct set -mp0` creates AND formats in one step
|
|
- **Fresh CT has no SSH server** — must `apt install openssh-server` + set root password before rsync
|
|
- **rsync over SSH needs sshpass in `-e` flag** — `rsync -e "sshpass -p PASS ssh"`
|
|
- **ZFS `du` hangs under rsync I/O load** — use `find -printf` or wait for rsync to complete
|
|
- **`pct push` writes null bytes when source file is missing** — verify with `cat` after push
|
|
- **Shell `$` in SHA512-CRYPT hashes eaten through nested bash** — use base64 round-trip
|
|
- **`pct create` with nonexistent template name** — check `pveam list hdd_templates` first
|
|
|
|
## Password Reset Session (2026-07-04)
|
|
|
|
All 6 mailbox accounts were reset to a uniform password using `doveadm pw`:
|
|
|
|
```bash
|
|
# Generate hash inside CT
|
|
HASH=$(pct exec 137 -- doveadm pw -s SHA512-CRYPT -p "NEW_PASSWORD")
|
|
# Output: {SHA512-CRYPT}$6$xMdJTDRO/SDIbrdP$qONDQi...
|
|
|
|
# Build users file locally (same hash for all accounts)
|
|
for user in dominik sarah newsletter dokumente sarah-coc honig; do
|
|
echo "${user}@familie-schoen.com:${HASH}"
|
|
done > /tmp/dovecot-users
|
|
|
|
# Base64 round-trip to CT (preserves $ characters)
|
|
B64=$(base64 /tmp/dovecot-users | tr -d '\n')
|
|
pct exec 137 -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"
|
|
pct exec 137 -- systemctl restart dovecot
|
|
|
|
# Verify login
|
|
pct exec 137 -- bash -c \
|
|
"echo -e '1 LOGIN dominik@familie-schoen.com NEW_PASSWORD\n2 LIST \"\" \"INBOX\"\n3 LOGOUT' | timeout 5 nc localhost 143"
|
|
```
|
|
|
|
### Seafile Version Identified
|
|
|
|
The Seafile installation on 10.0.30.100 was `seafileltd/seafile-mc:11.0-latest` (Docker image, 20 months old). Found via `docker images | grep seafile` after daemon recovery.
|
|
|
|
## Post-Migration Cleanup (2026-07-04)
|
|
|
|
### Source Data Deletion on 10.0.30.100
|
|
|
|
After successful migration to CT 136 (media) and CT 137 (IMAP), source data was deleted from 10.0.30.100:
|
|
|
|
| Path Deleted | Size | Method | Duration |
|
|
|-------------|------|--------|----------|
|
|
| `/var/lib/docker/volumes/mailserver/mail/vhosts/` | 11 GB | `rm -rf` | ~30s (99k small files) |
|
|
| `/pool01_n2_redundant/Download/Filme/` | 3.5 GB | `rm -rf` | ~5s |
|
|
| `/pool01_n2_redundant/Download/Serien/` | 33 GB | `rm -rf` | ~10s |
|
|
| `/pool01_n2_redundant/Download/XXX/` | 329 GB | `rm -rf` (background) | ~15 min |
|
|
| `/pool01_n2_redundant/Download/complete/` | 746 MB | `rm -rf` (background) | ~5s |
|
|
|
|
**ZFS async free lag:** After `rm -rf` completed, `zfs list` still showed 290 GB USED on the Download dataset, but `du -sh` showed only 3.1 GB live. ZFS frees blocks asynchronously via TXG commits — the gap is expected and resolves over time.
|
|
|
|
### ZFS Dataset Destruction
|
|
|
|
The entire `pool01_n2_redundant/Download` dataset was destroyed to reclaim all space:
|
|
|
|
```bash
|
|
# 1. Find blocking processes (cwd in dataset)
|
|
lsof +D /pool01_n2_redundant/Download/
|
|
fuser -mv /pool01_n2_redundant/Download/
|
|
|
|
# 2. Kill ALL blockers (including background monitor processes!)
|
|
kill -9 <PID1> <PID2> ...
|
|
|
|
# 3. Destroy dataset with all snapshots
|
|
zfs destroy -r pool01_n2_redundant/Download
|
|
# ⚠️ Enters D-state (uninterruptible I/O) — cannot be killed
|
|
# ⚠️ On degraded RAIDZ2 with defective disks: 10+ minutes
|
|
```
|
|
|
|
**Pitfalls encountered:**
|
|
- Background monitoring processes (polling `zfs list` in a loop) had cwd in the dataset, blocking `zfs destroy`. Must kill ALL processes before destroy.
|
|
- `zfs destroy` entered D-state for 10+ minutes on the degraded RAIDZ2 pool (2 defective disks). This is normal I/O wait, not a hang.
|
|
- `zpool sync` also hangs during heavy async free operations — do not rely on it for progress monitoring.
|
|
|
|
### Ceph Pool-Full Incident After Migration
|
|
|
|
Adding CT 137's 20 GB RBD to `media_ec` pushed osd.10 to exactly 95% (`full_ratio`), triggering a cluster-wide pool-full condition. See Section 6.12 of the SKILL.md for the full diagnosis and fix.
|
|
|
|
### Download Dataset Contents (Final Inventory Before Deletion)
|
|
|
|
Remaining items in `/pool01_n2_redundant/Download/` after media migration (all <3 GB, kept until dataset destroyed):
|
|
|
|
| Path | Size | Description |
|
|
|------|------|-------------|
|
|
| VirtualBox VMs/ | 2.7 GB | Old VB-VM from 2017 |
|
|
| Windows 8.1.vmwarevm/ | 270 MB | Old VMware VM (1 VMDK) |
|
|
| Circuit.dmg | 83 MB | macOS app |
|
|
| NZBget dirs (incomplete/nzb/output/queue/tmp/vm) | 0 | Empty |
|
|
| complete/ | 746 MB | NZBget output (old macOS software, 1 film) |
|