hermes-skills/devops/proxmox-ve-administration/references/ceph-osd-create-hotplug-2026-07.md

Creating a Ceph OSD via Hot-Plug on Non-PVE Nodes (n5pro / Minisforum)

When to Use

Adding a new HDD OSD to a Ceph cluster by physically hot-plugging a SATA drive into a running node that is not a standard PVE host (e.g. Minisforum N5 PRO with JMB58x SATA controller, CRUSH host n5pro).

Prerequisites Checklist

• Drive is physically inserted (hot-plug or powered-off install)
• Node has ceph, ceph-volume, pveceph installed
• Node has /etc/ceph/ceph.conf and ceph.client.admin.keyring
• /etc/ceph/ceph.conf is a symlink to /etc/pve/ceph.conf

Step-by-Step

1. Detect the Hot-Plugged Drive

If the drive was inserted while the node was running, the SATA bus may not auto-scan. Trigger a manual rescan:

for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
  echo "- - -" > ${h}/scan
done
sleep 2
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop

Verify the drive appears (e.g. /dev/sdb, MODEL=WDC WD30EFZX).

2. Copy Ceph Config to Non-PVE Node (if missing)

If the node lacks /etc/ceph/ceph.conf or the admin keyring:

# From PVE master (10.0.20.10), copy to target node:
scp /etc/ceph/ceph.conf root@<node>:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@<node>:/etc/ceph/
ssh root@<node> 'chmod 644 /etc/ceph/ceph.conf && chmod 600 /etc/ceph/ceph.client.admin.keyring'

3. Fix ceph.conf Symlink (CRITICAL — pveceph requires it)

pveceph osd create refuses to run if /etc/ceph/ceph.conf is a regular file rather than a symlink to /etc/pve/ceph.conf:

file '/etc/ceph/ceph.conf' already exists and is not a symlink to /etc/pve/ceph.conf

Fix:

rm /etc/ceph/ceph.conf
ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf

4. Zap the Drive Clean

Remove any leftover partitions/filesystem signatures from prior use:

sgdisk --zap-all /dev/sdX
wipefs -a /dev/sdX
parted /dev/sdX mklabel gpt
lsblk /dev/sdX -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT  # verify clean

5. Create the OSD

pveceph osd create /dev/sdX

This runs ceph-volume lvm create internally: creates a VG + LV on the disk, formats BlueStore, registers with the MON, enables systemd services (ceph-volume@lvm-<id>-<uuid>.service, ceph-osd@<id>.service), and starts the OSD.

6. Wait for OSD to Come Up (10-15 seconds)

The newly created OSD will initially show as down in ceph osd tree. The ceph-osd@<id> service is running but waits for the OSD map to load. Wait ~15 seconds, then verify:

ceph osd tree | grep osd.<id>
# Should show: up   1.00000

If still down after 30s, check:

journalctl -u ceph-osd@<id> --no-pager -n 30
systemctl status ceph-osd@<id>

Common non-fatal log lines:

• failed to load OSD map for epoch N, got 0 bytes — transient, resolves
• set_numa_affinity unable to identify public interface — cosmetic

7. Verify Cluster Integration

ceph osd crush get-device-class osd.<id>  # should report 'hdd'
ceph -s                                   # OSD count increased
ceph df                                   # raw capacity increased

Rebalancing begins automatically — expect a spike in remapped_pgs (normal).

Pitfalls

pveceph osd create fails: "already exists and is not a symlink"

Symptom: pveceph osd create /dev/sdb exits with code 22 and the message file '/etc/ceph/ceph.conf' already exists and is not a symlink.

Cause: On non-standard PVE nodes (e.g. n5pro), /etc/ceph/ceph.conf was manually placed as a regular file. pveceph insists on it being a symlink to /etc/pve/ceph.conf.

Fix: Remove the file and create the symlink (step 3 above).

Hot-Plugged SATA Drive Not Detected

Symptom: Drive physically inserted but lsblk doesn't show it.

Cause: The JMB58x AHCI controller does not expose hot_plug sysfs attribute, and Linux may not auto-scan the port.

Fix: Manual SCSI host rescan (step 1 above). The JMB58x does support AHCI hot-plug electrically even though the sysfs attribute is absent.

OSD Shows "down" Immediately After Creation

Symptom: ceph osd tree shows the new OSD as down with weight 0.

Cause: The ceph-osd daemon takes 10-15 seconds to load the OSD map and register with the MONs. This is normal startup latency, not a failure.

Fix: Wait. Check systemctl status ceph-osd@<id> — if it says active (running), the OSD will come up shortly.

Bootstrap Keyring Warning During ceph-volume

Symptom: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring

Cause: The bootstrap-osd keyring lives elsewhere on some nodes. The warning is non-fatal — ceph-volume falls back to the admin keyring.

Fix: None needed. OSD creation succeeds despite this warning.

Hardware Notes: Minisforum N5 PRO

Spec	Value
SATA Controller	JMicron JMB58x AHCI, 5 ports
AHCI Version	1.0301, 32 command slots, 6 Gbps
AHCI Flags	ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh
Hot-Plug sysfs	hot_plug attribute NOT exposed
Actual Hot-Plug	Works — bus rescan detects the drive
CRUSH Host Name	n5pro (own bucket, NOT ubuntu)
Existing OSDs	NONE — n5pro starts with zero OSDs
NVMe	AirDisk 128GB SSD (boot + LVM VG pve with WAL/DB LVs)

CRITICAL — n5pro ≠ ubuntu: Host ubuntu (IP 10.0.30.100) is a SEPARATE machine with its own OSDs (osd.6, osd.8). n5pro (IP 10.0.20.91) must have its own CRUSH host bucket named n5pro. Do NOT place n5pro OSDs under ubuntu — this would falsely suggest geographic separation for replicas that are actually on the same physical node.

Adding SSD WAL/DB to an OSD on Non-PVE Nodes

When the OSD's HDD is on a node that also has an NVMe SSD, placing the BlueStore DB (and collocated WAL) on the SSD dramatically improves write performance. The n5pro has a 128 GB AirDisk NVMe (/dev/nvme0n1) with an LVM VG pve that can host WAL/DB LVs.

Creating WAL/DB LVs on the NVMe

# Check free space in the pve VG
vgs pve   # look at VFree

# Create 30 GB LV for WAL/DB (one per OSD)
lvcreate -L 30G -n wal-db-osd-a pve
# Optional second LV for a future OSD:
lvcreate -L 30G -n wal-db-osd-b pve

⚠️ pveceph Cannot Use LVM Devices for WAL

pveceph osd create /dev/sdb --wal_dev /dev/pve/wal-db-osd-a FAILS:

unable to get device info for '/dev/dm-2' for type wal_dev

pveceph expects a raw block device, not a device-mapper LV. Use ceph-volume directly instead.

⚠️ BlueStore Cannot Share One LV For Both WAL and DB Separately

Specifying --block.wal AND --block.db pointing to the SAME LV fails:

bdev(...) open got: (16) Device or resource busy
bluestore _minimal_open_bluefs add block device(block.wal) returned: (16) Device or resource busy
OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error

BlueStore opens the WAL device exclusively — it cannot be the same underlying device as DB. The rollback also destroys the WAL LV.

✅ Correct Method: ceph-volume with --block.db Only

Specify ONLY --block.db — BlueStore collocates WAL on the DB device automatically. This is the standard approach for a single shared SSD:

# 1. Ensure the WAL/DB LV exists
lvcreate -L 30G -n wal-db-osd-a pve

# 2. Prepare the OSD with DB on NVMe (WAL collocated)
ceph-volume lvm prepare --bluestore \
  --data /dev/sdb \
  --block.db /dev/pve/wal-db-osd-a

# 3. Activate the OSD (replace UUID from prepare output)
ceph-volume lvm activate <osd_id> <osd_uuid>

# 4. Verify DB/WAL placement
ceph-volume lvm list <osd_id>
# Should show:
#   [block]     /dev/ceph-<vg>/osd-block-<uuid>   (data on HDD)
#   [db]        /dev/pve/wal-db-osd-a              (DB on NVMe)
#   db device   /dev/pve/wal-db-osd-a
#   devices     /dev/nvme0n1p3

Destroy and Recreate an OSD With WAL

If an OSD was already created WITHOUT WAL and needs to be rebuilt:

# 1. Stop the OSD service
systemctl stop ceph-osd@<id>

# 2. Mark down and destroy
ceph osd down <id>
ceph osd destroy <id> --yes-i-really-mean-it

# 3. Zap the disk (removes VG, LV, BlueStore signatures)
ceph-volume lvm zap /dev/sdX --destroy

# 4. Recreate with WAL/DB as above

CRUSH Host Assignment for Non-PVE Nodes

When using ceph-volume (not pveceph), the OSD may land in the wrong CRUSH host or under a generic bucket. Verify and fix:

# Check which host the OSD landed in
ceph osd tree | grep osd.<id>

# If in wrong host, move it:
ceph osd crush set osd.<id> <weight> root=default host=<correct_hostname>

# Set device class if not auto-detected
ceph osd crush set-device-class hdd osd.<id>

# Remove old destroyed OSD from CRUSH
ceph osd crush remove osd.<old_id>
ceph osd rm <old_id>

Session Log — 2026-07-13

Phase 1: Initial OSD Creation (osd.9, no WAL)

Added osd.9 (WD Red Plus 3 TB, WD-WX22DA0D7EP3) to n5pro via hot-plug. Drive was removed from CT 10.0.30.100 and physically inserted into n5pro while running. Bus rescan detected it on ata5. Created as BlueStore OSD with pveceph osd create /dev/sdb. Cluster went from 10→11 OSDs. backfill_toofull PGs dropped from 9→1 immediately. Remapped PGs spiked to 317 (expected rebalance). Device class: hdd.

Phase 2: CRUSH Host Correction

Mistake: Initially moved osd.6+osd.8 from host ubuntu to n5pro, assuming they were on the same machine. User corrected: osd.6 and osd.8 belong to host ubuntu (10.0.30.100), a separate physical machine. n5pro (10.0.20.91) only hosts osd.9 (later osd.11).

Fixed by recreating the ubuntu bucket and moving osd.6+osd.8 back:

ceph osd crush add-bucket ubuntu host
ceph osd crush move ubuntu root=default
ceph osd crush set osd.6 2.75829 root=default host=ubuntu
ceph osd crush set osd.8 2.72899 root=default host=ubuntu

Phase 3: Rebuild with SSD WAL/DB (osd.9 → osd.11)

Destroyed osd.9 (created without WAL) and rebuilt as osd.11 with DB on NVMe SSD. Key steps:

1. systemctl stop ceph-osd@9 → ceph osd destroy 9 → ceph-volume lvm zap /dev/sdb --destroy
2. Attempted ceph-volume lvm prepare --bluestore --data /dev/sdb --block.wal /dev/pve/wal-db-osd-a --block.db /dev/pve/wal-db-osd-a → FAILED (device busy, same LV for WAL+DB)
3. Rollback deleted wal-db-osd-a LV — had to recreate with lvcreate -L 30G -n wal-db-osd-a pve
4. Succeeded with --block.db /dev/pve/wal-db-osd-a only (WAL collocated on DB)
5. Activated, verified DB on NVMe, set CRUSH host to n5pro, device class hdd
6. Removed old osd.9 from CRUSH: ceph osd crush remove osd.9 + ceph osd rm 9

Final osd.11: data on /dev/sdb (3 TB HDD), DB+WAL on /dev/pve/wal-db-osd-a (30 GB NVMe LV), CRUSH host n5pro, weight 2.76, device class hdd.

Related: references/hdd-selection-for-osd-2026-07.md (drive selection), references/ceph-pg-osd-management-2026-07.md (rebalancing), references/ceph-recovery-acceleration-2026-07.md (recovery tuning).