hermes-skills/devops/proxmox-ve-administration/references/ceph-osd-create-hotplug-2026-07.md

# Creating a Ceph OSD via Hot-Plug on Non-PVE Nodes (n5pro / Minisforum)

## When to Use

Adding a new HDD OSD to a Ceph cluster by physically hot-plugging a SATA
drive into a running node that is not a standard PVE host (e.g. Minisforum
N5 PRO with JMB58x SATA controller, CRUSH host `n5pro`).

## Prerequisites Checklist

- [ ] Drive is physically inserted (hot-plug or powered-off install)
- [ ] Node has `ceph`, `ceph-volume`, `pveceph` installed
- [ ] Node has `/etc/ceph/ceph.conf` and `ceph.client.admin.keyring`
- [ ] `/etc/ceph/ceph.conf` is a **symlink to `/etc/pve/ceph.conf`**

## Step-by-Step

### 1. Detect the Hot-Plugged Drive

If the drive was inserted while the node was running, the SATA bus may
not auto-scan. Trigger a manual rescan:

```bash
for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
  echo "- - -" > ${h}/scan
done
sleep 2
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop
```

Verify the drive appears (e.g. `/dev/sdb`, MODEL=WDC WD30EFZX).

### 2. Copy Ceph Config to Non-PVE Node (if missing)

If the node lacks `/etc/ceph/ceph.conf` or the admin keyring:

```bash
# From PVE master (10.0.20.10), copy to target node:
scp /etc/ceph/ceph.conf root@<node>:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@<node>:/etc/ceph/
ssh root@<node> 'chmod 644 /etc/ceph/ceph.conf && chmod 600 /etc/ceph/ceph.client.admin.keyring'
```

### 3. Fix ceph.conf Symlink (CRITICAL — pveceph requires it)

`pveceph osd create` refuses to run if `/etc/ceph/ceph.conf` is a regular
file rather than a symlink to `/etc/pve/ceph.conf`:

```
file '/etc/ceph/ceph.conf' already exists and is not a symlink to /etc/pve/ceph.conf
```

Fix:

```bash
rm /etc/ceph/ceph.conf
ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
```

### 4. Zap the Drive Clean

Remove any leftover partitions/filesystem signatures from prior use:

```bash
sgdisk --zap-all /dev/sdX
wipefs -a /dev/sdX
parted /dev/sdX mklabel gpt
lsblk /dev/sdX -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT  # verify clean
```

### 5. Create the OSD

```bash
pveceph osd create /dev/sdX
```

This runs `ceph-volume lvm create` internally: creates a VG + LV on the
disk, formats BlueStore, registers with the MON, enables systemd services
(`ceph-volume@lvm-<id>-<uuid>.service`, `ceph-osd@<id>.service`), and
starts the OSD.

### 6. Wait for OSD to Come Up (10-15 seconds)

The newly created OSD will initially show as **down** in `ceph osd tree`.
The `ceph-osd@<id>` service is running but waits for the OSD map to load.
Wait ~15 seconds, then verify:

```bash
ceph osd tree | grep osd.<id>
# Should show: up   1.00000
```

If still down after 30s, check:

```bash
journalctl -u ceph-osd@<id> --no-pager -n 30
systemctl status ceph-osd@<id>
```

Common non-fatal log lines:
- `failed to load OSD map for epoch N, got 0 bytes` — transient, resolves
- `set_numa_affinity unable to identify public interface` — cosmetic

### 7. Verify Cluster Integration

```bash
ceph osd crush get-device-class osd.<id>  # should report 'hdd'
ceph -s                                   # OSD count increased
ceph df                                   # raw capacity increased
```

Rebalancing begins automatically — expect a spike in `remapped_pgs` (normal).

## Pitfalls

### pveceph osd create fails: "already exists and is not a symlink"

**Symptom**: `pveceph osd create /dev/sdb` exits with code 22 and the
message `file '/etc/ceph/ceph.conf' already exists and is not a symlink`.

**Cause**: On non-standard PVE nodes (e.g. n5pro), `/etc/ceph/ceph.conf`
was manually placed as a regular file. `pveceph` insists on it being a
symlink to `/etc/pve/ceph.conf`.

**Fix**: Remove the file and create the symlink (step 3 above).

### Hot-Plugged SATA Drive Not Detected

**Symptom**: Drive physically inserted but `lsblk` doesn't show it.

**Cause**: The JMB58x AHCI controller does not expose `hot_plug` sysfs
attribute, and Linux may not auto-scan the port.

**Fix**: Manual SCSI host rescan (step 1 above). The JMB58x does support
AHCI hot-plug electrically even though the sysfs attribute is absent.

### OSD Shows "down" Immediately After Creation

**Symptom**: `ceph osd tree` shows the new OSD as `down` with weight 0.

**Cause**: The `ceph-osd` daemon takes 10-15 seconds to load the OSD map
and register with the MONs. This is normal startup latency, not a failure.

**Fix**: Wait. Check `systemctl status ceph-osd@<id>` — if it says
`active (running)`, the OSD will come up shortly.

### Bootstrap Keyring Warning During ceph-volume

**Symptom**: `unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring`

**Cause**: The bootstrap-osd keyring lives elsewhere on some nodes. The
warning is non-fatal — `ceph-volume` falls back to the admin keyring.

**Fix**: None needed. OSD creation succeeds despite this warning.

## Hardware Notes: Minisforum N5 PRO

| Spec | Value |
|------|-------|
| SATA Controller | JMicron JMB58x AHCI, 5 ports |
| AHCI Version | 1.0301, 32 command slots, 6 Gbps |
| AHCI Flags | ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh |
| Hot-Plug sysfs | `hot_plug` attribute NOT exposed |
| Actual Hot-Plug | Works — bus rescan detects the drive |
| CRUSH Host Name | `n5pro` (own bucket, NOT `ubuntu`) |
| Existing OSDs | NONE — n5pro starts with zero OSDs |
| NVMe | AirDisk 128GB SSD (boot + LVM VG `pve` with WAL/DB LVs) |

> **CRITICAL — n5pro ≠ ubuntu**: Host `ubuntu` (IP 10.0.30.100) is a SEPARATE
> machine with its own OSDs (osd.6, osd.8). n5pro (IP 10.0.20.91) must have
> its own CRUSH host bucket named `n5pro`. Do NOT place n5pro OSDs under
> `ubuntu` — this would falsely suggest geographic separation for replicas
> that are actually on the same physical node.

## Adding SSD WAL/DB to an OSD on Non-PVE Nodes

When the OSD's HDD is on a node that also has an NVMe SSD, placing the
BlueStore DB (and collocated WAL) on the SSD dramatically improves write
performance. The n5pro has a 128 GB AirDisk NVMe (`/dev/nvme0n1`) with
an LVM VG `pve` that can host WAL/DB LVs.

### Creating WAL/DB LVs on the NVMe

```bash
# Check free space in the pve VG
vgs pve   # look at VFree

# Create 30 GB LV for WAL/DB (one per OSD)
lvcreate -L 30G -n wal-db-osd-a pve
# Optional second LV for a future OSD:
lvcreate -L 30G -n wal-db-osd-b pve
```

### ⚠️ pveceph Cannot Use LVM Devices for WAL

`pveceph osd create /dev/sdb --wal_dev /dev/pve/wal-db-osd-a` FAILS:

```
unable to get device info for '/dev/dm-2' for type wal_dev
```

pveceph expects a raw block device, not a device-mapper LV. Use
`ceph-volume` directly instead.

### ⚠️ BlueStore Cannot Share One LV For Both WAL and DB Separately

Specifying `--block.wal` AND `--block.db` pointing to the SAME LV fails:

```
bdev(...) open got: (16) Device or resource busy
bluestore _minimal_open_bluefs add block device(block.wal) returned: (16) Device or resource busy
OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
```

BlueStore opens the WAL device exclusively — it cannot be the same
underlying device as DB. The rollback also destroys the WAL LV.

### ✅ Correct Method: ceph-volume with --block.db Only

Specify ONLY `--block.db` — BlueStore collocates WAL on the DB device
automatically. This is the standard approach for a single shared SSD:

```bash
# 1. Ensure the WAL/DB LV exists
lvcreate -L 30G -n wal-db-osd-a pve

# 2. Prepare the OSD with DB on NVMe (WAL collocated)
ceph-volume lvm prepare --bluestore \
  --data /dev/sdb \
  --block.db /dev/pve/wal-db-osd-a

# 3. Activate the OSD (replace UUID from prepare output)
ceph-volume lvm activate <osd_id> <osd_uuid>

# 4. Verify DB/WAL placement
ceph-volume lvm list <osd_id>
# Should show:
#   [block]     /dev/ceph-<vg>/osd-block-<uuid>   (data on HDD)
#   [db]        /dev/pve/wal-db-osd-a              (DB on NVMe)
#   db device   /dev/pve/wal-db-osd-a
#   devices     /dev/nvme0n1p3
```

### Destroy and Recreate an OSD With WAL

If an OSD was already created WITHOUT WAL and needs to be rebuilt:

```bash
# 1. Stop the OSD service
systemctl stop ceph-osd@<id>

# 2. Mark down and destroy
ceph osd down <id>
ceph osd destroy <id> --yes-i-really-mean-it

# 3. Zap the disk (removes VG, LV, BlueStore signatures)
ceph-volume lvm zap /dev/sdX --destroy

# 4. Recreate with WAL/DB as above
```

### CRUSH Host Assignment for Non-PVE Nodes

When using `ceph-volume` (not `pveceph`), the OSD may land in the wrong
CRUSH host or under a generic bucket. Verify and fix:

```bash
# Check which host the OSD landed in
ceph osd tree | grep osd.<id>

# If in wrong host, move it:
ceph osd crush set osd.<id> <weight> root=default host=<correct_hostname>

# Set device class if not auto-detected
ceph osd crush set-device-class hdd osd.<id>

# Remove old destroyed OSD from CRUSH
ceph osd crush remove osd.<old_id>
ceph osd rm <old_id>
```

## Session Log — 2026-07-13

### Phase 1: Initial OSD Creation (osd.9, no WAL)

Added osd.9 (WD Red Plus 3 TB, WD-WX22DA0D7EP3) to n5pro via hot-plug.
Drive was removed from CT 10.0.30.100 and physically inserted into n5pro
while running. Bus rescan detected it on ata5. Created as BlueStore OSD
with `pveceph osd create /dev/sdb`. Cluster went from 10→11 OSDs.
`backfill_toofull` PGs dropped from 9→1 immediately. Remapped PGs spiked
to 317 (expected rebalance). Device class: hdd.

### Phase 2: CRUSH Host Correction

**Mistake**: Initially moved osd.6+osd.8 from host `ubuntu` to `n5pro`,
assuming they were on the same machine. **User corrected**: osd.6 and
osd.8 belong to host `ubuntu` (10.0.30.100), a separate physical machine.
n5pro (10.0.20.91) only hosts osd.9 (later osd.11).

Fixed by recreating the `ubuntu` bucket and moving osd.6+osd.8 back:
```bash
ceph osd crush add-bucket ubuntu host
ceph osd crush move ubuntu root=default
ceph osd crush set osd.6 2.75829 root=default host=ubuntu
ceph osd crush set osd.8 2.72899 root=default host=ubuntu
```

### Phase 3: Rebuild with SSD WAL/DB (osd.9 → osd.11)

Destroyed osd.9 (created without WAL) and rebuilt as osd.11 with DB on
NVMe SSD. Key steps:
1. `systemctl stop ceph-osd@9` → `ceph osd destroy 9` → `ceph-volume lvm zap /dev/sdb --destroy`
2. Attempted `ceph-volume lvm prepare --bluestore --data /dev/sdb --block.wal /dev/pve/wal-db-osd-a --block.db /dev/pve/wal-db-osd-a` → FAILED (device busy, same LV for WAL+DB)
3. Rollback deleted `wal-db-osd-a` LV — had to recreate with `lvcreate -L 30G -n wal-db-osd-a pve`
4. Succeeded with `--block.db /dev/pve/wal-db-osd-a` only (WAL collocated on DB)
5. Activated, verified DB on NVMe, set CRUSH host to `n5pro`, device class `hdd`
6. Removed old osd.9 from CRUSH: `ceph osd crush remove osd.9` + `ceph osd rm 9`

Final osd.11: data on /dev/sdb (3 TB HDD), DB+WAL on /dev/pve/wal-db-osd-a
(30 GB NVMe LV), CRUSH host `n5pro`, weight 2.76, device class hdd.

Related: `references/hdd-selection-for-osd-2026-07.md` (drive selection),
`references/ceph-pg-osd-management-2026-07.md` (rebalancing),
`references/ceph-recovery-acceleration-2026-07.md` (recovery tuning).