feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
@@ -0,0 +1,324 @@
|
||||
# Creating a Ceph OSD via Hot-Plug on Non-PVE Nodes (n5pro / Minisforum)
|
||||
|
||||
## When to Use
|
||||
|
||||
Adding a new HDD OSD to a Ceph cluster by physically hot-plugging a SATA
|
||||
drive into a running node that is not a standard PVE host (e.g. Minisforum
|
||||
N5 PRO with JMB58x SATA controller, CRUSH host `n5pro`).
|
||||
|
||||
## Prerequisites Checklist
|
||||
|
||||
- [ ] Drive is physically inserted (hot-plug or powered-off install)
|
||||
- [ ] Node has `ceph`, `ceph-volume`, `pveceph` installed
|
||||
- [ ] Node has `/etc/ceph/ceph.conf` and `ceph.client.admin.keyring`
|
||||
- [ ] `/etc/ceph/ceph.conf` is a **symlink to `/etc/pve/ceph.conf`**
|
||||
|
||||
## Step-by-Step
|
||||
|
||||
### 1. Detect the Hot-Plugged Drive
|
||||
|
||||
If the drive was inserted while the node was running, the SATA bus may
|
||||
not auto-scan. Trigger a manual rescan:
|
||||
|
||||
```bash
|
||||
for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
|
||||
echo "- - -" > ${h}/scan
|
||||
done
|
||||
sleep 2
|
||||
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop
|
||||
```
|
||||
|
||||
Verify the drive appears (e.g. `/dev/sdb`, MODEL=WDC WD30EFZX).
|
||||
|
||||
### 2. Copy Ceph Config to Non-PVE Node (if missing)
|
||||
|
||||
If the node lacks `/etc/ceph/ceph.conf` or the admin keyring:
|
||||
|
||||
```bash
|
||||
# From PVE master (10.0.20.10), copy to target node:
|
||||
scp /etc/ceph/ceph.conf root@<node>:/etc/ceph/
|
||||
scp /etc/ceph/ceph.client.admin.keyring root@<node>:/etc/ceph/
|
||||
ssh root@<node> 'chmod 644 /etc/ceph/ceph.conf && chmod 600 /etc/ceph/ceph.client.admin.keyring'
|
||||
```
|
||||
|
||||
### 3. Fix ceph.conf Symlink (CRITICAL — pveceph requires it)
|
||||
|
||||
`pveceph osd create` refuses to run if `/etc/ceph/ceph.conf` is a regular
|
||||
file rather than a symlink to `/etc/pve/ceph.conf`:
|
||||
|
||||
```
|
||||
file '/etc/ceph/ceph.conf' already exists and is not a symlink to /etc/pve/ceph.conf
|
||||
```
|
||||
|
||||
Fix:
|
||||
|
||||
```bash
|
||||
rm /etc/ceph/ceph.conf
|
||||
ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
|
||||
```
|
||||
|
||||
### 4. Zap the Drive Clean
|
||||
|
||||
Remove any leftover partitions/filesystem signatures from prior use:
|
||||
|
||||
```bash
|
||||
sgdisk --zap-all /dev/sdX
|
||||
wipefs -a /dev/sdX
|
||||
parted /dev/sdX mklabel gpt
|
||||
lsblk /dev/sdX -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT # verify clean
|
||||
```
|
||||
|
||||
### 5. Create the OSD
|
||||
|
||||
```bash
|
||||
pveceph osd create /dev/sdX
|
||||
```
|
||||
|
||||
This runs `ceph-volume lvm create` internally: creates a VG + LV on the
|
||||
disk, formats BlueStore, registers with the MON, enables systemd services
|
||||
(`ceph-volume@lvm-<id>-<uuid>.service`, `ceph-osd@<id>.service`), and
|
||||
starts the OSD.
|
||||
|
||||
### 6. Wait for OSD to Come Up (10-15 seconds)
|
||||
|
||||
The newly created OSD will initially show as **down** in `ceph osd tree`.
|
||||
The `ceph-osd@<id>` service is running but waits for the OSD map to load.
|
||||
Wait ~15 seconds, then verify:
|
||||
|
||||
```bash
|
||||
ceph osd tree | grep osd.<id>
|
||||
# Should show: up 1.00000
|
||||
```
|
||||
|
||||
If still down after 30s, check:
|
||||
|
||||
```bash
|
||||
journalctl -u ceph-osd@<id> --no-pager -n 30
|
||||
systemctl status ceph-osd@<id>
|
||||
```
|
||||
|
||||
Common non-fatal log lines:
|
||||
- `failed to load OSD map for epoch N, got 0 bytes` — transient, resolves
|
||||
- `set_numa_affinity unable to identify public interface` — cosmetic
|
||||
|
||||
### 7. Verify Cluster Integration
|
||||
|
||||
```bash
|
||||
ceph osd crush get-device-class osd.<id> # should report 'hdd'
|
||||
ceph -s # OSD count increased
|
||||
ceph df # raw capacity increased
|
||||
```
|
||||
|
||||
Rebalancing begins automatically — expect a spike in `remapped_pgs` (normal).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
### pveceph osd create fails: "already exists and is not a symlink"
|
||||
|
||||
**Symptom**: `pveceph osd create /dev/sdb` exits with code 22 and the
|
||||
message `file '/etc/ceph/ceph.conf' already exists and is not a symlink`.
|
||||
|
||||
**Cause**: On non-standard PVE nodes (e.g. n5pro), `/etc/ceph/ceph.conf`
|
||||
was manually placed as a regular file. `pveceph` insists on it being a
|
||||
symlink to `/etc/pve/ceph.conf`.
|
||||
|
||||
**Fix**: Remove the file and create the symlink (step 3 above).
|
||||
|
||||
### Hot-Plugged SATA Drive Not Detected
|
||||
|
||||
**Symptom**: Drive physically inserted but `lsblk` doesn't show it.
|
||||
|
||||
**Cause**: The JMB58x AHCI controller does not expose `hot_plug` sysfs
|
||||
attribute, and Linux may not auto-scan the port.
|
||||
|
||||
**Fix**: Manual SCSI host rescan (step 1 above). The JMB58x does support
|
||||
AHCI hot-plug electrically even though the sysfs attribute is absent.
|
||||
|
||||
### OSD Shows "down" Immediately After Creation
|
||||
|
||||
**Symptom**: `ceph osd tree` shows the new OSD as `down` with weight 0.
|
||||
|
||||
**Cause**: The `ceph-osd` daemon takes 10-15 seconds to load the OSD map
|
||||
and register with the MONs. This is normal startup latency, not a failure.
|
||||
|
||||
**Fix**: Wait. Check `systemctl status ceph-osd@<id>` — if it says
|
||||
`active (running)`, the OSD will come up shortly.
|
||||
|
||||
### Bootstrap Keyring Warning During ceph-volume
|
||||
|
||||
**Symptom**: `unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring`
|
||||
|
||||
**Cause**: The bootstrap-osd keyring lives elsewhere on some nodes. The
|
||||
warning is non-fatal — `ceph-volume` falls back to the admin keyring.
|
||||
|
||||
**Fix**: None needed. OSD creation succeeds despite this warning.
|
||||
|
||||
## Hardware Notes: Minisforum N5 PRO
|
||||
|
||||
| Spec | Value |
|
||||
|------|-------|
|
||||
| SATA Controller | JMicron JMB58x AHCI, 5 ports |
|
||||
| AHCI Version | 1.0301, 32 command slots, 6 Gbps |
|
||||
| AHCI Flags | ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh |
|
||||
| Hot-Plug sysfs | `hot_plug` attribute NOT exposed |
|
||||
| Actual Hot-Plug | Works — bus rescan detects the drive |
|
||||
| CRUSH Host Name | `n5pro` (own bucket, NOT `ubuntu`) |
|
||||
| Existing OSDs | NONE — n5pro starts with zero OSDs |
|
||||
| NVMe | AirDisk 128GB SSD (boot + LVM VG `pve` with WAL/DB LVs) |
|
||||
|
||||
> **CRITICAL — n5pro ≠ ubuntu**: Host `ubuntu` (IP 10.0.30.100) is a SEPARATE
|
||||
> machine with its own OSDs (osd.6, osd.8). n5pro (IP 10.0.20.91) must have
|
||||
> its own CRUSH host bucket named `n5pro`. Do NOT place n5pro OSDs under
|
||||
> `ubuntu` — this would falsely suggest geographic separation for replicas
|
||||
> that are actually on the same physical node.
|
||||
|
||||
## Adding SSD WAL/DB to an OSD on Non-PVE Nodes
|
||||
|
||||
When the OSD's HDD is on a node that also has an NVMe SSD, placing the
|
||||
BlueStore DB (and collocated WAL) on the SSD dramatically improves write
|
||||
performance. The n5pro has a 128 GB AirDisk NVMe (`/dev/nvme0n1`) with
|
||||
an LVM VG `pve` that can host WAL/DB LVs.
|
||||
|
||||
### Creating WAL/DB LVs on the NVMe
|
||||
|
||||
```bash
|
||||
# Check free space in the pve VG
|
||||
vgs pve # look at VFree
|
||||
|
||||
# Create 30 GB LV for WAL/DB (one per OSD)
|
||||
lvcreate -L 30G -n wal-db-osd-a pve
|
||||
# Optional second LV for a future OSD:
|
||||
lvcreate -L 30G -n wal-db-osd-b pve
|
||||
```
|
||||
|
||||
### ⚠️ pveceph Cannot Use LVM Devices for WAL
|
||||
|
||||
`pveceph osd create /dev/sdb --wal_dev /dev/pve/wal-db-osd-a` FAILS:
|
||||
|
||||
```
|
||||
unable to get device info for '/dev/dm-2' for type wal_dev
|
||||
```
|
||||
|
||||
pveceph expects a raw block device, not a device-mapper LV. Use
|
||||
`ceph-volume` directly instead.
|
||||
|
||||
### ⚠️ BlueStore Cannot Share One LV For Both WAL and DB Separately
|
||||
|
||||
Specifying `--block.wal` AND `--block.db` pointing to the SAME LV fails:
|
||||
|
||||
```
|
||||
bdev(...) open got: (16) Device or resource busy
|
||||
bluestore _minimal_open_bluefs add block device(block.wal) returned: (16) Device or resource busy
|
||||
OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
|
||||
```
|
||||
|
||||
BlueStore opens the WAL device exclusively — it cannot be the same
|
||||
underlying device as DB. The rollback also destroys the WAL LV.
|
||||
|
||||
### ✅ Correct Method: ceph-volume with --block.db Only
|
||||
|
||||
Specify ONLY `--block.db` — BlueStore collocates WAL on the DB device
|
||||
automatically. This is the standard approach for a single shared SSD:
|
||||
|
||||
```bash
|
||||
# 1. Ensure the WAL/DB LV exists
|
||||
lvcreate -L 30G -n wal-db-osd-a pve
|
||||
|
||||
# 2. Prepare the OSD with DB on NVMe (WAL collocated)
|
||||
ceph-volume lvm prepare --bluestore \
|
||||
--data /dev/sdb \
|
||||
--block.db /dev/pve/wal-db-osd-a
|
||||
|
||||
# 3. Activate the OSD (replace UUID from prepare output)
|
||||
ceph-volume lvm activate <osd_id> <osd_uuid>
|
||||
|
||||
# 4. Verify DB/WAL placement
|
||||
ceph-volume lvm list <osd_id>
|
||||
# Should show:
|
||||
# [block] /dev/ceph-<vg>/osd-block-<uuid> (data on HDD)
|
||||
# [db] /dev/pve/wal-db-osd-a (DB on NVMe)
|
||||
# db device /dev/pve/wal-db-osd-a
|
||||
# devices /dev/nvme0n1p3
|
||||
```
|
||||
|
||||
### Destroy and Recreate an OSD With WAL
|
||||
|
||||
If an OSD was already created WITHOUT WAL and needs to be rebuilt:
|
||||
|
||||
```bash
|
||||
# 1. Stop the OSD service
|
||||
systemctl stop ceph-osd@<id>
|
||||
|
||||
# 2. Mark down and destroy
|
||||
ceph osd down <id>
|
||||
ceph osd destroy <id> --yes-i-really-mean-it
|
||||
|
||||
# 3. Zap the disk (removes VG, LV, BlueStore signatures)
|
||||
ceph-volume lvm zap /dev/sdX --destroy
|
||||
|
||||
# 4. Recreate with WAL/DB as above
|
||||
```
|
||||
|
||||
### CRUSH Host Assignment for Non-PVE Nodes
|
||||
|
||||
When using `ceph-volume` (not `pveceph`), the OSD may land in the wrong
|
||||
CRUSH host or under a generic bucket. Verify and fix:
|
||||
|
||||
```bash
|
||||
# Check which host the OSD landed in
|
||||
ceph osd tree | grep osd.<id>
|
||||
|
||||
# If in wrong host, move it:
|
||||
ceph osd crush set osd.<id> <weight> root=default host=<correct_hostname>
|
||||
|
||||
# Set device class if not auto-detected
|
||||
ceph osd crush set-device-class hdd osd.<id>
|
||||
|
||||
# Remove old destroyed OSD from CRUSH
|
||||
ceph osd crush remove osd.<old_id>
|
||||
ceph osd rm <old_id>
|
||||
```
|
||||
|
||||
## Session Log — 2026-07-13
|
||||
|
||||
### Phase 1: Initial OSD Creation (osd.9, no WAL)
|
||||
|
||||
Added osd.9 (WD Red Plus 3 TB, WD-WX22DA0D7EP3) to n5pro via hot-plug.
|
||||
Drive was removed from CT 10.0.30.100 and physically inserted into n5pro
|
||||
while running. Bus rescan detected it on ata5. Created as BlueStore OSD
|
||||
with `pveceph osd create /dev/sdb`. Cluster went from 10→11 OSDs.
|
||||
`backfill_toofull` PGs dropped from 9→1 immediately. Remapped PGs spiked
|
||||
to 317 (expected rebalance). Device class: hdd.
|
||||
|
||||
### Phase 2: CRUSH Host Correction
|
||||
|
||||
**Mistake**: Initially moved osd.6+osd.8 from host `ubuntu` to `n5pro`,
|
||||
assuming they were on the same machine. **User corrected**: osd.6 and
|
||||
osd.8 belong to host `ubuntu` (10.0.30.100), a separate physical machine.
|
||||
n5pro (10.0.20.91) only hosts osd.9 (later osd.11).
|
||||
|
||||
Fixed by recreating the `ubuntu` bucket and moving osd.6+osd.8 back:
|
||||
```bash
|
||||
ceph osd crush add-bucket ubuntu host
|
||||
ceph osd crush move ubuntu root=default
|
||||
ceph osd crush set osd.6 2.75829 root=default host=ubuntu
|
||||
ceph osd crush set osd.8 2.72899 root=default host=ubuntu
|
||||
```
|
||||
|
||||
### Phase 3: Rebuild with SSD WAL/DB (osd.9 → osd.11)
|
||||
|
||||
Destroyed osd.9 (created without WAL) and rebuilt as osd.11 with DB on
|
||||
NVMe SSD. Key steps:
|
||||
1. `systemctl stop ceph-osd@9` → `ceph osd destroy 9` → `ceph-volume lvm zap /dev/sdb --destroy`
|
||||
2. Attempted `ceph-volume lvm prepare --bluestore --data /dev/sdb --block.wal /dev/pve/wal-db-osd-a --block.db /dev/pve/wal-db-osd-a` → FAILED (device busy, same LV for WAL+DB)
|
||||
3. Rollback deleted `wal-db-osd-a` LV — had to recreate with `lvcreate -L 30G -n wal-db-osd-a pve`
|
||||
4. Succeeded with `--block.db /dev/pve/wal-db-osd-a` only (WAL collocated on DB)
|
||||
5. Activated, verified DB on NVMe, set CRUSH host to `n5pro`, device class `hdd`
|
||||
6. Removed old osd.9 from CRUSH: `ceph osd crush remove osd.9` + `ceph osd rm 9`
|
||||
|
||||
Final osd.11: data on /dev/sdb (3 TB HDD), DB+WAL on /dev/pve/wal-db-osd-a
|
||||
(30 GB NVMe LV), CRUSH host `n5pro`, weight 2.76, device class hdd.
|
||||
|
||||
Related: `references/hdd-selection-for-osd-2026-07.md` (drive selection),
|
||||
`references/ceph-pg-osd-management-2026-07.md` (rebalancing),
|
||||
`references/ceph-recovery-acceleration-2026-07.md` (recovery tuning).
|
||||
Reference in New Issue
Block a user