hermes-skills/devops/ceph-cluster-administration/references/bluefs-db-expansion-2026-07.md

BlueFS DB Expansion — Session Log (2026-07-13)

Scenario

osd.7 (982 GiB HDD, proxmox7) triggered BLUEFS_SPILLOVER warning — RocksDB grew beyond its 50 GiB DB LV on NVMe and spilled onto the slow HDD device. DB was 50 GiB, only 2.2 GiB actually used, but BlueFS spillover is about the RocksDB logical size exceeding the DB device, not physical usage.

Diagnosis

# Check spillover target
ceph health detail | grep -A2 SPILLOVER
# Shows which OSD spilled and how much

# Check DB device size and usage
ceph daemon osd.7 bluefs stats
# Line 1: DB device size + usage
# Line 2: Slow (HDD) device size + usage
# If DB device is full → spillover to slow device

# Check DB LV and VG free space
ssh root@<osd-host> 'lvs --units g | grep ceph-db && vgs --units g | grep pve'

Assessment Heuristic

Spillover < 1 MiB with >50% DB free → cosmetic, do NOT expand. The spillover is often a tiny RocksDB metadata overflow that will self-resolve on compaction. Stopping an OSD for bluefs-bdev-expand causes unnecessary downtime (PGs degraded, recovery triggered).

Spillover > 1 GiB OR DB device >80% full → expand. This indicates sustained RocksDB growth that won't self-resolve.

Expansion Procedure (Verified Working)

osd.7 on proxmox7 (10.0.20.70), DB LV = pve/ceph-db:

# 1. Extend LV (online, no OSD stop needed for LVM)
lvextend -L +20G pve/ceph-db
# Output: "Size of logical volume pve/ceph-db changed from 50.00 GiB to 70.00 GiB"

# 2. Stop OSD — bluefs-bdev-expand CANNOT run while OSD is active
systemctl stop ceph-osd@7
sleep 2

# 3. Expand BlueFS awareness
ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-7
# Output:
# "1 : device size 0x1180000000(70 GiB) : using 0x89000000(2.1 GiB)"
# "Expanding DB/WAL..."
# "1 : Expanding to 0x1180000000(70 GiB)"
# "1 : size updated to 0x1180000000(70 GiB)"

# 4. Restart OSD
systemctl start ceph-osd@7

# 5. Wait for OSD to come up in cluster (10-15 seconds)
ceph osd tree | grep "osd.7"
# Should show "up"

# 6. Verify BlueFS sees new size
ceph daemon osd.7 bluefs stats | head -3
# Should show: "1 : device size 0x1180000000(70 GiB)"

What Happened After

• osd.7 spillover cleared — warning moved to osd.8 (128 KiB, cosmetic)
• osd.7 utilization dropped from 86.1% → 78.9% (backfill continued, OSD healthier)
• osd.8 spillover assessed: 128 KiB spill, 28 GiB DB free → NO action needed
• BlueFS spillover warning persists in ceph health until RocksDB compacts back onto DB device — background process, minutes to hours

Pitfall: Online bluefs-bdev-expand Crashes

Running ceph-bluestore-tool bluefs-bdev-expand while OSD is active:

bdev(...) open open got: (16) Device or resource busy
bluestore(...) _lock_fsid failed to lock fsid (is another ceph-osd still running?)
FAILED ceph_assert(r == 0)

The tool tries to open the block device exclusively, which the running OSD already holds. Must stop OSD first.

Pitfall: ceph daemon admin socket Has No Expand Command

ceph daemon osd.7 help shows bluefs commands but NO expand or bdev-expand option. Available bluefs commands: bluefs stats, bluefs files list, bluefs debug_inject_read_zeros. The only path to expand is the offline ceph-bluestore-tool.

osd.8 Case Study — When NOT to Expand

osd.8 on host ubuntu (10.0.30.100):

• DB LV: ceph-db-vg/osd-8-db = 30 GiB on Crucial MX300 SSD (sda3)
• DB usage: 2.0 GiB of 30 GiB (93% free)
• Spillover: 128 KiB (trivial)
• VG free: 35.67 GiB across two PVs (sda3 1.89 GiB free, sda4 33.78 GiB free)

Decision: NO expansion. 128 KiB spillover with 28 GiB DB free is cosmetic. Stopping the OSD would trigger PG degradation and recovery I/O for zero practical benefit. The spillover will likely self-resolve on next RocksDB compaction.

Cross-Host SSH Pattern

OSDs span multiple PVE hosts. The Ceph commands (ceph -s, ceph osd df) run from any monitor node (10.0.20.10). But ceph-bluestore-tool and systemctl must run on the OSD's host. Use ceph osd metadata N | grep hostname to find the host, then SSH there:

HOST=$(ssh root@10.0.20.10 "ceph osd metadata 7 | python3 -c \"import sys,json; print(json.load(sys.stdin)['hostname'])\"")
ssh root@$HOST 'systemctl stop ceph-osd@7 && ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-7 && systemctl start ceph-osd@7'