Dominik Schön
5cf6a573f1
New patterns: - PAT-008: Ansible default_ipv6 fact missing on fresh VMs - PAT-009: Stale NBD devices after RBD volume swap Updated: - index.md: added PAT-008, PAT-009 - log.md: retrospective entry with 4 solution docs + 2 patterns - Solution docs dispatched to ~/docs/solutions/
2.2 KiB
2.2 KiB
pattern_id, title, category, severity, status, first_observed, last_updated, related_systems, related_solution_docs, related_skills
| pattern_id | title | category | severity | status | first_observed | last_updated | related_systems | related_solution_docs | related_skills | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PAT-009 | Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown | infrastructure | high | active | 2026-08-01 | 2026-08-30 |
|
|
Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
Symptom
After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating images), pods fail to mount with errors like:
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
desc = rbd: map failed with error: /dev/nbd0 already in use
The NBD device is held by a stale mapping from the old RBD image, even though the new image has the same name.
Root Cause
When an RBD image is recreated (delete + create with same name), the Ceph CSI
driver's NBD mappings from the old image remain active. The Linux NBD layer
holds /dev/nbdX open, blocking new mounts to the same device path.
The Kubernetes VolumeAttachment object also references the old volume handle, preventing the CSI driver from cleanly attaching the new volume.
Mitigation
Three-step teardown procedure:
# 1. Delete the VolumeAttachment (allows CSI driver to release)
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
# 2. Disconnect the stale NBD device on the target node
ssh <node> 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0
# 3. Restart the CSI node plugin to pick up clean state
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
# (DaemonSet will respawn it)
After this, the pod can remount with the new RBD image.
Prevention
- Before deleting RBD images, ensure all pods using them are scaled to 0
- Delete VolumeAttachments BEFORE deleting RBD images
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
- Document this in the K8s disaster recovery runbook
Evidence
- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
- Part of the full cluster rebuild procedure