Dominik Schön
5cf6a573f1
New patterns: - PAT-008: Ansible default_ipv6 fact missing on fresh VMs - PAT-009: Stale NBD devices after RBD volume swap Updated: - index.md: added PAT-008, PAT-009 - log.md: retrospective entry with 4 solution docs + 2 patterns - Solution docs dispatched to ~/docs/solutions/
67 lines
2.2 KiB
Markdown
67 lines
2.2 KiB
Markdown
---
|
|
pattern_id: PAT-009
|
|
title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
|
|
category: infrastructure
|
|
severity: high
|
|
status: active
|
|
first_observed: 2026-08-01
|
|
last_updated: 2026-08-30
|
|
related_systems: [rke2-kubernetes, ceph-cluster]
|
|
related_solution_docs:
|
|
- docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
|
|
related_skills: []
|
|
---
|
|
|
|
# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
|
|
|
|
## Symptom
|
|
|
|
After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
|
|
images), pods fail to mount with errors like:
|
|
```
|
|
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
|
|
desc = rbd: map failed with error: /dev/nbd0 already in use
|
|
```
|
|
|
|
The NBD device is held by a stale mapping from the old RBD image, even though the
|
|
new image has the same name.
|
|
|
|
## Root Cause
|
|
|
|
When an RBD image is recreated (delete + create with same name), the Ceph CSI
|
|
driver's NBD mappings from the old image remain active. The Linux NBD layer
|
|
holds `/dev/nbdX` open, blocking new mounts to the same device path.
|
|
|
|
The Kubernetes VolumeAttachment object also references the old volume handle,
|
|
preventing the CSI driver from cleanly attaching the new volume.
|
|
|
|
## Mitigation
|
|
|
|
Three-step teardown procedure:
|
|
```bash
|
|
# 1. Delete the VolumeAttachment (allows CSI driver to release)
|
|
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
|
|
|
|
# 2. Disconnect the stale NBD device on the target node
|
|
ssh <node> 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0
|
|
|
|
# 3. Restart the CSI node plugin to pick up clean state
|
|
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
|
|
# (DaemonSet will respawn it)
|
|
```
|
|
|
|
After this, the pod can remount with the new RBD image.
|
|
|
|
## Prevention
|
|
|
|
- Before deleting RBD images, ensure all pods using them are scaled to 0
|
|
- Delete VolumeAttachments BEFORE deleting RBD images
|
|
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
|
|
- Document this in the K8s disaster recovery runbook
|
|
|
|
## Evidence
|
|
|
|
- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
|
|
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
|
|
- Part of the full cluster rebuild procedure
|