---
pattern_id: PAT-009
title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
category: infrastructure
severity: high
status: active
first_observed: 2026-08-01
last_updated: 2026-08-30
related_systems: [rke2-kubernetes, ceph-cluster]
related_solution_docs:
  - docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
related_skills: []
---

# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown

## Symptom

After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
images), pods fail to mount with errors like:
```
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
desc = rbd: map failed with error: /dev/nbd0 already in use
```

The NBD device is held by a stale mapping from the old RBD image, even though the
new image has the same name.

## Root Cause

When an RBD image is recreated (delete + create with same name), the Ceph CSI
driver's NBD mappings from the old image remain active. The Linux NBD layer
holds `/dev/nbdX` open, blocking new mounts to the same device path.

The Kubernetes VolumeAttachment object also references the old volume handle,
preventing the CSI driver from cleanly attaching the new volume.

## Mitigation

Three-step teardown procedure:
```bash
# 1. Delete the VolumeAttachment (allows CSI driver to release)
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>

# 2. Disconnect the stale NBD device on the target node
ssh <node> 'nbd-client -d /dev/nbd0'  # or: qemu-nbd --disconnect /dev/nbd0

# 3. Restart the CSI node plugin to pick up clean state
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
# (DaemonSet will respawn it)
```

After this, the pod can remount with the new RBD image.

## Prevention

- Before deleting RBD images, ensure all pods using them are scaled to 0
- Delete VolumeAttachments BEFORE deleting RBD images
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
- Document this in the K8s disaster recovery runbook

## Evidence

- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
- Part of the full cluster rebuild procedure