retro: compound-learning 30-day retrospective + 2 new patterns

New patterns:
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
- PAT-009: Stale NBD devices after RBD volume swap

Updated:
- index.md: added PAT-008, PAT-009
- log.md: retrospective entry with 4 solution docs + 2 patterns
- Solution docs dispatched to ~/docs/solutions/

patterns/k8s-stale-nbd-devices.md

@@ -0,0 +1,66 @@
+---
+pattern_id: PAT-009
+title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
+category: infrastructure
+severity: high
+status: active
+first_observed: 2026-08-01
+last_updated: 2026-08-30
+related_systems: [rke2-kubernetes, ceph-cluster]
+related_solution_docs:
+  - docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
+related_skills: []
+---
+
+# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
+
+## Symptom
+
+After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
+images), pods fail to mount with errors like:
+```
+MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
+desc = rbd: map failed with error: /dev/nbd0 already in use
+```
+
+The NBD device is held by a stale mapping from the old RBD image, even though the
+new image has the same name.
+
+## Root Cause
+
+When an RBD image is recreated (delete + create with same name), the Ceph CSI
+driver's NBD mappings from the old image remain active. The Linux NBD layer
+holds `/dev/nbdX` open, blocking new mounts to the same device path.
+
+The Kubernetes VolumeAttachment object also references the old volume handle,
+preventing the CSI driver from cleanly attaching the new volume.
+
+## Mitigation
+
+Three-step teardown procedure:
+```bash
+# 1. Delete the VolumeAttachment (allows CSI driver to release)
+kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
+
+# 2. Disconnect the stale NBD device on the target node
+ssh <node> 'nbd-client -d /dev/nbd0'  # or: qemu-nbd --disconnect /dev/nbd0
+
+# 3. Restart the CSI node plugin to pick up clean state
+kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
+# (DaemonSet will respawn it)
+```
+
+After this, the pod can remount with the new RBD image.
+
+## Prevention
+
+- Before deleting RBD images, ensure all pods using them are scaled to 0
+- Delete VolumeAttachments BEFORE deleting RBD images
+- After RBD image recreation, restart CSI plugins on all nodes that had mounts
+- Document this in the K8s disaster recovery runbook
+
+## Evidence
+
+- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
+- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
+- Part of the full cluster rebuild procedure