retro: compound-learning 30-day retrospective + 2 new patterns
New patterns: - PAT-008: Ansible default_ipv6 fact missing on fresh VMs - PAT-009: Stale NBD devices after RBD volume swap Updated: - index.md: added PAT-008, PAT-009 - log.md: retrospective entry with 4 solution docs + 2 patterns - Solution docs dispatched to ~/docs/solutions/
This commit is contained in:
Dominik Schön
@@ -0,0 +1,59 @@
|
||||
---
|
||||
pattern_id: PAT-008
|
||||
title: "Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails"
|
||||
category: tooling
|
||||
severity: medium
|
||||
status: active
|
||||
first_observed: 2026-07-21
|
||||
last_updated: 2026-08-30
|
||||
related_systems: [rke2-kubernetes]
|
||||
related_solution_docs:
|
||||
- docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails
|
||||
|
||||
## Symptom
|
||||
|
||||
The `lablabs.rke2` Ansible role fails on freshly provisioned VMs with a Jinja2 error:
|
||||
```
|
||||
ansible_facts['default_ipv6']['address']
|
||||
```
|
||||
The `default_ipv6` fact doesn't exist on VMs without IPv6 configured, causing the
|
||||
`meta/argument_specs.yml` validation to crash.
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `lablabs.rke2` role's `meta/argument_specs.yml` references
|
||||
`ansible_facts['default_ipv6']['address']` with a hardcoded default. On fresh VMs
|
||||
without IPv6, `ansible_facts['default_ipv6']` is undefined → Jinja2 raises
|
||||
`UndefinedError`.
|
||||
|
||||
A `pre_task` setting `default_ipv6` via `combine()` doesn't help because the
|
||||
argument_specs validation runs BEFORE pre_tasks — it re-collects facts and
|
||||
overwrites the pre_task fix.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Patch `meta/argument_specs.yml` directly in the role:
|
||||
```yaml
|
||||
# Replace:
|
||||
default: "{{ ansible_facts['default_ipv6']['address'] }}"
|
||||
# With:
|
||||
default: "{{ ansible_facts.default_ipv6.address | default(None) }}"
|
||||
```
|
||||
|
||||
Or: enable IPv6 on the target VMs (faster workaround, no role patching needed).
|
||||
|
||||
## Prevention
|
||||
|
||||
- Pin Ansible roles and patch argument_specs when they assume facts that may not exist
|
||||
- Test roles on fresh VMs without IPv6 before production use
|
||||
- Consider forking the role with the fix upstream
|
||||
|
||||
## Evidence
|
||||
|
||||
- Observed during GPU worker provisioning (worker-04/05, Jul 2026)
|
||||
- Solution doc: `docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md`
|
||||
- Session: @session:default/20260721_115501_78b25032
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
pattern_id: PAT-009
|
||||
title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
|
||||
category: infrastructure
|
||||
severity: high
|
||||
status: active
|
||||
first_observed: 2026-08-01
|
||||
last_updated: 2026-08-30
|
||||
related_systems: [rke2-kubernetes, ceph-cluster]
|
||||
related_solution_docs:
|
||||
- docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
|
||||
|
||||
## Symptom
|
||||
|
||||
After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
|
||||
images), pods fail to mount with errors like:
|
||||
```
|
||||
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
|
||||
desc = rbd: map failed with error: /dev/nbd0 already in use
|
||||
```
|
||||
|
||||
The NBD device is held by a stale mapping from the old RBD image, even though the
|
||||
new image has the same name.
|
||||
|
||||
## Root Cause
|
||||
|
||||
When an RBD image is recreated (delete + create with same name), the Ceph CSI
|
||||
driver's NBD mappings from the old image remain active. The Linux NBD layer
|
||||
holds `/dev/nbdX` open, blocking new mounts to the same device path.
|
||||
|
||||
The Kubernetes VolumeAttachment object also references the old volume handle,
|
||||
preventing the CSI driver from cleanly attaching the new volume.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Three-step teardown procedure:
|
||||
```bash
|
||||
# 1. Delete the VolumeAttachment (allows CSI driver to release)
|
||||
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
|
||||
|
||||
# 2. Disconnect the stale NBD device on the target node
|
||||
ssh <node> 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0
|
||||
|
||||
# 3. Restart the CSI node plugin to pick up clean state
|
||||
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
|
||||
# (DaemonSet will respawn it)
|
||||
```
|
||||
|
||||
After this, the pod can remount with the new RBD image.
|
||||
|
||||
## Prevention
|
||||
|
||||
- Before deleting RBD images, ensure all pods using them are scaled to 0
|
||||
- Delete VolumeAttachments BEFORE deleting RBD images
|
||||
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
|
||||
- Document this in the K8s disaster recovery runbook
|
||||
|
||||
## Evidence
|
||||
|
||||
- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
|
||||
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
|
||||
- Part of the full cluster rebuild procedure
|
||||
Reference in New Issue
Block a user