retro: compound-learning 30-day retrospective + 2 new patterns
New patterns: - PAT-008: Ansible default_ipv6 fact missing on fresh VMs - PAT-009: Stale NBD devices after RBD volume swap Updated: - index.md: added PAT-008, PAT-009 - log.md: retrospective entry with 4 solution docs + 2 patterns - Solution docs dispatched to ~/docs/solutions/
This commit is contained in:
Dominik Schön
@@ -52,6 +52,8 @@
|
||||
- [[patterns/finanzblick-sync-waf]] — POST /sync WAF-blocked, UI Modal Sequence (PAT-005)
|
||||
- [[patterns/linkedin-react-input]] — nativeInputSetter, Session Expiry (PAT-006)
|
||||
- [[patterns/ceph-ssd-wear-ec-pool]] — EC Pool unusable + SSD Wear-Level (PAT-007)
|
||||
- [[patterns/ansible-default-ipv6]] — lablabs.rke2 role fails on IPv6-less VMs (PAT-008)
|
||||
- [[patterns/k8s-stale-nbd-devices]] — RBD swap leaves stale NBD mappings (PAT-009)
|
||||
- [[patterns/skill-impact]] — Skill Modification Audit Trail
|
||||
|
||||
## Memory Layer Architektur
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
# Memory Log
|
||||
|
||||
## [2026-08-30] retro | Compound Learning Retrospective (Last 30 Days)
|
||||
- Reviewed sessions from Jul 31 – Aug 30, 2026
|
||||
- **4 new solution docs** written by subagents:
|
||||
- `bug-fixes/2026-08-05-ceph-squid-ec-pool-mark-complete-bug.md` — EC pool mark-complete fails in Squid
|
||||
- `bug-fixes/2026-08-06-traefik-cross-namespace-routing-pitfall.md` — Two Traefik 404 incidents
|
||||
- `architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md` — Full RKE2 rebuild sequence
|
||||
- `architecture/2026-08-01-immich-data-loss-no-backup.md` — Ceph redundancy ≠ backup
|
||||
- **2 new patterns** extracted:
|
||||
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
|
||||
- PAT-009: Stale NBD devices after RBD volume swap
|
||||
- **4 Hindsight entries** indexed with solution summaries
|
||||
- Key themes: K8s cluster disaster recovery, Ceph EC pool unrecoverability, Traefik routing complexity, backup gap identification
|
||||
|
||||
## [2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker
|
||||
- Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution)
|
||||
- **New `patterns/` directory** in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
pattern_id: PAT-008
|
||||
title: "Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails"
|
||||
category: tooling
|
||||
severity: medium
|
||||
status: active
|
||||
first_observed: 2026-07-21
|
||||
last_updated: 2026-08-30
|
||||
related_systems: [rke2-kubernetes]
|
||||
related_solution_docs:
|
||||
- docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails
|
||||
|
||||
## Symptom
|
||||
|
||||
The `lablabs.rke2` Ansible role fails on freshly provisioned VMs with a Jinja2 error:
|
||||
```
|
||||
ansible_facts['default_ipv6']['address']
|
||||
```
|
||||
The `default_ipv6` fact doesn't exist on VMs without IPv6 configured, causing the
|
||||
`meta/argument_specs.yml` validation to crash.
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `lablabs.rke2` role's `meta/argument_specs.yml` references
|
||||
`ansible_facts['default_ipv6']['address']` with a hardcoded default. On fresh VMs
|
||||
without IPv6, `ansible_facts['default_ipv6']` is undefined → Jinja2 raises
|
||||
`UndefinedError`.
|
||||
|
||||
A `pre_task` setting `default_ipv6` via `combine()` doesn't help because the
|
||||
argument_specs validation runs BEFORE pre_tasks — it re-collects facts and
|
||||
overwrites the pre_task fix.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Patch `meta/argument_specs.yml` directly in the role:
|
||||
```yaml
|
||||
# Replace:
|
||||
default: "{{ ansible_facts['default_ipv6']['address'] }}"
|
||||
# With:
|
||||
default: "{{ ansible_facts.default_ipv6.address | default(None) }}"
|
||||
```
|
||||
|
||||
Or: enable IPv6 on the target VMs (faster workaround, no role patching needed).
|
||||
|
||||
## Prevention
|
||||
|
||||
- Pin Ansible roles and patch argument_specs when they assume facts that may not exist
|
||||
- Test roles on fresh VMs without IPv6 before production use
|
||||
- Consider forking the role with the fix upstream
|
||||
|
||||
## Evidence
|
||||
|
||||
- Observed during GPU worker provisioning (worker-04/05, Jul 2026)
|
||||
- Solution doc: `docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md`
|
||||
- Session: @session:default/20260721_115501_78b25032
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
pattern_id: PAT-009
|
||||
title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
|
||||
category: infrastructure
|
||||
severity: high
|
||||
status: active
|
||||
first_observed: 2026-08-01
|
||||
last_updated: 2026-08-30
|
||||
related_systems: [rke2-kubernetes, ceph-cluster]
|
||||
related_solution_docs:
|
||||
- docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
|
||||
|
||||
## Symptom
|
||||
|
||||
After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
|
||||
images), pods fail to mount with errors like:
|
||||
```
|
||||
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
|
||||
desc = rbd: map failed with error: /dev/nbd0 already in use
|
||||
```
|
||||
|
||||
The NBD device is held by a stale mapping from the old RBD image, even though the
|
||||
new image has the same name.
|
||||
|
||||
## Root Cause
|
||||
|
||||
When an RBD image is recreated (delete + create with same name), the Ceph CSI
|
||||
driver's NBD mappings from the old image remain active. The Linux NBD layer
|
||||
holds `/dev/nbdX` open, blocking new mounts to the same device path.
|
||||
|
||||
The Kubernetes VolumeAttachment object also references the old volume handle,
|
||||
preventing the CSI driver from cleanly attaching the new volume.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Three-step teardown procedure:
|
||||
```bash
|
||||
# 1. Delete the VolumeAttachment (allows CSI driver to release)
|
||||
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
|
||||
|
||||
# 2. Disconnect the stale NBD device on the target node
|
||||
ssh <node> 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0
|
||||
|
||||
# 3. Restart the CSI node plugin to pick up clean state
|
||||
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
|
||||
# (DaemonSet will respawn it)
|
||||
```
|
||||
|
||||
After this, the pod can remount with the new RBD image.
|
||||
|
||||
## Prevention
|
||||
|
||||
- Before deleting RBD images, ensure all pods using them are scaled to 0
|
||||
- Delete VolumeAttachments BEFORE deleting RBD images
|
||||
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
|
||||
- Document this in the K8s disaster recovery runbook
|
||||
|
||||
## Evidence
|
||||
|
||||
- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
|
||||
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
|
||||
- Part of the full cluster rebuild procedure
|
||||
Reference in New Issue
Block a user