retro: compound-learning 30-day retrospective + 2 new patterns

New patterns:
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
- PAT-009: Stale NBD devices after RBD volume swap

Updated:
- index.md: added PAT-008, PAT-009
- log.md: retrospective entry with 4 solution docs + 2 patterns
- Solution docs dispatched to ~/docs/solutions/
This commit is contained in:
Dominik Schön
2026-08-30 11:56:47 +00:00
parent bbbe4f8985
commit 5cf6a573f1
4 changed files with 140 additions and 0 deletions
+2
View File
@@ -52,6 +52,8 @@
- [[patterns/finanzblick-sync-waf]] — POST /sync WAF-blocked, UI Modal Sequence (PAT-005) - [[patterns/finanzblick-sync-waf]] — POST /sync WAF-blocked, UI Modal Sequence (PAT-005)
- [[patterns/linkedin-react-input]] — nativeInputSetter, Session Expiry (PAT-006) - [[patterns/linkedin-react-input]] — nativeInputSetter, Session Expiry (PAT-006)
- [[patterns/ceph-ssd-wear-ec-pool]] — EC Pool unusable + SSD Wear-Level (PAT-007) - [[patterns/ceph-ssd-wear-ec-pool]] — EC Pool unusable + SSD Wear-Level (PAT-007)
- [[patterns/ansible-default-ipv6]] — lablabs.rke2 role fails on IPv6-less VMs (PAT-008)
- [[patterns/k8s-stale-nbd-devices]] — RBD swap leaves stale NBD mappings (PAT-009)
- [[patterns/skill-impact]] — Skill Modification Audit Trail - [[patterns/skill-impact]] — Skill Modification Audit Trail
## Memory Layer Architektur ## Memory Layer Architektur
+13
View File
@@ -1,5 +1,18 @@
# Memory Log # Memory Log
## [2026-08-30] retro | Compound Learning Retrospective (Last 30 Days)
- Reviewed sessions from Jul 31 Aug 30, 2026
- **4 new solution docs** written by subagents:
- `bug-fixes/2026-08-05-ceph-squid-ec-pool-mark-complete-bug.md` — EC pool mark-complete fails in Squid
- `bug-fixes/2026-08-06-traefik-cross-namespace-routing-pitfall.md` — Two Traefik 404 incidents
- `architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md` — Full RKE2 rebuild sequence
- `architecture/2026-08-01-immich-data-loss-no-backup.md` — Ceph redundancy ≠ backup
- **2 new patterns** extracted:
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
- PAT-009: Stale NBD devices after RBD volume swap
- **4 Hindsight entries** indexed with solution summaries
- Key themes: K8s cluster disaster recovery, Ceph EC pool unrecoverability, Traefik routing complexity, backup gap identification
## [2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker ## [2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker
- Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution) - Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution)
- **New `patterns/` directory** in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer - **New `patterns/` directory** in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer
+59
View File
@@ -0,0 +1,59 @@
---
pattern_id: PAT-008
title: "Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails"
category: tooling
severity: medium
status: active
first_observed: 2026-07-21
last_updated: 2026-08-30
related_systems: [rke2-kubernetes]
related_solution_docs:
- docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md
related_skills: []
---
# Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails
## Symptom
The `lablabs.rke2` Ansible role fails on freshly provisioned VMs with a Jinja2 error:
```
ansible_facts['default_ipv6']['address']
```
The `default_ipv6` fact doesn't exist on VMs without IPv6 configured, causing the
`meta/argument_specs.yml` validation to crash.
## Root Cause
The `lablabs.rke2` role's `meta/argument_specs.yml` references
`ansible_facts['default_ipv6']['address']` with a hardcoded default. On fresh VMs
without IPv6, `ansible_facts['default_ipv6']` is undefined → Jinja2 raises
`UndefinedError`.
A `pre_task` setting `default_ipv6` via `combine()` doesn't help because the
argument_specs validation runs BEFORE pre_tasks — it re-collects facts and
overwrites the pre_task fix.
## Mitigation
Patch `meta/argument_specs.yml` directly in the role:
```yaml
# Replace:
default: "{{ ansible_facts['default_ipv6']['address'] }}"
# With:
default: "{{ ansible_facts.default_ipv6.address | default(None) }}"
```
Or: enable IPv6 on the target VMs (faster workaround, no role patching needed).
## Prevention
- Pin Ansible roles and patch argument_specs when they assume facts that may not exist
- Test roles on fresh VMs without IPv6 before production use
- Consider forking the role with the fix upstream
## Evidence
- Observed during GPU worker provisioning (worker-04/05, Jul 2026)
- Solution doc: `docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md`
- Session: @session:default/20260721_115501_78b25032
+66
View File
@@ -0,0 +1,66 @@
---
pattern_id: PAT-009
title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown"
category: infrastructure
severity: high
status: active
first_observed: 2026-08-01
last_updated: 2026-08-30
related_systems: [rke2-kubernetes, ceph-cluster]
related_solution_docs:
- docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md
related_skills: []
---
# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown
## Symptom
After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating
images), pods fail to mount with errors like:
```
MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal
desc = rbd: map failed with error: /dev/nbd0 already in use
```
The NBD device is held by a stale mapping from the old RBD image, even though the
new image has the same name.
## Root Cause
When an RBD image is recreated (delete + create with same name), the Ceph CSI
driver's NBD mappings from the old image remain active. The Linux NBD layer
holds `/dev/nbdX` open, blocking new mounts to the same device path.
The Kubernetes VolumeAttachment object also references the old volume handle,
preventing the CSI driver from cleanly attaching the new volume.
## Mitigation
Three-step teardown procedure:
```bash
# 1. Delete the VolumeAttachment (allows CSI driver to release)
kubectl delete volumeattachment csi-cephfsplugin-<node>-<volume-handle>
# 2. Disconnect the stale NBD device on the target node
ssh <node> 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0
# 3. Restart the CSI node plugin to pick up clean state
kubectl delete pod -n kube-system csi-cephfsplugin-<node-id>
# (DaemonSet will respawn it)
```
After this, the pod can remount with the new RBD image.
## Prevention
- Before deleting RBD images, ensure all pods using them are scaled to 0
- Delete VolumeAttachments BEFORE deleting RBD images
- After RBD image recreation, restart CSI plugins on all nodes that had mounts
- Document this in the K8s disaster recovery runbook
## Evidence
- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping
- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls)
- Part of the full cluster rebuild procedure