diff --git a/index.md b/index.md index ae951d5..dcde658 100644 --- a/index.md +++ b/index.md @@ -52,6 +52,8 @@ - [[patterns/finanzblick-sync-waf]] — POST /sync WAF-blocked, UI Modal Sequence (PAT-005) - [[patterns/linkedin-react-input]] — nativeInputSetter, Session Expiry (PAT-006) - [[patterns/ceph-ssd-wear-ec-pool]] — EC Pool unusable + SSD Wear-Level (PAT-007) +- [[patterns/ansible-default-ipv6]] — lablabs.rke2 role fails on IPv6-less VMs (PAT-008) +- [[patterns/k8s-stale-nbd-devices]] — RBD swap leaves stale NBD mappings (PAT-009) - [[patterns/skill-impact]] — Skill Modification Audit Trail ## Memory Layer Architektur diff --git a/log.md b/log.md index 86f986e..26a99aa 100644 --- a/log.md +++ b/log.md @@ -1,5 +1,18 @@ # Memory Log +## [2026-08-30] retro | Compound Learning Retrospective (Last 30 Days) +- Reviewed sessions from Jul 31 – Aug 30, 2026 +- **4 new solution docs** written by subagents: + - `bug-fixes/2026-08-05-ceph-squid-ec-pool-mark-complete-bug.md` — EC pool mark-complete fails in Squid + - `bug-fixes/2026-08-06-traefik-cross-namespace-routing-pitfall.md` — Two Traefik 404 incidents + - `architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md` — Full RKE2 rebuild sequence + - `architecture/2026-08-01-immich-data-loss-no-backup.md` — Ceph redundancy ≠ backup +- **2 new patterns** extracted: + - PAT-008: Ansible default_ipv6 fact missing on fresh VMs + - PAT-009: Stale NBD devices after RBD volume swap +- **4 Hindsight entries** indexed with solution summaries +- Key themes: K8s cluster disaster recovery, Ceph EC pool unrecoverability, Traefik routing complexity, backup gap identification + ## [2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker - Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution) - **New `patterns/` directory** in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer diff --git a/patterns/ansible-default-ipv6.md b/patterns/ansible-default-ipv6.md new file mode 100644 index 0000000..8e020b2 --- /dev/null +++ b/patterns/ansible-default-ipv6.md @@ -0,0 +1,59 @@ +--- +pattern_id: PAT-008 +title: "Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails" +category: tooling +severity: medium +status: active +first_observed: 2026-07-21 +last_updated: 2026-08-30 +related_systems: [rke2-kubernetes] +related_solution_docs: + - docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md +related_skills: [] +--- + +# Ansible default_ipv6 fact missing on fresh VMs → lablabs.rke2 role fails + +## Symptom + +The `lablabs.rke2` Ansible role fails on freshly provisioned VMs with a Jinja2 error: +``` +ansible_facts['default_ipv6']['address'] +``` +The `default_ipv6` fact doesn't exist on VMs without IPv6 configured, causing the +`meta/argument_specs.yml` validation to crash. + +## Root Cause + +The `lablabs.rke2` role's `meta/argument_specs.yml` references +`ansible_facts['default_ipv6']['address']` with a hardcoded default. On fresh VMs +without IPv6, `ansible_facts['default_ipv6']` is undefined → Jinja2 raises +`UndefinedError`. + +A `pre_task` setting `default_ipv6` via `combine()` doesn't help because the +argument_specs validation runs BEFORE pre_tasks — it re-collects facts and +overwrites the pre_task fix. + +## Mitigation + +Patch `meta/argument_specs.yml` directly in the role: +```yaml +# Replace: +default: "{{ ansible_facts['default_ipv6']['address'] }}" +# With: +default: "{{ ansible_facts.default_ipv6.address | default(None) }}" +``` + +Or: enable IPv6 on the target VMs (faster workaround, no role patching needed). + +## Prevention + +- Pin Ansible roles and patch argument_specs when they assume facts that may not exist +- Test roles on fresh VMs without IPv6 before production use +- Consider forking the role with the fix upstream + +## Evidence + +- Observed during GPU worker provisioning (worker-04/05, Jul 2026) +- Solution doc: `docs/solutions/bug-fixes/2026-07-21-ansible-rke2-default-ipv6-bug.md` +- Session: @session:default/20260721_115501_78b25032 diff --git a/patterns/k8s-stale-nbd-devices.md b/patterns/k8s-stale-nbd-devices.md new file mode 100644 index 0000000..e7e3e09 --- /dev/null +++ b/patterns/k8s-stale-nbd-devices.md @@ -0,0 +1,66 @@ +--- +pattern_id: PAT-009 +title: "Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown" +category: infrastructure +severity: high +status: active +first_observed: 2026-08-01 +last_updated: 2026-08-30 +related_systems: [rke2-kubernetes, ceph-cluster] +related_solution_docs: + - docs/solutions/architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md +related_skills: [] +--- + +# Stale NBD devices after RBD volume swap — delete VolumeAttachment + nbd teardown + +## Symptom + +After swapping RBD images for PVCs (e.g. migrating to new Ceph pool or recreating +images), pods fail to mount with errors like: +``` +MountVolume.MountDevice failed for volume "pvc-xxx" : rpc error: code = Internal +desc = rbd: map failed with error: /dev/nbd0 already in use +``` + +The NBD device is held by a stale mapping from the old RBD image, even though the +new image has the same name. + +## Root Cause + +When an RBD image is recreated (delete + create with same name), the Ceph CSI +driver's NBD mappings from the old image remain active. The Linux NBD layer +holds `/dev/nbdX` open, blocking new mounts to the same device path. + +The Kubernetes VolumeAttachment object also references the old volume handle, +preventing the CSI driver from cleanly attaching the new volume. + +## Mitigation + +Three-step teardown procedure: +```bash +# 1. Delete the VolumeAttachment (allows CSI driver to release) +kubectl delete volumeattachment csi-cephfsplugin-- + +# 2. Disconnect the stale NBD device on the target node +ssh 'nbd-client -d /dev/nbd0' # or: qemu-nbd --disconnect /dev/nbd0 + +# 3. Restart the CSI node plugin to pick up clean state +kubectl delete pod -n kube-system csi-cephfsplugin- +# (DaemonSet will respawn it) +``` + +After this, the pod can remount with the new RBD image. + +## Prevention + +- Before deleting RBD images, ensure all pods using them are scaled to 0 +- Delete VolumeAttachments BEFORE deleting RBD images +- After RBD image recreation, restart CSI plugins on all nodes that had mounts +- Document this in the K8s disaster recovery runbook + +## Evidence + +- Observed during K8s cluster rebuild (Aug 2026) — 13 RBD volumes needed swapping +- Session: @session:default/20260731_113701_74fd2814 (250+ tool calls) +- Part of the full cluster rebuild procedure