dominik/memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7c36a7600a258d864ca21b10d7f678ba680d63ae
memory/log.md
T
Dominik Schön 5cf6a573f1 retro: compound-learning 30-day retrospective + 2 new patterns 
New patterns:
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
- PAT-009: Stale NBD devices after RBD volume swap

Updated:
- index.md: added PAT-008, PAT-009
- log.md: retrospective entry with 4 solution docs + 2 patterns
- Solution docs dispatched to ~/docs/solutions/
2026-08-30 11:56:47 +00:00

12 KiB
Raw Blame History

Memory Log

[2026-08-30] retro | Compound Learning Retrospective (Last 30 Days)

  • Reviewed sessions from Jul 31 – Aug 30, 2026
  • 4 new solution docs written by subagents:
    • bug-fixes/2026-08-05-ceph-squid-ec-pool-mark-complete-bug.md — EC pool mark-complete fails in Squid
    • bug-fixes/2026-08-06-traefik-cross-namespace-routing-pitfall.md — Two Traefik 404 incidents
    • architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md — Full RKE2 rebuild sequence
    • architecture/2026-08-01-immich-data-loss-no-backup.md — Ceph redundancy ≠ backup
  • 2 new patterns extracted:
    • PAT-008: Ansible default_ipv6 fact missing on fresh VMs
    • PAT-009: Stale NBD devices after RBD volume swap
  • 4 Hindsight entries indexed with solution summaries
  • Key themes: K8s cluster disaster recovery, Ceph EC pool unrecoverability, Traefik routing complexity, backup gap identification

[2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker

  • Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution)
  • New patterns/ directory in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer
  • Created 7 initial pattern pages from existing MEMORY.md entries + solution docs:
    • PAT-001: Galera DDL TOI Deadlock
    • PAT-002: Traefik reload unreliable
    • PAT-003: 1Password account-level rate-limit
    • PAT-004: VFIO GPU passthrough race condition
    • PAT-005: Finanzblick sync WAF-blocked
    • PAT-006: LinkedIn React nativeInputSetter
    • PAT-007: Ceph EC pool + SSD wear-level
  • Skill-Impact Tracker (patterns/skill-impact.md) — audit trail for skill modifications (accept/reject history)
  • Template (patterns/_template.md) for future pattern creation
  • compound-learning skill patched — added Phase 4.8 (Pattern Extraction) + Phase 4.9 (Skill-Impact Tracking)
  • Updated index.md with Patterns section
  • Hindsight: indexed with tags architecture, wikiskill, patterns, skill-evolution

[2026-08-13] feat | Memory Architecture Enhancement (arxiv 2602.06052v4)

  • Implemented 3 new automated memory capabilities from survey paper analysis:
  • Decay & Importance Scoring (hindsight_decay_scoring.py): Monthly cron (1st 03:30). Score = log(1+access)/log(51) × 0.5^(age/90d). Archives score<0.15 with 0 accesses. First run: 200 evaluated, 50 archived.
  • User Drift Detection (user_drift_detection.py): Weekly cron (Mon 04:00). Compares 14d vs 90d baseline for message length, frustration, corrections, topic shifts, style changes. First run: detected 3 drifts (41% longer msgs, new focus gitops, declining ai_ml).
  • Skill Health Check (skill_health_check.py): Weekly cron (Mon 04:30). Scans 187 active SKILL.md files for stubs, broken refs, missing pitfalls, staleness, duplicates. First run: 70 healthy, 117 warnings, 0 critical.
  • Cron jobs: 112c106c0954, 280458a2b035, 761b6694d99b (all no_agent, deliver=origin)
  • Wiki updated: systems/hindsight.md (meta-memory automation table expanded)
  • Hindsight: indexed with tags architecture, memory-system, cron, arxiv-2602.06052

[2026-08-13] fix | Seafile Recovery + Paperless Upgrade + Authelia Ingress

  • Gitea SSH access: No SSH key on Hermes VM was authorized for Gitea. HTTP port 3000 has no external IngressRoute. Generated dedicated ED25519 key (id_ed25519_gitea-hermes-push), stored in 1Password (ID: lqaebdtewj5xatvpzfrkag73v4), registered in Gitea as key ID 4 for user dominik. SSH via 10.0.30.202:22 (LoadBalancer) is the reliable push path.
  • Paperless v3 IngressRoute: Host was dokumente.familie-schoen.com instead of dokumente-neu.familie-schoen.com. Two-track fix: kubectl patch (immediate) + Git commit fa4184b (permanent via ArgoCD self-heal). Backtick escaping in Traefik Host() match requires --patch-file not inline --patch.
  • Wiki updated: systems/gitea.md (SSH push key section), reference/ssh-keys.md (new key entry)
  • Solution docs: workflows/2026-07-26-gitea-ssh-access-for-hermes.md, bug-fixes/2026-07-26-paperless-ingressroute-hostname-fix.md
  • Hindsight: both solutions indexed with tags

[2026-07-25] fix | Loki 500 Error + CNPG Leader Election + Stale Pods Cleanup

  • Loki HTTP 500: replication_factor: 3 in hash ring with only 1 SingleBinary instance → "too many unhealthy instances in the ring". Chart v6.42.0 ignores loki.common.replication_factor — correct path is loki.commonConfig.replication_factor. Fix: commit cb70d36
  • Memcached caches disabled: chunksCache.enabled: false, resultsCache.enabled: false (SingleBinary doesn't need them). Commit f8852e8
  • Fluent Bit: cascading failure from Loki 500s — fixed automatically once Loki accepted pushes
  • CNPG 28+129 restarts: Leader election lease renewal failed during API server latency spikes (Ceph recovery I/O). Default 15s/10s too short. Fix: --leader-lease-duration=60 --leader-renew-deadline=40 via additionalArgs. Commit 6ed4a53
  • 17 stale node-debugger pods deleted from default namespace
  • OSDs 0+2 destroyed+purged (proxmox2, SSDs with 92% wear + slow ops). CRUSH host proxmox2 removed. 13 OSDs remaining on 8 hosts.
  • Wiki updated: ceph-cluster.md, loki-fluentbit.md

[2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate

  • Git push failed: remote URL used http:// but Gitea redirects to https:// — git doesn't follow auth redirects
  • Token itself was valid (same as ArgoCD argocd-repo-credentials), just wrong protocol
  • Fixed: git remote set-url to https://
  • Eliminated duplicate: systems/ceph.md (82 lines, no frontmatter) merged into systems/ceph-cluster.md (canonical, referenced in index.md)
  • Fixed sync script: removed --allow-empty flag, improved change detection

[2026-07-24] fix | Velero CSI Volume Snapshots Three-Layer Fix

  • Velero backups PartiallyFailed for 121 days — all PVCs skipped
  • Fix 1: Created VolumeSnapshotClass ceph-rbd-snapclass (rbd.csi.ceph.com)
  • Fix 2: Migrated ArgoCD repoURLs from dead Gitea (10.0.30.105) to new (ssh://git@10.0.30.202:22)
    • Gitea SSH user is git not gitea; git.schoen.codes → Traefik (10.0.30.208), SSH on 10.0.30.202
    • New SSH deploy key generated, added to Gitea repo
  • Fix 3: features: EnableCSI must be under configuration: in Velero Helm values (not top-level)
    • Must be string, not array — array form breaks Helm template
  • Verified: test backup created VolumeSnapshot + VolumeSnapshotContent, uploaded to S3 ✅
  • Solution doc: docs/solutions/bug-fixes/2026-07-24-velero-csi-snapshots-three-layer-fix.md
  • Commits: 5ebfa30, 19f35bc, cd18c58, baf85f5, f652bfb

[2026-07-24] fix | VFIO Module Loading Race Condition RCA + Fix

  • RCA: GPU passthrough failed on ms-a2-1 due to missing modprobe config (NOT kernel version)
  • Root cause: Missing softdep amdgpu pre: vfio-pci + incomplete DRM blacklist → race condition (302s vs 2.4s bind time)
  • Fix applied to ms-a2-1: Added blacklist drm, blacklist drm_kms_helper, softdep amdgpu pre: vfio-pci, rebuilt initramfs
  • Verification pending (reboot required, 4 VMs + OSD 13 on ms-a2-1)
  • Solution doc: docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md
  • Also documented: K8s persistent storage architecture (all PVCs on Ceph RBD, 3x replication)
  • Solution doc: docs/solutions/architecture/2026-07-24-k8s-persistent-storage-ceph-rbd.md

[2026-07-24] fix | worker-05 GPU Passthrough — Move to ms-a2-2

  • Problem: worker-05 (VM 102) on ms-a2-1 had kernel BUG with 1002:13c0 GPU — /dev/dri/renderD128 missing
  • ms-a2-1 and ms-a2-2 have identical hardware (AMD Ryzen 9 9955HX, GPU 1002:13c0)
  • Solution: Moved VM 102 from ms-a2-1 to ms-a2-2 (offline migration, Ceph RBD shared storage)
  • Result: renderD128 ✅ visible in guest, amdgpu driver loaded, all pods running
  • IaC: main.tf cleaned up — single unified rke2_worker block, worker-05 node=ms-a2-2, worker-06 removed
  • PCI mapping: amd-gpu now includes n5pro + ms-a2-1 + ms-a2-2
  • Commit: 21bd335
  • Also: ms-a2-2 upgraded PVE 9.1.1→9.2.5, kernel 7.0.14-6-pve, VFIO persistent after reboot, HA integrated

[2026-07-24] init | ms-a2-2 Node Initialization

  • New Proxmox node ms-a2-2 (10.0.20.93, nodeid 1) joined cluster
  • Hardware: AMD Ryzen 9 9955HX (32 threads), 91GB RAM, 1.8TB NVMe
  • GPU: AMD Radeon 1002:13c0 (identical to ms-a2-1) — VFIO passthrough configured
  • PCI mapping: amd-gpu-ms-a2-2 created, added to generic amd-gpu mapping
  • Ceph OSD 14: created on nvme0n1 (1.8TB SSD), class ssd, weight 1.82
  • Fluent-bit: installed + configured (hostname adapted to ms-a2-2)
  • APT sources: fixed from enterprise to no-subscription (ceph + pve)
  • LLM Wiki: ip-map.md, ceph.md updated; AGENTS.md updated (9 nodes)
  • Note: PVE 9.1.1 on ms-a2-2 (rest of cluster is 9.2.3) — upgrade pending

[2026-07-24] cleanup | proxmox1 Permanent Removal + Ceph PG Fixes

  • Removed proxmox1 from Ceph CRUSH (empty host bucket, weight 0)
  • Also removed stale empty buckets: proxmox, px-tmp20
  • Updated corosync.conf: removed proxmox1 node entry, config_version 14→15, expected_votes 9→8
  • Removed stale /etc/pve/nodes/{proxmox1,proxmox,pve,px-tmp20}
  • Updated IaC: proxmox_api_url 10.0.20.10→10.0.20.91, template_node proxmox1→n5pro, target_nodes list updated
  • Updated AGENTS.md topology (9→8 nodes), LLM Wiki ip-map.md + ceph.md
  • Commit: 469d29d
  • Additional: Manual pg-upmap for stuck PGs 5.13→[1,6,7], 6.6c→[11,8,7] — all clean+remapped eliminated

[2026-07-24] fix | Ceph Remapped PGs — Weight Imbalance + pg_num Mismatch

  • RCA: 11 active+clean+remapped PGs caused by CRUSH placement failure
  • Root cause 1: Pool 6 (hdd_disk) pg_num=120 ≠ pgp_num=112 (autoscaler mid-merge)
  • Root cause 2: Extreme HDD host weight imbalance (n5pro=10TB vs proxmox6=0.3TB)
  • Fix: Equalized pg_num→112, set nopgchange=true, reweighted osd.7 0.80→1.0
  • Result: 9/11 PGs recovering (backfilling at 36MiB/s), 2 cosmetic clean+remapped remain
  • Created systems/ceph.md with full OSD/pool/CRUSH documentation

[2026-07-24] update | Gitea Actions K8s Runner Bug + DinD Pitfalls

  • Documented Gitea 1.27.0 bug: actions_ready_job queue doesn't assign tasks to K8s-based runners
  • Updated systems/gitea.md with known issue, DinD pitfalls, runner management via DB
  • Solution doc: docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md
  • IaC committed: runner-deployment.yaml + runner-config.yaml (DinD TCP sidecar pattern)

[2026-04-28] init | Memory System Created

  • Created memory directory structure at ~/.hermes/memory
  • Installed qmd 2.1.0 (Query Markup Documents)
  • Created initial index.md with entities, concepts, and sources
  • Created Entity pages: Infrastructure, Email-System, Health-Fitness, Team-Structure, Memory-System
  • Created Concept pages: Network-Architecture, Email-Organization, Monitoring-System, Proxmox-Cluster, qmd-Knowledge-Base
  • Initialized git repo, pushed to origin/main

[2026-04-29] fix | Memory Process Established

  • Corrected stale memory-entry-001.md
  • Established protocol: memory update + git sync on every session

[2026-04-30] ingest | cloud.familie-schoen.com

  • Full port scan, DNS, SSL, HTTP headers, API analysis
  • Identified: Seafile 13.0.19 on nginx, Let's Encrypt SSL

[2026-07-24] restructure | Full Wiki Restructuring

  • Deleted 5 root-duplicate files (dominik-schoen, email-organization, team-structure, llm-wiki-pattern, embedding-model)
  • Deleted obsolete Entities/Memory-System.md, Concepts/qmd-Knowledge-Base.md
  • Deleted raw/articles/ and memory-entry-001/002.md (stale)
  • Renamed Entities/ → entities/, Concepts/ → concepts/ (lowercase)
  • Created new directories: systems/, reference/
  • Moved Proxmox-Cluster, Monitoring-System, seafile to systems/
  • Updated all existing pages with current data (April → July 2026):
    • Infrastructure: 2 Nodes → 9 Nodes, added Ceph/RKE2/Galera/Loki
    • Proxmox-Cluster: PVE 9.2.3, 9 Nodes, Fluent Bit details, PVE Tasks, Ceph Audit
    • Network-Architecture: cleaned up, links to reference pages
    • Monitoring: Prometheus/Grafana/Loki/HolmesGPT (was just "weekly network scans")
  • Created 6 new system pages: ceph-cluster, rke2-kubernetes, galera-maxscale, loki-fluentbit, gitea, hindsight
  • Created 3 reference pages: ip-map, ssh-keys, ports
  • Created 2 concept pages: gitops-workflow, credential-policy
  • Rewrote index.md with new structure + Memory Layer Architecture table
  • Total: 18 pages (was 21 with dupes, now 18 clean unique pages)
Reference in New Issue View Git Blame Copy Permalink