From 92a13fdfa4384bfab04dcfe6296fe588305c8674 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dominik=20Sch=C3=B6n?= Date: Sat, 25 Jul 2026 09:27:19 +0000 Subject: [PATCH] fix: update ceph-cluster + loki-fluentbit + log for 2026-07-25 fixes (OSD purge, replication_factor, CNPG leader election) --- log.md | 9 +++++++++ systems/ceph-cluster.md | 12 ++++++------ systems/loki-fluentbit.md | 13 +++++++++++-- 3 files changed, 26 insertions(+), 8 deletions(-) diff --git a/log.md b/log.md index 339533b..b96b2d1 100644 --- a/log.md +++ b/log.md @@ -1,5 +1,14 @@ # Memory Log +## [2026-07-25] fix | Loki 500 Error + CNPG Leader Election + Stale Pods Cleanup +- **Loki HTTP 500**: `replication_factor: 3` in hash ring with only 1 SingleBinary instance → "too many unhealthy instances in the ring". Chart v6.42.0 ignores `loki.common.replication_factor` — correct path is `loki.commonConfig.replication_factor`. Fix: commit `cb70d36` +- **Memcached caches disabled**: `chunksCache.enabled: false`, `resultsCache.enabled: false` (SingleBinary doesn't need them). Commit `f8852e8` +- **Fluent Bit**: cascading failure from Loki 500s — fixed automatically once Loki accepted pushes +- **CNPG 28+129 restarts**: Leader election lease renewal failed during API server latency spikes (Ceph recovery I/O). Default 15s/10s too short. Fix: `--leader-lease-duration=60 --leader-renew-deadline=40` via `additionalArgs`. Commit `6ed4a53` +- **17 stale node-debugger pods** deleted from default namespace +- **OSDs 0+2 destroyed+purged** (proxmox2, SSDs with 92% wear + slow ops). CRUSH host proxmox2 removed. 13 OSDs remaining on 8 hosts. +- Wiki updated: ceph-cluster.md, loki-fluentbit.md + ## [2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate - Git push failed: remote URL used `http://` but Gitea redirects to `https://` — git doesn't follow auth redirects - Token itself was valid (same as ArgoCD `argocd-repo-credentials`), just wrong protocol diff --git a/systems/ceph-cluster.md b/systems/ceph-cluster.md index ccfaf80..29db407 100644 --- a/systems/ceph-cluster.md +++ b/systems/ceph-cluster.md @@ -10,18 +10,18 @@ modified: "2026-07-25" ## Overview - **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 -- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s) -- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5 -- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in -- **Capacity**: 6.0 TiB used / 21 TiB total +- **Health**: HEALTH_OK (recovery complete after OSD 0+2 drain, 0.3% misplaced settling) +- **Nodes**: 8 Proxmox hosts (proxmox2 removed from CRUSH 2026-07-25), 3 MONs (proxmox5/7/4), MGR on proxmox5 +- **OSDs**: 13 (8 SSD, 5 HDD), all up/in — OSDs 0+2 destroyed+purged 2026-07-25 +- **Capacity**: ~22 TiB total, 6.0 TiB used ## OSD Layout | OSD | Class | Size | Host | Reweight | Notes | |-----|-------|------|------|----------|-------| -| 0 | ssd | 188 GB | proxmox2 | 0.50 | 92% Wear, Critical Warning | +| 0 | ssd | 188 GB | proxmox2 | — | **DESTROYED 2026-07-25** (92% wear) | | 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD | -| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full | +| 2 | ssd | 233 GB | proxmox2 | — | **DESTROYED 2026-07-25** (slow ops, 81% full) | | 3 | ssd | 238 GB | proxmox4 | 0.95 | | | 4 | ssd | 233 GB | proxmox3 | 0.95 | | | 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full | diff --git a/systems/loki-fluentbit.md b/systems/loki-fluentbit.md index a155ab3..3ff53d0 100644 --- a/systems/loki-fluentbit.md +++ b/systems/loki-fluentbit.md @@ -3,15 +3,24 @@ title: Loki + Fluent Bit (Logging Stack) category: systems tags: [loki, fluent-bit, logging, observability] created: "2026-07-24" -modified: "2026-07-24" +modified: "2026-07-25" --- # Loki + Fluent Bit (Logging Stack) ## Loki - **Namespace:** logging (K8s) +- **Mode:** SingleBinary, `replication_factor: 1` (fix 2026-07-25) +- **Chart:** grafana/loki v6.42.0 via ArgoCD GitOps +- **Caches:** memcached disabled (`chunksCache.enabled: false`, `resultsCache.enabled: false`) - **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100` -- **Caches:** memcached (results-cache, `-m 512 -I 2m`, requests 256Mi/limits 512Mi) + +### Known Issues (Fixed 2026-07-25) +| Issue | Root Cause | Fix | +|-------|-----------|-----| +| HTTP 500 on push | `replication_factor: 3` in hash ring with only 1 instance → "too many unhealthy instances in the ring" | `loki.commonConfig.replication_factor: 1` (NOT `loki.common` — chart ignores that path) | +| Memcached CrashLoop | Caches allocated 0 memory | Disabled caches entirely (SingleBinary doesn't need them) | +| Fluent Bit connection failures | Cascading from Loki 500s | Fixed by Loki replication_factor fix | ## Fluent Bit v5.0.9