dominik/memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: update ceph-cluster + loki-fluentbit + log for 2026-07-25 fixes (OSD purge, replication_factor, CNPG leader election)

Browse Source
This commit is contained in:
Dominik Schön
2026-07-25 09:27:19 +00:00
parent dc316f7b26
commit 92a13fdfa4
3 changed files with 26 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
log.md
+9
View File
@@ -1,5 +1,14 @@
# Memory Log
## [2026-07-25] fix | Loki 500 Error + CNPG Leader Election + Stale Pods Cleanup
- **Loki HTTP 500**: `replication_factor: 3` in hash ring with only 1 SingleBinary instance → "too many unhealthy instances in the ring". Chart v6.42.0 ignores `loki.common.replication_factor` — correct path is `loki.commonConfig.replication_factor`. Fix: commit `cb70d36`
- **Memcached caches disabled**: `chunksCache.enabled: false`, `resultsCache.enabled: false` (SingleBinary doesn't need them). Commit `f8852e8`
- **Fluent Bit**: cascading failure from Loki 500s — fixed automatically once Loki accepted pushes
- **CNPG 28+129 restarts**: Leader election lease renewal failed during API server latency spikes (Ceph recovery I/O). Default 15s/10s too short. Fix: `--leader-lease-duration=60 --leader-renew-deadline=40` via `additionalArgs`. Commit `6ed4a53`
- **17 stale node-debugger pods** deleted from default namespace
- **OSDs 0+2 destroyed+purged** (proxmox2, SSDs with 92% wear + slow ops). CRUSH host proxmox2 removed. 13 OSDs remaining on 8 hosts.
- Wiki updated: ceph-cluster.md, loki-fluentbit.md
## [2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate
- Git push failed: remote URL used `http://` but Gitea redirects to `https://` — git doesn't follow auth redirects
- Token itself was valid (same as ArgoCD `argocd-repo-credentials`), just wrong protocol