dominik/memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: update ceph-cluster + loki-fluentbit + log for 2026-07-25 fixes (OSD purge, replication_factor, CNPG leader election)

Browse Source
This commit is contained in:
Dominik Schön
2026-07-25 09:27:19 +00:00
parent dc316f7b26
commit 92a13fdfa4
3 changed files with 26 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
log.md
+9
View File
@@ -1,5 +1,14 @@
# Memory Log # Memory Log
## [2026-07-25] fix | Loki 500 Error + CNPG Leader Election + Stale Pods Cleanup
- **Loki HTTP 500**: `replication_factor: 3` in hash ring with only 1 SingleBinary instance → "too many unhealthy instances in the ring". Chart v6.42.0 ignores `loki.common.replication_factor` — correct path is `loki.commonConfig.replication_factor`. Fix: commit `cb70d36`
- **Memcached caches disabled**: `chunksCache.enabled: false`, `resultsCache.enabled: false` (SingleBinary doesn't need them). Commit `f8852e8`
- **Fluent Bit**: cascading failure from Loki 500s — fixed automatically once Loki accepted pushes
- **CNPG 28+129 restarts**: Leader election lease renewal failed during API server latency spikes (Ceph recovery I/O). Default 15s/10s too short. Fix: `--leader-lease-duration=60 --leader-renew-deadline=40` via `additionalArgs`. Commit `6ed4a53`
- **17 stale node-debugger pods** deleted from default namespace
- **OSDs 0+2 destroyed+purged** (proxmox2, SSDs with 92% wear + slow ops). CRUSH host proxmox2 removed. 13 OSDs remaining on 8 hosts.
- Wiki updated: ceph-cluster.md, loki-fluentbit.md
## [2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate ## [2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate
- Git push failed: remote URL used `http://` but Gitea redirects to `https://` — git doesn't follow auth redirects - Git push failed: remote URL used `http://` but Gitea redirects to `https://` — git doesn't follow auth redirects
- Token itself was valid (same as ArgoCD `argocd-repo-credentials`), just wrong protocol - Token itself was valid (same as ArgoCD `argocd-repo-credentials`), just wrong protocol
systems/ceph-cluster.md
+6 -6
View File
@@ -10,18 +10,18 @@ modified: "2026-07-25"
## Overview ## Overview
- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 - **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s) - **Health**: HEALTH_OK (recovery complete after OSD 0+2 drain, 0.3% misplaced settling)
- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5 - **Nodes**: 8 Proxmox hosts (proxmox2 removed from CRUSH 2026-07-25), 3 MONs (proxmox5/7/4), MGR on proxmox5
- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in - **OSDs**: 13 (8 SSD, 5 HDD), all up/in — OSDs 0+2 destroyed+purged 2026-07-25
- **Capacity**: 6.0 TiB used / 21 TiB total - **Capacity**: ~22 TiB total, 6.0 TiB used
## OSD Layout ## OSD Layout
| OSD | Class | Size | Host | Reweight | Notes | | OSD | Class | Size | Host | Reweight | Notes |
|-----|-------|------|------|----------|-------| |-----|-------|------|------|----------|-------|
| 0 | ssd | 188 GB | proxmox2 | 0.50 | 92% Wear, Critical Warning | | 0 | ssd | 188 GB | proxmox2 | — | **DESTROYED 2026-07-25** (92% wear) |
| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD | | 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD |
| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full | | 2 | ssd | 233 GB | proxmox2 | — | **DESTROYED 2026-07-25** (slow ops, 81% full) |
| 3 | ssd | 238 GB | proxmox4 | 0.95 | | | 3 | ssd | 238 GB | proxmox4 | 0.95 | |
| 4 | ssd | 233 GB | proxmox3 | 0.95 | | | 4 | ssd | 233 GB | proxmox3 | 0.95 | |
| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full | | 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full |
systems/loki-fluentbit.md
+11 -2
View File
@@ -3,15 +3,24 @@ title: Loki + Fluent Bit (Logging Stack)
category: systems category: systems
tags: [loki, fluent-bit, logging, observability] tags: [loki, fluent-bit, logging, observability]
created: "2026-07-24" created: "2026-07-24"
modified: "2026-07-24" modified: "2026-07-25"
--- ---
# Loki + Fluent Bit (Logging Stack) # Loki + Fluent Bit (Logging Stack)
## Loki ## Loki
- **Namespace:** logging (K8s) - **Namespace:** logging (K8s)
- **Mode:** SingleBinary, `replication_factor: 1` (fix 2026-07-25)
- **Chart:** grafana/loki v6.42.0 via ArgoCD GitOps
- **Caches:** memcached disabled (`chunksCache.enabled: false`, `resultsCache.enabled: false`)
- **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100` - **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100`
- **Caches:** memcached (results-cache, `-m 512 -I 2m`, requests 256Mi/limits 512Mi)
### Known Issues (Fixed 2026-07-25)
| Issue | Root Cause | Fix |
|-------|-----------|-----|
| HTTP 500 on push | `replication_factor: 3` in hash ring with only 1 instance → "too many unhealthy instances in the ring" | `loki.commonConfig.replication_factor: 1` (NOT `loki.common` — chart ignores that path) |
| Memcached CrashLoop | Caches allocated 0 memory | Disabled caches entirely (SingleBinary doesn't need them) |
| Fluent Bit connection failures | Cascading from Loki 500s | Fixed by Loki replication_factor fix |
## Fluent Bit v5.0.9 ## Fluent Bit v5.0.9