Files
memory/systems/loki-fluentbit.md
T
2026-10-09 22:00:10 +00:00

2.7 KiB

title, category, tags, created, modified
title category tags created modified
Loki + Fluent Bit (Logging Stack) systems
loki
fluent-bit
logging
observability
2026-07-24 2026-10-09

Loki + Fluent Bit (Logging Stack)

Loki

  • Namespace: logging (K8s)
  • Mode: SingleBinary, replication_factor: 1 (fix 2026-07-25)
  • Chart: grafana/loki v6.42.0 via ArgoCD GitOps
  • Caches: memcached disabled (chunksCache.enabled: false, resultsCache.enabled: false)
  • External Service: loki-lb, LoadBalancer 10.0.30.207:3100 (restauriert 2026-10-09, Commit fb4d8b9, Manifest clusters/main/logging/loki-lb.yaml, App logging-config) — Vorgänger loki-external war MANUELL kubectl'd, nie in Git, starb unbemerkt bei Helm-Iterationen → Fleet-weiter Journal-Spam. Seit Restaurierung GitOps-gemanagt.

Known Issues (Fixed 2026-10-09: LB-VIP-Totalausfall)

Symptom Root Cause Fix
fluent-bit Fleet (9 Hosts + LXCs) fluutet Journals mit „No route to host 10.0.30.207:3100" seit ~7. Okt.; CT111-Journal 4,1 GB (95 % Spam), Disk 99 % Manueller SVC loki-external (LB .207) verschwand bei Loki-Helm-Iterations-Recreates — keine GitOps-Deckung loki-lb Manifest in Git (Selector single-binary auf loki-0), ArgoCD-App logging-config synced es; VIP sofort wieder im Cilium-Pool

Known Issues (Fixed 2026-07-25)

Issue Root Cause Fix
HTTP 500 on push replication_factor: 3 in hash ring with only 1 instance → "too many unhealthy instances in the ring" loki.commonConfig.replication_factor: 1 (NOT loki.common — chart ignores that path)
Memcached CrashLoop Caches allocated 0 memory Disabled caches entirely (SingleBinary doesn't need them)
Fluent Bit connection failures Cascading from Loki 500s Fixed by Loki replication_factor fix

Fluent Bit v5.0.9

Coverage

Target How Status
9 PVE Hosts install-host.sh auf VM200 ✅ aktiv
20 LXC Containers install-lxc.sh ✅ aktiv (außer CT110 stopped)
5 QEMU VMs qm guest exec ✅ aktiv (VM300/301/302/310/311)
6 K8s Nodes K8s DaemonSet ✅ aktiv
VM200 bereits vorhanden ✅ aktiv

Inputs (Proxmox Hosts)

  • systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd)
  • auth.log
  • pveproxy/access.log
  • pvedaemon.log
  • cluster.log
  • PVE Tasks (/var/log/pve/tasks/index) — UPID-Format
  • Ceph Audit (/var/log/ceph/ceph.audit.log) — JSON

Scripts

  • logging/fluent-bit-proxmox/ im iac-homelab Repo

Query

logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox'