memory/systems/loki-fluentbit.md

title, category, tags, created, modified

title	category	tags	created	modified
Loki + Fluent Bit (Logging Stack)	systems	loki fluent-bit logging observability	2026-07-24	2026-07-25

Loki + Fluent Bit (Logging Stack)

Loki

• Namespace: logging (K8s)
• Mode: SingleBinary, replication_factor: 1 (fix 2026-07-25)
• Chart: grafana/loki v6.42.0 via ArgoCD GitOps
• Caches: memcached disabled (chunksCache.enabled: false, resultsCache.enabled: false)
• External Service: loki-external, LoadBalancer 10.0.30.207:3100

Known Issues (Fixed 2026-07-25)

Issue	Root Cause	Fix
HTTP 500 on push	replication_factor: 3 in hash ring with only 1 instance → "too many unhealthy instances in the ring"	loki.commonConfig.replication_factor: 1 (NOT loki.common — chart ignores that path)
Memcached CrashLoop	Caches allocated 0 memory	Disabled caches entirely (SingleBinary doesn't need them)
Fluent Bit connection failures	Cascading from Loki 500s	Fixed by Loki replication_factor fix

Fluent Bit v5.0.9

Coverage

Target	How	Status
9 PVE Hosts	install-host.sh auf VM200	✅ aktiv
20 LXC Containers	install-lxc.sh	✅ aktiv (außer CT110 stopped)
5 QEMU VMs	qm guest exec	✅ aktiv (VM300/301/302/310/311)
6 K8s Nodes	K8s DaemonSet	✅ aktiv
VM200	bereits vorhanden	✅ aktiv

Inputs (Proxmox Hosts)

• systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd)
• auth.log
• pveproxy/access.log
• pvedaemon.log
• cluster.log
• PVE Tasks (/var/log/pve/tasks/index) — UPID-Format
• Ceph Audit (/var/log/ceph/ceph.audit.log) — JSON

Scripts

• logging/fluent-bit-proxmox/ im iac-homelab Repo

Query

logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox'

Related

• systems/proxmox-cluster
• systems/monitoring
• systems/rke2-kubernetes