57 lines
1.9 KiB
Markdown
57 lines
1.9 KiB
Markdown
---
|
|
title: Loki + Fluent Bit (Logging Stack)
|
|
category: systems
|
|
tags: [loki, fluent-bit, logging, observability]
|
|
created: "2026-07-24"
|
|
modified: "2026-07-25"
|
|
---
|
|
|
|
# Loki + Fluent Bit (Logging Stack)
|
|
|
|
## Loki
|
|
- **Namespace:** logging (K8s)
|
|
- **Mode:** SingleBinary, `replication_factor: 1` (fix 2026-07-25)
|
|
- **Chart:** grafana/loki v6.42.0 via ArgoCD GitOps
|
|
- **Caches:** memcached disabled (`chunksCache.enabled: false`, `resultsCache.enabled: false`)
|
|
- **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100`
|
|
|
|
### Known Issues (Fixed 2026-07-25)
|
|
| Issue | Root Cause | Fix |
|
|
|-------|-----------|-----|
|
|
| HTTP 500 on push | `replication_factor: 3` in hash ring with only 1 instance → "too many unhealthy instances in the ring" | `loki.commonConfig.replication_factor: 1` (NOT `loki.common` — chart ignores that path) |
|
|
| Memcached CrashLoop | Caches allocated 0 memory | Disabled caches entirely (SingleBinary doesn't need them) |
|
|
| Fluent Bit connection failures | Cascading from Loki 500s | Fixed by Loki replication_factor fix |
|
|
|
|
## Fluent Bit v5.0.9
|
|
|
|
### Coverage
|
|
| Target | How | Status |
|
|
|--------|-----|--------|
|
|
| 9 PVE Hosts | `install-host.sh` auf VM200 | ✅ aktiv |
|
|
| 20 LXC Containers | `install-lxc.sh` | ✅ aktiv (außer CT110 stopped) |
|
|
| 5 QEMU VMs | `qm guest exec` | ✅ aktiv (VM300/301/302/310/311) |
|
|
| 6 K8s Nodes | K8s DaemonSet | ✅ aktiv |
|
|
| VM200 | bereits vorhanden | ✅ aktiv |
|
|
|
|
### Inputs (Proxmox Hosts)
|
|
- systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd)
|
|
- auth.log
|
|
- pveproxy/access.log
|
|
- pvedaemon.log
|
|
- cluster.log
|
|
- **PVE Tasks** (`/var/log/pve/tasks/index`) — UPID-Format
|
|
- **Ceph Audit** (`/var/log/ceph/ceph.audit.log`) — JSON
|
|
|
|
### Scripts
|
|
- `logging/fluent-bit-proxmox/` im iac-homelab Repo
|
|
|
|
## Query
|
|
```bash
|
|
logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox'
|
|
```
|
|
|
|
## Related
|
|
- [[systems/proxmox-cluster]]
|
|
- [[systems/monitoring]]
|
|
- [[systems/rke2-kubernetes]]
|