35 lines
1.0 KiB
Markdown
35 lines
1.0 KiB
Markdown
---
|
|
title: Monitoring & Observability
|
|
category: systems
|
|
tags: [monitoring, prometheus, grafana, loki, holmesgpt, alerting]
|
|
created: "2026-04-28"
|
|
modified: "2026-07-24"
|
|
---
|
|
|
|
# Monitoring & Observability
|
|
|
|
## Prometheus + Grafana
|
|
- **Prometheus**: K8s (kube-prometheus-stack)
|
|
- **Grafana**: K8s, Ingress grafana.familie-schoen.com (.205)
|
|
- **Alertmanager**: K8s, Routes → Telegram
|
|
|
|
## Loki Stack (Centralized Logging)
|
|
- **Loki**: K8s ns=logging, LB 10.0.30.207:3100
|
|
- **Fluent Bit v5.0.9**: auf 9 PVE Hosts + 20 LXC + 5 QEMU + 6 K8s Nodes
|
|
- **Inputs**: PVE tasks, Ceph audit, journald, auth, pveproxy, pvedaemon, cluster
|
|
- Siehe [[systems/loki-fluentbit]]
|
|
|
|
## HolmesGPT (SRE)
|
|
- **Namespace**: holmes
|
|
- **Pipeline**: Alertmanager → adapter (9095) → HolmesGPT → TG @schoen_sre_bot
|
|
- **Model**: noris-glm (via ai.noris.de)
|
|
- RKE2 ctrl-plane alerts disabled
|
|
|
|
## Alerting Rules
|
|
- Via IaC Git Repo deployt (Ansible für CT141, ArgoCD für K8s)
|
|
- RKE2 control-plane alerts disabled
|
|
|
|
## Related
|
|
- [[systems/loki-fluentbit]]
|
|
- [[systems/rke2-kubernetes]]
|