dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/proxmox-ve-administration/references/infra-monitoring-2026-07.md
+815
View File
@@ -0,0 +1,815 @@
# Infrastructure Monitoring: Prometheus + Grafana Stack
## When to Use
Deploying a full monitoring stack for Proxmox VE + Ceph + PBS + Galera/MaxScale
infrastructure. Covers container provisioning, exporter deployment, alerting
via Telegram, and Grafana dashboards.
## Architecture
```
CT 141 (monitoring, 10.0.30.141/24, VLAN 30, 4CPU/8GB/30GB)
├── Prometheus (:9090) — 90-day TSDB retention
├── Grafana (:3000) — dashboards, via Traefik → grafana.familie-schoen.com
├── Alertmanager (:9093) — routes alerts to Telegram bridge
├── blackbox_exporter (:9115) — ICMP + HTTP probes for uptime
├── pve-exporter (:9221) — PVE API metrics (VM/CT status, resources)
├── telegram-bridge (:9099) — Flask webhook → Telegram Bot API
└── node_exporter (:9100) — local CT metrics
Exporters on targets:
├── 6× PVE Nodes — node_exporter :9100 (systemd service)
├── Ceph mgr — :9283 (built-in prometheus module, active mgr node)
├── 3× Galera — mysqld_exporter :9104 (MySQL/Galera metrics)
├── PBS Local (CT 116) — node_exporter :9100
└── PBS Remote (213.95.54.60) — node_exporter :9100 (UFW restricted)
```
## Container Creation
```bash
pct create 141 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname monitoring \
--memory 8192 --swap 4096 --cores 4 \
--rootfs vm_disks:vm-141-disk-0,size=30G \
--net0 name=eth0,bridge=vmbr0,ip=10.0.30.141/24,gw=10.0.30.1,tag=30,type=veth \
--features nesting=1 \
--onboot 1 --start 0
```
Install Docker inside the CT (standard Docker CE bookworm repo).
## Exporter Deployment
### node_exporter on PVE Nodes
Deploy via systemd service on each PVE node. Binary from GitHub releases
(v1.8.2). Service file at `/etc/systemd/system/node_exporter.service`.
Enable `--collector.systemd` and `--collector.textfile.directory` for
custom textfile collectors.
PVE node IPs (VLAN 20 management network) — 8 nodes total:
- proxmox1: 10.0.20.10
- proxmox2: 10.0.20.20
- proxmox3: 10.0.20.30
- proxmox4: 10.0.20.40
- proxmox5: 10.0.20.50
- proxmox6: 10.0.20.60
- proxmox7: 10.0.20.70 (primary jump host for SSH)
- n5pro: 10.0.20.91
⚠️ **Do NOT confuse node names with IPs** — the numeric suffix in the
hostname does NOT match the last octet. proxmox4 = .40, proxmox7 = .70,
n5pro = .91. Always verify with `hostname` after SSH.
### Ceph Prometheus Module
Built-in, just needs enabling:
```bash
ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/scrape_interval 15
```
Metrics available at `http://<active-mgr-ip>:9283/metrics`.
Active mgr can fail over — add all potential mgr nodes as scrape targets.
### mysqld_exporter on Galera
Requires exporter MySQL user on each Galera node:
```sql
CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
```
Binary: mysqld_exporter v0.15.1. Config at `/etc/mysqld_exporter/my.cnf`.
Galera SSH access: 1Password 'SSH-Key Galera' (debian user, then sudo).
### node_exporter on PBS Remote
Remote PBS is on public IP (213.95.54.60). **Must restrict UFW** to PVE
WAN IP only — node_exporter exposes system information to anyone.
```bash
sudo ufw allow from <PVE_WAN_IP> to any port 9100
```
### MaxScale Prometheus Metrics
MaxScale has a built-in Prometheus endpoint at `:8189/metrics`. No
additional exporter needed — just add it as a scrape target.
```yaml
# In prometheus.yml
- job_name: maxscale
static_configs:
- targets: ["10.0.30.81:8189"]
```
⚠️ **MaxScale VM 310 was unreachable as of 2026-07-05** — PVE shows
`status: running` but no ping, no SSH, QGA not running. The VM may have
hung at the bootloader or lost its network config. Until fixed, this
target will show as `down` in Prometheus — which is actually useful as
an alert. The scrape config should be deployed regardless; it becomes
functional as soon as the VM is reachable.
MaxScale VM details:
- VM 310 on proxmox4 (10.0.20.40)
- IP: 10.0.30.81 (VLAN 30)
- Metrics: http://10.0.30.81:8189/metrics
- REST API: http://10.0.30.81:8989/
## Docker Compose Stack
Services: prometheus, grafana, alertmanager, blackbox-exporter,
telegram-bridge, pve-exporter.
Key config files:
- `prometheus/prometheus.yml` — scrape configs for all targets
- `prometheus/rules/*.yml` — alerting rules (general, ceph, galera, backup)
- `alertmanager/alertmanager.yml` — routing to Telegram bridge
- `blackbox/blackbox.yml` — ICMP + HTTP probe modules
- `grafana/provisioning/` — datasource + dashboard auto-provisioning
### Prometheus Scrape Targets
| Job | Target | Interval | Notes |
|-----|--------|----------|-------|
| node_exporter | 7× :9100 | 30s | Named `node_exporter` (not `pve_nodes`) for dashboard compat |
| pve_api | :9221 | 30s | `metrics_path: /pve`, params: `cluster=1, node=1, target=<api-ip>:8006` |
| ceph | :9283 | 30s | Active mgr node |
| galera | 3× :9104 | 30s | UFW rule needed (allow from monitoring CT IP) |
| pbs-remote | :9101 | 30s | Via SSH tunnel, use Docker gateway IP `172.18.0.1:9101` |
| blackbox_icmp | via :9115 | 30s | ICMP probes to all infra nodes |
| blackbox_http | via :9115 | 30s | HTTP probes to external services |
| prometheus | :9090 | 30s | Self-monitoring |
## Telegram Alerting
Alertmanager doesn't natively support Telegram. Deploy a lightweight Flask
bridge container that:
1. Receives Alertmanager webhooks (POST to :9099)
2. Formats alerts as Markdown messages
3. Sends via Telegram Bot API `sendMessage`
Bot token from Hermes `.env` file (`TELEGRAM_BOT_TOKEN`). Chat ID: 223926918
(user's Telegram ID, same as `TELEGRAM_HOME_CHANNEL`).
Alert routing:
- **Critical** (host down, Ceph ERR, Galera split, PBS unreachable): immediate,
repeat every 1h
- **Warning** (disk >80%, Ceph WARN, slow ops): grouped 5min, repeat every 4h
## Alert Rules Summary
| Rule | Expression | Severity |
|------|-----------|----------|
| HostDown | `up{job="pve-nodes"} == 0` for 2m | critical |
| HighDiskUsage | disk >80% for 10m | warning |
| HighMemoryUsage | mem >85% for 10m | warning |
| ServiceUnreachable | `probe_success == 0` for 2m | critical |
| CephHealthError | `ceph_health_status == 2` for 5m | critical |
| CephHealthWarning | `ceph_health_status == 1` for 15m | warning |
| CephOSDDown | `ceph_osd_up < 1` for 5m | warning |
| GaleraNodeDown | `wsrep_ready == 0` for 2m | critical |
| GaleraClusterSize | `wsrep_cluster_size < 3` for 2m | critical |
| PBSLocalDown | `up{job="pbs-local"} == 0` for 5m | critical |
| PBSRemoteDown | `up{job="pbs-remote"} == 0` for 5m | critical |
| MaxScaleDown | `up{job="maxscale"} == 0` for 5m | warning |
## Grafana Dashboards
Auto-provisioned community dashboards (downloaded from grafana.com API):
- **Node Exporter Full** (ID 1860) — system metrics, uses `$job` + `$node` variables
- **Ceph Cluster** (ID 2842) — cluster overview, PATCHED for Ceph Squid (see pitfalls 32+)
- **MySQL Overview** (ID 7362) — Galera metrics, uses `$host` variable
- **Proxmox VE** (ID 10347) — PVE API metrics, uses `$instance` variable
Download command:
```bash
curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" \
-H "Accept: application/json" -o "dashboard_<ID>.json"
```
⚠️ Community dashboards may reference metrics that don't exist in your
exporter version. Always verify metric availability via the Prometheus
API before importing. Patch dashboard JSON in-place when metrics are
missing (see pitfall 33).
## Traefik Integration
Expose Grafana externally via Traefik (CT 99999) at
`grafana.familie-schoen.com`. Add dynamic config route to
`http://10.0.30.141:3000` with Let's Encrypt TLS.
## PVE Native Notifications
PVE has its own notification system. Forward PVE backup failures and HA
state changes to the Telegram bridge via webhook endpoint
(`http://10.0.30.141:9099/pve`). The bridge needs a separate handler for
PVE's webhook format (different from Alertmanager's).
## Pitfalls
1. **Remote PBS node_exporter on public IP** — Must restrict UFW to PVE WAN
IP. Without restriction, anyone can scrape system metrics.
2. **Ceph mgr failover moves :9283** — Active mgr can change. Add all
potential mgr nodes as scrape targets; Prometheus handles dedup.
3. **Alertmanager can't send to Telegram natively** — Need a bridge
container. The bridge must handle both Alertmanager and PVE webhook formats.
4. **PVE backup job "skip external VMs" masks failures** — See
`references/pbs-lxc-setup-2026-07.md` pitfall #12. A backup job showing
`status=OK` may have skipped most VMs/CTs on other nodes.
5. **PBS owner mismatch silently kills backups** — See
`references/pbs-lxc-setup-2026-07.md` pitfall #11. Always verify owner
files match the PVE storage username after credential changes.
6. **PVE node hostname ≠ IP last octet** — `proxmox4` is at `.40` not
`.70`, `proxmox7` is at `.70` not `.91`, `n5pro` is at `.91`. The
numeric suffix in the hostname does NOT correspond to the IP octet.
Always use the IP→name mapping from `pvecm nodes` + corosync.conf, or
verify with `hostname` after SSH. Wasted significant time SSH-ing to
wrong nodes during monitoring setup.
7. **VMs can show "running" in PVE but be network-dead** — MaxScale VM
310 showed `status: running` in PVE API but was completely unreachable
(no ping, no SSH, QGA not running). The VM may have hung at boot or
lost its network config. Don't assume a VM is functional just because
PVE says it's running — always verify network reachability separately.
This is exactly the kind of failure monitoring catches: deploy the
scrape config anyway so Prometheus alerts on the unreachable target.
8. **Docker container name conflicts after compose down** — When
`docker compose up` times out mid-pull and you retry, containers can
get stuck in "Created" state. `docker compose down` reports success
but `docker rm -f <name>` says "No such container" — yet the names
remain reserved and `compose up` fails with "container name already
in use". Fix: `systemctl restart docker` inside the CT, then
`docker compose up -d`. The daemon restart clears the stale name
reservations. Do NOT waste time trying to `docker rm` by truncated ID
— the IDs from the error message don't exist anymore.
9. **Prometheus permission denied on mounted config** — The Prometheus
container runs as UID 65534 (nobody) by default. Config files
created by root on the host with mode 600 cause `permission denied`
on startup, and the container enters a restart loop. Fix: either
`chmod 644` all config files on the host, or add `user: root` to
the prometheus service in docker-compose.yml. The latter is simpler
for a trusted internal monitoring CT.
10. **pve-exporter config mount path** — The `prompve/prometheus-pve-exporter`
Docker image hardcodes the config lookup at `/etc/prometheus/pve.yml`.
Even if you set `PVE_EXPORTER_CONFIG` env var to a different path,
the binary's default still looks for `/etc/prometheus/pve.yml` first.
Always mount the config to `/etc/prometheus/pve.yml:ro` inside the
container, regardless of the env var.
11. **PBS Remote behind OpenStack security group** — The remote PBS
(213.95.54.60) runs on an OpenStack VM. Port 9100 (node_exporter)
was blocked by the OpenStack security group, NOT by UFW (which was
inactive). Since we had no OpenStack SG access, the fix was an SSH
tunnel via autossh from the monitoring CT: `autossh -N -L
9101:localhost:9100 ubuntu@213.95.54.60`. Prometheus scrapes
`localhost:9101` instead of the remote directly. The SSH key must
be copied into the CT (it lives on the Hermes host, not on the PVE
node). Pattern: scp key to PVE jump host → `pct push` into CT.
12. **Grafana first-start SQLite migrations** — On first launch with a
fresh persistent volume, Grafana runs dozens of SQLite migration
steps that take 1–2+ minutes. During this time, HTTP requests to
:3000 return connection refused (curl `%{http_code}` = 000). This
is NOT a crash — check `docker logs grafana` for ongoing migration
log lines. Just wait and re-check. Do not restart the container.
13. **SCP to PVE jump host needs -i flag** — When copying files TO a
PVE node that requires a specific SSH key, `scp` also needs the
`-i` flag (same as ssh). `scp file root@10.0.20.70:/path` fails;
`scp -i ~/.ssh/id_ed25519_proxmox file root@10.0.20.70:/path` works.
14. **PVE /tmp can be full on jump hosts** — proxmox7 (10.0.20.70) had
a full /tmp partition. SCP to `/tmp/` failed with "No space left on
device". Use `/root/` or another writable path for temporary file
transfers through PVE jump hosts.
15. **Docker image pulls on slow CT connections** — Pulling 5+ Docker
images (Prometheus, Grafana, Alertmanager, Blackbox, PVE exporter,
Python base for telegram bridge) on a CT with limited bandwidth can
take 5–10+ minutes. The 300s default terminal timeout is insufficient.
Either set timeout=600 or run `docker compose pull` separately first,
then `docker compose up -d`. The `--build` flag for the telegram
bridge adds another 2+ minutes for pip install.
## Working Templates
- `templates/monitoring-docker-compose.yml` — Production-tested compose file with all fixes (user:root for Prometheus, correct pve-exporter mount path, custom telegram-bridge build)
- `templates/telegram-bridge.py` — Flask webhook bridge for Alertmanager → Telegram Bot API
## Related References
- `references/pbs-lxc-setup-2026-07.md` — PBS setup, PVE integration, owner mismatch pitfall
- `references/pbs-sync-pipeline-2026-07.md` — PBS sync pipeline, direct vzdump vs sync
## Additional Pitfalls (Session 2 — 2026-07-05)
16. **Galera UFW blocks mysqld_exporter from outside** — mysqld_exporter
binds to `*:9104` and works from `localhost`, but Galera nodes have UFW
active which blocks 9104 from other subnets. Prometheus scrapes fail
silently (empty response, no error). Fix: `ufw allow from
<monitoring-ct-ip> to any port 9104 proto tcp` on each Galera node.
Must be done via the Galera SSH key (1Password 'SSH-Key Galera',
debian user → sudo).
17. **Docker `localhost` inside a container ≠ CT host** — When Prometheus
runs in a Docker bridge network, `localhost:9101` in a scrape target
refers to the *container's* loopback, not the CT host. An SSH tunnel
listening on the CT's `0.0.0.0:9101` is invisible to the container.
Fix: use the Docker bridge gateway IP (`172.18.0.1:9101`) as the
scrape target. Find it with `docker network inspect
<network_name> | grep Gateway`. Alternatively, use `network_mode:
host` on the Prometheus container, but that breaks container
isolation and inter-container DNS.
18. **Grafana provisioning files need chmod 644** — Grafana container
runs as UID 472. Provisioning YAML files created by root with mode
600 cause silent startup failures (HTTP 000, no error in logs —
Grafana just hangs). The earlier `find . -name *.yml -exec chmod 644`
fix didn't catch files in `grafana/provisioning/` because the glob
only searched from the prometheus subdir. Fix: explicitly `chmod 644
/opt/monitoring/grafana/provisioning/**/*.yml` or use `find
/opt/monitoring -name "*.yml" -exec chmod 644 {} +`.
19. **PBS Local is on n5pro, not CT 116** — The reference doc originally
listed "PBS Local (CT 116)". In reality, CT 116 is on n5pro
(10.0.20.91) per `ha-manager status`. The earlier `pct list` on
proxmox7 didn't show it because CT 116 runs on n5pro, not proxmox7.
Always check `ha-manager status` for the *actual* node assignment
before trying to `pct exec` into a CT.
20. **PVE notification webhook setup** — PVE's native notification system
supports webhook endpoints. Creating the endpoint alone is NOT enough —
you also need a matcher to route notifications to it. Two steps:
```bash
# Step 1: Create the webhook endpoint
pvesh create /cluster/notifications/endpoints/webhook \
--name telegram-bridge \
--url http://10.0.30.141:9099/pve \
--method post
# ⚠️ --method must be LOWERCASE (post/put/get). UPPERCASE "POST" fails
# with: "value 'POST' does not have a value in the enumeration 'post, put, get'"
# Step 2: Create a matcher that routes all notifications to this endpoint
pvesh create /cluster/notifications/matchers \
--name telegram-route \
--target telegram-bridge \
--mode all \
--comment "Route all PVE notifications to Telegram"
```
Without Step 2, the endpoint exists but no notifications are routed to
it — PVE's default matcher only targets `mail-to-root`. The bridge's
`/pve` endpoint handles PVE's format (title/message/severity fields),
which differs from Alertmanager's alerts array.
⚠️ Ensure `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` are set in the
docker-compose env (or `.env` file) — empty values cause the bridge
to accept the webhook (HTTP 200) but silently fail the Telegram send.
⚠️ Telegram bots cannot initiate conversations — the user MUST send
`/start` to the bot first. Until then, `sendMessage` returns
`400 Bad Request: chat not found`. After `/start`, the bot can send
messages freely.
21. **Traefik on CT 99999, not a VM** — Traefik runs as a native binary
(systemd service) inside CT 99999 on proxmox7. Config at
`/etc/traefik/traefik.yaml` with dynamic configs in
`/etc/traefik/conf.d/`. Adding a new route is as simple as dropping
a YAML file into `conf.d/` — Traefik watches the directory and
auto-reloads. No restart needed. The ACME cert resolver
(`letsencrypt`) handles TLS automatically for new domains.
22. **Prometheus target list cleanup** — Dead targets (pbs-local CT 116
on n5pro, maxscale VIP .81 unreachable) should be removed from
prometheus.yml to avoid noise. But keep scrape configs for targets
that are temporarily down but expected to come back (maxscale) —
the `down` state itself is a useful alert. Removed pbs-local entirely
since CT 116 is being decommissioned. Kept pbs-remote via SSH tunnel.
## Additional Pitfalls (Session 3 — Dashboard Fixes 2026-07-05)
23. **pve-exporter config key is NOT `url` — it's implicit `host`** —
The `prompve/prometheus-pve-exporter` Docker image uses `proxmoxer`
which expects `host` as the first positional arg to `ProxmoxAPI()`.
The YAML config must NOT include a `url:` or `host:` key — the
exporter passes `host` from the scrape URL's `target` query param.
Correct config (`pve-exporter.yml`):
```yaml
default:
user: exporter@pve
token_name: monitoring
token_value: <token-uuid>
verify_ssl: false
```
If you include `url:` → `TypeError: unexpected keyword argument 'url'`.
If you include `host:` → `TypeError: multiple values for argument 'host'`.
Neither error is obvious from the generic 500 HTML response. Check
`docker logs pve-exporter` for the traceback.
24. **pve-exporter needs `target` param in scrape URL** — Without
`?target=<pve-api-host:port>` in the Prometheus scrape URL, the
exporter defaults to `localhost:8006` and gets connection refused.
The Prometheus job config must be:
```yaml
- job_name: pve_api
metrics_path: /pve
params:
cluster: [1]
node: [1]
target: ['10.0.20.70:8006']
static_configs:
- targets: ['pve-exporter:9221']
```
Without `metrics_path: /pve`, Prometheus scrapes `/metrics` (only
5 collector self-metrics). Without `cluster=1&node=1`, the cluster
and node collectors are skipped. Without `target=`, it tries
localhost:8006 and fails. All three params are required.
25. **Community Grafana dashboards hardcode job names** — Dashboard 1860
(Node Exporter Full) uses `$job` variable populated by
`label_values(node_uname_info, job)`. If your Prometheus job is
named `pve_nodes` instead of `node_exporter`, the dropdown populates
with `pve_nodes` but panel queries that hardcode `job="node_exporter"`
show no data. Fix: name the Prometheus job `node_exporter` (the
conventional name) so community dashboards work without modification.
Same principle for other dashboards: Galera dashboard 7362 uses
`label_values(mysql_up, instance)` — works as long as `mysql_up`
exists regardless of job name. PVE dashboard 10347 uses
`label_values(pve_node_info, instance)` — works once pve-exporter
is properly configured (see pitfalls 23–24).
26. **Prometheus reload via POST /-/reload sometimes silently fails** —
After editing `prometheus.yml`, `curl -X POST
http://localhost:9090/-/reload` returns success but the config
doesn't change (verified by checking scrape URLs via the targets
API). This happens when the YAML has a subtle issue (duplicate
keys, control characters from copy-paste) that Prometheus rejects
on parse but doesn't surface as an error to the reload endpoint.
Fix: `docker restart prometheus` forces a full config reload. If
that still doesn't work, exec into the container and run
`promtool check config /etc/prometheus/prometheus.yml` to find
the validation error.
27. **Old job labels persist in Prometheus after rename** — Renaming
a job from `pve_nodes` to `node_exporter` causes BOTH labels to
coexist in query results until the old TSDB samples age out
(retention period). `node_uname_info` will show series with
`job=pve_nodes` (cached) and `job=node_exporter` (new). This is
cosmetic — the old data expires naturally. Don't try to force-
purge; just wait.
28. **pve-exporter Docker image has split Python environments** — The
`prompve/prometheus-pve-exporter` image installs the exporter in
`/opt/prometheus-pve-exporter/` (virtualenv), NOT in the system
Python. `docker exec pve-exporter python3 -c "import ..."` uses
the wrong interpreter and gets ModuleNotFoundError. Use
`docker exec pve-exporter /opt/prometheus-pve-exporter/bin/python3`
or `docker exec pve-exporter /opt/prometheus-pve-exporter/bin/pve_exporter`
for introspection. Similarly, `pip list` shows only setuptools —
the real packages are in the venv's pip.
29. **PVE exporter collectors are URL-param gated, not CLI-flag gated** —
Unlike most Prometheus exporters where collectors are enabled via
`--collector.xxx` CLI flags, pve-exporter gates them via URL query
params: `?cluster=1` enables status/version/node/cluster/resources/
backup-info/qdevice collectors, `?node=1` enables config/replication/
subscription collectors. Without these params, you get only 5
self-metrics (collector duration + request errors). The `--help`
output lists `--collector.status` etc. but these are NOT how you
enable them in practice — the URL params are the mechanism.
30. **Grafana dashboard provisioning requires `provider.yml`** —
Dashboard JSON files in `/var/lib/grafana/dashboards/` are NOT
auto-discovered. Grafana needs a provisioning provider config at
`/etc/grafana/provisioning/dashboards/provider.yml` that points to
the dashboard directory. Without it, dashboards exist on disk but
never appear in the UI. The provider config specifies `path:
/var/lib/grafana/dashboards` (inside the container), which must
match the volume mount in docker-compose.yml.
31. **Verifying dashboard data availability** — Before declaring
dashboards fixed, query the Prometheus API for the specific metric
names the dashboard uses. For Node Exporter: `node_uname_info`,
`node_load1`. For Ceph: `ceph_health_status`, `ceph_pool_metadata`.
For Galera: `mysql_up`, `mysql_galera_status_info`. For PVE:
`pve_node_info`, `pve_cpu_usage_ratio`, `pve_up`. If the metric
exists in `label_values` API and has the expected labels, the
dashboard will populate. If not, trace the gap: exporter not
running → exporter running but wrong config → Prometheus not
scraping → scraping but wrong job name → job name correct but
dashboard template variable doesn't match.
32. **Ceph Squid prometheus module lacks OSD op counters** — Ceph 19.2
(Squid) prometheus module does NOT export `ceph_osd_op_r`,
`ceph_osd_op_w`, `ceph_osd_op_r_out_bytes`, `ceph_osd_op_w_in_bytes`,
`ceph_osd_op_r/w_latency_sum/count` by default. There is no config
option to enable them (`mgr/prometheus/experimental_perf_counters`
doesn't exist in Squid). Disabling/re-enabling the module doesn't help.
Available OSD metrics: `ceph_osd_apply_latency_ms`,
`ceph_osd_commit_latency_ms`, `ceph_osd_up`, `ceph_osd_in`,
`ceph_osd_weight`, flags, metadata. Available pool-level I/O:
`ceph_pool_rd`, `ceph_pool_wr`, `ceph_pool_rd_bytes`,
`ceph_pool_wr_bytes` (all counters, usable with `irate()`/`rate()`).
Community Ceph dashboards (ID 2842) that use `ceph_osd_op_*` show
"No data". Fix: patch the dashboard JSON to substitute pool-level
metrics. Mapping table:
- `ceph_osd_op_w_in_bytes` → `ceph_pool_wr_bytes`
- `ceph_osd_op_r_out_bytes` → `ceph_pool_rd_bytes`
- `ceph_osd_op_w` → `ceph_pool_wr`
- `ceph_osd_op_r` → `ceph_pool_rd`
- `ceph_osd_op_r/w_latency_sum/count` → `ceph_osd_commit_latency_ms`
or `ceph_osd_apply_latency_ms` (these are gauges, not sum/count
pairs — remove the `rate(sum)/rate(count)` formula and use `avg()`)
- `ceph_mon_num_sessions` → `count(ceph_mon_quorum_status)`
- `ceph_osd_numpg` → `sum(ceph_pg_total)`
33. **Patching Grafana dashboard JSON in-place** — When community
dashboards reference missing metrics, the fastest fix is to download
the dashboard JSON, patch all `expr` fields with `json.load` +
recursive walk + `str.replace`, and push it back to the
provisioning directory. Steps:
1. `curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" -o dashboard_<ID>.json`
2. Python: load JSON, walk all dicts looking for `"expr"` keys,
apply replacements, save
3. `scp` to PVE jump host → `pct push` into CT → `chmod 644`
4. Grafana auto-reloads provisioned dashboards every 30s
(`updateIntervalSeconds: 30` in provider.yml). If it doesn't,
`docker restart grafana` forces a reload.
5. Verify via browser: `document.body.innerText.match(/No data/gi).length`
in the Grafana page console — should be 0.
34. **Grafana dashboard provisioning reload timing** — After updating
a dashboard JSON file in the provisioning directory, Grafana picks
it up within `updateIntervalSeconds` (default 30s). But if the
file was modified while Grafana was running, it may not detect the
change immediately. `docker restart grafana` guarantees a reload
but takes 15-20s for startup + SQLite checks. After restart, verify
with the Grafana API: `curl -u admin:<pass>
http://localhost:3000/api/search` to confirm dashboards are listed.
35. **Browser-based Grafana dashboard verification** — When verifying
dashboards remotely, use the browser tools to navigate to the
dashboard URL, login with credentials, then run this in the page
console to count "No data" panels:
```javascript
(() => {
const txt = document.body.innerText;
const noData = (txt.match(/No data/gi) || []).length;
return JSON.stringify({noDataCount: noData});
})()
```
To identify WHICH panels show no data:
```javascript
(() => {
const p = document.querySelectorAll('.react-grid-item, [class*="panel-container"]');
const nd = [];
for (const panel of p) {
if (panel.innerText.includes('No data')) {
const t = panel.querySelector('.panel-title, h2, [class*="title"]');
nd.push(t ? t.textContent.trim() : '?');
}
}
return JSON.stringify(nd);
})()
```
This is faster than screenshots for identifying broken panels.
Note: panels inside collapsed rows won't be detected — expand all
rows first.
36. **Patching dashboards with latency formulas** — When substituting
`ceph_osd_op_r_latency_sum/count` (histogram-style sum+count pair)
with `ceph_osd_commit_latency_ms` (a simple gauge), the original
formula `rate(sum[5m]) / rate(count[5m])` must be replaced with
just `avg(ceph_osd_commit_latency_ms{})`. Keeping the division
formula with non-histogram metrics produces nonsensical values or
NaN. Always check the metric TYPE (counter vs gauge vs histogram)
before patching — `ceph_pool_rd` is a counter (use with `irate`),
`ceph_osd_commit_latency_ms` is a gauge (use directly or with
`avg()`).
## Dashboard Patching Reference
### Ceph Dashboard 2842 — Metric Substitution Table
| Original (missing) | Replacement (available) | Type Change |
|---|---|---|
| `ceph_osd_op_w_in_bytes` | `ceph_pool_wr_bytes` | counter → counter |
| `ceph_osd_op_r_out_bytes` | `ceph_pool_rd_bytes` | counter → counter |
| `ceph_osd_op_w` | `ceph_pool_wr` | counter → counter |
| `ceph_osd_op_r` | `ceph_pool_rd` | counter → counter |
| `ceph_osd_op_r_latency_sum/count` | `ceph_osd_commit_latency_ms` | histogram → gauge |
| `ceph_osd_op_w_latency_sum/count` | `ceph_osd_apply_latency_ms` | histogram → gauge |
| `ceph_mon_num_sessions` | `count(ceph_mon_quorum_status)` | gauge → count |
| `ceph_osd_numpg` | `sum(ceph_pg_total)` | gauge → sum |
### Verification Workflow
1. Query Prometheus API for available metrics:
`curl -s http://localhost:9090/api/v1/label/__name__/values | python3 -c "..."`
2. Check specific metric existence and label values
3. Test patched PromQL expressions directly via API
4. Push patched dashboard JSON to CT
5. Browser verify: login to Grafana, navigate to dashboard, run console
JS to count "No data" panels
6. If panels still show no data, check:
- Metric exists in Prometheus? (`label_values` API)
- Metric has expected labels? (`query` API)
- Dashboard template variable populates? (check dropdown)
- Panel query matches available data? (test in Grafana Explore)
## Additional Pitfalls (Session 4 — MySQL Dashboard Fix 2026-07-05)
37. **Imported dashboards with `__inputs` leave `${DS_PROMETHEUS}` unresolved**
— Community dashboards downloaded from grafana.com often ship with an
`__inputs` section declaring a datasource variable (e.g.
`{"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus"}`).
When imported via file provisioning (not the Grafana import UI), the
`${DS_PROMETHEUS}` placeholder in panel targets and template variables
is NEVER resolved — panels show "No data" because the datasource ref is
invalid. Fix: download the JSON, replace ALL `${DS_PROMETHEUS}`
references with the actual Prometheus datasource UID (found via
`GET /api/datasources`), then remove the `__inputs` section entirely.
The replacement must handle both string-format (`"${DS_PROMETHEUS}"`)
and object-format (`{"uid":"${DS_PROMETHEUS}"}`) datasource references.
Python walk-and-replace pattern:
```python
def fix_ds(obj):
if isinstance(obj, dict):
for k, v in obj.items():
if k == "datasource":
if isinstance(v, str) and "${DS_PROMETHEUS}" in v:
obj[k] = DS_UID
elif isinstance(v, dict) and v.get("uid") == "${DS_PROMETHEUS}":
obj[k] = {"type": "prometheus", "uid": DS_UID}
else:
fix_ds(v)
elif isinstance(obj, list):
for item in obj:
fix_ds(item)
```
38. **Template variables with empty `current` value produce no data** —
Even after fixing the datasource, a query-type template variable
(e.g. `host = label_values(mysql_up, instance)`) may have
`current: null` and `options: []` in the provisioned JSON. This means
`$host` expands to empty string in all panel queries
(`{instance=""}`), returning no data. The Grafana UI populates these
on first load, but file-provisioned dashboards skip that step. Fix:
pre-populate the `current` and `options` fields in the JSON with
known values BEFORE pushing to the provisioning directory:
```python
t["current"] = {"text": "10.0.30.71:9104", "value": "10.0.30.71:9104"}
t["options"] = [
{"selected": True, "text": "10.0.30.71:9104", "value": "10.0.30.71:9104"},
{"selected": False, "text": "10.0.30.72:9104", "value": "10.0.30.72:9104"},
{"selected": False, "text": "10.0.30.73:9104", "value": "10.0.30.73:9104"},
]
```
Grafana will still query the variable dynamically on page load, but
the pre-populated `current` ensures the initial render has data.
Users can then switch via the dropdown.
39. **Cross-network metric joins fail when instances differ** — The
MySQL Overview dashboard (ID 7362) has a "Buffer Pool Size of Total
RAM" panel that joins `mysql_global_variables_innodb_buffer_pool_size`
with `node_memory_MemTotal_bytes` using `on(instance)`. This fails
because MySQL exporter instances (`10.0.30.71-73:9104`) and
node_exporter instances (`10.0.20.x:9100`) are on completely
different networks — Galera nodes don't run node_exporter at all.
The `on(instance)` join matches nothing. Fix options:
a. Install node_exporter on Galera nodes (adds operational burden).
b. Use `label_replace` to strip ports and match by IP only (still
fails because IPs differ: 10.0.30.x vs 10.0.20.x).
c. Replace with a static constant if hardware specs are known (e.g.
`(buffer_pool_size * 100) / 8589934592` for 8GB RAM nodes).
Option (c) is the pragmatic fix for homelab clusters with uniform
hardware. Document the assumption in the panel description.
### MySQL Dashboard 7362 — Fixes Applied
| Issue | Fix |
|---|---|
| `${DS_PROMETHEUS}` unresolved in all panels | Replace with UID `PBFA97CFB590B2093`, remove `__inputs` |
| `$host` variable empty (`current: null`) | Pre-populate with 3 Galera instances, default `10.0.30.71:9104` |
| Buffer Pool % of RAM join fails (no node_exporter on Galera) | Static 8GB constant: `(buf_pool * 100) / 8589934592` |
### General Dashboard Import Checklist
When importing ANY community Grafana dashboard via file provisioning:
1. Check for `__inputs` section → remove it, hardcode datasource UID
2. Check all template variables → pre-populate `current` + `options` if empty
3. Check for cross-datasource joins (e.g. mysql + node) → verify both sides exist
4. Check for hardcoded job names (e.g. `job="node_exporter"`) → match your Prometheus config
5. Push to provisioning dir, `chmod 644`, restart Grafana
6. Browser verify: count "No data" panels via console JS
## Additional Pitfalls (Session 5 — Resume & Dashboard Variable Fixes 2026-07-05)
40. **Base64-through-SSH file transfer to CTs** — When you need to get a
Python script (or any file) into a CT but `pct push` fails because the
file isn't on the PVE host yet, and direct SCP to the PVE host `/tmp/`
may fail (permissions, disk space), use base64 encoding piped through
SSH directly into the CT:
```bash
SCRIPT=$(base64 -w0 /tmp/local_script.py)
ssh -i ~/.ssh/key root@10.0.20.70 "echo '$SCRIPT' | base64 -d | pct exec 141 -- tee /tmp/script.py >/dev/null && pct exec 141 -- python3 /tmp/script.py"
```
This bypasses the two-step (scp to PVE host → pct push into CT) entirely.
The `>/dev/null` suppresses the tee output. Works for any text file.
For binary files, use base64 with `--decode` on the receiving end.
41. **Grafana REST API for dashboard variable fixes** — Instead of
downloading dashboard JSON, patching it locally, and re-provisioning
via file (pitfalls 33-34, 37-38), you can fix dashboard variables
directly via the Grafana REST API. This is faster for surgical fixes:
```python
import json, urllib.request, base64
BASE = "http://127.0.0.1:3000"
AUTH = base64.b64encode(b"admin:Grafana2026!").decode()
PROM_UID = "PBFA97CFB590B2093"
def api_get(path):
req = urllib.request.Request(f"{BASE}{path}")
req.add_header("Authorization", f"Basic {AUTH}")
with urllib.request.urlopen(req) as resp:
return json.loads(resp.read())
def api_post(path, data):
body = json.dumps(data).encode()
req = urllib.request.Request(f"{BASE}{path}", data=body, method="POST")
req.add_header("Authorization", f"Basic {AUTH}")
req.add_header("Content-Type", "application/json")
with urllib.request.urlopen(req) as resp:
return json.loads(resp.read())
# GET dashboard → modify → POST back
d = api_get("/api/dashboards/uid/Dp7Cd57Zza")
dash = d.get("dashboard", {})
templating = dash.get("templating", {}).get("list", [])
# Add missing DS_PROMETHEUS variable
if "DS_PROMETHEUS" not in [v.get("name") for v in templating]:
templating.insert(0, {
"name": "DS_PROMETHEUS", "type": "datasource",
"query": "prometheus",
"current": {"text": "Prometheus", "value": PROM_UID},
})
# Fix empty current value on existing variable
for v in templating:
if v.get("name") == "ds_prometheus" and not v.get("current"):
v["current"] = {"text": "Prometheus", "value": PROM_UID}
result = api_post("/api/dashboards/db", {
"dashboard": dash, "message": "Fix datasource vars", "overwrite": True
})
```
Key endpoints:
- `GET /api/dashboards/uid/{uid}` — fetch dashboard JSON
- `POST /api/dashboards/db` — save modified dashboard (requires
`"overwrite": True` if the dashboard already exists)
- `GET /api/datasources` — list datasources to find the Prometheus UID
The dashboard JSON path is `response["dashboard"]` (NOT
`response["data"]["dashboard"]` — that's the file-provisioning format).
42. **Python `urllib` + `localhost` DNS resolution failure with embedded
credentials** — Using `urllib.request.urlopen` with a URL like
`http://admin:pass@localhost:3000/api/...` fails with
`socket.gaierror: [Errno -2] Name or service not known` in some
environments. The URL-embedded credentials cause urllib to parse
`localhost` as a hostname for DNS lookup rather than resolving it to
127.0.0.1. Fix: use `127.0.0.1` explicitly and pass credentials via
the Authorization header instead:
```python
import base64, urllib.request
AUTH = base64.b64encode(b"admin:password").decode()
req = urllib.request.Request("http://127.0.0.1:3000/api/...")
req.add_header("Authorization", f"Basic {AUTH}")
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read())
```
This is more robust than URL-embedded credentials and works in all
environments. The same issue affects `requests` library — always
prefer explicit headers over URL-embedded auth.
43. **Echo with parentheses breaks bash in nested SSH+pct exec** — When
running multi-line Python or bash scripts through nested SSH
(`ssh root@pve 'pct exec 141 -- bash -c "..."'`), `echo` statements
containing parentheses like `echo === Node Exporter (dashboard 1860)
===` cause `syntax error near unexpected token '('`. Bash interprets
the parentheses as subshell syntax. Fix: either escape parentheses
(`echo === Node Exporter \(dashboard 1860\) ===`), avoid parentheses
in echo strings, or — preferably — write the script to a local file,
base64-transfer it (pitfall 40), and execute it as a unit. The
base64 approach eliminates all quoting/escaping issues in nested SSH.
## Related References
- `references/openstack-offsite-pbs-2026-07.md` — Remote PBS provisioning
- `references/infra-monitoring-2026-07.md` — Full monitoring stack setup, dashboard patching guide (pitfalls 23-43), Ceph Squid metric substitution table, MySQL dashboard import checklist, Grafana API dashboard fixes