dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

40 KiB

Raw Blame History

Infrastructure Monitoring: Prometheus + Grafana Stack

When to Use

Deploying a full monitoring stack for Proxmox VE + Ceph + PBS + Galera/MaxScale infrastructure. Covers container provisioning, exporter deployment, alerting via Telegram, and Grafana dashboards.

Architecture

CT 141 (monitoring, 10.0.30.141/24, VLAN 30, 4CPU/8GB/30GB)
├── Prometheus (:9090) — 90-day TSDB retention
├── Grafana (:3000) — dashboards, via Traefik → grafana.familie-schoen.com
├── Alertmanager (:9093) — routes alerts to Telegram bridge
├── blackbox_exporter (:9115) — ICMP + HTTP probes for uptime
├── pve-exporter (:9221) — PVE API metrics (VM/CT status, resources)
├── telegram-bridge (:9099) — Flask webhook → Telegram Bot API
└── node_exporter (:9100) — local CT metrics

Exporters on targets:
├── 6× PVE Nodes — node_exporter :9100 (systemd service)
├── Ceph mgr — :9283 (built-in prometheus module, active mgr node)
├── 3× Galera — mysqld_exporter :9104 (MySQL/Galera metrics)
├── PBS Local (CT 116) — node_exporter :9100
└── PBS Remote (213.95.54.60) — node_exporter :9100 (UFW restricted)

Container Creation

pct create 141 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname monitoring \
  --memory 8192 --swap 4096 --cores 4 \
  --rootfs vm_disks:vm-141-disk-0,size=30G \
  --net0 name=eth0,bridge=vmbr0,ip=10.0.30.141/24,gw=10.0.30.1,tag=30,type=veth \
  --features nesting=1 \
  --onboot 1 --start 0

Install Docker inside the CT (standard Docker CE bookworm repo).

Exporter Deployment

node_exporter on PVE Nodes

Deploy via systemd service on each PVE node. Binary from GitHub releases (v1.8.2). Service file at /etc/systemd/system/node_exporter.service. Enable --collector.systemd and --collector.textfile.directory for custom textfile collectors.

PVE node IPs (VLAN 20 management network) — 8 nodes total:

proxmox1: 10.0.20.10
proxmox2: 10.0.20.20
proxmox3: 10.0.20.30
proxmox4: 10.0.20.40
proxmox5: 10.0.20.50
proxmox6: 10.0.20.60
proxmox7: 10.0.20.70 (primary jump host for SSH)
n5pro: 10.0.20.91

⚠️ Do NOT confuse node names with IPs — the numeric suffix in the hostname does NOT match the last octet. proxmox4 = .40, proxmox7 = .70, n5pro = .91. Always verify with hostname after SSH.

Ceph Prometheus Module

Built-in, just needs enabling:

ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/scrape_interval 15

Metrics available at http://<active-mgr-ip>:9283/metrics. Active mgr can fail over — add all potential mgr nodes as scrape targets.

mysqld_exporter on Galera

Requires exporter MySQL user on each Galera node:

CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

Binary: mysqld_exporter v0.15.1. Config at /etc/mysqld_exporter/my.cnf. Galera SSH access: 1Password 'SSH-Key Galera' (debian user, then sudo).

node_exporter on PBS Remote

Remote PBS is on public IP (213.95.54.60). Must restrict UFW to PVE WAN IP only — node_exporter exposes system information to anyone.

sudo ufw allow from <PVE_WAN_IP> to any port 9100

MaxScale Prometheus Metrics

MaxScale has a built-in Prometheus endpoint at :8189/metrics. No additional exporter needed — just add it as a scrape target.

# In prometheus.yml
  - job_name: maxscale
    static_configs:
      - targets: ["10.0.30.81:8189"]

⚠️ MaxScale VM 310 was unreachable as of 2026-07-05 — PVE shows status: running but no ping, no SSH, QGA not running. The VM may have hung at the bootloader or lost its network config. Until fixed, this target will show as down in Prometheus — which is actually useful as an alert. The scrape config should be deployed regardless; it becomes functional as soon as the VM is reachable.

MaxScale VM details:

VM 310 on proxmox4 (10.0.20.40)
IP: 10.0.30.81 (VLAN 30)
Metrics: http://10.0.30.81:8189/metrics
REST API: http://10.0.30.81:8989/

Docker Compose Stack

Services: prometheus, grafana, alertmanager, blackbox-exporter, telegram-bridge, pve-exporter.

Key config files:

prometheus/prometheus.yml — scrape configs for all targets
prometheus/rules/*.yml — alerting rules (general, ceph, galera, backup)
alertmanager/alertmanager.yml — routing to Telegram bridge
blackbox/blackbox.yml — ICMP + HTTP probe modules
grafana/provisioning/ — datasource + dashboard auto-provisioning

Prometheus Scrape Targets

Job	Target	Interval	Notes
node_exporter	7× :9100	30s	Named `node_exporter` (not `pve_nodes`) for dashboard compat
pve_api	:9221	30s	`metrics_path: /pve`, params: `cluster=1, node=1, target=<api-ip>:8006`
ceph	:9283	30s	Active mgr node
galera	3× :9104	30s	UFW rule needed (allow from monitoring CT IP)
pbs-remote	:9101	30s	Via SSH tunnel, use Docker gateway IP `172.18.0.1:9101`
blackbox_icmp	via :9115	30s	ICMP probes to all infra nodes
blackbox_http	via :9115	30s	HTTP probes to external services
prometheus	:9090	30s	Self-monitoring

Telegram Alerting

Alertmanager doesn't natively support Telegram. Deploy a lightweight Flask bridge container that:

Receives Alertmanager webhooks (POST to :9099)
Formats alerts as Markdown messages
Sends via Telegram Bot API sendMessage

Bot token from Hermes .env file (TELEGRAM_BOT_TOKEN). Chat ID: 223926918 (user's Telegram ID, same as TELEGRAM_HOME_CHANNEL).

Alert routing:

Critical (host down, Ceph ERR, Galera split, PBS unreachable): immediate, repeat every 1h
Warning (disk >80%, Ceph WARN, slow ops): grouped 5min, repeat every 4h

Alert Rules Summary

Rule	Expression	Severity
HostDown	`up{job="pve-nodes"} == 0` for 2m	critical
HighDiskUsage	disk >80% for 10m	warning
HighMemoryUsage	mem >85% for 10m	warning
ServiceUnreachable	`probe_success == 0` for 2m	critical
CephHealthError	`ceph_health_status == 2` for 5m	critical
CephHealthWarning	`ceph_health_status == 1` for 15m	warning
CephOSDDown	`ceph_osd_up < 1` for 5m	warning
GaleraNodeDown	`wsrep_ready == 0` for 2m	critical
GaleraClusterSize	`wsrep_cluster_size < 3` for 2m	critical
PBSLocalDown	`up{job="pbs-local"} == 0` for 5m	critical
PBSRemoteDown	`up{job="pbs-remote"} == 0` for 5m	critical
MaxScaleDown	`up{job="maxscale"} == 0` for 5m	warning

Grafana Dashboards

Auto-provisioned community dashboards (downloaded from grafana.com API):

Node Exporter Full (ID 1860) — system metrics, uses $job + $node variables
Ceph Cluster (ID 2842) — cluster overview, PATCHED for Ceph Squid (see pitfalls 32+)
MySQL Overview (ID 7362) — Galera metrics, uses $host variable
Proxmox VE (ID 10347) — PVE API metrics, uses $instance variable

Download command:

curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" \
  -H "Accept: application/json" -o "dashboard_<ID>.json"

⚠️ Community dashboards may reference metrics that don't exist in your exporter version. Always verify metric availability via the Prometheus API before importing. Patch dashboard JSON in-place when metrics are missing (see pitfall 33).

Traefik Integration

Expose Grafana externally via Traefik (CT 99999) at grafana.familie-schoen.com. Add dynamic config route to http://10.0.30.141:3000 with Let's Encrypt TLS.

PVE Native Notifications

PVE has its own notification system. Forward PVE backup failures and HA state changes to the Telegram bridge via webhook endpoint (http://10.0.30.141:9099/pve). The bridge needs a separate handler for PVE's webhook format (different from Alertmanager's).

Pitfalls

Remote PBS node_exporter on public IP — Must restrict UFW to PVE WAN IP. Without restriction, anyone can scrape system metrics.
Ceph mgr failover moves :9283 — Active mgr can change. Add all potential mgr nodes as scrape targets; Prometheus handles dedup.
Alertmanager can't send to Telegram natively — Need a bridge container. The bridge must handle both Alertmanager and PVE webhook formats.
PVE backup job "skip external VMs" masks failures — See references/pbs-lxc-setup-2026-07.md pitfall #12. A backup job showing status=OK may have skipped most VMs/CTs on other nodes.
PBS owner mismatch silently kills backups — See references/pbs-lxc-setup-2026-07.md pitfall #11. Always verify owner files match the PVE storage username after credential changes.
PVE node hostname ≠ IP last octet — proxmox4 is at .40 not .70, proxmox7 is at .70 not .91, n5pro is at .91. The numeric suffix in the hostname does NOT correspond to the IP octet. Always use the IP→name mapping from pvecm nodes + corosync.conf, or verify with hostname after SSH. Wasted significant time SSH-ing to wrong nodes during monitoring setup.
VMs can show "running" in PVE but be network-dead — MaxScale VM 310 showed status: running in PVE API but was completely unreachable (no ping, no SSH, QGA not running). The VM may have hung at boot or lost its network config. Don't assume a VM is functional just because PVE says it's running — always verify network reachability separately. This is exactly the kind of failure monitoring catches: deploy the scrape config anyway so Prometheus alerts on the unreachable target.
Docker container name conflicts after compose down — When docker compose up times out mid-pull and you retry, containers can get stuck in "Created" state. docker compose down reports success but docker rm -f <name> says "No such container" — yet the names remain reserved and compose up fails with "container name already in use". Fix: systemctl restart docker inside the CT, then docker compose up -d. The daemon restart clears the stale name reservations. Do NOT waste time trying to docker rm by truncated ID — the IDs from the error message don't exist anymore.
Prometheus permission denied on mounted config — The Prometheus container runs as UID 65534 (nobody) by default. Config files created by root on the host with mode 600 cause permission denied on startup, and the container enters a restart loop. Fix: either chmod 644 all config files on the host, or add user: root to the prometheus service in docker-compose.yml. The latter is simpler for a trusted internal monitoring CT.
pve-exporter config mount path — The prompve/prometheus-pve-exporter Docker image hardcodes the config lookup at /etc/prometheus/pve.yml. Even if you set PVE_EXPORTER_CONFIG env var to a different path, the binary's default still looks for /etc/prometheus/pve.yml first. Always mount the config to /etc/prometheus/pve.yml:ro inside the container, regardless of the env var.
PBS Remote behind OpenStack security group — The remote PBS (213.95.54.60) runs on an OpenStack VM. Port 9100 (node_exporter) was blocked by the OpenStack security group, NOT by UFW (which was inactive). Since we had no OpenStack SG access, the fix was an SSH tunnel via autossh from the monitoring CT: autossh -N -L 9101:localhost:9100 ubuntu@213.95.54.60. Prometheus scrapes localhost:9101 instead of the remote directly. The SSH key must be copied into the CT (it lives on the Hermes host, not on the PVE node). Pattern: scp key to PVE jump host → pct push into CT.
Grafana first-start SQLite migrations — On first launch with a fresh persistent volume, Grafana runs dozens of SQLite migration steps that take 1–2+ minutes. During this time, HTTP requests to :3000 return connection refused (curl %{http_code} = 000). This is NOT a crash — check docker logs grafana for ongoing migration log lines. Just wait and re-check. Do not restart the container.
SCP to PVE jump host needs -i flag — When copying files TO a PVE node that requires a specific SSH key, scp also needs the -i flag (same as ssh). scp file root@10.0.20.70:/path fails; scp -i ~/.ssh/id_ed25519_proxmox file root@10.0.20.70:/path works.
PVE /tmp can be full on jump hosts — proxmox7 (10.0.20.70) had a full /tmp partition. SCP to /tmp/ failed with "No space left on device". Use /root/ or another writable path for temporary file transfers through PVE jump hosts.
Docker image pulls on slow CT connections — Pulling 5+ Docker images (Prometheus, Grafana, Alertmanager, Blackbox, PVE exporter, Python base for telegram bridge) on a CT with limited bandwidth can take 5–10+ minutes. The 300s default terminal timeout is insufficient. Either set timeout=600 or run docker compose pull separately first, then docker compose up -d. The --build flag for the telegram bridge adds another 2+ minutes for pip install.

Working Templates

templates/monitoring-docker-compose.yml — Production-tested compose file with all fixes (user:root for Prometheus, correct pve-exporter mount path, custom telegram-bridge build)
templates/telegram-bridge.py — Flask webhook bridge for Alertmanager → Telegram Bot API

references/pbs-lxc-setup-2026-07.md — PBS setup, PVE integration, owner mismatch pitfall
references/pbs-sync-pipeline-2026-07.md — PBS sync pipeline, direct vzdump vs sync

Additional Pitfalls (Session 2 — 2026-07-05)

Galera UFW blocks mysqld_exporter from outside — mysqld_exporter binds to *:9104 and works from localhost, but Galera nodes have UFW active which blocks 9104 from other subnets. Prometheus scrapes fail silently (empty response, no error). Fix: ufw allow from <monitoring-ct-ip> to any port 9104 proto tcp on each Galera node. Must be done via the Galera SSH key (1Password 'SSH-Key Galera', debian user → sudo).
Docker localhost inside a container ≠ CT host — When Prometheus runs in a Docker bridge network, localhost:9101 in a scrape target refers to the container's loopback, not the CT host. An SSH tunnel listening on the CT's 0.0.0.0:9101 is invisible to the container. Fix: use the Docker bridge gateway IP (172.18.0.1:9101) as the scrape target. Find it with docker network inspect <network_name> | grep Gateway. Alternatively, use network_mode: host on the Prometheus container, but that breaks container isolation and inter-container DNS.
Grafana provisioning files need chmod 644 — Grafana container runs as UID 472. Provisioning YAML files created by root with mode 600 cause silent startup failures (HTTP 000, no error in logs — Grafana just hangs). The earlier find . -name *.yml -exec chmod 644 fix didn't catch files in grafana/provisioning/ because the glob only searched from the prometheus subdir. Fix: explicitly chmod 644 /opt/monitoring/grafana/provisioning/**/*.yml or use find /opt/monitoring -name "*.yml" -exec chmod 644 {} +.
PBS Local is on n5pro, not CT 116 — The reference doc originally listed "PBS Local (CT 116)". In reality, CT 116 is on n5pro (10.0.20.91) per ha-manager status. The earlier pct list on proxmox7 didn't show it because CT 116 runs on n5pro, not proxmox7. Always check ha-manager status for the actual node assignment before trying to pct exec into a CT.
PVE notification webhook setup — PVE's native notification system supports webhook endpoints. Creating the endpoint alone is NOT enough — you also need a matcher to route notifications to it. Two steps:
```
# Step 1: Create the webhook endpoint
pvesh create /cluster/notifications/endpoints/webhook \
  --name telegram-bridge \
  --url http://10.0.30.141:9099/pve \
  --method post
# ⚠️ --method must be LOWERCASE (post/put/get). UPPERCASE "POST" fails
# with: "value 'POST' does not have a value in the enumeration 'post, put, get'"

# Step 2: Create a matcher that routes all notifications to this endpoint
pvesh create /cluster/notifications/matchers \
  --name telegram-route \
  --target telegram-bridge \
  --mode all \
  --comment "Route all PVE notifications to Telegram"
```
Without Step 2, the endpoint exists but no notifications are routed to it — PVE's default matcher only targets mail-to-root. The bridge's /pve endpoint handles PVE's format (title/message/severity fields), which differs from Alertmanager's alerts array. ⚠️ Ensure TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID are set in the docker-compose env (or .env file) — empty values cause the bridge to accept the webhook (HTTP 200) but silently fail the Telegram send. ⚠️ Telegram bots cannot initiate conversations — the user MUST send /start to the bot first. Until then, sendMessage returns 400 Bad Request: chat not found. After /start, the bot can send messages freely.
Traefik on CT 99999, not a VM — Traefik runs as a native binary (systemd service) inside CT 99999 on proxmox7. Config at /etc/traefik/traefik.yaml with dynamic configs in /etc/traefik/conf.d/. Adding a new route is as simple as dropping a YAML file into conf.d/ — Traefik watches the directory and auto-reloads. No restart needed. The ACME cert resolver (letsencrypt) handles TLS automatically for new domains.
Prometheus target list cleanup — Dead targets (pbs-local CT 116 on n5pro, maxscale VIP .81 unreachable) should be removed from prometheus.yml to avoid noise. But keep scrape configs for targets that are temporarily down but expected to come back (maxscale) — the down state itself is a useful alert. Removed pbs-local entirely since CT 116 is being decommissioned. Kept pbs-remote via SSH tunnel.

Additional Pitfalls (Session 3 — Dashboard Fixes 2026-07-05)

pve-exporter config key is NOT url — it's implicit host — The prompve/prometheus-pve-exporter Docker image uses proxmoxer which expects host as the first positional arg to ProxmoxAPI(). The YAML config must NOT include a url: or host: key — the exporter passes host from the scrape URL's target query param. Correct config (pve-exporter.yml):
```
default:
  user: exporter@pve
  token_name: monitoring
  token_value: <token-uuid>
  verify_ssl: false
```
If you include url: → TypeError: unexpected keyword argument 'url'. If you include host: → TypeError: multiple values for argument 'host'. Neither error is obvious from the generic 500 HTML response. Check docker logs pve-exporter for the traceback.
pve-exporter needs target param in scrape URL — Without ?target=<pve-api-host:port> in the Prometheus scrape URL, the exporter defaults to localhost:8006 and gets connection refused. The Prometheus job config must be:
```
- job_name: pve_api
  metrics_path: /pve
  params:
    cluster: [1]
    node: [1]
    target: ['10.0.20.70:8006']
  static_configs:
    - targets: ['pve-exporter:9221']
```
Without metrics_path: /pve, Prometheus scrapes /metrics (only 5 collector self-metrics). Without cluster=1&node=1, the cluster and node collectors are skipped. Without target=, it tries localhost:8006 and fails. All three params are required.
Community Grafana dashboards hardcode job names — Dashboard 1860 (Node Exporter Full) uses $job variable populated by label_values(node_uname_info, job). If your Prometheus job is named pve_nodes instead of node_exporter, the dropdown populates with pve_nodes but panel queries that hardcode job="node_exporter" show no data. Fix: name the Prometheus job node_exporter (the conventional name) so community dashboards work without modification. Same principle for other dashboards: Galera dashboard 7362 uses label_values(mysql_up, instance) — works as long as mysql_up exists regardless of job name. PVE dashboard 10347 uses label_values(pve_node_info, instance) — works once pve-exporter is properly configured (see pitfalls 23–24).
Prometheus reload via POST /-/reload sometimes silently fails — After editing prometheus.yml, curl -X POST http://localhost:9090/-/reload returns success but the config doesn't change (verified by checking scrape URLs via the targets API). This happens when the YAML has a subtle issue (duplicate keys, control characters from copy-paste) that Prometheus rejects on parse but doesn't surface as an error to the reload endpoint. Fix: docker restart prometheus forces a full config reload. If that still doesn't work, exec into the container and run promtool check config /etc/prometheus/prometheus.yml to find the validation error.
Old job labels persist in Prometheus after rename — Renaming a job from pve_nodes to node_exporter causes BOTH labels to coexist in query results until the old TSDB samples age out (retention period). node_uname_info will show series with job=pve_nodes (cached) and job=node_exporter (new). This is cosmetic — the old data expires naturally. Don't try to force- purge; just wait.
pve-exporter Docker image has split Python environments — The prompve/prometheus-pve-exporter image installs the exporter in /opt/prometheus-pve-exporter/ (virtualenv), NOT in the system Python. docker exec pve-exporter python3 -c "import ..." uses the wrong interpreter and gets ModuleNotFoundError. Use docker exec pve-exporter /opt/prometheus-pve-exporter/bin/python3 or docker exec pve-exporter /opt/prometheus-pve-exporter/bin/pve_exporter for introspection. Similarly, pip list shows only setuptools — the real packages are in the venv's pip.
PVE exporter collectors are URL-param gated, not CLI-flag gated — Unlike most Prometheus exporters where collectors are enabled via --collector.xxx CLI flags, pve-exporter gates them via URL query params: ?cluster=1 enables status/version/node/cluster/resources/ backup-info/qdevice collectors, ?node=1 enables config/replication/ subscription collectors. Without these params, you get only 5 self-metrics (collector duration + request errors). The --help output lists --collector.status etc. but these are NOT how you enable them in practice — the URL params are the mechanism.
Grafana dashboard provisioning requires provider.yml — Dashboard JSON files in /var/lib/grafana/dashboards/ are NOT auto-discovered. Grafana needs a provisioning provider config at /etc/grafana/provisioning/dashboards/provider.yml that points to the dashboard directory. Without it, dashboards exist on disk but never appear in the UI. The provider config specifies path: /var/lib/grafana/dashboards (inside the container), which must match the volume mount in docker-compose.yml.
Verifying dashboard data availability — Before declaring dashboards fixed, query the Prometheus API for the specific metric names the dashboard uses. For Node Exporter: node_uname_info, node_load1. For Ceph: ceph_health_status, ceph_pool_metadata. For Galera: mysql_up, mysql_galera_status_info. For PVE: pve_node_info, pve_cpu_usage_ratio, pve_up. If the metric exists in label_values API and has the expected labels, the dashboard will populate. If not, trace the gap: exporter not running → exporter running but wrong config → Prometheus not scraping → scraping but wrong job name → job name correct but dashboard template variable doesn't match.
Ceph Squid prometheus module lacks OSD op counters — Ceph 19.2 (Squid) prometheus module does NOT export ceph_osd_op_r, ceph_osd_op_w, ceph_osd_op_r_out_bytes, ceph_osd_op_w_in_bytes, ceph_osd_op_r/w_latency_sum/count by default. There is no config option to enable them (mgr/prometheus/experimental_perf_counters doesn't exist in Squid). Disabling/re-enabling the module doesn't help. Available OSD metrics: ceph_osd_apply_latency_ms, ceph_osd_commit_latency_ms, ceph_osd_up, ceph_osd_in, ceph_osd_weight, flags, metadata. Available pool-level I/O: ceph_pool_rd, ceph_pool_wr, ceph_pool_rd_bytes, ceph_pool_wr_bytes (all counters, usable with irate()/rate()). Community Ceph dashboards (ID 2842) that use ceph_osd_op_* show "No data". Fix: patch the dashboard JSON to substitute pool-level metrics. Mapping table:
- ceph_osd_op_w_in_bytes → ceph_pool_wr_bytes
- ceph_osd_op_r_out_bytes → ceph_pool_rd_bytes
- ceph_osd_op_w → ceph_pool_wr
- ceph_osd_op_r → ceph_pool_rd
- ceph_osd_op_r/w_latency_sum/count → ceph_osd_commit_latency_ms or ceph_osd_apply_latency_ms (these are gauges, not sum/count pairs — remove the rate(sum)/rate(count) formula and use avg())
- ceph_mon_num_sessions → count(ceph_mon_quorum_status)
- ceph_osd_numpg → sum(ceph_pg_total)
Patching Grafana dashboard JSON in-place — When community dashboards reference missing metrics, the fastest fix is to download the dashboard JSON, patch all expr fields with json.load + recursive walk + str.replace, and push it back to the provisioning directory. Steps:
1. curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" -o dashboard_<ID>.json
2. Python: load JSON, walk all dicts looking for "expr" keys, apply replacements, save
3. scp to PVE jump host → pct push into CT → chmod 644
4. Grafana auto-reloads provisioned dashboards every 30s (updateIntervalSeconds: 30 in provider.yml). If it doesn't, docker restart grafana forces a reload.
5. Verify via browser: document.body.innerText.match(/No data/gi).length in the Grafana page console — should be 0.
Grafana dashboard provisioning reload timing — After updating a dashboard JSON file in the provisioning directory, Grafana picks it up within updateIntervalSeconds (default 30s). But if the file was modified while Grafana was running, it may not detect the change immediately. docker restart grafana guarantees a reload but takes 15-20s for startup + SQLite checks. After restart, verify with the Grafana API: curl -u admin:<pass> http://localhost:3000/api/search to confirm dashboards are listed.

Browser-based Grafana dashboard verification — When verifying dashboards remotely, use the browser tools to navigate to the dashboard URL, login with credentials, then run this in the page console to count "No data" panels:

(() => {
  const txt = document.body.innerText;
  const noData = (txt.match(/No data/gi) || []).length;
  return JSON.stringify({noDataCount: noData});
})()

To identify WHICH panels show no data:

(() => {
  const p = document.querySelectorAll('.react-grid-item, [class*="panel-container"]');
  const nd = [];
  for (const panel of p) {
    if (panel.innerText.includes('No data')) {
      const t = panel.querySelector('.panel-title, h2, [class*="title"]');
      nd.push(t ? t.textContent.trim() : '?');
    }
  }
  return JSON.stringify(nd);
})()

This is faster than screenshots for identifying broken panels. Note: panels inside collapsed rows won't be detected — expand all rows first.

Patching dashboards with latency formulas — When substituting ceph_osd_op_r_latency_sum/count (histogram-style sum+count pair) with ceph_osd_commit_latency_ms (a simple gauge), the original formula rate(sum[5m]) / rate(count[5m]) must be replaced with just avg(ceph_osd_commit_latency_ms{}). Keeping the division formula with non-histogram metrics produces nonsensical values or NaN. Always check the metric TYPE (counter vs gauge vs histogram) before patching — ceph_pool_rd is a counter (use with irate), ceph_osd_commit_latency_ms is a gauge (use directly or with avg()).

Dashboard Patching Reference

Ceph Dashboard 2842 — Metric Substitution Table

Original (missing)	Replacement (available)	Type Change
`ceph_osd_op_w_in_bytes`	`ceph_pool_wr_bytes`	counter → counter
`ceph_osd_op_r_out_bytes`	`ceph_pool_rd_bytes`	counter → counter
`ceph_osd_op_w`	`ceph_pool_wr`	counter → counter
`ceph_osd_op_r`	`ceph_pool_rd`	counter → counter
`ceph_osd_op_r_latency_sum/count`	`ceph_osd_commit_latency_ms`	histogram → gauge
`ceph_osd_op_w_latency_sum/count`	`ceph_osd_apply_latency_ms`	histogram → gauge
`ceph_mon_num_sessions`	`count(ceph_mon_quorum_status)`	gauge → count
`ceph_osd_numpg`	`sum(ceph_pg_total)`	gauge → sum

Verification Workflow

Query Prometheus API for available metrics: curl -s http://localhost:9090/api/v1/label/__name__/values | python3 -c "..."
Check specific metric existence and label values
Test patched PromQL expressions directly via API
Push patched dashboard JSON to CT
Browser verify: login to Grafana, navigate to dashboard, run console JS to count "No data" panels
If panels still show no data, check:
- Metric exists in Prometheus? (label_values API)
- Metric has expected labels? (query API)
- Dashboard template variable populates? (check dropdown)
- Panel query matches available data? (test in Grafana Explore)

Additional Pitfalls (Session 4 — MySQL Dashboard Fix 2026-07-05)

Imported dashboards with __inputs leave ${DS_PROMETHEUS} unresolved — Community dashboards downloaded from grafana.com often ship with an __inputs section declaring a datasource variable (e.g. {"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus"}). When imported via file provisioning (not the Grafana import UI), the ${DS_PROMETHEUS} placeholder in panel targets and template variables is NEVER resolved — panels show "No data" because the datasource ref is invalid. Fix: download the JSON, replace ALL ${DS_PROMETHEUS} references with the actual Prometheus datasource UID (found via GET /api/datasources), then remove the __inputs section entirely. The replacement must handle both string-format ("${DS_PROMETHEUS}") and object-format ({"uid":"${DS_PROMETHEUS}"}) datasource references. Python walk-and-replace pattern:
```
def fix_ds(obj):
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == "datasource":
                if isinstance(v, str) and "${DS_PROMETHEUS}" in v:
                    obj[k] = DS_UID
                elif isinstance(v, dict) and v.get("uid") == "${DS_PROMETHEUS}":
                    obj[k] = {"type": "prometheus", "uid": DS_UID}
            else:
                fix_ds(v)
    elif isinstance(obj, list):
        for item in obj:
            fix_ds(item)
```
Template variables with empty current value produce no data — Even after fixing the datasource, a query-type template variable (e.g. host = label_values(mysql_up, instance)) may have current: null and options: [] in the provisioned JSON. This means $host expands to empty string in all panel queries ({instance=""}), returning no data. The Grafana UI populates these on first load, but file-provisioned dashboards skip that step. Fix: pre-populate the current and options fields in the JSON with known values BEFORE pushing to the provisioning directory:
```
t["current"] = {"text": "10.0.30.71:9104", "value": "10.0.30.71:9104"}
t["options"] = [
    {"selected": True, "text": "10.0.30.71:9104", "value": "10.0.30.71:9104"},
    {"selected": False, "text": "10.0.30.72:9104", "value": "10.0.30.72:9104"},
    {"selected": False, "text": "10.0.30.73:9104", "value": "10.0.30.73:9104"},
]
```
Grafana will still query the variable dynamically on page load, but the pre-populated current ensures the initial render has data. Users can then switch via the dropdown.
Cross-network metric joins fail when instances differ — The MySQL Overview dashboard (ID 7362) has a "Buffer Pool Size of Total RAM" panel that joins mysql_global_variables_innodb_buffer_pool_size with node_memory_MemTotal_bytes using on(instance). This fails because MySQL exporter instances (10.0.30.71-73:9104) and node_exporter instances (10.0.20.x:9100) are on completely different networks — Galera nodes don't run node_exporter at all. The on(instance) join matches nothing. Fix options: a. Install node_exporter on Galera nodes (adds operational burden). b. Use label_replace to strip ports and match by IP only (still fails because IPs differ: 10.0.30.x vs 10.0.20.x). c. Replace with a static constant if hardware specs are known (e.g. (buffer_pool_size * 100) / 8589934592 for 8GB RAM nodes). Option (c) is the pragmatic fix for homelab clusters with uniform hardware. Document the assumption in the panel description.

MySQL Dashboard 7362 — Fixes Applied

Issue	Fix
`${DS_PROMETHEUS}` unresolved in all panels	Replace with UID `PBFA97CFB590B2093`, remove `__inputs`
`$host` variable empty (`current: null`)	Pre-populate with 3 Galera instances, default `10.0.30.71:9104`
Buffer Pool % of RAM join fails (no node_exporter on Galera)	Static 8GB constant: `(buf_pool * 100) / 8589934592`

General Dashboard Import Checklist

When importing ANY community Grafana dashboard via file provisioning:

Check for __inputs section → remove it, hardcode datasource UID
Check all template variables → pre-populate current + options if empty
Check for cross-datasource joins (e.g. mysql + node) → verify both sides exist
Check for hardcoded job names (e.g. job="node_exporter") → match your Prometheus config
Push to provisioning dir, chmod 644, restart Grafana
Browser verify: count "No data" panels via console JS

Additional Pitfalls (Session 5 — Resume & Dashboard Variable Fixes 2026-07-05)

Base64-through-SSH file transfer to CTs — When you need to get a Python script (or any file) into a CT but pct push fails because the file isn't on the PVE host yet, and direct SCP to the PVE host /tmp/ may fail (permissions, disk space), use base64 encoding piped through SSH directly into the CT:
```
SCRIPT=$(base64 -w0 /tmp/local_script.py)
ssh -i ~/.ssh/key root@10.0.20.70 "echo '$SCRIPT' | base64 -d | pct exec 141 -- tee /tmp/script.py >/dev/null && pct exec 141 -- python3 /tmp/script.py"
```
This bypasses the two-step (scp to PVE host → pct push into CT) entirely. The >/dev/null suppresses the tee output. Works for any text file. For binary files, use base64 with --decode on the receiving end.

Grafana REST API for dashboard variable fixes — Instead of downloading dashboard JSON, patching it locally, and re-provisioning via file (pitfalls 33-34, 37-38), you can fix dashboard variables directly via the Grafana REST API. This is faster for surgical fixes:

import json, urllib.request, base64

BASE = "http://127.0.0.1:3000"
AUTH = base64.b64encode(b"admin:Grafana2026!").decode()
PROM_UID = "PBFA97CFB590B2093"

def api_get(path):
    req = urllib.request.Request(f"{BASE}{path}")
    req.add_header("Authorization", f"Basic {AUTH}")
    with urllib.request.urlopen(req) as resp:
        return json.loads(resp.read())

def api_post(path, data):
    body = json.dumps(data).encode()
    req = urllib.request.Request(f"{BASE}{path}", data=body, method="POST")
    req.add_header("Authorization", f"Basic {AUTH}")
    req.add_header("Content-Type", "application/json")
    with urllib.request.urlopen(req) as resp:
        return json.loads(resp.read())

# GET dashboard → modify → POST back
d = api_get("/api/dashboards/uid/Dp7Cd57Zza")
dash = d.get("dashboard", {})
templating = dash.get("templating", {}).get("list", [])

# Add missing DS_PROMETHEUS variable
if "DS_PROMETHEUS" not in [v.get("name") for v in templating]:
    templating.insert(0, {
        "name": "DS_PROMETHEUS", "type": "datasource",
        "query": "prometheus",
        "current": {"text": "Prometheus", "value": PROM_UID},
    })

# Fix empty current value on existing variable
for v in templating:
    if v.get("name") == "ds_prometheus" and not v.get("current"):
        v["current"] = {"text": "Prometheus", "value": PROM_UID}

result = api_post("/api/dashboards/db", {
    "dashboard": dash, "message": "Fix datasource vars", "overwrite": True
})

Key endpoints:

GET /api/dashboards/uid/{uid} — fetch dashboard JSON
POST /api/dashboards/db — save modified dashboard (requires "overwrite": True if the dashboard already exists)
GET /api/datasources — list datasources to find the Prometheus UID The dashboard JSON path is response["dashboard"] (NOT response["data"]["dashboard"] — that's the file-provisioning format).

Python urllib + localhost DNS resolution failure with embedded credentials — Using urllib.request.urlopen with a URL like http://admin:pass@localhost:3000/api/... fails with socket.gaierror: [Errno -2] Name or service not known in some environments. The URL-embedded credentials cause urllib to parse localhost as a hostname for DNS lookup rather than resolving it to 127.0.0.1. Fix: use 127.0.0.1 explicitly and pass credentials via the Authorization header instead:
```
import base64, urllib.request
AUTH = base64.b64encode(b"admin:password").decode()
req = urllib.request.Request("http://127.0.0.1:3000/api/...")
req.add_header("Authorization", f"Basic {AUTH}")
with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
```
This is more robust than URL-embedded credentials and works in all environments. The same issue affects requests library — always prefer explicit headers over URL-embedded auth.
Echo with parentheses breaks bash in nested SSH+pct exec — When running multi-line Python or bash scripts through nested SSH (ssh root@pve 'pct exec 141 -- bash -c "..."'), echo statements containing parentheses like echo === Node Exporter (dashboard 1860) === cause syntax error near unexpected token '('. Bash interprets the parentheses as subshell syntax. Fix: either escape parentheses (echo === Node Exporter $dashboard 1860$ ===), avoid parentheses in echo strings, or — preferably — write the script to a local file, base64-transfer it (pitfall 40), and execute it as a unit. The base64 approach eliminates all quoting/escaping issues in nested SSH.

references/openstack-offsite-pbs-2026-07.md — Remote PBS provisioning
references/infra-monitoring-2026-07.md — Full monitoring stack setup, dashboard patching guide (pitfalls 23-43), Ceph Squid metric substitution table, MySQL dashboard import checklist, Grafana API dashboard fixes

40 KiB Raw Blame History Unescape Escape