# Infrastructure Monitoring: Prometheus + Grafana Stack

## When to Use

Deploying a full monitoring stack for Proxmox VE + Ceph + PBS + Galera/MaxScale
infrastructure. Covers container provisioning, exporter deployment, alerting
via Telegram, and Grafana dashboards.

## Architecture

```
CT 141 (monitoring, 10.0.30.141/24, VLAN 30, 4CPU/8GB/30GB)
├── Prometheus (:9090) — 90-day TSDB retention
├── Grafana (:3000) — dashboards, via Traefik → grafana.familie-schoen.com
├── Alertmanager (:9093) — routes alerts to Telegram bridge
├── blackbox_exporter (:9115) — ICMP + HTTP probes for uptime
├── pve-exporter (:9221) — PVE API metrics (VM/CT status, resources)
├── telegram-bridge (:9099) — Flask webhook → Telegram Bot API
└── node_exporter (:9100) — local CT metrics

Exporters on targets:
├── 6× PVE Nodes — node_exporter :9100 (systemd service)
├── Ceph mgr — :9283 (built-in prometheus module, active mgr node)
├── 3× Galera — mysqld_exporter :9104 (MySQL/Galera metrics)
├── PBS Local (CT 116) — node_exporter :9100
└── PBS Remote (213.95.54.60) — node_exporter :9100 (UFW restricted)
```

## Container Creation

```bash
pct create 141 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname monitoring \
  --memory 8192 --swap 4096 --cores 4 \
  --rootfs vm_disks:vm-141-disk-0,size=30G \
  --net0 name=eth0,bridge=vmbr0,ip=10.0.30.141/24,gw=10.0.30.1,tag=30,type=veth \
  --features nesting=1 \
  --onboot 1 --start 0
```

Install Docker inside the CT (standard Docker CE bookworm repo).

## Exporter Deployment

### node_exporter on PVE Nodes

Deploy via systemd service on each PVE node. Binary from GitHub releases
(v1.8.2). Service file at `/etc/systemd/system/node_exporter.service`.
Enable `--collector.systemd` and `--collector.textfile.directory` for
custom textfile collectors.

PVE node IPs (VLAN 20 management network) — 8 nodes total:
- proxmox1: 10.0.20.10
- proxmox2: 10.0.20.20
- proxmox3: 10.0.20.30
- proxmox4: 10.0.20.40
- proxmox5: 10.0.20.50
- proxmox6: 10.0.20.60
- proxmox7: 10.0.20.70  (primary jump host for SSH)
- n5pro: 10.0.20.91

⚠️ **Do NOT confuse node names with IPs** — the numeric suffix in the
hostname does NOT match the last octet. proxmox4 = .40, proxmox7 = .70,
n5pro = .91. Always verify with `hostname` after SSH.

### Ceph Prometheus Module

Built-in, just needs enabling:
```bash
ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/scrape_interval 15
```
Metrics available at `http://<active-mgr-ip>:9283/metrics`.
Active mgr can fail over — add all potential mgr nodes as scrape targets.

### mysqld_exporter on Galera

Requires exporter MySQL user on each Galera node:
```sql
CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
```
Binary: mysqld_exporter v0.15.1. Config at `/etc/mysqld_exporter/my.cnf`.
Galera SSH access: 1Password 'SSH-Key Galera' (debian user, then sudo).

### node_exporter on PBS Remote

Remote PBS is on public IP (213.95.54.60). **Must restrict UFW** to PVE
WAN IP only — node_exporter exposes system information to anyone.

```bash
sudo ufw allow from <PVE_WAN_IP> to any port 9100
```

### MaxScale Prometheus Metrics

MaxScale has a built-in Prometheus endpoint at `:8189/metrics`. No
additional exporter needed — just add it as a scrape target.

```yaml
# In prometheus.yml
  - job_name: maxscale
    static_configs:
      - targets: ["10.0.30.81:8189"]
```

⚠️ **MaxScale VM 310 was unreachable as of 2026-07-05** — PVE shows
`status: running` but no ping, no SSH, QGA not running. The VM may have
hung at the bootloader or lost its network config. Until fixed, this
target will show as `down` in Prometheus — which is actually useful as
an alert. The scrape config should be deployed regardless; it becomes
functional as soon as the VM is reachable.

MaxScale VM details:
- VM 310 on proxmox4 (10.0.20.40)
- IP: 10.0.30.81 (VLAN 30)
- Metrics: http://10.0.30.81:8189/metrics
- REST API: http://10.0.30.81:8989/

## Docker Compose Stack

Services: prometheus, grafana, alertmanager, blackbox-exporter,
telegram-bridge, pve-exporter.

Key config files:
- `prometheus/prometheus.yml` — scrape configs for all targets
- `prometheus/rules/*.yml` — alerting rules (general, ceph, galera, backup)
- `alertmanager/alertmanager.yml` — routing to Telegram bridge
- `blackbox/blackbox.yml` — ICMP + HTTP probe modules
- `grafana/provisioning/` — datasource + dashboard auto-provisioning

### Prometheus Scrape Targets

| Job | Target | Interval | Notes |
|-----|--------|----------|-------|
| node_exporter | 7× :9100 | 30s | Named `node_exporter` (not `pve_nodes`) for dashboard compat |
| pve_api | :9221 | 30s | `metrics_path: /pve`, params: `cluster=1, node=1, target=<api-ip>:8006` |
| ceph | :9283 | 30s | Active mgr node |
| galera | 3× :9104 | 30s | UFW rule needed (allow from monitoring CT IP) |
| pbs-remote | :9101 | 30s | Via SSH tunnel, use Docker gateway IP `172.18.0.1:9101` |
| blackbox_icmp | via :9115 | 30s | ICMP probes to all infra nodes |
| blackbox_http | via :9115 | 30s | HTTP probes to external services |
| prometheus | :9090 | 30s | Self-monitoring |

## Telegram Alerting

Alertmanager doesn't natively support Telegram. Deploy a lightweight Flask
bridge container that:
1. Receives Alertmanager webhooks (POST to :9099)
2. Formats alerts as Markdown messages
3. Sends via Telegram Bot API `sendMessage`

Bot token from Hermes `.env` file (`TELEGRAM_BOT_TOKEN`). Chat ID: 223926918
(user's Telegram ID, same as `TELEGRAM_HOME_CHANNEL`).

Alert routing:
- **Critical** (host down, Ceph ERR, Galera split, PBS unreachable): immediate,
  repeat every 1h
- **Warning** (disk >80%, Ceph WARN, slow ops): grouped 5min, repeat every 4h

## Alert Rules Summary

| Rule | Expression | Severity |
|------|-----------|----------|
| HostDown | `up{job="pve-nodes"} == 0` for 2m | critical |
| HighDiskUsage | disk >80% for 10m | warning |
| HighMemoryUsage | mem >85% for 10m | warning |
| ServiceUnreachable | `probe_success == 0` for 2m | critical |
| CephHealthError | `ceph_health_status == 2` for 5m | critical |
| CephHealthWarning | `ceph_health_status == 1` for 15m | warning |
| CephOSDDown | `ceph_osd_up < 1` for 5m | warning |
| GaleraNodeDown | `wsrep_ready == 0` for 2m | critical |
| GaleraClusterSize | `wsrep_cluster_size < 3` for 2m | critical |
| PBSLocalDown | `up{job="pbs-local"} == 0` for 5m | critical |
| PBSRemoteDown | `up{job="pbs-remote"} == 0` for 5m | critical |
| MaxScaleDown | `up{job="maxscale"} == 0` for 5m | warning |

## Grafana Dashboards

Auto-provisioned community dashboards (downloaded from grafana.com API):
- **Node Exporter Full** (ID 1860) — system metrics, uses `$job` + `$node` variables
- **Ceph Cluster** (ID 2842) — cluster overview, PATCHED for Ceph Squid (see pitfalls 32+)
- **MySQL Overview** (ID 7362) — Galera metrics, uses `$host` variable
- **Proxmox VE** (ID 10347) — PVE API metrics, uses `$instance` variable

Download command:
```bash
curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" \
  -H "Accept: application/json" -o "dashboard_<ID>.json"
```

⚠️ Community dashboards may reference metrics that don't exist in your
exporter version. Always verify metric availability via the Prometheus
API before importing. Patch dashboard JSON in-place when metrics are
missing (see pitfall 33).

## Traefik Integration

Expose Grafana externally via Traefik (CT 99999) at
`grafana.familie-schoen.com`. Add dynamic config route to
`http://10.0.30.141:3000` with Let's Encrypt TLS.

## PVE Native Notifications

PVE has its own notification system. Forward PVE backup failures and HA
state changes to the Telegram bridge via webhook endpoint
(`http://10.0.30.141:9099/pve`). The bridge needs a separate handler for
PVE's webhook format (different from Alertmanager's).

## Pitfalls

1. **Remote PBS node_exporter on public IP** — Must restrict UFW to PVE WAN
   IP. Without restriction, anyone can scrape system metrics.

2. **Ceph mgr failover moves :9283** — Active mgr can change. Add all
   potential mgr nodes as scrape targets; Prometheus handles dedup.

3. **Alertmanager can't send to Telegram natively** — Need a bridge
   container. The bridge must handle both Alertmanager and PVE webhook formats.

4. **PVE backup job "skip external VMs" masks failures** — See
   `references/pbs-lxc-setup-2026-07.md` pitfall #12. A backup job showing
   `status=OK` may have skipped most VMs/CTs on other nodes.

5. **PBS owner mismatch silently kills backups** — See
   `references/pbs-lxc-setup-2026-07.md` pitfall #11. Always verify owner
   files match the PVE storage username after credential changes.

6. **PVE node hostname ≠ IP last octet** — `proxmox4` is at `.40` not
   `.70`, `proxmox7` is at `.70` not `.91`, `n5pro` is at `.91`. The
   numeric suffix in the hostname does NOT correspond to the IP octet.
   Always use the IP→name mapping from `pvecm nodes` + corosync.conf, or
   verify with `hostname` after SSH. Wasted significant time SSH-ing to
   wrong nodes during monitoring setup.

7. **VMs can show "running" in PVE but be network-dead** — MaxScale VM
   310 showed `status: running` in PVE API but was completely unreachable
   (no ping, no SSH, QGA not running). The VM may have hung at boot or
   lost its network config. Don't assume a VM is functional just because
   PVE says it's running — always verify network reachability separately.
   This is exactly the kind of failure monitoring catches: deploy the
   scrape config anyway so Prometheus alerts on the unreachable target.

8. **Docker container name conflicts after compose down** — When
   `docker compose up` times out mid-pull and you retry, containers can
   get stuck in "Created" state. `docker compose down` reports success
   but `docker rm -f <name>` says "No such container" — yet the names
   remain reserved and `compose up` fails with "container name already
   in use". Fix: `systemctl restart docker` inside the CT, then
   `docker compose up -d`. The daemon restart clears the stale name
   reservations. Do NOT waste time trying to `docker rm` by truncated ID
   — the IDs from the error message don't exist anymore.

9. **Prometheus permission denied on mounted config** — The Prometheus
    container runs as UID 65534 (nobody) by default. Config files
    created by root on the host with mode 600 cause `permission denied`
    on startup, and the container enters a restart loop. Fix: either
    `chmod 644` all config files on the host, or add `user: root` to
    the prometheus service in docker-compose.yml. The latter is simpler
    for a trusted internal monitoring CT.

10. **pve-exporter config mount path** — The `prompve/prometheus-pve-exporter`
    Docker image hardcodes the config lookup at `/etc/prometheus/pve.yml`.
    Even if you set `PVE_EXPORTER_CONFIG` env var to a different path,
    the binary's default still looks for `/etc/prometheus/pve.yml` first.
    Always mount the config to `/etc/prometheus/pve.yml:ro` inside the
    container, regardless of the env var.

11. **PBS Remote behind OpenStack security group** — The remote PBS
    (213.95.54.60) runs on an OpenStack VM. Port 9100 (node_exporter)
    was blocked by the OpenStack security group, NOT by UFW (which was
    inactive). Since we had no OpenStack SG access, the fix was an SSH
    tunnel via autossh from the monitoring CT: `autossh -N -L
    9101:localhost:9100 ubuntu@213.95.54.60`. Prometheus scrapes
    `localhost:9101` instead of the remote directly. The SSH key must
    be copied into the CT (it lives on the Hermes host, not on the PVE
    node). Pattern: scp key to PVE jump host → `pct push` into CT.

12. **Grafana first-start SQLite migrations** — On first launch with a
    fresh persistent volume, Grafana runs dozens of SQLite migration
    steps that take 1–2+ minutes. During this time, HTTP requests to
    :3000 return connection refused (curl `%{http_code}` = 000). This
    is NOT a crash — check `docker logs grafana` for ongoing migration
    log lines. Just wait and re-check. Do not restart the container.

13. **SCP to PVE jump host needs -i flag** — When copying files TO a
    PVE node that requires a specific SSH key, `scp` also needs the
    `-i` flag (same as ssh). `scp file root@10.0.20.70:/path` fails;
    `scp -i ~/.ssh/id_ed25519_proxmox file root@10.0.20.70:/path` works.

14. **PVE /tmp can be full on jump hosts** — proxmox7 (10.0.20.70) had
    a full /tmp partition. SCP to `/tmp/` failed with "No space left on
    device". Use `/root/` or another writable path for temporary file
    transfers through PVE jump hosts.

15. **Docker image pulls on slow CT connections** — Pulling 5+ Docker
    images (Prometheus, Grafana, Alertmanager, Blackbox, PVE exporter,
    Python base for telegram bridge) on a CT with limited bandwidth can
    take 5–10+ minutes. The 300s default terminal timeout is insufficient.
    Either set timeout=600 or run `docker compose pull` separately first,
    then `docker compose up -d`. The `--build` flag for the telegram
    bridge adds another 2+ minutes for pip install.

## Working Templates

- `templates/monitoring-docker-compose.yml` — Production-tested compose file with all fixes (user:root for Prometheus, correct pve-exporter mount path, custom telegram-bridge build)
- `templates/telegram-bridge.py` — Flask webhook bridge for Alertmanager → Telegram Bot API

## Related References

- `references/pbs-lxc-setup-2026-07.md` — PBS setup, PVE integration, owner mismatch pitfall
- `references/pbs-sync-pipeline-2026-07.md` — PBS sync pipeline, direct vzdump vs sync

## Additional Pitfalls (Session 2 — 2026-07-05)

16. **Galera UFW blocks mysqld_exporter from outside** — mysqld_exporter
    binds to `*:9104` and works from `localhost`, but Galera nodes have UFW
    active which blocks 9104 from other subnets. Prometheus scrapes fail
    silently (empty response, no error). Fix: `ufw allow from
    <monitoring-ct-ip> to any port 9104 proto tcp` on each Galera node.
    Must be done via the Galera SSH key (1Password 'SSH-Key Galera',
    debian user → sudo).

17. **Docker `localhost` inside a container ≠ CT host** — When Prometheus
    runs in a Docker bridge network, `localhost:9101` in a scrape target
    refers to the *container's* loopback, not the CT host. An SSH tunnel
    listening on the CT's `0.0.0.0:9101` is invisible to the container.
    Fix: use the Docker bridge gateway IP (`172.18.0.1:9101`) as the
    scrape target. Find it with `docker network inspect
    <network_name> | grep Gateway`. Alternatively, use `network_mode:
    host` on the Prometheus container, but that breaks container
    isolation and inter-container DNS.

18. **Grafana provisioning files need chmod 644** — Grafana container
    runs as UID 472. Provisioning YAML files created by root with mode
    600 cause silent startup failures (HTTP 000, no error in logs —
    Grafana just hangs). The earlier `find . -name *.yml -exec chmod 644`
    fix didn't catch files in `grafana/provisioning/` because the glob
    only searched from the prometheus subdir. Fix: explicitly `chmod 644
    /opt/monitoring/grafana/provisioning/**/*.yml` or use `find
    /opt/monitoring -name "*.yml" -exec chmod 644 {} +`.

19. **PBS Local is on n5pro, not CT 116** — The reference doc originally
    listed "PBS Local (CT 116)". In reality, CT 116 is on n5pro
    (10.0.20.91) per `ha-manager status`. The earlier `pct list` on
    proxmox7 didn't show it because CT 116 runs on n5pro, not proxmox7.
    Always check `ha-manager status` for the *actual* node assignment
    before trying to `pct exec` into a CT.

20. **PVE notification webhook setup** — PVE's native notification system
    supports webhook endpoints. Creating the endpoint alone is NOT enough —
    you also need a matcher to route notifications to it. Two steps:
    ```bash
    # Step 1: Create the webhook endpoint
    pvesh create /cluster/notifications/endpoints/webhook \
      --name telegram-bridge \
      --url http://10.0.30.141:9099/pve \
      --method post
    # ⚠️ --method must be LOWERCASE (post/put/get). UPPERCASE "POST" fails
    # with: "value 'POST' does not have a value in the enumeration 'post, put, get'"

    # Step 2: Create a matcher that routes all notifications to this endpoint
    pvesh create /cluster/notifications/matchers \
      --name telegram-route \
      --target telegram-bridge \
      --mode all \
      --comment "Route all PVE notifications to Telegram"
    ```
    Without Step 2, the endpoint exists but no notifications are routed to
    it — PVE's default matcher only targets `mail-to-root`. The bridge's
    `/pve` endpoint handles PVE's format (title/message/severity fields),
    which differs from Alertmanager's alerts array.
    ⚠️ Ensure `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` are set in the
    docker-compose env (or `.env` file) — empty values cause the bridge
    to accept the webhook (HTTP 200) but silently fail the Telegram send.
    ⚠️ Telegram bots cannot initiate conversations — the user MUST send
    `/start` to the bot first. Until then, `sendMessage` returns
    `400 Bad Request: chat not found`. After `/start`, the bot can send
    messages freely.

21. **Traefik on CT 99999, not a VM** — Traefik runs as a native binary
    (systemd service) inside CT 99999 on proxmox7. Config at
    `/etc/traefik/traefik.yaml` with dynamic configs in
    `/etc/traefik/conf.d/`. Adding a new route is as simple as dropping
    a YAML file into `conf.d/` — Traefik watches the directory and
    auto-reloads. No restart needed. The ACME cert resolver
    (`letsencrypt`) handles TLS automatically for new domains.

22. **Prometheus target list cleanup** — Dead targets (pbs-local CT 116
    on n5pro, maxscale VIP .81 unreachable) should be removed from
    prometheus.yml to avoid noise. But keep scrape configs for targets
    that are temporarily down but expected to come back (maxscale) —
    the `down` state itself is a useful alert. Removed pbs-local entirely
    since CT 116 is being decommissioned. Kept pbs-remote via SSH tunnel.

## Additional Pitfalls (Session 3 — Dashboard Fixes 2026-07-05)

23. **pve-exporter config key is NOT `url` — it's implicit `host`** —
    The `prompve/prometheus-pve-exporter` Docker image uses `proxmoxer`
    which expects `host` as the first positional arg to `ProxmoxAPI()`.
    The YAML config must NOT include a `url:` or `host:` key — the
    exporter passes `host` from the scrape URL's `target` query param.
    Correct config (`pve-exporter.yml`):
    ```yaml
    default:
      user: exporter@pve
      token_name: monitoring
      token_value: <token-uuid>
      verify_ssl: false
    ```
    If you include `url:` → `TypeError: unexpected keyword argument 'url'`.
    If you include `host:` → `TypeError: multiple values for argument 'host'`.
    Neither error is obvious from the generic 500 HTML response. Check
    `docker logs pve-exporter` for the traceback.

24. **pve-exporter needs `target` param in scrape URL** — Without
    `?target=<pve-api-host:port>` in the Prometheus scrape URL, the
    exporter defaults to `localhost:8006` and gets connection refused.
    The Prometheus job config must be:
    ```yaml
    - job_name: pve_api
      metrics_path: /pve
      params:
        cluster: [1]
        node: [1]
        target: ['10.0.20.70:8006']
      static_configs:
        - targets: ['pve-exporter:9221']
    ```
    Without `metrics_path: /pve`, Prometheus scrapes `/metrics` (only
    5 collector self-metrics). Without `cluster=1&node=1`, the cluster
    and node collectors are skipped. Without `target=`, it tries
    localhost:8006 and fails. All three params are required.

25. **Community Grafana dashboards hardcode job names** — Dashboard 1860
    (Node Exporter Full) uses `$job` variable populated by
    `label_values(node_uname_info, job)`. If your Prometheus job is
    named `pve_nodes` instead of `node_exporter`, the dropdown populates
    with `pve_nodes` but panel queries that hardcode `job="node_exporter"`
    show no data. Fix: name the Prometheus job `node_exporter` (the
    conventional name) so community dashboards work without modification.
    Same principle for other dashboards: Galera dashboard 7362 uses
    `label_values(mysql_up, instance)` — works as long as `mysql_up`
    exists regardless of job name. PVE dashboard 10347 uses
    `label_values(pve_node_info, instance)` — works once pve-exporter
    is properly configured (see pitfalls 23–24).

26. **Prometheus reload via POST /-/reload sometimes silently fails** —
    After editing `prometheus.yml`, `curl -X POST
    http://localhost:9090/-/reload` returns success but the config
    doesn't change (verified by checking scrape URLs via the targets
    API). This happens when the YAML has a subtle issue (duplicate
    keys, control characters from copy-paste) that Prometheus rejects
    on parse but doesn't surface as an error to the reload endpoint.
    Fix: `docker restart prometheus` forces a full config reload. If
    that still doesn't work, exec into the container and run
    `promtool check config /etc/prometheus/prometheus.yml` to find
    the validation error.

27. **Old job labels persist in Prometheus after rename** — Renaming
    a job from `pve_nodes` to `node_exporter` causes BOTH labels to
    coexist in query results until the old TSDB samples age out
    (retention period). `node_uname_info` will show series with
    `job=pve_nodes` (cached) and `job=node_exporter` (new). This is
    cosmetic — the old data expires naturally. Don't try to force-
    purge; just wait.

28. **pve-exporter Docker image has split Python environments** — The
    `prompve/prometheus-pve-exporter` image installs the exporter in
    `/opt/prometheus-pve-exporter/` (virtualenv), NOT in the system
    Python. `docker exec pve-exporter python3 -c "import ..."` uses
    the wrong interpreter and gets ModuleNotFoundError. Use
    `docker exec pve-exporter /opt/prometheus-pve-exporter/bin/python3`
    or `docker exec pve-exporter /opt/prometheus-pve-exporter/bin/pve_exporter`
    for introspection. Similarly, `pip list` shows only setuptools —
    the real packages are in the venv's pip.

29. **PVE exporter collectors are URL-param gated, not CLI-flag gated** —
    Unlike most Prometheus exporters where collectors are enabled via
    `--collector.xxx` CLI flags, pve-exporter gates them via URL query
    params: `?cluster=1` enables status/version/node/cluster/resources/
    backup-info/qdevice collectors, `?node=1` enables config/replication/
    subscription collectors. Without these params, you get only 5
    self-metrics (collector duration + request errors). The `--help`
    output lists `--collector.status` etc. but these are NOT how you
    enable them in practice — the URL params are the mechanism.

30. **Grafana dashboard provisioning requires `provider.yml`** —
    Dashboard JSON files in `/var/lib/grafana/dashboards/` are NOT
    auto-discovered. Grafana needs a provisioning provider config at
    `/etc/grafana/provisioning/dashboards/provider.yml` that points to
    the dashboard directory. Without it, dashboards exist on disk but
    never appear in the UI. The provider config specifies `path:
    /var/lib/grafana/dashboards` (inside the container), which must
    match the volume mount in docker-compose.yml.

31. **Verifying dashboard data availability** — Before declaring
    dashboards fixed, query the Prometheus API for the specific metric
    names the dashboard uses. For Node Exporter: `node_uname_info`,
    `node_load1`. For Ceph: `ceph_health_status`, `ceph_pool_metadata`.
    For Galera: `mysql_up`, `mysql_galera_status_info`. For PVE:
    `pve_node_info`, `pve_cpu_usage_ratio`, `pve_up`. If the metric
    exists in `label_values` API and has the expected labels, the
    dashboard will populate. If not, trace the gap: exporter not
    running → exporter running but wrong config → Prometheus not
    scraping → scraping but wrong job name → job name correct but
    dashboard template variable doesn't match.
32. **Ceph Squid prometheus module lacks OSD op counters** — Ceph 19.2
    (Squid) prometheus module does NOT export `ceph_osd_op_r`,
    `ceph_osd_op_w`, `ceph_osd_op_r_out_bytes`, `ceph_osd_op_w_in_bytes`,
    `ceph_osd_op_r/w_latency_sum/count` by default. There is no config
    option to enable them (`mgr/prometheus/experimental_perf_counters`
    doesn't exist in Squid). Disabling/re-enabling the module doesn't help.
    Available OSD metrics: `ceph_osd_apply_latency_ms`,
    `ceph_osd_commit_latency_ms`, `ceph_osd_up`, `ceph_osd_in`,
    `ceph_osd_weight`, flags, metadata. Available pool-level I/O:
    `ceph_pool_rd`, `ceph_pool_wr`, `ceph_pool_rd_bytes`,
    `ceph_pool_wr_bytes` (all counters, usable with `irate()`/`rate()`).
    Community Ceph dashboards (ID 2842) that use `ceph_osd_op_*` show
    "No data". Fix: patch the dashboard JSON to substitute pool-level
    metrics. Mapping table:
    - `ceph_osd_op_w_in_bytes` → `ceph_pool_wr_bytes`
    - `ceph_osd_op_r_out_bytes` → `ceph_pool_rd_bytes`
    - `ceph_osd_op_w` → `ceph_pool_wr`
    - `ceph_osd_op_r` → `ceph_pool_rd`
    - `ceph_osd_op_r/w_latency_sum/count` → `ceph_osd_commit_latency_ms`
      or `ceph_osd_apply_latency_ms` (these are gauges, not sum/count
      pairs — remove the `rate(sum)/rate(count)` formula and use `avg()`)
    - `ceph_mon_num_sessions` → `count(ceph_mon_quorum_status)`
    - `ceph_osd_numpg` → `sum(ceph_pg_total)`

33. **Patching Grafana dashboard JSON in-place** — When community
    dashboards reference missing metrics, the fastest fix is to download
    the dashboard JSON, patch all `expr` fields with `json.load` +
    recursive walk + `str.replace`, and push it back to the
    provisioning directory. Steps:
    1. `curl -s "https://grafana.com/api/dashboards/<ID>/revisions/latest/download" -o dashboard_<ID>.json`
    2. Python: load JSON, walk all dicts looking for `"expr"` keys,
       apply replacements, save
    3. `scp` to PVE jump host → `pct push` into CT → `chmod 644`
    4. Grafana auto-reloads provisioned dashboards every 30s
       (`updateIntervalSeconds: 30` in provider.yml). If it doesn't,
       `docker restart grafana` forces a reload.
    5. Verify via browser: `document.body.innerText.match(/No data/gi).length`
       in the Grafana page console — should be 0.

34. **Grafana dashboard provisioning reload timing** — After updating
    a dashboard JSON file in the provisioning directory, Grafana picks
    it up within `updateIntervalSeconds` (default 30s). But if the
    file was modified while Grafana was running, it may not detect the
    change immediately. `docker restart grafana` guarantees a reload
    but takes 15-20s for startup + SQLite checks. After restart, verify
    with the Grafana API: `curl -u admin:<pass>
    http://localhost:3000/api/search` to confirm dashboards are listed.

35. **Browser-based Grafana dashboard verification** — When verifying
    dashboards remotely, use the browser tools to navigate to the
    dashboard URL, login with credentials, then run this in the page
    console to count "No data" panels:
    ```javascript
    (() => {
      const txt = document.body.innerText;
      const noData = (txt.match(/No data/gi) || []).length;
      return JSON.stringify({noDataCount: noData});
    })()
    ```
    To identify WHICH panels show no data:
    ```javascript
    (() => {
      const p = document.querySelectorAll('.react-grid-item, [class*="panel-container"]');
      const nd = [];
      for (const panel of p) {
        if (panel.innerText.includes('No data')) {
          const t = panel.querySelector('.panel-title, h2, [class*="title"]');
          nd.push(t ? t.textContent.trim() : '?');
        }
      }
      return JSON.stringify(nd);
    })()
    ```
    This is faster than screenshots for identifying broken panels.
    Note: panels inside collapsed rows won't be detected — expand all
    rows first.

36. **Patching dashboards with latency formulas** — When substituting
    `ceph_osd_op_r_latency_sum/count` (histogram-style sum+count pair)
    with `ceph_osd_commit_latency_ms` (a simple gauge), the original
    formula `rate(sum[5m]) / rate(count[5m])` must be replaced with
    just `avg(ceph_osd_commit_latency_ms{})`. Keeping the division
    formula with non-histogram metrics produces nonsensical values or
    NaN. Always check the metric TYPE (counter vs gauge vs histogram)
    before patching — `ceph_pool_rd` is a counter (use with `irate`),
    `ceph_osd_commit_latency_ms` is a gauge (use directly or with
    `avg()`).

## Dashboard Patching Reference

### Ceph Dashboard 2842 — Metric Substitution Table

| Original (missing) | Replacement (available) | Type Change |
|---|---|---|
| `ceph_osd_op_w_in_bytes` | `ceph_pool_wr_bytes` | counter → counter |
| `ceph_osd_op_r_out_bytes` | `ceph_pool_rd_bytes` | counter → counter |
| `ceph_osd_op_w` | `ceph_pool_wr` | counter → counter |
| `ceph_osd_op_r` | `ceph_pool_rd` | counter → counter |
| `ceph_osd_op_r_latency_sum/count` | `ceph_osd_commit_latency_ms` | histogram → gauge |
| `ceph_osd_op_w_latency_sum/count` | `ceph_osd_apply_latency_ms` | histogram → gauge |
| `ceph_mon_num_sessions` | `count(ceph_mon_quorum_status)` | gauge → count |
| `ceph_osd_numpg` | `sum(ceph_pg_total)` | gauge → sum |

### Verification Workflow

1. Query Prometheus API for available metrics:
   `curl -s http://localhost:9090/api/v1/label/__name__/values | python3 -c "..."`
2. Check specific metric existence and label values
3. Test patched PromQL expressions directly via API
4. Push patched dashboard JSON to CT
5. Browser verify: login to Grafana, navigate to dashboard, run console
   JS to count "No data" panels
6. If panels still show no data, check:
   - Metric exists in Prometheus? (`label_values` API)
   - Metric has expected labels? (`query` API)
   - Dashboard template variable populates? (check dropdown)
   - Panel query matches available data? (test in Grafana Explore)

## Additional Pitfalls (Session 4 — MySQL Dashboard Fix 2026-07-05)

37. **Imported dashboards with `__inputs` leave `${DS_PROMETHEUS}` unresolved**
    — Community dashboards downloaded from grafana.com often ship with an
    `__inputs` section declaring a datasource variable (e.g.
    `{"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus"}`).
    When imported via file provisioning (not the Grafana import UI), the
    `${DS_PROMETHEUS}` placeholder in panel targets and template variables
    is NEVER resolved — panels show "No data" because the datasource ref is
    invalid. Fix: download the JSON, replace ALL `${DS_PROMETHEUS}`
    references with the actual Prometheus datasource UID (found via
    `GET /api/datasources`), then remove the `__inputs` section entirely.
    The replacement must handle both string-format (`"${DS_PROMETHEUS}"`)
    and object-format (`{"uid":"${DS_PROMETHEUS}"}`) datasource references.
    Python walk-and-replace pattern:
    ```python
    def fix_ds(obj):
        if isinstance(obj, dict):
            for k, v in obj.items():
                if k == "datasource":
                    if isinstance(v, str) and "${DS_PROMETHEUS}" in v:
                        obj[k] = DS_UID
                    elif isinstance(v, dict) and v.get("uid") == "${DS_PROMETHEUS}":
                        obj[k] = {"type": "prometheus", "uid": DS_UID}
                else:
                    fix_ds(v)
        elif isinstance(obj, list):
            for item in obj:
                fix_ds(item)
    ```

38. **Template variables with empty `current` value produce no data** —
    Even after fixing the datasource, a query-type template variable
    (e.g. `host = label_values(mysql_up, instance)`) may have
    `current: null` and `options: []` in the provisioned JSON. This means
    `$host` expands to empty string in all panel queries
    (`{instance=""}`), returning no data. The Grafana UI populates these
    on first load, but file-provisioned dashboards skip that step. Fix:
    pre-populate the `current` and `options` fields in the JSON with
    known values BEFORE pushing to the provisioning directory:
    ```python
    t["current"] = {"text": "10.0.30.71:9104", "value": "10.0.30.71:9104"}
    t["options"] = [
        {"selected": True, "text": "10.0.30.71:9104", "value": "10.0.30.71:9104"},
        {"selected": False, "text": "10.0.30.72:9104", "value": "10.0.30.72:9104"},
        {"selected": False, "text": "10.0.30.73:9104", "value": "10.0.30.73:9104"},
    ]
    ```
    Grafana will still query the variable dynamically on page load, but
    the pre-populated `current` ensures the initial render has data.
    Users can then switch via the dropdown.

39. **Cross-network metric joins fail when instances differ** — The
    MySQL Overview dashboard (ID 7362) has a "Buffer Pool Size of Total
    RAM" panel that joins `mysql_global_variables_innodb_buffer_pool_size`
    with `node_memory_MemTotal_bytes` using `on(instance)`. This fails
    because MySQL exporter instances (`10.0.30.71-73:9104`) and
    node_exporter instances (`10.0.20.x:9100`) are on completely
    different networks — Galera nodes don't run node_exporter at all.
    The `on(instance)` join matches nothing. Fix options:
    a. Install node_exporter on Galera nodes (adds operational burden).
    b. Use `label_replace` to strip ports and match by IP only (still
       fails because IPs differ: 10.0.30.x vs 10.0.20.x).
    c. Replace with a static constant if hardware specs are known (e.g.
       `(buffer_pool_size * 100) / 8589934592` for 8GB RAM nodes).
    Option (c) is the pragmatic fix for homelab clusters with uniform
    hardware. Document the assumption in the panel description.

### MySQL Dashboard 7362 — Fixes Applied

| Issue | Fix |
|---|---|
| `${DS_PROMETHEUS}` unresolved in all panels | Replace with UID `PBFA97CFB590B2093`, remove `__inputs` |
| `$host` variable empty (`current: null`) | Pre-populate with 3 Galera instances, default `10.0.30.71:9104` |
| Buffer Pool % of RAM join fails (no node_exporter on Galera) | Static 8GB constant: `(buf_pool * 100) / 8589934592` |

### General Dashboard Import Checklist

When importing ANY community Grafana dashboard via file provisioning:
1. Check for `__inputs` section → remove it, hardcode datasource UID
2. Check all template variables → pre-populate `current` + `options` if empty
3. Check for cross-datasource joins (e.g. mysql + node) → verify both sides exist
4. Check for hardcoded job names (e.g. `job="node_exporter"`) → match your Prometheus config
5. Push to provisioning dir, `chmod 644`, restart Grafana
6. Browser verify: count "No data" panels via console JS
## Additional Pitfalls (Session 5 — Resume & Dashboard Variable Fixes 2026-07-05)

40. **Base64-through-SSH file transfer to CTs** — When you need to get a
    Python script (or any file) into a CT but `pct push` fails because the
    file isn't on the PVE host yet, and direct SCP to the PVE host `/tmp/`
    may fail (permissions, disk space), use base64 encoding piped through
    SSH directly into the CT:
    ```bash
    SCRIPT=$(base64 -w0 /tmp/local_script.py)
    ssh -i ~/.ssh/key root@10.0.20.70 "echo '$SCRIPT' | base64 -d | pct exec 141 -- tee /tmp/script.py >/dev/null && pct exec 141 -- python3 /tmp/script.py"
    ```
    This bypasses the two-step (scp to PVE host → pct push into CT) entirely.
    The `>/dev/null` suppresses the tee output. Works for any text file.
    For binary files, use base64 with `--decode` on the receiving end.

41. **Grafana REST API for dashboard variable fixes** — Instead of
    downloading dashboard JSON, patching it locally, and re-provisioning
    via file (pitfalls 33-34, 37-38), you can fix dashboard variables
    directly via the Grafana REST API. This is faster for surgical fixes:
    ```python
    import json, urllib.request, base64

    BASE = "http://127.0.0.1:3000"
    AUTH = base64.b64encode(b"admin:Grafana2026!").decode()
    PROM_UID = "PBFA97CFB590B2093"

    def api_get(path):
        req = urllib.request.Request(f"{BASE}{path}")
        req.add_header("Authorization", f"Basic {AUTH}")
        with urllib.request.urlopen(req) as resp:
            return json.loads(resp.read())

    def api_post(path, data):
        body = json.dumps(data).encode()
        req = urllib.request.Request(f"{BASE}{path}", data=body, method="POST")
        req.add_header("Authorization", f"Basic {AUTH}")
        req.add_header("Content-Type", "application/json")
        with urllib.request.urlopen(req) as resp:
            return json.loads(resp.read())

    # GET dashboard → modify → POST back
    d = api_get("/api/dashboards/uid/Dp7Cd57Zza")
    dash = d.get("dashboard", {})
    templating = dash.get("templating", {}).get("list", [])

    # Add missing DS_PROMETHEUS variable
    if "DS_PROMETHEUS" not in [v.get("name") for v in templating]:
        templating.insert(0, {
            "name": "DS_PROMETHEUS", "type": "datasource",
            "query": "prometheus",
            "current": {"text": "Prometheus", "value": PROM_UID},
        })

    # Fix empty current value on existing variable
    for v in templating:
        if v.get("name") == "ds_prometheus" and not v.get("current"):
            v["current"] = {"text": "Prometheus", "value": PROM_UID}

    result = api_post("/api/dashboards/db", {
        "dashboard": dash, "message": "Fix datasource vars", "overwrite": True
    })
    ```
    Key endpoints:
    - `GET /api/dashboards/uid/{uid}` — fetch dashboard JSON
    - `POST /api/dashboards/db` — save modified dashboard (requires
      `"overwrite": True` if the dashboard already exists)
    - `GET /api/datasources` — list datasources to find the Prometheus UID
    The dashboard JSON path is `response["dashboard"]` (NOT
    `response["data"]["dashboard"]` — that's the file-provisioning format).

42. **Python `urllib` + `localhost` DNS resolution failure with embedded
    credentials** — Using `urllib.request.urlopen` with a URL like
    `http://admin:pass@localhost:3000/api/...` fails with
    `socket.gaierror: [Errno -2] Name or service not known` in some
    environments. The URL-embedded credentials cause urllib to parse
    `localhost` as a hostname for DNS lookup rather than resolving it to
    127.0.0.1. Fix: use `127.0.0.1` explicitly and pass credentials via
    the Authorization header instead:
    ```python
    import base64, urllib.request
    AUTH = base64.b64encode(b"admin:password").decode()
    req = urllib.request.Request("http://127.0.0.1:3000/api/...")
    req.add_header("Authorization", f"Basic {AUTH}")
    with urllib.request.urlopen(req) as resp:
        data = json.loads(resp.read())
    ```
    This is more robust than URL-embedded credentials and works in all
    environments. The same issue affects `requests` library — always
    prefer explicit headers over URL-embedded auth.

43. **Echo with parentheses breaks bash in nested SSH+pct exec** — When
    running multi-line Python or bash scripts through nested SSH
    (`ssh root@pve 'pct exec 141 -- bash -c "..."'`), `echo` statements
    containing parentheses like `echo === Node Exporter (dashboard 1860)
    ===` cause `syntax error near unexpected token '('`. Bash interprets
    the parentheses as subshell syntax. Fix: either escape parentheses
    (`echo === Node Exporter \(dashboard 1860\) ===`), avoid parentheses
    in echo strings, or — preferably — write the script to a local file,
    base64-transfer it (pitfall 40), and execute it as a unit. The
    base64 approach eliminates all quoting/escaping issues in nested SSH.

## Related References

- `references/openstack-offsite-pbs-2026-07.md` — Remote PBS provisioning
- `references/infra-monitoring-2026-07.md` — Full monitoring stack setup, dashboard patching guide (pitfalls 23-43), Ceph Squid metric substitution table, MySQL dashboard import checklist, Grafana API dashboard fixes
