56 lines
3.3 KiB
Markdown
56 lines
3.3 KiB
Markdown
---
|
||
pattern_id: PAT-003
|
||
title: "1Password CLI rate-limit is account-level — stop ESO, wait 1 hour"
|
||
category: tooling
|
||
severity: high
|
||
status: active
|
||
first_observed: 2026-07
|
||
last_updated: 2026-08-30
|
||
related_systems: [rke2-kubernetes]
|
||
related_solution_docs:
|
||
- docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md
|
||
- docs/solutions/bug-fixes/2026-07-14-1password-token-telegram-corruption.md
|
||
related_skills: [1password-cli]
|
||
---
|
||
|
||
# 1Password CLI rate-limit is account-level — stop ESO, wait 1 hour
|
||
|
||
## Symptom
|
||
|
||
All 1Password CLI (`op`) calls suddenly fail with rate-limit errors. This affects
|
||
both Hermes tools and ExternalSecrets Operator (ESO) in K8s simultaneously.
|
||
The outage lasts approximately 1 hour.
|
||
|
||
## Root Cause
|
||
|
||
The 1Password CLI enforces rate limits at the **account level**, not per-token or
|
||
per-client. When ESO polls aggressively (multiple SecretStores + frequent sync intervals),
|
||
it exhausts the quota for the ENTIRE account, blocking all other `op` callers including
|
||
Hermes automation.
|
||
|
||
## Mitigation
|
||
|
||
1. **Stop ESO immediately** — scale down the ESO deployment to halt API calls:
|
||
```bash
|
||
kubectl scale deploy -n external-secrets external-secrets-operator --replicas=0
|
||
```
|
||
2. **Wait 1 hour** — the rate limit resets automatically. No polling needed.
|
||
3. **Resume ESO** with reduced polling frequency afterward
|
||
|
||
## Prevention
|
||
|
||
- Reduce ESO `refreshInterval` to 10+ minutes (not the default 1 minute)
|
||
- Limit the number of SecretStores that reference 1Password
|
||
- Consider caching secrets locally to reduce API pressure
|
||
- Never run `op` in tight loops — always add delays for batch operations
|
||
- **Nie `op read` pro Watchdog-Tick:** statische Secrets (HA-Long-Lived-Token, API-Keys) in einen lokalen 0600-Cache legen, `op` nur on-miss/401. Ein 15-Minuten-Watchdog macht sonst 96 Reads/Tag und wird bei jedem Limit blind (2026-11-06: free@home-Watchdog meldete „BLIND (RuntimeError)").
|
||
- **Bei anhaltendem Limit (>10 min ohne lokalen Verursacher) zuerst ESO prüfen:** `kubectl logs -n external-secrets deploy/external-secrets --since=10m | grep -c "rate limit exceeded"`. Fehlschlagende ExternalSecrets mit kurzem `refreshInterval` erzeugen einen Retry-Sturm, der die Quote dauerhaft verbraucht.
|
||
- 5 ES mit `refreshInterval: 1h` (authelia-oidc-jwks/-secrets, postgres-main-db, paperless-ai-secret/-oidc-secret) sind die üblichen Verdächtigen — Rest steht auf 12h.
|
||
|
||
## Evidence
|
||
|
||
- Hit twice: Jul 2026 (initial discovery) and during token corruption investigation
|
||
- Solution doc: `docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md`
|
||
- MEMORY.md entry: "1P CLI:3 vaults R/W. Rate-limit=acct-level. Stop ESO, wait 1hr, no polling."
|
||
- **2026-11-06:** free@home-Watchdog (Cron `43612c6d52b6`) täglich „BLIND: kann HA-Status nicht prüfen (RuntimeError)" — Ursache `op read` pro 15-min-Tick. Parallel: ESO (`onepasswordSDK`, ClusterSecretStore `onepassword-store`) im Retry-Sturm — 38 × „rate limit exceeded" in 10 min, 5 ExternalSecrets `SecretSyncedError` bei `refreshInterval: 1h`. HA selbst gesund (Entry `freeathome` = `loaded`, SysAP HTTP 200). Fix Watchdog: Cache-first-Token + dynamischer PVE-Node (VM 106 war von ms-a2-2 auf proxmox3 gewandert, hardcodierter Node → Rettungs-Reboot unmöglich). SolDoc `docs/solutions/bug-fixes/2026-11-06-fah-watchdog-blind-1p-ratelimit-node.md`.
|