Files
memory/patterns/1password-rate-limit.md
T
2026-11-06 22:00:26 +00:00

56 lines
3.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
pattern_id: PAT-003
title: "1Password CLI rate-limit is account-level — stop ESO, wait 1 hour"
category: tooling
severity: high
status: active
first_observed: 2026-07
last_updated: 2026-08-30
related_systems: [rke2-kubernetes]
related_solution_docs:
- docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md
- docs/solutions/bug-fixes/2026-07-14-1password-token-telegram-corruption.md
related_skills: [1password-cli]
---
# 1Password CLI rate-limit is account-level — stop ESO, wait 1 hour
## Symptom
All 1Password CLI (`op`) calls suddenly fail with rate-limit errors. This affects
both Hermes tools and ExternalSecrets Operator (ESO) in K8s simultaneously.
The outage lasts approximately 1 hour.
## Root Cause
The 1Password CLI enforces rate limits at the **account level**, not per-token or
per-client. When ESO polls aggressively (multiple SecretStores + frequent sync intervals),
it exhausts the quota for the ENTIRE account, blocking all other `op` callers including
Hermes automation.
## Mitigation
1. **Stop ESO immediately** — scale down the ESO deployment to halt API calls:
```bash
kubectl scale deploy -n external-secrets external-secrets-operator --replicas=0
```
2. **Wait 1 hour** — the rate limit resets automatically. No polling needed.
3. **Resume ESO** with reduced polling frequency afterward
## Prevention
- Reduce ESO `refreshInterval` to 10+ minutes (not the default 1 minute)
- Limit the number of SecretStores that reference 1Password
- Consider caching secrets locally to reduce API pressure
- Never run `op` in tight loops — always add delays for batch operations
- **Nie `op read` pro Watchdog-Tick:** statische Secrets (HA-Long-Lived-Token, API-Keys) in einen lokalen 0600-Cache legen, `op` nur on-miss/401. Ein 15-Minuten-Watchdog macht sonst 96 Reads/Tag und wird bei jedem Limit blind (2026-11-06: free@home-Watchdog meldete „BLIND (RuntimeError)").
- **Bei anhaltendem Limit (>10 min ohne lokalen Verursacher) zuerst ESO prüfen:** `kubectl logs -n external-secrets deploy/external-secrets --since=10m | grep -c "rate limit exceeded"`. Fehlschlagende ExternalSecrets mit kurzem `refreshInterval` erzeugen einen Retry-Sturm, der die Quote dauerhaft verbraucht.
- 5 ES mit `refreshInterval: 1h` (authelia-oidc-jwks/-secrets, postgres-main-db, paperless-ai-secret/-oidc-secret) sind die üblichen Verdächtigen — Rest steht auf 12h.
## Evidence
- Hit twice: Jul 2026 (initial discovery) and during token corruption investigation
- Solution doc: `docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md`
- MEMORY.md entry: "1P CLI:3 vaults R/W. Rate-limit=acct-level. Stop ESO, wait 1hr, no polling."
- **2026-11-06:** free@home-Watchdog (Cron `43612c6d52b6`) täglich „BLIND: kann HA-Status nicht prüfen (RuntimeError)" — Ursache `op read` pro 15-min-Tick. Parallel: ESO (`onepasswordSDK`, ClusterSecretStore `onepassword-store`) im Retry-Sturm — 38 × „rate limit exceeded" in 10 min, 5 ExternalSecrets `SecretSyncedError` bei `refreshInterval: 1h`. HA selbst gesund (Entry `freeathome` = `loaded`, SysAP HTTP 200). Fix Watchdog: Cache-first-Token + dynamischer PVE-Node (VM 106 war von ms-a2-2 auf proxmox3 gewandert, hardcodierter Node → Rettungs-Reboot unmöglich). SolDoc `docs/solutions/bug-fixes/2026-11-06-fah-watchdog-blind-1p-ratelimit-node.md`.