Files
memory/patterns/1password-rate-limit.md
T
2026-11-06 22:00:26 +00:00

3.3 KiB
Raw Blame History

pattern_id, title, category, severity, status, first_observed, last_updated, related_systems, related_solution_docs, related_skills
pattern_id title category severity status first_observed last_updated related_systems related_solution_docs related_skills
PAT-003 1Password CLI rate-limit is account-level — stop ESO, wait 1 hour tooling high active 2026-07 2026-08-30
rke2-kubernetes
docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md
docs/solutions/bug-fixes/2026-07-14-1password-token-telegram-corruption.md
1password-cli

1Password CLI rate-limit is account-level — stop ESO, wait 1 hour

Symptom

All 1Password CLI (op) calls suddenly fail with rate-limit errors. This affects both Hermes tools and ExternalSecrets Operator (ESO) in K8s simultaneously. The outage lasts approximately 1 hour.

Root Cause

The 1Password CLI enforces rate limits at the account level, not per-token or per-client. When ESO polls aggressively (multiple SecretStores + frequent sync intervals), it exhausts the quota for the ENTIRE account, blocking all other op callers including Hermes automation.

Mitigation

  1. Stop ESO immediately — scale down the ESO deployment to halt API calls:
    kubectl scale deploy -n external-secrets external-secrets-operator --replicas=0
    
  2. Wait 1 hour — the rate limit resets automatically. No polling needed.
  3. Resume ESO with reduced polling frequency afterward

Prevention

  • Reduce ESO refreshInterval to 10+ minutes (not the default 1 minute)
  • Limit the number of SecretStores that reference 1Password
  • Consider caching secrets locally to reduce API pressure
  • Never run op in tight loops — always add delays for batch operations
  • Nie op read pro Watchdog-Tick: statische Secrets (HA-Long-Lived-Token, API-Keys) in einen lokalen 0600-Cache legen, op nur on-miss/401. Ein 15-Minuten-Watchdog macht sonst 96 Reads/Tag und wird bei jedem Limit blind (2026-11-06: free@home-Watchdog meldete „BLIND (RuntimeError)").
  • Bei anhaltendem Limit (>10 min ohne lokalen Verursacher) zuerst ESO prüfen: kubectl logs -n external-secrets deploy/external-secrets --since=10m | grep -c "rate limit exceeded". Fehlschlagende ExternalSecrets mit kurzem refreshInterval erzeugen einen Retry-Sturm, der die Quote dauerhaft verbraucht.
  • 5 ES mit refreshInterval: 1h (authelia-oidc-jwks/-secrets, postgres-main-db, paperless-ai-secret/-oidc-secret) sind die üblichen Verdächtigen — Rest steht auf 12h.

Evidence

  • Hit twice: Jul 2026 (initial discovery) and during token corruption investigation
  • Solution doc: docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md
  • MEMORY.md entry: "1P CLI:3 vaults R/W. Rate-limit=acct-level. Stop ESO, wait 1hr, no polling."
  • 2026-11-06: free@home-Watchdog (Cron 43612c6d52b6) täglich „BLIND: kann HA-Status nicht prüfen (RuntimeError)" — Ursache op read pro 15-min-Tick. Parallel: ESO (onepasswordSDK, ClusterSecretStore onepassword-store) im Retry-Sturm — 38 × „rate limit exceeded" in 10 min, 5 ExternalSecrets SecretSyncedError bei refreshInterval: 1h. HA selbst gesund (Entry freeathome = loaded, SysAP HTTP 200). Fix Watchdog: Cache-first-Token + dynamischer PVE-Node (VM 106 war von ms-a2-2 auf proxmox3 gewandert, hardcodierter Node → Rettungs-Reboot unmöglich). SolDoc docs/solutions/bug-fixes/2026-11-06-fah-watchdog-blind-1p-ratelimit-node.md.