Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,91 @@
+# Chefkoch Scraper Session: v4 First Production Deploy
+
+## Session Date: 2026-06-18
+## Trigger: User requested continuous scraping to 25,000 recipes
+## Final State: Scraper running as nohup daemon (PID 80297) + watchdog cronjob
+
+---
+
+## Problem Discovery Log (Failure Chain)
+
+### Failure 1: LLM-based cronjob fails on provider mismatch
+- Cronjob `chefkoch-real-recipes-refill` (job_id: c77127dd9130)
+- 06:01: `Error 400: model is required` — no model configured on job
+- Fix attempted: set `model="moonshotai/kimi-k2.6"` — BUT omitted `provider`
+- 06:17: `RuntimeError: No Anthropic credentials found` — system defaulted to anthropic provider
+
+**Lesson:** Model + provider MUST be set as a PAIR on agent cronjobs. Setting only one creates a mismatch.
+
+### Failure 2: `terminal(background=true)` silently killed scraper
+- Attempted to start scraper via `terminal(background=true)`
+- Result: Exit code 143 (SIGTERM after kill)
+- Also: `tcsetattr: Inappropriate ioctl for device` — pseudo-tty interferes with Playwright
+
+**Lesson:** Background process mode is NOT a nohup replacement. Use explicit `nohup` wrapper script.
+
+### Failure 3: Playwright browser not found
+- Playwright installed in nutrition-coach venv, but chromium not found
+- Error: `Executable doesn't exist at ...chrome-headless-shell`
+- Browser actually installed under `/home/debian/.cache/ms-playwright/` (installed globally)
+- venv looking elsewhere
+
+**Lesson:** Set `PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright` explicitly. Do NOT rely on auto-detection when multiple Python environments exist.
+
+---
+
+## Working Solution (Final)
+
+### 1. Standalone Scraper Script
+- `~/.hermes/scripts/chefkoch_scraper_v4.py` — headless, no LLM, asyncio
+- Reads/writes directly to `nutrition-coach` profile dir
+- Uses JSON-LD extraction (fast), Playwright only for URL discovery
+
+### 2. nohup Wrapper Script
+- `~/.hermes/scripts/start_scraper_v4.sh`
+- Sets `PLAYWRIGHT_BROWSERS_PATH` before launching
+- PID tracking in `/tmp/chefkoch_scraper_v4.pid`
+- Idempotent: checks if already running
+
+### 3. Watchdog Cronjob (no_agent)
+- `chefkoch-scraper-watchdog` (job_id: a884c342c49e)
+- Runs every 10 minutes
+- Calls wrapper script
+- `no_agent: true` — zero LLM cost
+- `deliver: local` — no notification spam
+
+### 4. Original Cronjob (agent-based) deprecated
+- Original `chefkoch-real-recipes-refill` still exists but should be paused
+- The LLM-based approach (checking count + starting scraper) is overkill
+- Direct nohup + watchdog is cheaper and more reliable
+
+---
+
+## Performance Observed
+
+- Discovery: ~50 URLs per batch from search pages
+- Extraction: ~60 recipes/minute with single context
+- 5,000 recipes in ~60–70 minutes
+- 25,000 recipe ETA: ~7~10 hours continuous
+
+---
+
+## Session-Context References
+
+- `real_recipes.jsonl` — append-only output file
+- `scraper_state.json` — tracks seed_cycle for resume
+- `scraper_v4.out` — live log file
+- `scraper.log` — legacy log from batch run (still there)
+
+---
+
+## New Pitfalls Captured in Skills
+
+1. **multi-source-recipe-scraper / infinite-refill-scraper-v4.md:**
+   - Browser path detection when multiple venvs exist
+   - `nohup` wrapper pattern for daemon mode
+   - `terminal(background=true)` tcsetattr conflict with Playwright
+   - Watchdog cronjob pattern
+
+2. **cronjob / SKILL.md:**
+   - Model+provider MUST be set as pair on agent jobs
+   - Custom provider mismatch error signature