Chefkoch Scraper Session: v4 First Production Deploy

Session Date: 2026-06-18

Trigger: User requested continuous scraping to 25,000 recipes

Final State: Scraper running as nohup daemon (PID 80297) + watchdog cronjob

Problem Discovery Log (Failure Chain)

Failure 1: LLM-based cronjob fails on provider mismatch

Cronjob chefkoch-real-recipes-refill (job_id: c77127dd9130)
06:01: Error 400: model is required — no model configured on job
Fix attempted: set model="moonshotai/kimi-k2.6" — BUT omitted provider
06:17: RuntimeError: No Anthropic credentials found — system defaulted to anthropic provider

Lesson: Model + provider MUST be set as a PAIR on agent cronjobs. Setting only one creates a mismatch.

Failure 2: `terminal(background=true)` silently killed scraper

Attempted to start scraper via terminal(background=true)
Result: Exit code 143 (SIGTERM after kill)
Also: tcsetattr: Inappropriate ioctl for device — pseudo-tty interferes with Playwright

Lesson: Background process mode is NOT a nohup replacement. Use explicit nohup wrapper script.

Failure 3: Playwright browser not found

Playwright installed in nutrition-coach venv, but chromium not found
Error: Executable doesn't exist at ...chrome-headless-shell
Browser actually installed under /home/debian/.cache/ms-playwright/ (installed globally)
venv looking elsewhere

Lesson: Set PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright explicitly. Do NOT rely on auto-detection when multiple Python environments exist.

Working Solution (Final)

1. Standalone Scraper Script

~/.hermes/scripts/chefkoch_scraper_v4.py — headless, no LLM, asyncio
Reads/writes directly to nutrition-coach profile dir
Uses JSON-LD extraction (fast), Playwright only for URL discovery

2. nohup Wrapper Script

~/.hermes/scripts/start_scraper_v4.sh
Sets PLAYWRIGHT_BROWSERS_PATH before launching
PID tracking in /tmp/chefkoch_scraper_v4.pid
Idempotent: checks if already running

3. Watchdog Cronjob (no_agent)

chefkoch-scraper-watchdog (job_id: a884c342c49e)
Runs every 10 minutes
Calls wrapper script
no_agent: true — zero LLM cost
deliver: local — no notification spam

4. Original Cronjob (agent-based) deprecated

Original chefkoch-real-recipes-refill still exists but should be paused
The LLM-based approach (checking count + starting scraper) is overkill
Direct nohup + watchdog is cheaper and more reliable

Performance Observed

Discovery: ~50 URLs per batch from search pages
Extraction: ~60 recipes/minute with single context
5,000 recipes in ~60–70 minutes
25,000 recipe ETA: 710 hours continuous

Session-Context References

real_recipes.jsonl — append-only output file
scraper_state.json — tracks seed_cycle for resume
scraper_v4.out — live log file
scraper.log — legacy log from batch run (still there)

New Pitfalls Captured in Skills

multi-source-recipe-scraper / infinite-refill-scraper-v4.md:
- Browser path detection when multiple venvs exist
- nohup wrapper pattern for daemon mode
- terminal(background=true) tcsetattr conflict with Playwright
- Watchdog cronjob pattern
cronjob / SKILL.md:
- Model+provider MUST be set as pair on agent jobs
- Custom provider mismatch error signature

3.3 KiB Raw Permalink Blame History Unescape Escape