# Chefkoch Scraper Session: v4 First Production Deploy

## Session Date: 2026-06-18
## Trigger: User requested continuous scraping to 25,000 recipes
## Final State: Scraper running as nohup daemon (PID 80297) + watchdog cronjob

---

## Problem Discovery Log (Failure Chain)

### Failure 1: LLM-based cronjob fails on provider mismatch
- Cronjob `chefkoch-real-recipes-refill` (job_id: c77127dd9130)
- 06:01: `Error 400: model is required` — no model configured on job
- Fix attempted: set `model="moonshotai/kimi-k2.6"` — BUT omitted `provider`
- 06:17: `RuntimeError: No Anthropic credentials found` — system defaulted to anthropic provider

**Lesson:** Model + provider MUST be set as a PAIR on agent cronjobs. Setting only one creates a mismatch.

### Failure 2: `terminal(background=true)` silently killed scraper
- Attempted to start scraper via `terminal(background=true)`
- Result: Exit code 143 (SIGTERM after kill)
- Also: `tcsetattr: Inappropriate ioctl for device` — pseudo-tty interferes with Playwright

**Lesson:** Background process mode is NOT a nohup replacement. Use explicit `nohup` wrapper script.

### Failure 3: Playwright browser not found
- Playwright installed in nutrition-coach venv, but chromium not found
- Error: `Executable doesn't exist at ...chrome-headless-shell`
- Browser actually installed under `/home/debian/.cache/ms-playwright/` (installed globally)
- venv looking elsewhere

**Lesson:** Set `PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright` explicitly. Do NOT rely on auto-detection when multiple Python environments exist.

---

## Working Solution (Final)

### 1. Standalone Scraper Script
- `~/.hermes/scripts/chefkoch_scraper_v4.py` — headless, no LLM, asyncio
- Reads/writes directly to `nutrition-coach` profile dir
- Uses JSON-LD extraction (fast), Playwright only for URL discovery

### 2. nohup Wrapper Script
- `~/.hermes/scripts/start_scraper_v4.sh`
- Sets `PLAYWRIGHT_BROWSERS_PATH` before launching
- PID tracking in `/tmp/chefkoch_scraper_v4.pid`
- Idempotent: checks if already running

### 3. Watchdog Cronjob (no_agent)
- `chefkoch-scraper-watchdog` (job_id: a884c342c49e)
- Runs every 10 minutes
- Calls wrapper script
- `no_agent: true` — zero LLM cost
- `deliver: local` — no notification spam

### 4. Original Cronjob (agent-based) deprecated
- Original `chefkoch-real-recipes-refill` still exists but should be paused
- The LLM-based approach (checking count + starting scraper) is overkill
- Direct nohup + watchdog is cheaper and more reliable

---

## Performance Observed

- Discovery: ~50 URLs per batch from search pages
- Extraction: ~60 recipes/minute with single context
- 5,000 recipes in ~60–70 minutes
- 25,000 recipe ETA: ~7~10 hours continuous

---

## Session-Context References

- `real_recipes.jsonl` — append-only output file
- `scraper_state.json` — tracks seed_cycle for resume
- `scraper_v4.out` — live log file
- `scraper.log` — legacy log from batch run (still there)

---

## New Pitfalls Captured in Skills

1. **multi-source-recipe-scraper / infinite-refill-scraper-v4.md:**
   - Browser path detection when multiple venvs exist
   - `nohup` wrapper pattern for daemon mode
   - `terminal(background=true)` tcsetattr conflict with Playwright
   - Watchdog cronjob pattern

2. **cronjob / SKILL.md:**
   - Model+provider MUST be set as pair on agent jobs
   - Custom provider mismatch error signature