92 lines
3.3 KiB
Markdown
92 lines
3.3 KiB
Markdown
# Chefkoch Scraper Session: v4 First Production Deploy
|
||
|
||
## Session Date: 2026-06-18
|
||
## Trigger: User requested continuous scraping to 25,000 recipes
|
||
## Final State: Scraper running as nohup daemon (PID 80297) + watchdog cronjob
|
||
|
||
---
|
||
|
||
## Problem Discovery Log (Failure Chain)
|
||
|
||
### Failure 1: LLM-based cronjob fails on provider mismatch
|
||
- Cronjob `chefkoch-real-recipes-refill` (job_id: c77127dd9130)
|
||
- 06:01: `Error 400: model is required` — no model configured on job
|
||
- Fix attempted: set `model="moonshotai/kimi-k2.6"` — BUT omitted `provider`
|
||
- 06:17: `RuntimeError: No Anthropic credentials found` — system defaulted to anthropic provider
|
||
|
||
**Lesson:** Model + provider MUST be set as a PAIR on agent cronjobs. Setting only one creates a mismatch.
|
||
|
||
### Failure 2: `terminal(background=true)` silently killed scraper
|
||
- Attempted to start scraper via `terminal(background=true)`
|
||
- Result: Exit code 143 (SIGTERM after kill)
|
||
- Also: `tcsetattr: Inappropriate ioctl for device` — pseudo-tty interferes with Playwright
|
||
|
||
**Lesson:** Background process mode is NOT a nohup replacement. Use explicit `nohup` wrapper script.
|
||
|
||
### Failure 3: Playwright browser not found
|
||
- Playwright installed in nutrition-coach venv, but chromium not found
|
||
- Error: `Executable doesn't exist at ...chrome-headless-shell`
|
||
- Browser actually installed under `/home/debian/.cache/ms-playwright/` (installed globally)
|
||
- venv looking elsewhere
|
||
|
||
**Lesson:** Set `PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright` explicitly. Do NOT rely on auto-detection when multiple Python environments exist.
|
||
|
||
---
|
||
|
||
## Working Solution (Final)
|
||
|
||
### 1. Standalone Scraper Script
|
||
- `~/.hermes/scripts/chefkoch_scraper_v4.py` — headless, no LLM, asyncio
|
||
- Reads/writes directly to `nutrition-coach` profile dir
|
||
- Uses JSON-LD extraction (fast), Playwright only for URL discovery
|
||
|
||
### 2. nohup Wrapper Script
|
||
- `~/.hermes/scripts/start_scraper_v4.sh`
|
||
- Sets `PLAYWRIGHT_BROWSERS_PATH` before launching
|
||
- PID tracking in `/tmp/chefkoch_scraper_v4.pid`
|
||
- Idempotent: checks if already running
|
||
|
||
### 3. Watchdog Cronjob (no_agent)
|
||
- `chefkoch-scraper-watchdog` (job_id: a884c342c49e)
|
||
- Runs every 10 minutes
|
||
- Calls wrapper script
|
||
- `no_agent: true` — zero LLM cost
|
||
- `deliver: local` — no notification spam
|
||
|
||
### 4. Original Cronjob (agent-based) deprecated
|
||
- Original `chefkoch-real-recipes-refill` still exists but should be paused
|
||
- The LLM-based approach (checking count + starting scraper) is overkill
|
||
- Direct nohup + watchdog is cheaper and more reliable
|
||
|
||
---
|
||
|
||
## Performance Observed
|
||
|
||
- Discovery: ~50 URLs per batch from search pages
|
||
- Extraction: ~60 recipes/minute with single context
|
||
- 5,000 recipes in ~60–70 minutes
|
||
- 25,000 recipe ETA: ~7~10 hours continuous
|
||
|
||
---
|
||
|
||
## Session-Context References
|
||
|
||
- `real_recipes.jsonl` — append-only output file
|
||
- `scraper_state.json` — tracks seed_cycle for resume
|
||
- `scraper_v4.out` — live log file
|
||
- `scraper.log` — legacy log from batch run (still there)
|
||
|
||
---
|
||
|
||
## New Pitfalls Captured in Skills
|
||
|
||
1. **multi-source-recipe-scraper / infinite-refill-scraper-v4.md:**
|
||
- Browser path detection when multiple venvs exist
|
||
- `nohup` wrapper pattern for daemon mode
|
||
- `terminal(background=true)` tcsetattr conflict with Playwright
|
||
- Watchdog cronjob pattern
|
||
|
||
2. **cronjob / SKILL.md:**
|
||
- Model+provider MUST be set as pair on agent jobs
|
||
- Custom provider mismatch error signature
|