hermes-skills/food-nutrition/meal-planning/references/multi-source-recipe-scraper/v4-known-issues-and-fixes.md

# v4 Known Issues & Fixes

Session-record of bugs found and fixed during production Chefkoch scraping runs.
This file is referenced from the main `infinite-refill-scraper-v4.md` reference.

## Environment: Context for these fixes

- Host: Debian container, Python 3.11
- Playwright 1.60.0 installed in profile venv
- Browser cache: `/home/debian/.cache/ms-playwright/chromium-1223/`
- Background process launcher: Hermes Agent `terminal(background=True)`
- Cron: `chefkoch-scraper-watchdog` every 10 minutes (`no_agent=true`)

---

## Fix 1 — Playwright browser path not inherited

### Failure signature

```
playwright._impl._api_types.Error: Executable doesn't exist at
/home/debian/.cache/ms-playwright/chromium-1071/chrome-linux/chrome
```

Note: The error references a **different** chromium version (`-1071`) than what is actually installed (`-1223`). This mismatch happens because Playwright falls back to its own lookup logic when `PLAYWRIGHT_BROWSERS_PATH` is absent.

### Why it happens

Setting the env var in the Hermes Agent shell workspace does **not** propagate to the background bash launched by `terminal(background=True)`. The background process inherits `PATH` and `VIRTUAL_ENV` but not custom `export FOO=bar` values.

### Fix options (ranked)

1. **Best:** Hardcode inside `chefkoch_scraper_v4.py` before any Playwright import:
   ```python
   import os
   os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/debian/.cache/ms-playwright")
   ```

2. **Acceptable:** Shell wrapper with `exec` and explicit `export`:
   ```bash
   #! /bin/bash
   export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
   exec python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1
   ```

3. **Do not:** Rely on cron env or parent-shell export alone.

---

## Fix 2 — `global SEED_CYCLE` missing

### Failure signature

Scraper runs every 10 minutes (cron restart) but recipe count never increases. Log shows:

```
START run | current: 12736/25000
```

State file `seed_cycle` is always `0`.

### Why it happens

Simplified pseudocode:

```python
SEED_CYCLE = 0    # module-level global

def _next_seeds(batch_size=50):
    global SEED_CYCLE
    for i in range(batch_size):
        # ... compute seed based on SEED_CYCLE ...
        SEED_CYCLE += 1

async def main():
    state = _load_state()
    SEED_CYCLE = state["seed_cycle"]   # ← BUG: creates a local variable in main()
    while ...:
        seeds = _next_seeds(50)        # reads/writes the GLOBAL SEED_CYCLE
        # but the global is still 0 because main()'s assignment was local!
```

Because `SEED_CYCLE` is assigned inside `main()` without `global SEED_CYCLE`, Python creates a local variable shadowing the module global. `_next_seeds()` then reads the global (still 0), increments it, but on the next cron restart `main()` sets the local back to whatever is in state (`0`). The seeds are the same every single run.

### Fix

```python
async def main():
    state = _load_state()
    state.setdefault("seed_cycle", 0)
    global SEED_CYCLE        # ← required
    SEED_CYCLE = state["seed_cycle"]
```

### Verification

```bash
python3 -c "
import chefkoch_scraper_v4 as m
m.main()  # or simulate the assignment
print('global SEED_CYCLE after main:', m.SEED_CYCLE)
"
```
Without `global` it prints `0`. With `global` it prints whatever was in state.

---

## Fix 3 — Stale `completed_seeds` locking the scraper

### Failure signature

Scraper starts, opens browser, visits search pages, but `new_urls` is always empty because all returned URLs are already in `existing`. No new recipes are appended.

### Why it happens

`scraper_state.json` accumulated a `completed_seeds` dict during a multi-worker batch run. The v4 infinite-refill script does not use `completed_seeds`, but the state file still carries it from an older script version. Certain seed terms are effectively blacklisted.

### Fix

```python
import json
s = json.load(open("scraper_state.json"))
s.pop("completed_seeds", None)
# Advance seed_cycle past known-exhausted 2-letter alphabet seeds
s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False, indent=2)
```

Setting `seed_cycle >= 1000` pushes the generator into the category+letter and category-only strategies, which yield far more fresh URLs than the `aa`, `ab`, `ac` alphabet exhaustion zone.

---

## Fix 4 — Dual script / dual Python trap (watchdog silently fails)

### Failure signature

Scraper cron log shows `Scraper started (PID NNN)` every 10 minutes. No errors ever appear. `scraper_v4.out` only contains `Starting: X/Y` line (the `print()` from `main()` before Playwright is used). Recipe count stays flat for days or weeks.

### Why it happens

The watchdog script (`~/.hermes/scripts/start_scraper_v4.sh`) uses:
1. **Global `python3`** → Playwright 1.59.0 (looks for `chromium-1071`, which does not exist in this environment).
2. **Global script** (`~/.hermes/scripts/chefkoch_scraper_v4.py`) → missing `global SEED_CYCLE` in `main()`.

Both of these individually produce crashes (Fix 1 + Fix 2 above). Combined, the global python immediately crashes on browser launch, outputting nothing useful to the log. The watchdog sees the PID exit, and restarts it again 10 minutes later. This loop repeats forever with zero recipes.

Meanwhile, a **working** copy exists at:
- `~/.hermes/profiles/nutrition-coach/.venv/bin/python3` (Playwright 1.60.0+, matches installed browsers)
- `~/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py` (has `global SEED_CYCLE`)

Both copies look very similar from a distance, making it easy to invoke the wrong pair.

### Fix

Replace the watchdog shell script with the version that explicitly points to the profile-local python and script:

```bash
nohup /home/debian/.hermes/profiles/nutrition-coach/.venv/bin/python3 \
  /home/debian/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py \
  --no-agent >> "$LOG" 2>&1 &
```

Also verify there is no stale PID file (otherwise the wrapper exits early assuming it's already running):

```bash
cat /tmp/chefkoch_scraper_v4.pid 2>/dev/null && (kill -0 "$(cat /tmp/chefkoch_scraper_v4.pid)" 2>/dev/null || rm -f /tmp/chefkoch_scraper_v4.pid)
```

### Detection

```bash
# 1. Check if log ever got past the first line
wc -l ~/.hermes/profiles/nutrition-coach/scraper_v4.out
# Unhealthy: 1 line forever

# 2. Check which python the startup script actually calls
grep 'python3' ~/.hermes/scripts/start_scraper_v4.sh
# Unhealthy: just "python3 ..." (no venv path)

# 3. Check which script path it points to
grep 'chefkoch_scraper' ~/.hermes/scripts/start_scraper_v4.sh
# Unhealthy: ~/.hermes/scripts/chefkoch_scraper_v4.py
```

### Recovery after fix

After updating the watchdog script:

```bash
# 1. Kill any stale scraper
pkill -f chefkoch_scraper_v4.py || true
rm -f /tmp/chefkoch_scraper_v4.pid

# 2. Advance state past exhaustion
python3 -c "
import json, os
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
s = json.load(open(p))
s.pop('completed_seeds', None)
s['seed_cycle'] = max(s.get('seed_cycle', 0), 2000)
json.dump(s, open(p, 'w'), ensure_ascii=False, indent=2)
print('State reset — seed_cycle now', s['seed_cycle'])
"

# 3. Start with fixed script
bash ~/.hermes/scripts/start_scraper_v4.sh

# 4. Verify within 2 minutes
sleep 120 && wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
```

---

## One-liner state reset (copy-paste)

```bash
python3 -c "
import json, os
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
s = json.load(open(p))
s.pop('completed_seeds', None)
s['seed_cycle'] = max(s.get('seed_cycle',0), 1000)
json.dump(s, open(p,'w'), ensure_ascii=False, indent=2)
print('Reset OK — seed_cycle now', s['seed_cycle'])
"
```

---

## Monitoring checklist (what to watch)

| Check | Command | Healthy / Unhealthy |
|-------|---------|---------------------|
| Recipe count moving | `wc -l real_recipes.jsonl` | Increases between checks |
| Process alive | `ps auxf \| grep chefkoch_scraper` | chromium + python present |
| State advancing | `python3 -c "... seed_cycle ..."` | value increases over time |
| Log not looping | `tail -n 5 scraper_v4.out` | NOT just "Starting: X/Y" only |
| Disk space | `df -h .` | > 1 GB free |