# Chefkoch Scraper Evolution: v4 Playwright → v5.2 API

Session learning from 2026-06-22: migrating the Chefkoch scraper from Playwright-based discovery to internal API discovery. This is a **10× speed increase** and removes all headless-browser fragility.

## Why v4 Stalled at ~25k

| Problem | Symptom | Root cause |
|---------|---------|-----------|
| Search exhaustion | Same seeds return same recipes | 2-letter alphabet combos cover finite space |
| Bot detection | Empty search pages, no links | Chefkoch serves JS shell to non-browser clients |
| Playwright fragility | `Executable doesn't exist`, EPIPE | Browser binary mismatches, env propagation issues |
| Seed cycle reset | `seed_cycle` stuck at 0 | Missing `global SEED_CYCLE` in `main()` |
| Cookie walls | Content blocked | GDPR consent banners block link extraction |
| Rate ~0.4/s | 100 recipes takes ~4 min | Browser overhead per search page |

The fundamental issue: **Chefkoch search pages require a real browser**, and the browser-based discovery approach has natural throughput limits and maintenance overhead.

## The API Discovery Path (v5.2)

### How we found it

During debugging, a test request to `api.chefkoch.de/v2/recipes?offset=0&limit=30` returned structured JSON with **384,252 recipes** listed. The endpoint is the same one Chefkoch's own frontend uses — no auth, no rate limiting observed.

### Architecture comparison

| Aspect | v4/v5.1 (Playwright) | v5.2 (API) |
|--------|----------------------|------------|
| Discovery | Browser-rendered search pages | HTTP listing API (100 recipes/page) |
| Detail fetch | JSON-LD regex from HTML | Structured JSON from API |
| Browser needed | Yes (Chromium binary) | No |
| Max reachable | ~25k (search exhaustion) | ~384k (full catalog) |
| Ingredients | Raw strings from JSON-LD | Objects with amount/unit/name |
| Avg speed | ~0.4 recipes/sec | ~5 recipes/sec |
| Headless issues | Yes (cookies, JS timing, EPIPE) | None |
| Maintenance | Browser updates, path management | Minimal (HTTP only) |

## v5.2 Architecture (query-rotation based)

Because the listing API has an offset ceiling (~2000 for unfiltered, ~1000 for search queries), v5.2 uses **query rotation** instead of simple offset pagination:

```python
QUERIES = [
    "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
    "Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
    "Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
    "Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
    "Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
    "Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
    "Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
    "Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
]

def _fetch_recipe_list(query, offset, limit=100):
    # Uses ?query=... not bare offset
    url = f"{API_BASE}?query={query}&offset={offset}&limit={limit}"
    ...
```

State tracks `query_cycle` (position in QUERIES list) not just `api_offset`.

## Migration notes

**State field change:** v4/v5.1 uses `seed_cycle` (integer, position in infinite sequence). v5.2 uses `api_offset` (integer, offset in API listing). The state file can carry both:

```json
{
  "seed_cycle": 9050,
  "api_offset": 25000,
  "source_counts": {"chefkoch": 25000, "hellofresh": 0},
  "migrated_from_v4": true
}
```

**Existing recipe IDs:** v4 IDs were `ck_347aaae514e25800` (16-char hex from URL). v5.2 IDs are `ck_4073161636112189` (numeric string from API). Both coexist in `real_recipes.jsonl` without conflict since dedup is per-ID.

**When to migrate from v4/v5.1 to v5.2:**
- Scraper is stuck at same count for multiple runs → migrate immediately
- Target > 25,000 → v5.2 is the only viable path (search exhaustion)
- Playwright/browser issues recurring → v5.2 eliminates browser dependency
- Building from scratch → start with v5.2

## Performance benchmark (observed)

| Metric | v5.1 (Playwright) | v5.2 (API) |
|--------|-------------------|------------|
| Recipes/hour | ~1,500 | ~5,000–9,000 |
| Recipes/min | ~25 | ~80–150 |
| Failure rate | ~5-10% (timeouts, empty pages) | ~0.1% |
| Memory footprint | High (Chromium process) | Low (pure Python + urllib) |
| CPU usage | High (browser rendering) | Low (JSON parse only) |
| Max reachable | ~25k (search exhaustion) | ~50k+ (query rotation) |

## Remaining work for 50k target

At 25,020 recipes (current), v5.2 needs to fetch ~25,000 more. At ~800 recipes/minute with 16 workers:
- **Time estimate:** ~35 minutes of continuous runtime
- **API call count:** ~250 listing fetches + 25,000 detail fetches  
- **Query rotation:** ~10 unique queries required (each yields ~2,500 unique IDs)
- **File size estimate:** ~25 MB of JSONL (assuming ~1 KB per recipe)

**Note on offsets:** Do NOT attempt to reach offset 25,000 on a single query. The unfiltered listing exhausts at ~2,000 and search queries at ~1,000. Use query rotation to distribute the target across many small offset windows instead.

**v5.3 refinements learned during deep scraping:**
- `api_offset` must advance by `limit` (e.g. 100) per batch, not by 1. Advancing by 1 re-fetches the same 100-recipe window with a 1-recipe offset shift, producing zero new recipes while appearing to "progress"
- 16 ThreadPool workers is the practical maximum; 32 workers showed no throughput improvement (~750 vs ~850 recipes/min) due to server-side latency becoming the dominant factor

## Pitfalls learned during migration

1. **Do not try `api.chefkoch.de/v2/recipes/{id}/ingredients`** — returns `resource_not_found`. Ingredients are inside the detail response under `ingredientGroups`.
2. **Tag filtering required** — API returns empty strings in `tags` array; always `strip()` and filter.
3. **Offset, not page** — The API uses `offset`, not `page`. Paginate with `offset += limit`, not `page += 1`.
4. **Count is approximate** — `count` field drifts; never use it as a hard loop bound.
5. **No `with=` query parameter works** — `with=ingredients,steps` is ignored; always fetch detail endpoint.
6. **Concurrent detail fetches** — ThreadPoolExecutor with 16 workers gives good throughput without hitting any observed limits.