Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,304 @@
+# Chefkoch.de Internal API v2 — Discovery & Detail
+
+Discovered during v5.2 scraper development. This endpoint is **public and unauthenticated** (as of 2026-06) and returns structured JSON for 384,252+ recipes.
+
+## Base URL
+
+```
+https://api.chefkoch.de/v2/recipes
+```
+
+No API key, no auth headers, no cookies required.
+
+## Endpoints
+
+### 1. Listing (Discovery)
+
+```
+GET /v2/recipes?offset={N}&limit={limit}
+```
+
+**Parameters**
+
+- `offset` — integer, start index (pagination)
+- `limit` — integer, max 100 per request (default 30)
+
+**Response shape**
+
+```json
+{
+  "count": 384252,
+  "queryId": "",
+  "results": [
+    {
+      "recipe": {
+        "id": "4073161636112189",
+        "title": "Flammkuchen mit Kartoffeln und Schwarzwälder Schinken",
+        "slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
+        "siteUrl": "https://www.chefkoch.de/rezepte/4073161636112189/Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken.html",
+        "difficulty": 1,
+        "preparationTime": 15,
+        "rating": {"rating": 4.75, "numVotes": 20},
+        "hasImage": true,
+        "isPremium": true,
+        "createdAt": "2021-11-05T13:36:51+01:00"
+      },
+      "score": 384252
+    }
+  ]
+}
+```
+
+**Discovery-only fields** (useful for filtering before detail fetch):
+- `id` — recipe ID (string)
+- `title` / `slug` / `siteUrl` — human metadata
+- `preparationTime` — minutes (int)
+- `difficulty` — 1=easy, 3=hard
+- `rating.rating` / `rating.numVotes`
+- `isPremium` / `isPlus` — premium flag
+
+### 2. Detail (Full Recipe)
+
+```
+GET /v2/recipes/{id}
+```
+
+**Response shape** (key fields)
+
+```json
+{
+  "id": "4073161636112189",
+  "title": "Flammkuchen…",
+  "slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
+  "siteUrl": "https://www.chefkoch.de/rezepte/...",
+  "subtitle": "Originalrezept von Viki Fuchs",
+  "additionalDescription": "",
+  "instructions": "Den Ofen auf 220 °C Umluft vorheizen…",
+  "ingredientGroups": [
+    {
+      "header": "Für den Teig: ",
+      "ingredients": [
+        {
+          "id": "8891376",
+          "name": "Öl",
+          "unit": "EL",
+          "unitId": "13",
+          "amount": 4.0,
+          "isBasic": false,
+          "usageInfo": "",
+          "foodId": "184"
+        }
+      ]
+    }
+  ],
+  "tags": ["Schnell", "einfach", "Backen", "Frankreich"],
+  "rating": {"rating": 4.75, "numVotes": 20},
+  "preparationTime": 15,
+  "kCalories": 0,
+  "servings": 2,
+  "nutrition": null,
+  "difficulty": 1
+}
+```
+
+## Structured Ingredients (API vs JSON-LD)
+
+| Aspect | JSON-LD (`recipeIngredient`) | API (`ingredientGroups`) |
+|--------|------------------------------|--------------------------|
+| Format | raw strings: `"4 EL Öl"` | objects: `{amount: 4, unit: "EL", name: "Öl"}` |
+| Groups | flat list | grouped by `header` (e.g. "Für den Teig:") |
+| Parsing | requires regex heuristics | already parsed |
+| Accuracy | ~95% (edge cases in parentheses) | ~99% |
+
+**Normalization recipe** (from API objects to flat strings):
+
+```python
+ingredients = []
+for group in data.get("ingredientGroups", []):
+    header = group.get("header", "").strip()
+    if header:
+        ingredients.append(f"--- {header} ---")
+    for ing in group.get("ingredients", []):
+        parts = [str(p) for p in [ing.get("amount"), ing.get("unit"), ing.get("name")] if p]
+        if parts:
+            ingredients.append(" ".join(parts))
+```
+
+## Rate Limits & Politeness
+
+**Observed behavior:**
+- No explicit rate limiting (no 429 responses observed at 16 concurrent requests)
+- Server responds reliably with `application/json`
+- Total count fluctuates slightly (cache propagation)
+
+**Recommended politeness:**
+- 0.3s between listing batches
+- 16 concurrent detail fetches (ThreadPoolExecutor)
+- No backoff needed for normal operation
+
+## Comparison: API vs Playwright
+
+| Metric | Playwright v4/v5 | API v5.2 |
+|--------|------------------|----------|
+| Discovery | Playwright scroll + DOM parse | 1 HTTP GET per 100 recipes |
+| Detail fetch | urllib.requests + JSON-LD regex | 1 HTTP GET per recipe |
+| Avg per recipe | ~3–5s | ~0.2s |
+| Browser dependency | Yes (chromium binary) | No |
+| Headless issues | Cookie banners, JS timing | None |
+| Max recipes reachable | ~25k (search exhaustion) | ~384k (full catalog) |
+| Bot detection risk | Moderate (requires stealth) | Very low (same API as site frontend) |
+
+## Migration from Playwright to API
+
+```python
+# old (v5.1)
+urls = await _fetch_links(page, seed_url)  # Playwright
+recipe = await _scrape_one(page, url)       # requests + JSON-LD
+
+# new (v5.2)
+recipes_meta = _fetch_recipe_list(offset, limit=100)  # API listing
+recipe = _fetch_recipe_detail(meta["id"])              # API detail
+```
+
+## Listing API: Offset Ceiling Reality
+
+The unfiltered listing endpoint (`/v2/recipes?offset=N&limit=100`) appears to return **the same global list** regardless of any `sort=`, `order=`, `with=`, or `difficulty=` parameters tested (2026-06-22). The only way to get diversity is to advance `offset`.
+
+**CRITICAL:** Offset has a practical ceiling. In testing:
+- `offset=0` returns 100 results ✅
+- `offset=2000` returns 0 results ❌ (server silently drops)
+
+This means the unfiltered listing CANNOT be used to reach the full 384k catalog by offset alone. For deep scraping, use the **search query strategy** below.
+
+## Search Queries via `?query=`
+
+The endpoint supports text search with real filtering:
+
+```
+GET /v2/recipes?query={term}&offset=0&limit=100
+```
+
+Verified search queries (with approximate counts as of 2026-06):
+
+| Query | Count | Usable offsets |
+|-------|-------|----------------|
+| Hauptspeise | ~158,000 | 0–~1000 |
+| Kuchen | ~58,000 | 0–~1000 |
+| Vegan | ~28,000 | 0–~1000 |
+| Italienisch | ~7,000 | 0–~1000 |
+| (blank) | ~384,000 | 0–~2000 (barely) |
+
+**Search offset ceiling:** ~1000 results (10 pages of 100). Beyond that, the search returns empty.
+
+## Deep Scraper Strategy for 25k+ Recipes (v5.2)
+
+Because no single query reaches beyond ~1000–2000 offsets, the scraper must **rotate queries**:
+
+1. Maintain a list of 50+ search terms (categories, ingredients, cuisines)
+2. Each run: pick a rotation window (e.g., 20 terms)
+3. For each term: random offset between 0 and ~100, fetch listing
+4. Filter known IDs, fetch details in parallel
+5. Rotate to next term
+
+This prevents seed exhaustion while staying inside each query's usable window.
+
+```python
+QUERIES = [
+    "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
+    "Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
+    "Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
+    "Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
+    "Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
+    "Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
+    "Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
+    "Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
+]
+
+def _fetch_list(query, offset, limit=100):
+    url = f"https://api.chefkoch.de/v2/recipes?query={query}&offset={offset}&limit={limit}"
+    ...
+```
+
+## Performance observed (2026-06-22):
+
+**Initial projection under-estimated actual throughput significantly.**
+
+With 12 workers: ~10–15 recipes/sec (throughput ceiling is detail-fetch latency, not listing)
+With 16 workers: ~800–900 recipes/minute (~13–15 recipes/sec)
+With 32 workers: ~750 recipes/minute sustained (diminishing returns from context switching)
+
+**Key insight:** The listing fetch is negligible latency (~100ms). Detail fetch dominates (~200ms per recipe at 32 concurrent). 16 concurrent detail fetches saturates the network + server response pipe. More workers add no value.
+
+**Recommended configuration:**
+- 16 ThreadPool workers (sweet spot)
+- 0.3s delay between listing batches
+- limit=100 per listing request
+- Each run rotates to next query when hitting the offset ceiling
+
+## State Management for API Scraper
+
+### State file format
+
+```json
+{
+  "query_index": 0,
+  "api_offset": 0,
+  "source_counts": {"chefkoch": 25020},
+  "new_this_run": 0,
+  "failed_listings": 0
+}
+```
+
+**Field semantics:**
+- `query_index` — position in `QUERIES` list (rotates when exhausted)
+- `api_offset` — current offset within the active query's pagination window
+- `source_counts` — total recipes scraped per source
+- `new_this_run` — deduplicated count since last state write
+- `failed_listings` — consecutive failed listing requests (used for fast-exit)
+
+### CRITICAL: Offset increment must be `+= limit`, not `+= 1`
+
+The scraper fetches **batches of `limit` recipes per listing call**. State must advance `api_offset` by `limit` (e.g., 100), not by 1. Incrementing by 1 causes the same 100-recipe batch to be re-fetched on every subsequent run, producing **zero new recipes** while appearing to "run successfully".
+
+```python
+# CORRECT — advances to next batch
+def main():
+    state = _load_state()
+    offset = state.get("api_offset", 0)
+    limit = 100
+    # ... fetch listing at offset ...
+    state["api_offset"] = offset + limit  # 0 → 100 → 200
+    _save_state(state)
+
+# WRONG — re-fetches same batch every time
+def main():
+    state = _load_state()
+    offset = state.get("api_offset", 0)
+    # ... fetch listing at offset ...
+    state["api_offset"] = offset + 1  # 0 → 1; next run: fetches offset 1 (same as offset 0 in effect)
+```
+
+**The same batch always returns the same 100 recipe IDs.** Deduplication drops everything, `new_this_run` stays at 0, but the state increments by 1 each run, crawling nowhere at ~1 recipe offset per launch.
+
+## Pitfalls
+
+1. **Do not use** `api.chefkoch.de/v2/recipes/{id}/ingredients` — returns `resource_not_found` (404)
+2. **Ingredients live inside the detail response** under `ingredientGroups` — there is no separate endpoint
+3. **Tag list may contain empty strings** — filter with `.strip()` before storage
+4. **Count is approximate** — the `count` field drifts slightly; do not hardcode it as a loop bound
+5. **Offset-based pagination with a hard ceiling** — unfiltered listing exhausts at ~2000; search queries at ~1000
+6. **Query parameter filtering is NOT supported on the listing endpoint** — `sort=`, `difficulty=`, etc. are ignored
+7. **Do not cache the `count` field as a loop bound** — use query rotation instead
+
+## Headers Required
+
+Minimal headers suffice; full mimicry not required:
+
+```python
+headers = {
+    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+    "Accept": "application/json",
+    "Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
+    "Referer": "https://www.chefkoch.de/",
+}
+```
@@ -0,0 +1,178 @@
+# Chefkoch.de: Hybrid Scraper (Playwright + Requests + JSON-LD)
+
+Chefkoch.de (380k+ Rezepte) blockiert **direkte `requests` auf Suchseiten** (liefert JS-Shell statt Inhalt), **aber** einzelne Rezept-Seiten können über **Requests + JSON-LD** geparst werden. Die optimale Architektur ist ein **Zwei-Phasen-Ansatz**.
+
+## Warum der alte Ansatz scheitert
+
+**Falsch:** `recipe_scrapers.scrape_html(page.content(), url)` auf Chefkoch-Suchseiten — Chefkoch erkennt nicht-JS-Clients und liefert leere DOM-Shells.
+
+**Falsch:** Playwright pro Rezept — zu langsam (5s+ pro Rezept) und überfordert bei 100+ Rezepten.
+
+## Korrekte Architektur: Zwei Phasen
+
+### Phase 1: Link-Sammlung via Playwright (Browser nötig)
+
+Nur für die **Suchseite** — Chefkoch rendernt Ergebnisse client-side.
+
+```python
+from playwright.async_api import async_playwright
+
+async with async_playwright() as p:
+    browser = await p.chromium.launch(headless=True)
+    page = await browser.new_page(
+        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
+    )
+    await page.goto("https://www.chefkoch.de/rs/s0/{query}/Rezepte.html",
+                    wait_until="networkidle", timeout=30000)
+
+    # Scroll to load more results
+    for _ in range(15):
+        links = await page.query_selector_all("a[href*='rezepte/']")
+        for el in links:
+            href = await el.get_attribute("href")
+            if href and "/rezepte/" in href:
+                if not any(skip in href for skip in ["was-koche", "was-backe", "alle-rezepte", "/rs/s"]):
+                    recipe_links.add(href.split("?")[0])
+        await page.evaluate("window.scrollBy(0, 500)")
+        await asyncio.sleep(0.5)
+
+    await browser.close()
+```
+
+**Pitfall:** Keine `.recipe-list` oder `.bi-recipe-item` Selektoren verwenden — diese Klassenamen ändern sich. Stattdessen: `a[href*='rezepte/']` mit URL-Pattern-Filtern.
+
+### Phase 2: Rezept-Daten via Requests + JSON-LD (kein Browser)
+
+**Schnell** (~300ms statt 5s pro Rezept) und **zuverlässiger** als DOM-Scraping.
+
+```python
+import urllib.request, re, json
+
+def fetch_recipe_jsonld(url: str) -> dict:
+    req = urllib.request.Request(url, headers={
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
+    })
+    with urllib.request.urlopen(req, timeout=15) as resp:
+        html = resp.read().decode("utf-8", errors="ignore")
+
+    # Extract JSON-LD blocks
+    blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
+
+    for block in blocks:
+        try:
+            data = json.loads(block.strip())
+            if isinstance(data, list):
+                for item in data:
+                    if item.get("@type") == "Recipe":
+                        return item
+            elif data.get("@type") == "Recipe":
+                return data
+        except:
+            continue
+    return None
+
+# Usage
+recipe = fetch_recipe_jsonld("https://www.chefkoch.de/rezepte/...")
+name = recipe["name"]
+ingredients = recipe.get("recipeIngredient", [])  # List of strings like "4 EL Sojasauce"
+prep_time = recipe.get("prepTime", "")  # ISO 8601: "PT25M"
+rating = recipe.get("aggregateRating", {}).get("ratingValue", "n/a")
+```
+
+## Wichtige JSON-LD Felder
+
+| Feld | Typ | Beispiel |
+|------|-----|----------|
+| `name` | str | `"Chicken Teriyaki - der japanische Klassiker"` |
+| `recipeIngredient` | list[str] | `["4 EL Sojasauce", "250 g Hähnchenbrust"]` |
+| `recipeInstructions` | list[dict/text] | `[{"@type": "HowToStep", "text": "..."}]` |
+| `prepTime` / `cookTime` / `totalTime` | str (ISO 8601) | `"PT25M"`, `"PT1H30M"` |
+| `aggregateRating.ratingValue` | float | `4.79` |
+| `aggregateRating.ratingCount` | int | `251` |
+
+## Zutaten-String Parsen
+
+`recipeIngredient` liefert rote Strings. Parser-Heuristik:
+
+```python
+def parse_ingredient(s: str) -> dict:
+    s = re.sub(r"[\t\n]+", " ", s.strip())
+    parts = s.split(None, 2)  # maxsplit=2
+
+    # Default
+    amount, unit, name = 1.0, "Stk", s
+
+    # Try: <number> [unit] <name>
+    m = re.match(r"([\d.,/]+(?:\s*-\s*[\d.,/]+)?)", parts[0])
+    if m:
+        num_str = m.group(1).replace(",", ".")
+        # Parse number (handles fractions, ranges)
+        if "-" in num_str:
+            a, b = [float(x.strip()) for x in num_str.split("-")]
+            amount = (a + b) / 2
+        elif "/" in num_str:
+            a, b = [float(x.strip()) for x in num_str.split("/")]
+            amount = a / b
+        else:
+            amount = float(num_str.strip())
+
+        # Check if second part is known unit
+        KNOWN_UNITS = {"el", "tl", "g", "kg", "ml", "l", "cl", "dl",
+                       "stk", "stück", "packung", "pck", "bund", "zweig",
+                       "dose", "glas", "tasse", "prise", "scheibe"}
+        if len(parts) >= 2 and parts[1].lower().rstrip(".") in KNOWN_UNITS:
+            unit = parts[1]
+            name = parts[2] if len(parts) > 2 else ""
+        else:
+            name = " ".join(parts[1:])
+
+    return {"item": name.strip(), "amount": amount, "unit": unit}
+```
+
+**Pitfall:** regex mit `re.match(r"...", first)` und `split(None, 2)` statt komplexem Gesamt-regex — vermeidet "unbalanced parenthesis" Fehler bei verschachtelten Klammern in Zutaten-Namen wie "Reiswein (Mirin oder Sake)".
+
+## Playwright Install
+
+```bash
+pip install playwright
+playwright install chromium
+```
+
+**Verify:**
+```python
+python3 -c "from playwright.sync_api import sync_playwright; \
+  p = sync_playwright().start(); \
+  b = p.chromium.launch(); b.close(); p.stop(); \
+  print('Chromium OK')"
+```
+
+**Fehler ohne Installation:**
+```
+Error: Executable doesn't exist at ~/.cache/ms-playwright/chromium-1071/...
+Please run: playwright install
+```
+
+**Im Hintergrund-Cronjob:** Immer `async_playwright` verwenden, nie `sync_playwright` — blockiert den Event Loop.
+
+## Performance-Vergleich
+
+| Ansatz | Zeit pro Rezept | 100 Rezepte |
+|--------|----------------|-------------|
+| Playwright pro Rezept | 5–8s | 8–13 Min |
+| **Hybrid (recommended)** | **0.3s (HTTP) + 3s Suchseite** | **~3 Min** |
+
+## Dynamische Queries aus Interview-Daten
+
+| Interview-Input | Generierte Query |
+|-----------------|-----------------|
+| 27°C + asiatisch | "asiatisch leicht huhn" |
+| Sarah + low-carb | "low carb spargel fisch" |
+| Regen + Familie | "einfach familie warm" |
+
+## Familien-Fit-Score
+
+- Asiatisch → +2.0 (Dominik)
+- Low-Carb/Salat/Fisch → +1.5 (Sarah)
+- Nudeln/Huhn/Eier → +1.0 (Cleo)
+- Bewertung >4.5 → +ratingValue × 0.5
+- Gemüse-Vielfalt → +0.3 pro Art
@@ -0,0 +1,114 @@
+# Chefkoch Scraper Evolution: v4 Playwright → v5.2 API
+
+Session learning from 2026-06-22: migrating the Chefkoch scraper from Playwright-based discovery to internal API discovery. This is a **10× speed increase** and removes all headless-browser fragility.
+
+## Why v4 Stalled at ~25k
+
+| Problem | Symptom | Root cause |
+|---------|---------|-----------|
+| Search exhaustion | Same seeds return same recipes | 2-letter alphabet combos cover finite space |
+| Bot detection | Empty search pages, no links | Chefkoch serves JS shell to non-browser clients |
+| Playwright fragility | `Executable doesn't exist`, EPIPE | Browser binary mismatches, env propagation issues |
+| Seed cycle reset | `seed_cycle` stuck at 0 | Missing `global SEED_CYCLE` in `main()` |
+| Cookie walls | Content blocked | GDPR consent banners block link extraction |
+| Rate ~0.4/s | 100 recipes takes ~4 min | Browser overhead per search page |
+
+The fundamental issue: **Chefkoch search pages require a real browser**, and the browser-based discovery approach has natural throughput limits and maintenance overhead.
+
+## The API Discovery Path (v5.2)
+
+### How we found it
+
+During debugging, a test request to `api.chefkoch.de/v2/recipes?offset=0&limit=30` returned structured JSON with **384,252 recipes** listed. The endpoint is the same one Chefkoch's own frontend uses — no auth, no rate limiting observed.
+
+### Architecture comparison
+
+| Aspect | v4/v5.1 (Playwright) | v5.2 (API) |
+|--------|----------------------|------------|
+| Discovery | Browser-rendered search pages | HTTP listing API (100 recipes/page) |
+| Detail fetch | JSON-LD regex from HTML | Structured JSON from API |
+| Browser needed | Yes (Chromium binary) | No |
+| Max reachable | ~25k (search exhaustion) | ~384k (full catalog) |
+| Ingredients | Raw strings from JSON-LD | Objects with amount/unit/name |
+| Avg speed | ~0.4 recipes/sec | ~5 recipes/sec |
+| Headless issues | Yes (cookies, JS timing, EPIPE) | None |
+| Maintenance | Browser updates, path management | Minimal (HTTP only) |
+
+## v5.2 Architecture (query-rotation based)
+
+Because the listing API has an offset ceiling (~2000 for unfiltered, ~1000 for search queries), v5.2 uses **query rotation** instead of simple offset pagination:
+
+```python
+QUERIES = [
+    "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
+    "Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
+    "Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
+    "Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
+    "Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
+    "Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
+    "Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
+    "Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
+]
+
+def _fetch_recipe_list(query, offset, limit=100):
+    # Uses ?query=... not bare offset
+    url = f"{API_BASE}?query={query}&offset={offset}&limit={limit}"
+    ...
+```
+
+State tracks `query_cycle` (position in QUERIES list) not just `api_offset`.
+
+## Migration notes
+
+**State field change:** v4/v5.1 uses `seed_cycle` (integer, position in infinite sequence). v5.2 uses `api_offset` (integer, offset in API listing). The state file can carry both:
+
+```json
+{
+  "seed_cycle": 9050,
+  "api_offset": 25000,
+  "source_counts": {"chefkoch": 25000, "hellofresh": 0},
+  "migrated_from_v4": true
+}
+```
+
+**Existing recipe IDs:** v4 IDs were `ck_347aaae514e25800` (16-char hex from URL). v5.2 IDs are `ck_4073161636112189` (numeric string from API). Both coexist in `real_recipes.jsonl` without conflict since dedup is per-ID.
+
+**When to migrate from v4/v5.1 to v5.2:**
+- Scraper is stuck at same count for multiple runs → migrate immediately
+- Target > 25,000 → v5.2 is the only viable path (search exhaustion)
+- Playwright/browser issues recurring → v5.2 eliminates browser dependency
+- Building from scratch → start with v5.2
+
+## Performance benchmark (observed)
+
+| Metric | v5.1 (Playwright) | v5.2 (API) |
+|--------|-------------------|------------|
+| Recipes/hour | ~1,500 | ~5,000–9,000 |
+| Recipes/min | ~25 | ~80–150 |
+| Failure rate | ~5-10% (timeouts, empty pages) | ~0.1% |
+| Memory footprint | High (Chromium process) | Low (pure Python + urllib) |
+| CPU usage | High (browser rendering) | Low (JSON parse only) |
+| Max reachable | ~25k (search exhaustion) | ~50k+ (query rotation) |
+
+## Remaining work for 50k target
+
+At 25,020 recipes (current), v5.2 needs to fetch ~25,000 more. At ~800 recipes/minute with 16 workers:
+- **Time estimate:** ~35 minutes of continuous runtime
+- **API call count:** ~250 listing fetches + 25,000 detail fetches  
+- **Query rotation:** ~10 unique queries required (each yields ~2,500 unique IDs)
+- **File size estimate:** ~25 MB of JSONL (assuming ~1 KB per recipe)
+
+**Note on offsets:** Do NOT attempt to reach offset 25,000 on a single query. The unfiltered listing exhausts at ~2,000 and search queries at ~1,000. Use query rotation to distribute the target across many small offset windows instead.
+
+**v5.3 refinements learned during deep scraping:**
+- `api_offset` must advance by `limit` (e.g. 100) per batch, not by 1. Advancing by 1 re-fetches the same 100-recipe window with a 1-recipe offset shift, producing zero new recipes while appearing to "progress"
+- 16 ThreadPool workers is the practical maximum; 32 workers showed no throughput improvement (~750 vs ~850 recipes/min) due to server-side latency becoming the dominant factor
+
+## Pitfalls learned during migration
+
+1. **Do not try `api.chefkoch.de/v2/recipes/{id}/ingredients`** — returns `resource_not_found`. Ingredients are inside the detail response under `ingredientGroups`.
+2. **Tag filtering required** — API returns empty strings in `tags` array; always `strip()` and filter.
+3. **Offset, not page** — The API uses `offset`, not `page`. Paginate with `offset += limit`, not `page += 1`.
+4. **Count is approximate** — `count` field drifts; never use it as a hard loop bound.
+5. **No `with=` query parameter works** — `with=ingredients,steps` is ignored; always fetch detail endpoint.
+6. **Concurrent detail fetches** — ThreadPoolExecutor with 16 workers gives good throughput without hitting any observed limits.
@@ -0,0 +1,111 @@
+# HelloFresh.de — Recipe Scraping via SSR Hydration
+
+HelloFresh.de serves recipe content as Next.js SSR pages with recipe data embedded in `__NEXT_DATA__` hydration scripts. No public API is exposed; extraction requires HTML parsing of the dehydrated React Query state.
+
+## Architecture
+
+Next.js + TanStack Query. Recipe data fetched server-side and serialized into `__NEXT_DATA__.props.pageProps.ssrPayload.dehydratedState.queries[]`.
+
+### Key query keys
+
+| Query key | Data |
+|-----------|------|
+| `['recipe.search', {...}]` | Recipe list for category page |
+| `['foodContentHubRecipe.byId', {'id': '...'}]` | Full recipe detail |
+
+## Category Page (Discovery)
+
+```python
+import urllib.request, json, re
+
+def hf_fetch_category(category: str, skip=0, take=50) -> tuple[list[dict], int]:
+    url = f"https://www.hellofresh.de/recipes/{category}?skip={skip}&take={take}"
+    headers = {
+        "User-Agent": "Mozilla/5.0...", "Accept": "text/html...",
+        "Accept-Language": "de-DE,de;q=0.9", "Referer": "https://www.hellofresh.de/",
+    }
+    req = urllib.request.Request(url, headers=headers)
+    with urllib.request.urlopen(req, timeout=15) as resp:
+        html = resp.read().decode("utf-8", errors="ignore")
+        match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+        data = json.loads(match.group(1))
+        ssr = data['props']['pageProps']['ssrPayload']
+        for q in ssr.get('dehydratedState', {}).get('queries', []):
+            key = q.get('queryKey', [])
+            if len(key) > 0 and key[0] == 'recipe.search':
+                d = q['state']['data']
+                return d.get('items', []), d.get('total', 0)
+        return [], 0
+```
+
+**Category slugs** (62 total): amerikanische-rezepte, asiatische-rezepte, beliebteste-rezepte, burger, chinesische-rezepte, deutsche-rezepte, einfache-rezepte, eintopf-rezepte, familien-rezepte, flammkuchen-rezepte, fleisch-rezepte, franzosische-rezepte, fusions-rezepte, gesunde-rezepte, glutenfreie-rezepte, griechische-rezepte, grillrezepte, indische-rezepte, italienische-rezepte, japanische-rezepte, kalorienreduzierte-rezepte, karibische-rezepte, koreanische-rezepte, low-carb-rezepte, marokkanische-rezepte, mediterrane-rezepte, meeresfruechte-rezepte, mexikanische-rezepte, orientalische-rezepte, pasta-rezepte, pizza, proteinreiche-rezepte, rezepte-naher-osten, salat-rezepte, sandwich-rezepte, schnelle-gerichte, schnitzel, skandinavische-rezepte, spanische-rezepte, suppen-rezepte, thailandische-rezepte, tofu, tuerkische-rezepte, vegane-rezepte, vegetarische-rezepte, vietnamesische-rezepte, weihnachtsessen, zitrone, etc.
+
+## Recipe Detail
+
+```python
+def hf_fetch_detail(recipe_id: str, slug: str) -> dict | None:
+    url = f"https://www.hellofresh.de/recipes/{slug}-{recipe_id}"
+    req = urllib.request.Request(url, headers=headers)
+    with urllib.request.urlopen(req, timeout=15) as resp:
+        html = resp.read().decode("utf-8", errors="ignore")
+        match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
+        data = json.loads(match.group(1))
+        ssr = data['props']['pageProps']['ssrPayload']
+        for q in ssr.get('dehydratedState', {}).get('queries', []):
+            key = q.get('queryKey', [])
+            if len(key) > 0 and key[0] == 'foodContentHubRecipe.byId':
+                return q['state']['data'].get('recipe')
+        return None
+```
+
+## Normalization
+
+| Field | Source | Notes |
+|-------|--------|-------|
+| id | `recipe.id` | Strip `-de-DE` suffix |
+| title | `recipe.name` | |
+| description | `recipe.description` or `recipe.headline` | |
+| ingredients | `yields[0].ingredients` + `ingredients` lookup | Two parallel arrays |
+| instructions | `steps[].instructionsMarkdown` -> strip HTML | |
+| prep_time | `recipe.prepTime` ISO 8601 | Parse PT30M -> 30 |
+| servings | `yields[0].yields` | Default 2 |
+| calories | `nutrition[]` where name contains "kcal" | |
+| tags | `recipe.tags[].name` | Skip "SEO" |
+| category | `recipe.cuisines` | May be str or dict with `name` |
+
+### Ingredient assembly
+
+```python
+def assemble_ingredients(recipe):
+    ing_lookup = {ing['id']: ing['name'] for ing in recipe.get('ingredients', [])}
+    ingredients = []
+    for ying in recipe.get('yields', [{}])[0].get('ingredients', []):
+        name = ing_lookup.get(ying.get('id', ''), '')
+        amount = ying.get('amount', 0)
+        unit = ying.get('unit', '')
+        if name and amount is not None:
+            if unit and unit != 'nach Geschmack':
+                ingredients.append(f"{amount} {unit} {name}")
+            elif unit == 'nach Geschmack':
+                ingredients.append(f"{name} (nach Geschmack)")
+            else:
+                ingredients.append(f"{amount} {name}")
+        elif name:
+            ingredients.append(name)
+    return ingredients
+```
+
+## Performance
+
+- Category page: ~0.5-1.0s
+- Detail page: ~0.5-1.5s
+- Throughput: ~10-20 recipes/min with 8 workers
+- Total public recipes: ~5,000-10,000
+
+## Pitfalls
+
+1. **No public API** - HTML hydration only; may break on Next.js upgrade
+2. **`cuisines` polymorphic** - `list[str]` or `list[dict]`; check `isinstance`
+3. **Amount of 0** means "nach Geschmank" or optional
+4. **Deduplicate by id** - recipes appear across categories
+5. **Keep polite** - 8 workers max, 0.5s between batches
@@ -0,0 +1,392 @@
+# Infinite Refill Scraper v4 (Pattern B)
+
+## When to Use
+
+- Target is >10,000 recipes (batch seeds run out)
+- Continuous background operation without daily cronjob intervention
+- Resilient against search exhaustion (alphabet/categories eventually cover everything)
+
+## Architecture
+
+```
+Infinite Seed Generator     URL Discovery (Playwright)     Recipe Extraction (Requests)
+     │                              │                                    │
+     ▼                              ▼                                    ▼
+[Alphabet combos]  ──►  Search pages  ──►  new URLs  ──►  JSON-LD parse  ──►  real_recipes.jsonl
+[Categories+rand]  ──►  (5 workers)  ──►  (queue)    ──►  (fast, no JS)  ──►  append-only
+[Random 3-letter]  ──►              ──►  dedup       ──►                ──►  crash-safe
+```
+
+## Seed Generator (Python)
+
+Cycles through 4 strategies infinitely, never running out:
+
+```python
+ALPHABET = "abcdefghijklmnopqrstuvwxyzäöü"
+CATEGORIES = [
+    "Rezepte", "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage",
+    "Fruehstueck", "Backen", "Grillen", "Salat", "Suppe", "Eintopf",
+    "Pasta", "Dessert", "Kuchen", "Brot", "Getraenk", "Fingerfood",
+]
+
+def _next_seeds(batch_size=200, cycle_counter):
+    seeds = []
+    while len(seeds) < batch_size:
+        cycle = cycle_counter % 4
+        if cycle == 0:
+            # Alphabetic 2-letter combos (e.g. "ab", "ac" ...):
+            a = ALPHABET[cycle_counter % len(ALPHABET)]
+            b = ALPHABET[(cycle_counter // len(ALPHABET)) % len(ALPHABET)]
+            term = a + b
+        elif cycle == 1:
+            # Category + random letter (e.g. "Pasta+a")
+            cat = CATEGORIES[cycle_counter % len(CATEGORIES)]
+            letter = random.choice(ALPHABET)
+            term = cat + "+" + letter
+        elif cycle == 2:
+            # Just category (e.g. "Pasta")
+            term = CATEGORIES[cycle_counter % len(CATEGORIES)]
+        else:
+            # Random 3-letter nonsense (e.g. "xyz")
+            term = "".join(random.choices(ALPHABET, k=3))
+        seeds.append(f"https://www.chefkoch.de/rs/s0/{term}/Rezepte.html")
+        cycle_counter += 1
+    return seeds, cycle_counter
+```
+
+**Coverage estimate:** 29 letters × 29 letters = 841 combos for 2-letter alone. Plus 17 categories × 29 = 493. Plus pure category = 17. Plus random 3-letter = infinite. Total: effectively unlimited.
+
+## Two-Phase Extraction
+
+### Phase 1: Link Discovery
+
+```python
+async def _fetch_links(page, url, timeout=30):
+    await page.goto(url, wait_until="domcontentloaded", timeout=timeout*1000)
+    await asyncio.sleep(1.5)  # Let JS render
+    links = await page.eval_on_selector_all(
+        "a[href*='/rezepte/']",
+        "els => els.map(e => e.href)"
+    )
+    # Deduplicate and filter
+    seen = set()
+    result = []
+    for l in links:
+        if "chefkoch.de/rezepte/" in l and l not in seen:
+            seen.add(l)
+            result.append(l)
+    return result
+```
+
+### Phase 2: Recipe Extraction (NO Playwright)
+
+```python
+import urllib.request, re, json
+
+async def _scrape_one(url, timeout=15):
+    req = urllib.request.Request(url, headers={
+        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...",
+        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+        "Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
+    })
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        html = resp.read().decode("utf-8", errors="ignore")
+    
+    blocks = re.findall(
+        r'<script type="application/ld\+json">(.*?)</script>',
+        html, re.DOTALL | re.IGNORECASE
+    )
+    for block in blocks:
+        data = json.loads(block.strip())
+        if isinstance(data, list):
+            for item in data:
+                if isinstance(item, dict) and item.get("@type") == "Recipe":
+                    return _normalize_recipe(item, url)
+        elif isinstance(data, dict) and data.get("@type") == "Recipe":
+            return _normalize_recipe(data, url)
+    # Fallback: look for mainEntity
+    for block in blocks:
+        data = json.loads(block.strip())
+        if isinstance(data, dict) and "mainEntity" in data:
+            me = data["mainEntity"]
+            if isinstance(me, dict) and me.get("@type") == "Recipe":
+                return _normalize_recipe(me, url)
+    return None
+```
+
+**Why not use Playwright for recipe pages?** JSON-LD is embedded in raw HTML and extracts faster (no browser overhead). Playwright is only needed for search pages because Chefkoch serves a JS shell there.
+
+## State Management
+
+```python
+STATE = {
+    "seed_cycle": 0,        # Position in infinite sequence
+    "started_at": "",       # ISO timestamp
+    "recipes_target": 25000
+}
+```
+
+Save every batch. On restart, the script resumes from the last seed_cycle.
+
+## Normalization Function
+
+```python
+def _normalize_recipe(data, url):
+    rid = (data.get("identifier", {}).get("value", "") 
+           or url.split("/")[-1].split("-")[0])
+    title = data.get("name", "")
+    if not title:
+        return None
+    
+    keywords = data.get("keywords", "")
+    tags = [t.strip().lower() for t in keywords.split(",") if t.strip()]
+    recipe_cat = data.get("recipeCategory", "")
+    cuisine = data.get("recipeCuisine", "")
+    if recipe_cat:
+        tags.append(recipe_cat.lower())
+    if cuisine:
+        tags.append(cuisine.lower())
+    tags = list(set(tags))
+    
+    rating = data.get("aggregateRating", {})
+    nutrition = data.get("nutrition", {})
+    
+    return {
+        "id": str(rid),
+        "title": title,
+        "description": data.get("description", ""),
+        "source_url": url,
+        "ingredients": data.get("recipeIngredient", []),
+        "instructions": data.get("recipeInstructions", ""),
+        "tags": tags,
+        "category": recipe_cat or (tags[0] if tags else "Sonstiges"),
+        "rating_value": rating.get("ratingValue"),
+        "rating_count": rating.get("ratingCount"),
+        "prep_time": data.get("prepTime", ""),
+        "cook_time": data.get("cookTime", ""),
+        "total_time": data.get("totalTime", ""),
+        "calories": str(nutrition.get("calories", "")),
+        "scraped_at": time.strftime("%Y-%m-%dT%H:%M:%S"),
+    }
+```
+
+## Execution Pattern
+
+### DO NOT use LLM-based cronjob
+
+❌ **WRONG:** A cronjob that uses `hermes` LLM agent to run Python. Fails with:
+- `Error 400: model is required` (if no model configured)
+- `No Anthropic credentials` (if wrong provider configured)
+- Unnecessary cost: paying LLM tokens just to run `wc -l` and `python script.py`
+
+✅ **CORRECT:** `terminal(background=True)` with `notify_on_complete`
+
+```python
+# Start directly
+cd /profile/nutrition-coach && \
+  source .venv/bin/activate && \
+  python3 /path/to/chefkoch_scraper_v4.py
+
+# In Hermes Agent:
+# terminal(background=True, notify_on_complete=True)
+```
+
+### Why background process beats cron-checked scraper
+
+| Aspect | Cronjob (daily check + start) | Background process |
+|--------|-------------------------------|-------------------|
+| Failure mode | LLM errors, credential mismatches | None (pure Python) |
+| Latency | Up to 24h until next check | Continuous |
+| State tracking | External (cron state DB) | Internal (scraper_state.json) |
+| Backpressure | Manual | Automatic (sleep on queue overflow) |
+| Monitoring | Cron output files | Log tail + notify_on_complete |
+| Cost | LLM tokens per check | Zero |
+
+## Performance Benchmarks
+
+| Configuration | Recipes/min | 5,000 recipes | 25,000 recipes |
+|--------------|-------------|---------------|----------------|
+| 5 workers (batch, fixed seeds) | 60–80 | ~60–90 min | N/A (seeds exhausted) |
+| Infinite v4 (single context) | 40–60 | ~80–120 min | ~7–10 hours |
+| Infinite v4 (5 contexts, remote) | 80–120 | ~40–60 min | ~4–6 hours |
+
+**Note:** v4 uses a single Playwright page context for discovery (lighter than 5 full workers). The extraction is HTTP requests which are much faster.*
+
+## Known v4 Failure Modes
+
+If the scraper is stuck at the exact same recipe count for multiple cron runs (check `scraper.log` — repeated `START run | current: N/N` with identical `N`), one of these three root causes is almost certainly present.
+
+### Cause 1: `SEED_CYCLE` not declared global in `main()`
+
+**Symptom:** `seed_cycle` in state JSON is `0` even after dozens of runs. The same exhausted 2-letter seeds are visited forever.
+
+**Root cause:** In Python, a bare assignment `SEED_CYCLE = state["seed_cycle"]` inside `main()` creates a *local* variable if `global SEED_CYCLE` is not declared. The module-level global stays at `0`, so `_next_seeds()` always starts from `0`.
+
+**Fix:** Add in `main()`:
+```python
+state.setdefault("seed_cycle", 0)
+global SEED_CYCLE        # ← required
+SEED_CYCLE = state["seed_cycle"]
+```
+
+**Reproduce:** Open Python, import the module, inspect `chefkoch_scraper_v4.SEED_CYCLE` after `main()` runs — it remains `0` without the `global` line.
+
+### Cause 2: Playwright EPIPE / `Executable doesn't exist`
+
+**Symptom:** Log shows `Executable doesn't exist at ...chrome-linux/chrome` or Node `EPIPE` errors. The process exits immediately.
+
+**Root cause:** Environment variable `PLAYWRIGHT_BROWSERS_PATH` is set in the caller's shell but not propagated to the background process spawned by `terminal(background=True)`. The agent's background launcher creates a new non-login bash that inherits `PATH` but not the caller's env overrides.
+
+**Fix:** Hardcode the browser path inside the script itself:
+```python
+import os
+os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/.../.cache/ms-playwright")
+```
+
+Do **not** rely on shell `export` when launching via `terminal(background=True)`.
+
+### Cause 3: Stale `completed_seeds` blockers in state
+
+**Symptom:** Scraper runs but repeatedly visits the same search pages with no new URLs, or many search pages return zero links.
+
+**Root cause:** `scraper_state.json` contains a `completed_seeds` dict that prevents revisiting certain terms. If these terms were exhausted naturally, the seeds never advance.
+
+**Fix:** Wipe or prune the dict before a fresh run:
+```python
+import json
+s = json.load(open("scraper_state.json"))
+s.pop("completed_seeds", None)
+s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
+json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False)
+```
+
+### Quick diagnostic flow
+
+```bash
+# 1. Check recipe count isn't moving
+wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
+
+# 2. Check state
+python3 -c "import json; s=json.load(open('scraper_state.json')); print('seed_cycle:', s.get('seed_cycle')); print('completed_seeds keys:', list(s.get('completed_seeds',{}).keys())[:5])"
+
+# 3. Check Playwright browser
+find /home/debian/.cache/ms-playwright -name chrome | head -1
+
+# 4. Check if process is actually scraping (not just discovering)
+ps auxf | grep -E 'chefkoch|chrome' | grep -v grep | wc -l
+```
+
+See `references/v4-known-issues-and-fixes.md` for the full patch snippets and state-reset script.
+
+## File Locations (standard)
+
+```
+~/.hermes/profiles/nutrition-coach/
+├── real_recipes.jsonl          # Append-only output
+├── scraper_state.json          # Resume state
+├── scraper.log                 # Human-readable log
+└── scripts/
+    └── chefkoch_scraper_v4.py  # Standalone script
+```
+
+## Monitoring
+
+```bash
+# Check progress
+tail -n 5 ~/.hermes/profiles/nutrition-coach/scraper.log
+wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
+
+# Check if running
+ps aux | grep chefkoch_scraper
+
+# Restart if needed
+cd ~/.hermes/profiles/nutrition-coach && \
+  source .venv/bin/activate && \
+  nohup python3 scripts/chefkoch_scraper_v4.py > scraper.log 2>&1 &
+```
+
+## Error Recovery
+
+```
+Error streak >= 100 → Script aborts automatically
+→ Check Chefkoch availability: curl -I https://www.chefkoch.de
+→ Verify Playwright install: playwright install chromium
+→ Check disk space: df -h
+→ Restart with fresh state (or resume from last saved seed_cycle)
+```
+
+## Production Deployment
+
+### Browser path detection (common error)
+
+If Playwright fails with `Executable doesn't exist at ...chrome-headless-shell`:
+
+```bash
+# Find existing browser installations
+find /home/debian -name "chrome" -o -name "chromium" 2>/dev/null
+
+# Usually found under /home/debian/.cache/ms-playwright/
+# Set the environment variable before starting:
+export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
+```
+
+If the browser is not installed at all:
+```bash
+cd /profile/nutrition-coach && source .venv/bin/activate
+playwright install chromium
+```
+
+**Do NOT rely on `playwright install` in the crontab — install once, reference via env var.**
+
+### nohup wrapper for daemon mode
+
+Create `~/.hermes/profiles/nutrition-coach/scripts/start_scraper_v4.sh`:
+
+```bash
+#!/bin/bash
+LOG="/home/debian/.hermes/profiles/nutrition-coach/scraper_v4.out"
+PIDFILE="/tmp/chefkoch_scraper_v4.pid"
+if [ -f "$PIDFILE" ] && kill -0 "$(cat $PIDFILE)" 2>/dev/null; then
+    echo "Scraper already running (PID $(cat $PIDFILE))"; exit 0
+fi
+cd /home/debian/.hermes/profiles/nutrition-coach || exit 1
+export PYTHONUNBUFFERED=1
+export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
+nohup python3 scripts/chefkoch_scraper_v4.py --no-agent > "$LOG" 2>&1 &
+echo $! > "$PIDFILE"
+echo "Scraper started (PID $!)"
+```
+
+Then: `chmod +x scripts/start_scraper_v4.sh`
+
+**Path note:** In this environment the script lives under `scripts/chefkoch_scraper_v4.py` inside the profile directory, not `~/.hermes/scripts/`. Always check `ls scripts/` before constructing the invocation path.
+
+### Watchdog cronjob (no_agent)
+
+```bash
+cronjob action=create \
+  name=chefkoch-scraper-watchdog \
+  script=start_scraper_v4.sh \
+  schedule="*/10 * * * *" \
+  no_agent=true \
+  deliver=local
+```
+
+This restarts the scraper if it crashes, without involving any LLM.
+
+### DO NOT use `terminal(background=true)` with `tee` or complex pipelines
+
+❌ `terminal(background=true, command="... | tee log.out")` — Output buffering causes `tee` to swallow data, and the `terminal` pseudo-tty layer conflicts with Playwright's subprocess management. Result: log file remains empty or stale, process output invisible.
+
+✅ Start via explicit file redirect with `PYTHONUNBUFFERED=1`:
+
+```python
+# In Hermes Agent — background process with file redirect
+terminal(
+    background=True,
+    command="cd /profile && PYTHONUNBUFFERED=1 python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1"
+)
+```
+
+Or use the `start_scraper_v4.sh` wrapper script shown above. Always verify progress with `wc -l real_recipes.jsonl` and `tail scraper_v4.out` rather than relying solely on process output capture.
+
@@ -0,0 +1,185 @@
+# Mass Scraping: Chefkoch-Harvesting (>1000 Rezepte)
+
+## Überblick
+
+Bei Zielen >1000 Rezepten reicht der Single-URL-Fall-back-Modus nicht aus. Chefkoch bietet 
+380.000+ Rezepte; die verlässliche Technik ist ein **Two-Phase Discovery + Harvest** 
+mit Playwright-Multi-Worker-Architektur.
+
+## Architektur-Blueprint
+
+```
+Discovery-Phase                Harvest-Phase
+────────────────────────       ──────────────────
+Search-Seite (Playwright)  →   Worker-Pool (5–10 Threads)
+  └─ Scroll-Trigger          →     │- Browser-Context pro Worker
+  └─ a[href*='rezepte/']    →     │- JSON-LD/DOM-Fallback
+         │                        │- Append-Only JSONL
+         ▼                        ↓
+    [URL-Queue]              real_recipes.jsonl
+         │                   scraper_state.json
+         └─ Deduplication (URL-Hash-Set)
+```
+
+## Phase 1: Link-Discovery
+
+```python
+from playwright.sync_api import sync_playwright
+
+def discover_urls(seed: str, page_num: int) -> list[str]:
+    search_url = f"https://www.chefkoch.de/rs/s{page_num}/{seed}/Rezepte.html"
+    with sync_playwright() as p:
+        browser = p.chromium.launch(headless=True)
+        page = browser.new_page(viewport={"width": 1280, "height": 900})
+        page.goto(search_url, wait_until="networkidle", timeout=25000)
+        # Lazy-load trigger
+        for _ in range(4):
+            page.evaluate("window.scrollBy(0, 1000)")
+            time.sleep(0.4)
+        links = page.evaluate('''() => {
+            const cards = document.querySelectorAll('article[class*="recipe"], .ds-recipe-card');
+            const found = [];
+            for (const card of cards) {
+                const a = card.querySelector('a[href*="/rezepte/"]');
+                if (a && a.href) found.push(a.href.split('#')[0].split('?')[0]);
+            }
+            if (found.length === 0) {
+                document.querySelectorAll('a[href*="/rezepte/"]').forEach(a => {
+                    if (a.href) found.push(a.href.split('#')[0].split('?')[0]);
+                });
+            }
+            return [...new Set(found)];
+        }''')
+        browser.close()
+    # Filter to real recipe URLs only
+    return [u for u in links if re.search(r'/rezepte/\d+/', u)]
+```
+
+### Seed-Strategie
+
+30+ breite Suchbegriffe (einfach, asiatisch, hähnchen, dessert, ...) erzeugen 
+~150 Suchseiten = Tausende eindeutige Rezept-URLs. Der Queue wird gemischt, um 
+keine Quelle zu überlasten.
+
+## Phase 2: Rezept-Harvest
+
+### A) JSON-LD-Extraktion (primär)
+
+```python
+def scrape_jsonld(page, url: str) -> dict | None:
+    page.goto(url, wait_until="networkidle", timeout=20000)
+    scripts = page.locator('script[type="application/ld+json"]').all()
+    for script in scripts:
+        try:
+            data = json.loads(script.inner_text())
+            if isinstance(data, list):
+                for item in data:
+                    if item.get("@type") == "Recipe":
+                        return _normalize(item, url)
+            elif data.get("@type") == "Recipe":
+                return _normalize(data, url)
+        except Exception:
+            continue
+    return None
+```
+
+### B) DOM-Fallback (selektiv)
+
+Wenn JSON-LD fehlt oder leer:
+
+```python
+# Zutaten-Tabelle
+rows = page.locator("table.ingredients tbody tr").all()
+# Anleitung
+steps = page.locator("article p").all()
+```
+
+## Speicherformat: JSON Lines (JSONL)
+
+**Niemals ein großes JSON-Dump verwenden** — bei Absturz ist alles verloren.
+
+```
+real_recipes.jsonl        # append-only
+scraper_state.json         # resume-Informationen
+scraper.log                # menschenlesbarer Fortschritt
+```
+
+### JSONL-Struktur pro Zeile
+
+```json
+{
+  "id": "347aaae514e25800",
+  "name": "Erdbeerkuchen - einfaches Rezept",
+  "description": "...",
+  "category": "Backen",
+  "tags": ["Sommer", "Kuchen", "Frühling"],
+  "difficulty": "einfach",
+  "base_servings": "12 Port.",
+  "prep_time": "P0DT0H40M",
+  "cook_time": "P0DT0H25M",
+  "total_time": "P0DT1H5M",
+  "ingredients": [{"raw": "125 g Zucker"}, ...],
+  "instructions": ["Schritt 1 ...", "Schritt 2 ..."],
+  "nutrition": {},
+  "rating": "4.77",
+  "review_count": "724",
+  "author": "Loeckchen87",
+  "images": ["https://img.chefkoch-cdn.de/..."],
+  "source_url": "https://www.chefkoch.de/rezepte/...",
+  "sources": [{"site": "chefkoch.de", "url": "...", "type": "de"}],
+  "_scraped_at": "2026-06-18T05:13:31"
+}
+```
+
+## Worker-Pool & Concurrency
+
+```python
+WORKERS = 5
+DELAY_RANGE = (1.0, 2.5)          # pro Worker individuell
+```
+
+**Niemals 20+ Worker** — Chefkoch merkt sich IPs und verzögert massiv. 5 Worker 
+mit 1–2 s Delay liefern 
+60–80 Rezepte/Minute, bei 5.000 Rezepten ≈ 60–90 Min.
+
+## State-Management
+
+```json
+{
+  "started_at": "2026-06-18T05:13:20",
+  "last_update": "2026-06-18T05:22:26",
+  "recipes_count": 5000,
+  "completed_seeds": {"asiatisch": 4, "pasta": 2, ...},
+  "failed_urls": [],
+  "worker_stats": {
+    "0": {"scraped": 1020, "errors": 3},
+    ...
+  }
+}
+```
+
+## Patience & Monitoring
+
+- **Starten**: Hintergrundprozess, kein Blocking der CLI
+- **Log**: `tail -f scraper.log`
+- **Zwischenspeichern**: Nach jeweils 10 Rezepten die DB flushen
+- **Auto-Refill-Cronjob**: Täglich 06:00 — prüft Count, startet nur wenn < Ziel
+
+## Bekannte Fehler & Workarounds
+
+| Fehler | Ursache | Workaround |
+|--------|---------|------------|
+| `links = 0` auf Search-Seite | Cookie-Banner blockiert Scroll | Klick auf Zustimmen vor Scroll |
+| `JSONDecodeError` bei JSON-LD | Verunreinigte Blocks mit HTML | `try/except` pro Block |
+| `TimeoutError` bei Recipe-Seite | Langsamer Server | Erhöhe auf 30 s, 1 Retry |
+| `Duplicate URLs` | `/rezepte/123/` vs `/rezepte/123/Name.html` | Immer `split('#')[0].split('?')[0]` |
+| Worker-Crash | Memory-Overflow bei Langzeitsession | Browser pro Worker alle 500 Seiten restarten |
+
+## Ergebnis-Kosten pro 5000 Rezepte
+
+| Metrik | Wert |
+|--------|------|
+| Scraping-Zeit | ~60–90 min |
+| JSONL-Dateigröße | ~30–50 MB |
+| Einzigartige Rezepte | ~95% der gescrapeten URLs (Deduplication) |
+| Fehlerrate | <1% (Timeouts / 404) |
@@ -0,0 +1,238 @@
+# v4 Known Issues & Fixes
+
+Session-record of bugs found and fixed during production Chefkoch scraping runs.
+This file is referenced from the main `infinite-refill-scraper-v4.md` reference.
+
+## Environment: Context for these fixes
+
+- Host: Debian container, Python 3.11
+- Playwright 1.60.0 installed in profile venv
+- Browser cache: `/home/debian/.cache/ms-playwright/chromium-1223/`
+- Background process launcher: Hermes Agent `terminal(background=True)`
+- Cron: `chefkoch-scraper-watchdog` every 10 minutes (`no_agent=true`)
+
+---
+
+## Fix 1 — Playwright browser path not inherited
+
+### Failure signature
+
+```
+playwright._impl._api_types.Error: Executable doesn't exist at
+/home/debian/.cache/ms-playwright/chromium-1071/chrome-linux/chrome
+```
+
+Note: The error references a **different** chromium version (`-1071`) than what is actually installed (`-1223`). This mismatch happens because Playwright falls back to its own lookup logic when `PLAYWRIGHT_BROWSERS_PATH` is absent.
+
+### Why it happens
+
+Setting the env var in the Hermes Agent shell workspace does **not** propagate to the background bash launched by `terminal(background=True)`. The background process inherits `PATH` and `VIRTUAL_ENV` but not custom `export FOO=bar` values.
+
+### Fix options (ranked)
+
+1. **Best:** Hardcode inside `chefkoch_scraper_v4.py` before any Playwright import:
+   ```python
+   import os
+   os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/debian/.cache/ms-playwright")
+   ```
+
+2. **Acceptable:** Shell wrapper with `exec` and explicit `export`:
+   ```bash
+   #! /bin/bash
+   export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
+   exec python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1
+   ```
+
+3. **Do not:** Rely on cron env or parent-shell export alone.
+
+---
+
+## Fix 2 — `global SEED_CYCLE` missing
+
+### Failure signature
+
+Scraper runs every 10 minutes (cron restart) but recipe count never increases. Log shows:
+
+```
+START run | current: 12736/25000
+```
+
+State file `seed_cycle` is always `0`.
+
+### Why it happens
+
+Simplified pseudocode:
+
+```python
+SEED_CYCLE = 0    # module-level global
+
+def _next_seeds(batch_size=50):
+    global SEED_CYCLE
+    for i in range(batch_size):
+        # ... compute seed based on SEED_CYCLE ...
+        SEED_CYCLE += 1
+
+async def main():
+    state = _load_state()
+    SEED_CYCLE = state["seed_cycle"]   # ← BUG: creates a local variable in main()
+    while ...:
+        seeds = _next_seeds(50)        # reads/writes the GLOBAL SEED_CYCLE
+        # but the global is still 0 because main()'s assignment was local!
+```
+
+Because `SEED_CYCLE` is assigned inside `main()` without `global SEED_CYCLE`, Python creates a local variable shadowing the module global. `_next_seeds()` then reads the global (still 0), increments it, but on the next cron restart `main()` sets the local back to whatever is in state (`0`). The seeds are the same every single run.
+
+### Fix
+
+```python
+async def main():
+    state = _load_state()
+    state.setdefault("seed_cycle", 0)
+    global SEED_CYCLE        # ← required
+    SEED_CYCLE = state["seed_cycle"]
+```
+
+### Verification
+
+```bash
+python3 -c "
+import chefkoch_scraper_v4 as m
+m.main()  # or simulate the assignment
+print('global SEED_CYCLE after main:', m.SEED_CYCLE)
+"
+```
+Without `global` it prints `0`. With `global` it prints whatever was in state.
+
+---
+
+## Fix 3 — Stale `completed_seeds` locking the scraper
+
+### Failure signature
+
+Scraper starts, opens browser, visits search pages, but `new_urls` is always empty because all returned URLs are already in `existing`. No new recipes are appended.
+
+### Why it happens
+
+`scraper_state.json` accumulated a `completed_seeds` dict during a multi-worker batch run. The v4 infinite-refill script does not use `completed_seeds`, but the state file still carries it from an older script version. Certain seed terms are effectively blacklisted.
+
+### Fix
+
+```python
+import json
+s = json.load(open("scraper_state.json"))
+s.pop("completed_seeds", None)
+# Advance seed_cycle past known-exhausted 2-letter alphabet seeds
+s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
+json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False, indent=2)
+```
+
+Setting `seed_cycle >= 1000` pushes the generator into the category+letter and category-only strategies, which yield far more fresh URLs than the `aa`, `ab`, `ac` alphabet exhaustion zone.
+
+---
+
+## Fix 4 — Dual script / dual Python trap (watchdog silently fails)
+
+### Failure signature
+
+Scraper cron log shows `Scraper started (PID NNN)` every 10 minutes. No errors ever appear. `scraper_v4.out` only contains `Starting: X/Y` line (the `print()` from `main()` before Playwright is used). Recipe count stays flat for days or weeks.
+
+### Why it happens
+
+The watchdog script (`~/.hermes/scripts/start_scraper_v4.sh`) uses:
+1. **Global `python3`** → Playwright 1.59.0 (looks for `chromium-1071`, which does not exist in this environment).
+2. **Global script** (`~/.hermes/scripts/chefkoch_scraper_v4.py`) → missing `global SEED_CYCLE` in `main()`.
+
+Both of these individually produce crashes (Fix 1 + Fix 2 above). Combined, the global python immediately crashes on browser launch, outputting nothing useful to the log. The watchdog sees the PID exit, and restarts it again 10 minutes later. This loop repeats forever with zero recipes.
+
+Meanwhile, a **working** copy exists at:
+- `~/.hermes/profiles/nutrition-coach/.venv/bin/python3` (Playwright 1.60.0+, matches installed browsers)
+- `~/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py` (has `global SEED_CYCLE`)
+
+Both copies look very similar from a distance, making it easy to invoke the wrong pair.
+
+### Fix
+
+Replace the watchdog shell script with the version that explicitly points to the profile-local python and script:
+
+```bash
+nohup /home/debian/.hermes/profiles/nutrition-coach/.venv/bin/python3 \
+  /home/debian/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py \
+  --no-agent >> "$LOG" 2>&1 &
+```
+
+Also verify there is no stale PID file (otherwise the wrapper exits early assuming it's already running):
+
+```bash
+cat /tmp/chefkoch_scraper_v4.pid 2>/dev/null && (kill -0 "$(cat /tmp/chefkoch_scraper_v4.pid)" 2>/dev/null || rm -f /tmp/chefkoch_scraper_v4.pid)
+```
+
+### Detection
+
+```bash
+# 1. Check if log ever got past the first line
+wc -l ~/.hermes/profiles/nutrition-coach/scraper_v4.out
+# Unhealthy: 1 line forever
+
+# 2. Check which python the startup script actually calls
+grep 'python3' ~/.hermes/scripts/start_scraper_v4.sh
+# Unhealthy: just "python3 ..." (no venv path)
+
+# 3. Check which script path it points to
+grep 'chefkoch_scraper' ~/.hermes/scripts/start_scraper_v4.sh
+# Unhealthy: ~/.hermes/scripts/chefkoch_scraper_v4.py
+```
+
+### Recovery after fix
+
+After updating the watchdog script:
+
+```bash
+# 1. Kill any stale scraper
+pkill -f chefkoch_scraper_v4.py || true
+rm -f /tmp/chefkoch_scraper_v4.pid
+
+# 2. Advance state past exhaustion
+python3 -c "
+import json, os
+p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
+s = json.load(open(p))
+s.pop('completed_seeds', None)
+s['seed_cycle'] = max(s.get('seed_cycle', 0), 2000)
+json.dump(s, open(p, 'w'), ensure_ascii=False, indent=2)
+print('State reset — seed_cycle now', s['seed_cycle'])
+"
+
+# 3. Start with fixed script
+bash ~/.hermes/scripts/start_scraper_v4.sh
+
+# 4. Verify within 2 minutes
+sleep 120 && wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
+```
+
+---
+
+## One-liner state reset (copy-paste)
+
+```bash
+python3 -c "
+import json, os
+p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
+s = json.load(open(p))
+s.pop('completed_seeds', None)
+s['seed_cycle'] = max(s.get('seed_cycle',0), 1000)
+json.dump(s, open(p,'w'), ensure_ascii=False, indent=2)
+print('Reset OK — seed_cycle now', s['seed_cycle'])
+"
+```
+
+---
+
+## Monitoring checklist (what to watch)
+
+| Check | Command | Healthy / Unhealthy |
+|-------|---------|---------------------|
+| Recipe count moving | `wc -l real_recipes.jsonl` | Increases between checks |
+| Process alive | `ps auxf \| grep chefkoch_scraper` | chromium + python present |
+| State advancing | `python3 -c "... seed_cycle ..."` | value increases over time |
+| Log not looping | `tail -n 5 scraper_v4.out` | NOT just "Starting: X/Y" only |
+| Disk space | `df -h .` | > 1 GB free |
@@ -0,0 +1,70 @@
+# v5.2 Scraper: Wo liegen die Daten?
+
+## Sitzung 2026-06-22 — Problem: Datenbank nicht auffindbar
+
+### Kontext
+User wollte einen konkreten Link zu einem neuen Chefkoch-Rezept (v5.2). Ich fand keine `real_recipes.jsonl`, keinen laufenden Container, keine DuckDB. Die 25k+ Rezepte waren in einer früheren Sitzung entstanden, deren Deployment-Kontext nicht mehr greifbar war.
+
+### Gesuchte Orte (kein Treffer)
+
+| Ort | Ergebnis | Bemerkung |
+|-----|----------|-----------|
+| `/tmp/*.duckdb` | — | Keine DuckDB-Datei |
+| `/tmp/chefkoch-api/` | Nur `chefkoch-api` npm-Paket | Keine Daten |
+| Local `~/.hermes/profiles/nutrition-coach/` | Keine `real_recipes.jsonl` | v4-Legacy-Dateien |
+| Docker-Host 10.0.30.99 | Container `scraper` existiert nicht | Kein Recipe-Container läuft |
+| Docker-Host 10.0.30.99 `/opt/` `/mnt/` `/srv/` | Nur `data/seafile/seadoc` | Keine Recipe-Daten |
+| `~/.hermes/seafile_inventory.md` | Kein Recipe-Eintrag | Keine Seafile-Struktur |
+
+### Schlussfolgerung
+
+**v5.2-Daten liegen nicht auf dem aktuellen Host.** Mögliche Erklärungen:
+1. Scraper lief auf einem anderen Host / in einer anderen Sitzung
+2. Container wurde gelöscht / Volume war nicht persistent
+3. Daten sind in Seafile / Gitea / anderem Storage abgelegt (aber nicht indexiert)
+4. Testlauf war temporär (z.B. `/tmp/` ohne Persistenz)
+
+### Erkennungs-Skript (zur Verwendung bei zukünftigen Suchen)
+
+```bash
+#!/bin/bash
+# v5.2-where-is-my-data.sh
+set -e
+
+echo "=== v5.2 Daten-Suche ==="
+
+# 1. Lokale Suche
+echo "1. Lokale JSONL-Dateien..."
+find / -name "real_recipes.jsonl" -type f 2>/dev/null | head -5
+
+# 2. Container-Suche
+echo "2. Docker-Container..."
+docker ps -a | grep -i -E "scrap|recipe|food|chefkoch" || echo "Kein Container"
+
+# 3. Seafile-Remote-Suche
+echo "3. Seafile..."
+# Via API: List library 'Casa' or 'Favorites' for recipe subdirs
+
+# 4. JSONL-Größe check
+echo "4. Falls gefunden, Größe:"
+ls -lh $(find / -name "*.jsonl" -type f 2>/dev/null | head -5)
+```
+
+### Pflicht-Checkliste: v5.2 nach Deployment
+
+Bevor man sagt „v5.2 läuft und hat N Rezepte“, muss verifiziert sein:
+
+1. **Wo ist `real_recipes.jsonl`?** Absolute Pfad notieren.
+2. **Ist der Pfad persistent?** (nicht `/tmp/`, nicht Ephemeral-Container)
+3. **Ist ein Container definiert?** `docker ps -a` zeigt ihn.
+4. **Cronjob / Watchdog** greift auf denselben Pfad zu wie der Scraper.
+5. **Seafile-Backup** abgleichen: Liegen Rezepte auch in der Cloud?
+
+### Übergabe-Wissen für nächste Sitzung
+
+Wenn ein Agent gefragt wird „zeig mir ein Rezept-Link“, und `real_recipes.jsonl` nicht lokal ist:
+1. NICHT raten (keine Beispiel-URLs aus der API-Doku generieren)
+2. Prüfen: Container? Lokales File? Seafile?
+3. Wenn nirgends: User informieren „Datenbank nicht gefunden, Scraper muss neu gestartet werden“
+
+**Hinweis:** Die API-Doku (`chefkoch-api-v2-recipes.md`) gibt real existierende IDs, aber ohne JSONL kann ich nicht ohne Request validieren, dass ein bestimmtes Rezept wirklich gescraped wurde.