305 lines
10 KiB
Markdown
305 lines
10 KiB
Markdown
# Chefkoch.de Internal API v2 — Discovery & Detail
|
||
|
||
Discovered during v5.2 scraper development. This endpoint is **public and unauthenticated** (as of 2026-06) and returns structured JSON for 384,252+ recipes.
|
||
|
||
## Base URL
|
||
|
||
```
|
||
https://api.chefkoch.de/v2/recipes
|
||
```
|
||
|
||
No API key, no auth headers, no cookies required.
|
||
|
||
## Endpoints
|
||
|
||
### 1. Listing (Discovery)
|
||
|
||
```
|
||
GET /v2/recipes?offset={N}&limit={limit}
|
||
```
|
||
|
||
**Parameters**
|
||
|
||
- `offset` — integer, start index (pagination)
|
||
- `limit` — integer, max 100 per request (default 30)
|
||
|
||
**Response shape**
|
||
|
||
```json
|
||
{
|
||
"count": 384252,
|
||
"queryId": "",
|
||
"results": [
|
||
{
|
||
"recipe": {
|
||
"id": "4073161636112189",
|
||
"title": "Flammkuchen mit Kartoffeln und Schwarzwälder Schinken",
|
||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||
"siteUrl": "https://www.chefkoch.de/rezepte/4073161636112189/Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken.html",
|
||
"difficulty": 1,
|
||
"preparationTime": 15,
|
||
"rating": {"rating": 4.75, "numVotes": 20},
|
||
"hasImage": true,
|
||
"isPremium": true,
|
||
"createdAt": "2021-11-05T13:36:51+01:00"
|
||
},
|
||
"score": 384252
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
**Discovery-only fields** (useful for filtering before detail fetch):
|
||
- `id` — recipe ID (string)
|
||
- `title` / `slug` / `siteUrl` — human metadata
|
||
- `preparationTime` — minutes (int)
|
||
- `difficulty` — 1=easy, 3=hard
|
||
- `rating.rating` / `rating.numVotes`
|
||
- `isPremium` / `isPlus` — premium flag
|
||
|
||
### 2. Detail (Full Recipe)
|
||
|
||
```
|
||
GET /v2/recipes/{id}
|
||
```
|
||
|
||
**Response shape** (key fields)
|
||
|
||
```json
|
||
{
|
||
"id": "4073161636112189",
|
||
"title": "Flammkuchen…",
|
||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||
"siteUrl": "https://www.chefkoch.de/rezepte/...",
|
||
"subtitle": "Originalrezept von Viki Fuchs",
|
||
"additionalDescription": "",
|
||
"instructions": "Den Ofen auf 220 °C Umluft vorheizen…",
|
||
"ingredientGroups": [
|
||
{
|
||
"header": "Für den Teig: ",
|
||
"ingredients": [
|
||
{
|
||
"id": "8891376",
|
||
"name": "Öl",
|
||
"unit": "EL",
|
||
"unitId": "13",
|
||
"amount": 4.0,
|
||
"isBasic": false,
|
||
"usageInfo": "",
|
||
"foodId": "184"
|
||
}
|
||
]
|
||
}
|
||
],
|
||
"tags": ["Schnell", "einfach", "Backen", "Frankreich"],
|
||
"rating": {"rating": 4.75, "numVotes": 20},
|
||
"preparationTime": 15,
|
||
"kCalories": 0,
|
||
"servings": 2,
|
||
"nutrition": null,
|
||
"difficulty": 1
|
||
}
|
||
```
|
||
|
||
## Structured Ingredients (API vs JSON-LD)
|
||
|
||
| Aspect | JSON-LD (`recipeIngredient`) | API (`ingredientGroups`) |
|
||
|--------|------------------------------|--------------------------|
|
||
| Format | raw strings: `"4 EL Öl"` | objects: `{amount: 4, unit: "EL", name: "Öl"}` |
|
||
| Groups | flat list | grouped by `header` (e.g. "Für den Teig:") |
|
||
| Parsing | requires regex heuristics | already parsed |
|
||
| Accuracy | ~95% (edge cases in parentheses) | ~99% |
|
||
|
||
**Normalization recipe** (from API objects to flat strings):
|
||
|
||
```python
|
||
ingredients = []
|
||
for group in data.get("ingredientGroups", []):
|
||
header = group.get("header", "").strip()
|
||
if header:
|
||
ingredients.append(f"--- {header} ---")
|
||
for ing in group.get("ingredients", []):
|
||
parts = [str(p) for p in [ing.get("amount"), ing.get("unit"), ing.get("name")] if p]
|
||
if parts:
|
||
ingredients.append(" ".join(parts))
|
||
```
|
||
|
||
## Rate Limits & Politeness
|
||
|
||
**Observed behavior:**
|
||
- No explicit rate limiting (no 429 responses observed at 16 concurrent requests)
|
||
- Server responds reliably with `application/json`
|
||
- Total count fluctuates slightly (cache propagation)
|
||
|
||
**Recommended politeness:**
|
||
- 0.3s between listing batches
|
||
- 16 concurrent detail fetches (ThreadPoolExecutor)
|
||
- No backoff needed for normal operation
|
||
|
||
## Comparison: API vs Playwright
|
||
|
||
| Metric | Playwright v4/v5 | API v5.2 |
|
||
|--------|------------------|----------|
|
||
| Discovery | Playwright scroll + DOM parse | 1 HTTP GET per 100 recipes |
|
||
| Detail fetch | urllib.requests + JSON-LD regex | 1 HTTP GET per recipe |
|
||
| Avg per recipe | ~3–5s | ~0.2s |
|
||
| Browser dependency | Yes (chromium binary) | No |
|
||
| Headless issues | Cookie banners, JS timing | None |
|
||
| Max recipes reachable | ~25k (search exhaustion) | ~384k (full catalog) |
|
||
| Bot detection risk | Moderate (requires stealth) | Very low (same API as site frontend) |
|
||
|
||
## Migration from Playwright to API
|
||
|
||
```python
|
||
# old (v5.1)
|
||
urls = await _fetch_links(page, seed_url) # Playwright
|
||
recipe = await _scrape_one(page, url) # requests + JSON-LD
|
||
|
||
# new (v5.2)
|
||
recipes_meta = _fetch_recipe_list(offset, limit=100) # API listing
|
||
recipe = _fetch_recipe_detail(meta["id"]) # API detail
|
||
```
|
||
|
||
## Listing API: Offset Ceiling Reality
|
||
|
||
The unfiltered listing endpoint (`/v2/recipes?offset=N&limit=100`) appears to return **the same global list** regardless of any `sort=`, `order=`, `with=`, or `difficulty=` parameters tested (2026-06-22). The only way to get diversity is to advance `offset`.
|
||
|
||
**CRITICAL:** Offset has a practical ceiling. In testing:
|
||
- `offset=0` returns 100 results ✅
|
||
- `offset=2000` returns 0 results ❌ (server silently drops)
|
||
|
||
This means the unfiltered listing CANNOT be used to reach the full 384k catalog by offset alone. For deep scraping, use the **search query strategy** below.
|
||
|
||
## Search Queries via `?query=`
|
||
|
||
The endpoint supports text search with real filtering:
|
||
|
||
```
|
||
GET /v2/recipes?query={term}&offset=0&limit=100
|
||
```
|
||
|
||
Verified search queries (with approximate counts as of 2026-06):
|
||
|
||
| Query | Count | Usable offsets |
|
||
|-------|-------|----------------|
|
||
| Hauptspeise | ~158,000 | 0–~1000 |
|
||
| Kuchen | ~58,000 | 0–~1000 |
|
||
| Vegan | ~28,000 | 0–~1000 |
|
||
| Italienisch | ~7,000 | 0–~1000 |
|
||
| (blank) | ~384,000 | 0–~2000 (barely) |
|
||
|
||
**Search offset ceiling:** ~1000 results (10 pages of 100). Beyond that, the search returns empty.
|
||
|
||
## Deep Scraper Strategy for 25k+ Recipes (v5.2)
|
||
|
||
Because no single query reaches beyond ~1000–2000 offsets, the scraper must **rotate queries**:
|
||
|
||
1. Maintain a list of 50+ search terms (categories, ingredients, cuisines)
|
||
2. Each run: pick a rotation window (e.g., 20 terms)
|
||
3. For each term: random offset between 0 and ~100, fetch listing
|
||
4. Filter known IDs, fetch details in parallel
|
||
5. Rotate to next term
|
||
|
||
This prevents seed exhaustion while staying inside each query's usable window.
|
||
|
||
```python
|
||
QUERIES = [
|
||
"Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
|
||
"Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
|
||
"Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
|
||
"Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
|
||
"Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
|
||
"Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
|
||
"Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
|
||
"Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
|
||
]
|
||
|
||
def _fetch_list(query, offset, limit=100):
|
||
url = f"https://api.chefkoch.de/v2/recipes?query={query}&offset={offset}&limit={limit}"
|
||
...
|
||
```
|
||
|
||
## Performance observed (2026-06-22):
|
||
|
||
**Initial projection under-estimated actual throughput significantly.**
|
||
|
||
With 12 workers: ~10–15 recipes/sec (throughput ceiling is detail-fetch latency, not listing)
|
||
With 16 workers: ~800–900 recipes/minute (~13–15 recipes/sec)
|
||
With 32 workers: ~750 recipes/minute sustained (diminishing returns from context switching)
|
||
|
||
**Key insight:** The listing fetch is negligible latency (~100ms). Detail fetch dominates (~200ms per recipe at 32 concurrent). 16 concurrent detail fetches saturates the network + server response pipe. More workers add no value.
|
||
|
||
**Recommended configuration:**
|
||
- 16 ThreadPool workers (sweet spot)
|
||
- 0.3s delay between listing batches
|
||
- limit=100 per listing request
|
||
- Each run rotates to next query when hitting the offset ceiling
|
||
|
||
## State Management for API Scraper
|
||
|
||
### State file format
|
||
|
||
```json
|
||
{
|
||
"query_index": 0,
|
||
"api_offset": 0,
|
||
"source_counts": {"chefkoch": 25020},
|
||
"new_this_run": 0,
|
||
"failed_listings": 0
|
||
}
|
||
```
|
||
|
||
**Field semantics:**
|
||
- `query_index` — position in `QUERIES` list (rotates when exhausted)
|
||
- `api_offset` — current offset within the active query's pagination window
|
||
- `source_counts` — total recipes scraped per source
|
||
- `new_this_run` — deduplicated count since last state write
|
||
- `failed_listings` — consecutive failed listing requests (used for fast-exit)
|
||
|
||
### CRITICAL: Offset increment must be `+= limit`, not `+= 1`
|
||
|
||
The scraper fetches **batches of `limit` recipes per listing call**. State must advance `api_offset` by `limit` (e.g., 100), not by 1. Incrementing by 1 causes the same 100-recipe batch to be re-fetched on every subsequent run, producing **zero new recipes** while appearing to "run successfully".
|
||
|
||
```python
|
||
# CORRECT — advances to next batch
|
||
def main():
|
||
state = _load_state()
|
||
offset = state.get("api_offset", 0)
|
||
limit = 100
|
||
# ... fetch listing at offset ...
|
||
state["api_offset"] = offset + limit # 0 → 100 → 200
|
||
_save_state(state)
|
||
|
||
# WRONG — re-fetches same batch every time
|
||
def main():
|
||
state = _load_state()
|
||
offset = state.get("api_offset", 0)
|
||
# ... fetch listing at offset ...
|
||
state["api_offset"] = offset + 1 # 0 → 1; next run: fetches offset 1 (same as offset 0 in effect)
|
||
```
|
||
|
||
**The same batch always returns the same 100 recipe IDs.** Deduplication drops everything, `new_this_run` stays at 0, but the state increments by 1 each run, crawling nowhere at ~1 recipe offset per launch.
|
||
|
||
## Pitfalls
|
||
|
||
1. **Do not use** `api.chefkoch.de/v2/recipes/{id}/ingredients` — returns `resource_not_found` (404)
|
||
2. **Ingredients live inside the detail response** under `ingredientGroups` — there is no separate endpoint
|
||
3. **Tag list may contain empty strings** — filter with `.strip()` before storage
|
||
4. **Count is approximate** — the `count` field drifts slightly; do not hardcode it as a loop bound
|
||
5. **Offset-based pagination with a hard ceiling** — unfiltered listing exhausts at ~2000; search queries at ~1000
|
||
6. **Query parameter filtering is NOT supported on the listing endpoint** — `sort=`, `difficulty=`, etc. are ignored
|
||
7. **Do not cache the `count` field as a loop bound** — use query rotation instead
|
||
|
||
## Headers Required
|
||
|
||
Minimal headers suffice; full mimicry not required:
|
||
|
||
```python
|
||
headers = {
|
||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||
"Accept": "application/json",
|
||
"Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
|
||
"Referer": "https://www.chefkoch.de/",
|
||
}
|
||
```
|