179 lines
6.4 KiB
Markdown
179 lines
6.4 KiB
Markdown
# Chefkoch.de: Hybrid Scraper (Playwright + Requests + JSON-LD)
|
||
|
||
Chefkoch.de (380k+ Rezepte) blockiert **direkte `requests` auf Suchseiten** (liefert JS-Shell statt Inhalt), **aber** einzelne Rezept-Seiten können über **Requests + JSON-LD** geparst werden. Die optimale Architektur ist ein **Zwei-Phasen-Ansatz**.
|
||
|
||
## Warum der alte Ansatz scheitert
|
||
|
||
**Falsch:** `recipe_scrapers.scrape_html(page.content(), url)` auf Chefkoch-Suchseiten — Chefkoch erkennt nicht-JS-Clients und liefert leere DOM-Shells.
|
||
|
||
**Falsch:** Playwright pro Rezept — zu langsam (5s+ pro Rezept) und überfordert bei 100+ Rezepten.
|
||
|
||
## Korrekte Architektur: Zwei Phasen
|
||
|
||
### Phase 1: Link-Sammlung via Playwright (Browser nötig)
|
||
|
||
Nur für die **Suchseite** — Chefkoch rendernt Ergebnisse client-side.
|
||
|
||
```python
|
||
from playwright.async_api import async_playwright
|
||
|
||
async with async_playwright() as p:
|
||
browser = await p.chromium.launch(headless=True)
|
||
page = await browser.new_page(
|
||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||
)
|
||
await page.goto("https://www.chefkoch.de/rs/s0/{query}/Rezepte.html",
|
||
wait_until="networkidle", timeout=30000)
|
||
|
||
# Scroll to load more results
|
||
for _ in range(15):
|
||
links = await page.query_selector_all("a[href*='rezepte/']")
|
||
for el in links:
|
||
href = await el.get_attribute("href")
|
||
if href and "/rezepte/" in href:
|
||
if not any(skip in href for skip in ["was-koche", "was-backe", "alle-rezepte", "/rs/s"]):
|
||
recipe_links.add(href.split("?")[0])
|
||
await page.evaluate("window.scrollBy(0, 500)")
|
||
await asyncio.sleep(0.5)
|
||
|
||
await browser.close()
|
||
```
|
||
|
||
**Pitfall:** Keine `.recipe-list` oder `.bi-recipe-item` Selektoren verwenden — diese Klassenamen ändern sich. Stattdessen: `a[href*='rezepte/']` mit URL-Pattern-Filtern.
|
||
|
||
### Phase 2: Rezept-Daten via Requests + JSON-LD (kein Browser)
|
||
|
||
**Schnell** (~300ms statt 5s pro Rezept) und **zuverlässiger** als DOM-Scraping.
|
||
|
||
```python
|
||
import urllib.request, re, json
|
||
|
||
def fetch_recipe_jsonld(url: str) -> dict:
|
||
req = urllib.request.Request(url, headers={
|
||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||
})
|
||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||
html = resp.read().decode("utf-8", errors="ignore")
|
||
|
||
# Extract JSON-LD blocks
|
||
blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
|
||
|
||
for block in blocks:
|
||
try:
|
||
data = json.loads(block.strip())
|
||
if isinstance(data, list):
|
||
for item in data:
|
||
if item.get("@type") == "Recipe":
|
||
return item
|
||
elif data.get("@type") == "Recipe":
|
||
return data
|
||
except:
|
||
continue
|
||
return None
|
||
|
||
# Usage
|
||
recipe = fetch_recipe_jsonld("https://www.chefkoch.de/rezepte/...")
|
||
name = recipe["name"]
|
||
ingredients = recipe.get("recipeIngredient", []) # List of strings like "4 EL Sojasauce"
|
||
prep_time = recipe.get("prepTime", "") # ISO 8601: "PT25M"
|
||
rating = recipe.get("aggregateRating", {}).get("ratingValue", "n/a")
|
||
```
|
||
|
||
## Wichtige JSON-LD Felder
|
||
|
||
| Feld | Typ | Beispiel |
|
||
|------|-----|----------|
|
||
| `name` | str | `"Chicken Teriyaki - der japanische Klassiker"` |
|
||
| `recipeIngredient` | list[str] | `["4 EL Sojasauce", "250 g Hähnchenbrust"]` |
|
||
| `recipeInstructions` | list[dict/text] | `[{"@type": "HowToStep", "text": "..."}]` |
|
||
| `prepTime` / `cookTime` / `totalTime` | str (ISO 8601) | `"PT25M"`, `"PT1H30M"` |
|
||
| `aggregateRating.ratingValue` | float | `4.79` |
|
||
| `aggregateRating.ratingCount` | int | `251` |
|
||
|
||
## Zutaten-String Parsen
|
||
|
||
`recipeIngredient` liefert rote Strings. Parser-Heuristik:
|
||
|
||
```python
|
||
def parse_ingredient(s: str) -> dict:
|
||
s = re.sub(r"[\t\n]+", " ", s.strip())
|
||
parts = s.split(None, 2) # maxsplit=2
|
||
|
||
# Default
|
||
amount, unit, name = 1.0, "Stk", s
|
||
|
||
# Try: <number> [unit] <name>
|
||
m = re.match(r"([\d.,/]+(?:\s*-\s*[\d.,/]+)?)", parts[0])
|
||
if m:
|
||
num_str = m.group(1).replace(",", ".")
|
||
# Parse number (handles fractions, ranges)
|
||
if "-" in num_str:
|
||
a, b = [float(x.strip()) for x in num_str.split("-")]
|
||
amount = (a + b) / 2
|
||
elif "/" in num_str:
|
||
a, b = [float(x.strip()) for x in num_str.split("/")]
|
||
amount = a / b
|
||
else:
|
||
amount = float(num_str.strip())
|
||
|
||
# Check if second part is known unit
|
||
KNOWN_UNITS = {"el", "tl", "g", "kg", "ml", "l", "cl", "dl",
|
||
"stk", "stück", "packung", "pck", "bund", "zweig",
|
||
"dose", "glas", "tasse", "prise", "scheibe"}
|
||
if len(parts) >= 2 and parts[1].lower().rstrip(".") in KNOWN_UNITS:
|
||
unit = parts[1]
|
||
name = parts[2] if len(parts) > 2 else ""
|
||
else:
|
||
name = " ".join(parts[1:])
|
||
|
||
return {"item": name.strip(), "amount": amount, "unit": unit}
|
||
```
|
||
|
||
**Pitfall:** regex mit `re.match(r"...", first)` und `split(None, 2)` statt komplexem Gesamt-regex — vermeidet "unbalanced parenthesis" Fehler bei verschachtelten Klammern in Zutaten-Namen wie "Reiswein (Mirin oder Sake)".
|
||
|
||
## Playwright Install
|
||
|
||
```bash
|
||
pip install playwright
|
||
playwright install chromium
|
||
```
|
||
|
||
**Verify:**
|
||
```python
|
||
python3 -c "from playwright.sync_api import sync_playwright; \
|
||
p = sync_playwright().start(); \
|
||
b = p.chromium.launch(); b.close(); p.stop(); \
|
||
print('Chromium OK')"
|
||
```
|
||
|
||
**Fehler ohne Installation:**
|
||
```
|
||
Error: Executable doesn't exist at ~/.cache/ms-playwright/chromium-1071/...
|
||
Please run: playwright install
|
||
```
|
||
|
||
**Im Hintergrund-Cronjob:** Immer `async_playwright` verwenden, nie `sync_playwright` — blockiert den Event Loop.
|
||
|
||
## Performance-Vergleich
|
||
|
||
| Ansatz | Zeit pro Rezept | 100 Rezepte |
|
||
|--------|----------------|-------------|
|
||
| Playwright pro Rezept | 5–8s | 8–13 Min |
|
||
| **Hybrid (recommended)** | **0.3s (HTTP) + 3s Suchseite** | **~3 Min** |
|
||
|
||
## Dynamische Queries aus Interview-Daten
|
||
|
||
| Interview-Input | Generierte Query |
|
||
|-----------------|-----------------|
|
||
| 27°C + asiatisch | "asiatisch leicht huhn" |
|
||
| Sarah + low-carb | "low carb spargel fisch" |
|
||
| Regen + Familie | "einfach familie warm" |
|
||
|
||
## Familien-Fit-Score
|
||
|
||
- Asiatisch → +2.0 (Dominik)
|
||
- Low-Carb/Salat/Fisch → +1.5 (Sarah)
|
||
- Nudeln/Huhn/Eier → +1.0 (Cleo)
|
||
- Bewertung >4.5 → +ratingValue × 0.5
|
||
- Gemüse-Vielfalt → +0.3 pro Art
|