186 lines
6.0 KiB
Markdown
186 lines
6.0 KiB
Markdown
# Mass Scraping: Chefkoch-Harvesting (>1000 Rezepte)
|
||
|
||
## Überblick
|
||
|
||
Bei Zielen >1000 Rezepten reicht der Single-URL-Fall-back-Modus nicht aus. Chefkoch bietet
|
||
380.000+ Rezepte; die verlässliche Technik ist ein **Two-Phase Discovery + Harvest**
|
||
mit Playwright-Multi-Worker-Architektur.
|
||
|
||
## Architektur-Blueprint
|
||
|
||
```
|
||
Discovery-Phase Harvest-Phase
|
||
──────────────────────── ──────────────────
|
||
Search-Seite (Playwright) → Worker-Pool (5–10 Threads)
|
||
└─ Scroll-Trigger → │- Browser-Context pro Worker
|
||
└─ a[href*='rezepte/'] → │- JSON-LD/DOM-Fallback
|
||
│ │- Append-Only JSONL
|
||
▼ ↓
|
||
[URL-Queue] real_recipes.jsonl
|
||
│ scraper_state.json
|
||
└─ Deduplication (URL-Hash-Set)
|
||
```
|
||
|
||
## Phase 1: Link-Discovery
|
||
|
||
```python
|
||
from playwright.sync_api import sync_playwright
|
||
|
||
def discover_urls(seed: str, page_num: int) -> list[str]:
|
||
search_url = f"https://www.chefkoch.de/rs/s{page_num}/{seed}/Rezepte.html"
|
||
with sync_playwright() as p:
|
||
browser = p.chromium.launch(headless=True)
|
||
page = browser.new_page(viewport={"width": 1280, "height": 900})
|
||
page.goto(search_url, wait_until="networkidle", timeout=25000)
|
||
# Lazy-load trigger
|
||
for _ in range(4):
|
||
page.evaluate("window.scrollBy(0, 1000)")
|
||
time.sleep(0.4)
|
||
links = page.evaluate('''() => {
|
||
const cards = document.querySelectorAll('article[class*="recipe"], .ds-recipe-card');
|
||
const found = [];
|
||
for (const card of cards) {
|
||
const a = card.querySelector('a[href*="/rezepte/"]');
|
||
if (a && a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||
}
|
||
if (found.length === 0) {
|
||
document.querySelectorAll('a[href*="/rezepte/"]').forEach(a => {
|
||
if (a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||
});
|
||
}
|
||
return [...new Set(found)];
|
||
}''')
|
||
browser.close()
|
||
# Filter to real recipe URLs only
|
||
return [u for u in links if re.search(r'/rezepte/\d+/', u)]
|
||
```
|
||
|
||
### Seed-Strategie
|
||
|
||
30+ breite Suchbegriffe (einfach, asiatisch, hähnchen, dessert, ...) erzeugen
|
||
~150 Suchseiten = Tausende eindeutige Rezept-URLs. Der Queue wird gemischt, um
|
||
keine Quelle zu überlasten.
|
||
|
||
## Phase 2: Rezept-Harvest
|
||
|
||
### A) JSON-LD-Extraktion (primär)
|
||
|
||
```python
|
||
def scrape_jsonld(page, url: str) -> dict | None:
|
||
page.goto(url, wait_until="networkidle", timeout=20000)
|
||
scripts = page.locator('script[type="application/ld+json"]').all()
|
||
for script in scripts:
|
||
try:
|
||
data = json.loads(script.inner_text())
|
||
if isinstance(data, list):
|
||
for item in data:
|
||
if item.get("@type") == "Recipe":
|
||
return _normalize(item, url)
|
||
elif data.get("@type") == "Recipe":
|
||
return _normalize(data, url)
|
||
except Exception:
|
||
continue
|
||
return None
|
||
```
|
||
|
||
### B) DOM-Fallback (selektiv)
|
||
|
||
Wenn JSON-LD fehlt oder leer:
|
||
|
||
```python
|
||
# Zutaten-Tabelle
|
||
rows = page.locator("table.ingredients tbody tr").all()
|
||
# Anleitung
|
||
steps = page.locator("article p").all()
|
||
```
|
||
|
||
## Speicherformat: JSON Lines (JSONL)
|
||
|
||
**Niemals ein großes JSON-Dump verwenden** — bei Absturz ist alles verloren.
|
||
|
||
```
|
||
real_recipes.jsonl # append-only
|
||
scraper_state.json # resume-Informationen
|
||
scraper.log # menschenlesbarer Fortschritt
|
||
```
|
||
|
||
### JSONL-Struktur pro Zeile
|
||
|
||
```json
|
||
{
|
||
"id": "347aaae514e25800",
|
||
"name": "Erdbeerkuchen - einfaches Rezept",
|
||
"description": "...",
|
||
"category": "Backen",
|
||
"tags": ["Sommer", "Kuchen", "Frühling"],
|
||
"difficulty": "einfach",
|
||
"base_servings": "12 Port.",
|
||
"prep_time": "P0DT0H40M",
|
||
"cook_time": "P0DT0H25M",
|
||
"total_time": "P0DT1H5M",
|
||
"ingredients": [{"raw": "125 g Zucker"}, ...],
|
||
"instructions": ["Schritt 1 ...", "Schritt 2 ..."],
|
||
"nutrition": {},
|
||
"rating": "4.77",
|
||
"review_count": "724",
|
||
"author": "Loeckchen87",
|
||
"images": ["https://img.chefkoch-cdn.de/..."],
|
||
"source_url": "https://www.chefkoch.de/rezepte/...",
|
||
"sources": [{"site": "chefkoch.de", "url": "...", "type": "de"}],
|
||
"_scraped_at": "2026-06-18T05:13:31"
|
||
}
|
||
```
|
||
|
||
## Worker-Pool & Concurrency
|
||
|
||
```python
|
||
WORKERS = 5
|
||
DELAY_RANGE = (1.0, 2.5) # pro Worker individuell
|
||
```
|
||
|
||
**Niemals 20+ Worker** — Chefkoch merkt sich IPs und verzögert massiv. 5 Worker
|
||
mit 1–2 s Delay liefern
|
||
60–80 Rezepte/Minute, bei 5.000 Rezepten ≈ 60–90 Min.
|
||
|
||
## State-Management
|
||
|
||
```json
|
||
{
|
||
"started_at": "2026-06-18T05:13:20",
|
||
"last_update": "2026-06-18T05:22:26",
|
||
"recipes_count": 5000,
|
||
"completed_seeds": {"asiatisch": 4, "pasta": 2, ...},
|
||
"failed_urls": [],
|
||
"worker_stats": {
|
||
"0": {"scraped": 1020, "errors": 3},
|
||
...
|
||
}
|
||
}
|
||
```
|
||
|
||
## Patience & Monitoring
|
||
|
||
- **Starten**: Hintergrundprozess, kein Blocking der CLI
|
||
- **Log**: `tail -f scraper.log`
|
||
- **Zwischenspeichern**: Nach jeweils 10 Rezepten die DB flushen
|
||
- **Auto-Refill-Cronjob**: Täglich 06:00 — prüft Count, startet nur wenn < Ziel
|
||
|
||
## Bekannte Fehler & Workarounds
|
||
|
||
| Fehler | Ursache | Workaround |
|
||
|--------|---------|------------|
|
||
| `links = 0` auf Search-Seite | Cookie-Banner blockiert Scroll | Klick auf Zustimmen vor Scroll |
|
||
| `JSONDecodeError` bei JSON-LD | Verunreinigte Blocks mit HTML | `try/except` pro Block |
|
||
| `TimeoutError` bei Recipe-Seite | Langsamer Server | Erhöhe auf 30 s, 1 Retry |
|
||
| `Duplicate URLs` | `/rezepte/123/` vs `/rezepte/123/Name.html` | Immer `split('#')[0].split('?')[0]` |
|
||
| Worker-Crash | Memory-Overflow bei Langzeitsession | Browser pro Worker alle 500 Seiten restarten |
|
||
|
||
## Ergebnis-Kosten pro 5000 Rezepte
|
||
|
||
| Metrik | Wert |
|
||
|--------|------|
|
||
| Scraping-Zeit | ~60–90 min |
|
||
| JSONL-Dateigröße | ~30–50 MB |
|
||
| Einzigartige Rezepte | ~95% der gescrapeten URLs (Deduplication) |
|
||
| Fehlerrate | <1% (Timeouts / 404) |
|