Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
+185
@@ -0,0 +1,185 @@
|
||||
# Mass Scraping: Chefkoch-Harvesting (>1000 Rezepte)
|
||||
|
||||
## Überblick
|
||||
|
||||
Bei Zielen >1000 Rezepten reicht der Single-URL-Fall-back-Modus nicht aus. Chefkoch bietet
|
||||
380.000+ Rezepte; die verlässliche Technik ist ein **Two-Phase Discovery + Harvest**
|
||||
mit Playwright-Multi-Worker-Architektur.
|
||||
|
||||
## Architektur-Blueprint
|
||||
|
||||
```
|
||||
Discovery-Phase Harvest-Phase
|
||||
──────────────────────── ──────────────────
|
||||
Search-Seite (Playwright) → Worker-Pool (5–10 Threads)
|
||||
└─ Scroll-Trigger → │- Browser-Context pro Worker
|
||||
└─ a[href*='rezepte/'] → │- JSON-LD/DOM-Fallback
|
||||
│ │- Append-Only JSONL
|
||||
▼ ↓
|
||||
[URL-Queue] real_recipes.jsonl
|
||||
│ scraper_state.json
|
||||
└─ Deduplication (URL-Hash-Set)
|
||||
```
|
||||
|
||||
## Phase 1: Link-Discovery
|
||||
|
||||
```python
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
def discover_urls(seed: str, page_num: int) -> list[str]:
|
||||
search_url = f"https://www.chefkoch.de/rs/s{page_num}/{seed}/Rezepte.html"
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 900})
|
||||
page.goto(search_url, wait_until="networkidle", timeout=25000)
|
||||
# Lazy-load trigger
|
||||
for _ in range(4):
|
||||
page.evaluate("window.scrollBy(0, 1000)")
|
||||
time.sleep(0.4)
|
||||
links = page.evaluate('''() => {
|
||||
const cards = document.querySelectorAll('article[class*="recipe"], .ds-recipe-card');
|
||||
const found = [];
|
||||
for (const card of cards) {
|
||||
const a = card.querySelector('a[href*="/rezepte/"]');
|
||||
if (a && a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
}
|
||||
if (found.length === 0) {
|
||||
document.querySelectorAll('a[href*="/rezepte/"]').forEach(a => {
|
||||
if (a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
});
|
||||
}
|
||||
return [...new Set(found)];
|
||||
}''')
|
||||
browser.close()
|
||||
# Filter to real recipe URLs only
|
||||
return [u for u in links if re.search(r'/rezepte/\d+/', u)]
|
||||
```
|
||||
|
||||
### Seed-Strategie
|
||||
|
||||
30+ breite Suchbegriffe (einfach, asiatisch, hähnchen, dessert, ...) erzeugen
|
||||
~150 Suchseiten = Tausende eindeutige Rezept-URLs. Der Queue wird gemischt, um
|
||||
keine Quelle zu überlasten.
|
||||
|
||||
## Phase 2: Rezept-Harvest
|
||||
|
||||
### A) JSON-LD-Extraktion (primär)
|
||||
|
||||
```python
|
||||
def scrape_jsonld(page, url: str) -> dict | None:
|
||||
page.goto(url, wait_until="networkidle", timeout=20000)
|
||||
scripts = page.locator('script[type="application/ld+json"]').all()
|
||||
for script in scripts:
|
||||
try:
|
||||
data = json.loads(script.inner_text())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return _normalize(item, url)
|
||||
elif data.get("@type") == "Recipe":
|
||||
return _normalize(data, url)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
```
|
||||
|
||||
### B) DOM-Fallback (selektiv)
|
||||
|
||||
Wenn JSON-LD fehlt oder leer:
|
||||
|
||||
```python
|
||||
# Zutaten-Tabelle
|
||||
rows = page.locator("table.ingredients tbody tr").all()
|
||||
# Anleitung
|
||||
steps = page.locator("article p").all()
|
||||
```
|
||||
|
||||
## Speicherformat: JSON Lines (JSONL)
|
||||
|
||||
**Niemals ein großes JSON-Dump verwenden** — bei Absturz ist alles verloren.
|
||||
|
||||
```
|
||||
real_recipes.jsonl # append-only
|
||||
scraper_state.json # resume-Informationen
|
||||
scraper.log # menschenlesbarer Fortschritt
|
||||
```
|
||||
|
||||
### JSONL-Struktur pro Zeile
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "347aaae514e25800",
|
||||
"name": "Erdbeerkuchen - einfaches Rezept",
|
||||
"description": "...",
|
||||
"category": "Backen",
|
||||
"tags": ["Sommer", "Kuchen", "Frühling"],
|
||||
"difficulty": "einfach",
|
||||
"base_servings": "12 Port.",
|
||||
"prep_time": "P0DT0H40M",
|
||||
"cook_time": "P0DT0H25M",
|
||||
"total_time": "P0DT1H5M",
|
||||
"ingredients": [{"raw": "125 g Zucker"}, ...],
|
||||
"instructions": ["Schritt 1 ...", "Schritt 2 ..."],
|
||||
"nutrition": {},
|
||||
"rating": "4.77",
|
||||
"review_count": "724",
|
||||
"author": "Loeckchen87",
|
||||
"images": ["https://img.chefkoch-cdn.de/..."],
|
||||
"source_url": "https://www.chefkoch.de/rezepte/...",
|
||||
"sources": [{"site": "chefkoch.de", "url": "...", "type": "de"}],
|
||||
"_scraped_at": "2026-06-18T05:13:31"
|
||||
}
|
||||
```
|
||||
|
||||
## Worker-Pool & Concurrency
|
||||
|
||||
```python
|
||||
WORKERS = 5
|
||||
DELAY_RANGE = (1.0, 2.5) # pro Worker individuell
|
||||
```
|
||||
|
||||
**Niemals 20+ Worker** — Chefkoch merkt sich IPs und verzögert massiv. 5 Worker
|
||||
mit 1–2 s Delay liefern
|
||||
60–80 Rezepte/Minute, bei 5.000 Rezepten ≈ 60–90 Min.
|
||||
|
||||
## State-Management
|
||||
|
||||
```json
|
||||
{
|
||||
"started_at": "2026-06-18T05:13:20",
|
||||
"last_update": "2026-06-18T05:22:26",
|
||||
"recipes_count": 5000,
|
||||
"completed_seeds": {"asiatisch": 4, "pasta": 2, ...},
|
||||
"failed_urls": [],
|
||||
"worker_stats": {
|
||||
"0": {"scraped": 1020, "errors": 3},
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Patience & Monitoring
|
||||
|
||||
- **Starten**: Hintergrundprozess, kein Blocking der CLI
|
||||
- **Log**: `tail -f scraper.log`
|
||||
- **Zwischenspeichern**: Nach jeweils 10 Rezepten die DB flushen
|
||||
- **Auto-Refill-Cronjob**: Täglich 06:00 — prüft Count, startet nur wenn < Ziel
|
||||
|
||||
## Bekannte Fehler & Workarounds
|
||||
|
||||
| Fehler | Ursache | Workaround |
|
||||
|--------|---------|------------|
|
||||
| `links = 0` auf Search-Seite | Cookie-Banner blockiert Scroll | Klick auf Zustimmen vor Scroll |
|
||||
| `JSONDecodeError` bei JSON-LD | Verunreinigte Blocks mit HTML | `try/except` pro Block |
|
||||
| `TimeoutError` bei Recipe-Seite | Langsamer Server | Erhöhe auf 30 s, 1 Retry |
|
||||
| `Duplicate URLs` | `/rezepte/123/` vs `/rezepte/123/Name.html` | Immer `split('#')[0].split('?')[0]` |
|
||||
| Worker-Crash | Memory-Overflow bei Langzeitsession | Browser pro Worker alle 500 Seiten restarten |
|
||||
|
||||
## Ergebnis-Kosten pro 5000 Rezepte
|
||||
|
||||
| Metrik | Wert |
|
||||
|--------|------|
|
||||
| Scraping-Zeit | ~60–90 min |
|
||||
| JSONL-Dateigröße | ~30–50 MB |
|
||||
| Einzigartige Rezepte | ~95% der gescrapeten URLs (Deduplication) |
|
||||
| Fehlerrate | <1% (Timeouts / 404) |
|
||||
Reference in New Issue
Block a user