Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
+178
@@ -0,0 +1,178 @@
|
||||
# Chefkoch.de: Hybrid Scraper (Playwright + Requests + JSON-LD)
|
||||
|
||||
Chefkoch.de (380k+ Rezepte) blockiert **direkte `requests` auf Suchseiten** (liefert JS-Shell statt Inhalt), **aber** einzelne Rezept-Seiten können über **Requests + JSON-LD** geparst werden. Die optimale Architektur ist ein **Zwei-Phasen-Ansatz**.
|
||||
|
||||
## Warum der alte Ansatz scheitert
|
||||
|
||||
**Falsch:** `recipe_scrapers.scrape_html(page.content(), url)` auf Chefkoch-Suchseiten — Chefkoch erkennt nicht-JS-Clients und liefert leere DOM-Shells.
|
||||
|
||||
**Falsch:** Playwright pro Rezept — zu langsam (5s+ pro Rezept) und überfordert bei 100+ Rezepten.
|
||||
|
||||
## Korrekte Architektur: Zwei Phasen
|
||||
|
||||
### Phase 1: Link-Sammlung via Playwright (Browser nötig)
|
||||
|
||||
Nur für die **Suchseite** — Chefkoch rendernt Ergebnisse client-side.
|
||||
|
||||
```python
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
async with async_playwright() as p:
|
||||
browser = await p.chromium.launch(headless=True)
|
||||
page = await browser.new_page(
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
)
|
||||
await page.goto("https://www.chefkoch.de/rs/s0/{query}/Rezepte.html",
|
||||
wait_until="networkidle", timeout=30000)
|
||||
|
||||
# Scroll to load more results
|
||||
for _ in range(15):
|
||||
links = await page.query_selector_all("a[href*='rezepte/']")
|
||||
for el in links:
|
||||
href = await el.get_attribute("href")
|
||||
if href and "/rezepte/" in href:
|
||||
if not any(skip in href for skip in ["was-koche", "was-backe", "alle-rezepte", "/rs/s"]):
|
||||
recipe_links.add(href.split("?")[0])
|
||||
await page.evaluate("window.scrollBy(0, 500)")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
await browser.close()
|
||||
```
|
||||
|
||||
**Pitfall:** Keine `.recipe-list` oder `.bi-recipe-item` Selektoren verwenden — diese Klassenamen ändern sich. Stattdessen: `a[href*='rezepte/']` mit URL-Pattern-Filtern.
|
||||
|
||||
### Phase 2: Rezept-Daten via Requests + JSON-LD (kein Browser)
|
||||
|
||||
**Schnell** (~300ms statt 5s pro Rezept) und **zuverlässiger** als DOM-Scraping.
|
||||
|
||||
```python
|
||||
import urllib.request, re, json
|
||||
|
||||
def fetch_recipe_jsonld(url: str) -> dict:
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
# Extract JSON-LD blocks
|
||||
blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
|
||||
|
||||
for block in blocks:
|
||||
try:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return item
|
||||
elif data.get("@type") == "Recipe":
|
||||
return data
|
||||
except:
|
||||
continue
|
||||
return None
|
||||
|
||||
# Usage
|
||||
recipe = fetch_recipe_jsonld("https://www.chefkoch.de/rezepte/...")
|
||||
name = recipe["name"]
|
||||
ingredients = recipe.get("recipeIngredient", []) # List of strings like "4 EL Sojasauce"
|
||||
prep_time = recipe.get("prepTime", "") # ISO 8601: "PT25M"
|
||||
rating = recipe.get("aggregateRating", {}).get("ratingValue", "n/a")
|
||||
```
|
||||
|
||||
## Wichtige JSON-LD Felder
|
||||
|
||||
| Feld | Typ | Beispiel |
|
||||
|------|-----|----------|
|
||||
| `name` | str | `"Chicken Teriyaki - der japanische Klassiker"` |
|
||||
| `recipeIngredient` | list[str] | `["4 EL Sojasauce", "250 g Hähnchenbrust"]` |
|
||||
| `recipeInstructions` | list[dict/text] | `[{"@type": "HowToStep", "text": "..."}]` |
|
||||
| `prepTime` / `cookTime` / `totalTime` | str (ISO 8601) | `"PT25M"`, `"PT1H30M"` |
|
||||
| `aggregateRating.ratingValue` | float | `4.79` |
|
||||
| `aggregateRating.ratingCount` | int | `251` |
|
||||
|
||||
## Zutaten-String Parsen
|
||||
|
||||
`recipeIngredient` liefert rote Strings. Parser-Heuristik:
|
||||
|
||||
```python
|
||||
def parse_ingredient(s: str) -> dict:
|
||||
s = re.sub(r"[\t\n]+", " ", s.strip())
|
||||
parts = s.split(None, 2) # maxsplit=2
|
||||
|
||||
# Default
|
||||
amount, unit, name = 1.0, "Stk", s
|
||||
|
||||
# Try: <number> [unit] <name>
|
||||
m = re.match(r"([\d.,/]+(?:\s*-\s*[\d.,/]+)?)", parts[0])
|
||||
if m:
|
||||
num_str = m.group(1).replace(",", ".")
|
||||
# Parse number (handles fractions, ranges)
|
||||
if "-" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("-")]
|
||||
amount = (a + b) / 2
|
||||
elif "/" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("/")]
|
||||
amount = a / b
|
||||
else:
|
||||
amount = float(num_str.strip())
|
||||
|
||||
# Check if second part is known unit
|
||||
KNOWN_UNITS = {"el", "tl", "g", "kg", "ml", "l", "cl", "dl",
|
||||
"stk", "stück", "packung", "pck", "bund", "zweig",
|
||||
"dose", "glas", "tasse", "prise", "scheibe"}
|
||||
if len(parts) >= 2 and parts[1].lower().rstrip(".") in KNOWN_UNITS:
|
||||
unit = parts[1]
|
||||
name = parts[2] if len(parts) > 2 else ""
|
||||
else:
|
||||
name = " ".join(parts[1:])
|
||||
|
||||
return {"item": name.strip(), "amount": amount, "unit": unit}
|
||||
```
|
||||
|
||||
**Pitfall:** regex mit `re.match(r"...", first)` und `split(None, 2)` statt komplexem Gesamt-regex — vermeidet "unbalanced parenthesis" Fehler bei verschachtelten Klammern in Zutaten-Namen wie "Reiswein (Mirin oder Sake)".
|
||||
|
||||
## Playwright Install
|
||||
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```python
|
||||
python3 -c "from playwright.sync_api import sync_playwright; \
|
||||
p = sync_playwright().start(); \
|
||||
b = p.chromium.launch(); b.close(); p.stop(); \
|
||||
print('Chromium OK')"
|
||||
```
|
||||
|
||||
**Fehler ohne Installation:**
|
||||
```
|
||||
Error: Executable doesn't exist at ~/.cache/ms-playwright/chromium-1071/...
|
||||
Please run: playwright install
|
||||
```
|
||||
|
||||
**Im Hintergrund-Cronjob:** Immer `async_playwright` verwenden, nie `sync_playwright` — blockiert den Event Loop.
|
||||
|
||||
## Performance-Vergleich
|
||||
|
||||
| Ansatz | Zeit pro Rezept | 100 Rezepte |
|
||||
|--------|----------------|-------------|
|
||||
| Playwright pro Rezept | 5–8s | 8–13 Min |
|
||||
| **Hybrid (recommended)** | **0.3s (HTTP) + 3s Suchseite** | **~3 Min** |
|
||||
|
||||
## Dynamische Queries aus Interview-Daten
|
||||
|
||||
| Interview-Input | Generierte Query |
|
||||
|-----------------|-----------------|
|
||||
| 27°C + asiatisch | "asiatisch leicht huhn" |
|
||||
| Sarah + low-carb | "low carb spargel fisch" |
|
||||
| Regen + Familie | "einfach familie warm" |
|
||||
|
||||
## Familien-Fit-Score
|
||||
|
||||
- Asiatisch → +2.0 (Dominik)
|
||||
- Low-Carb/Salat/Fisch → +1.5 (Sarah)
|
||||
- Nudeln/Huhn/Eier → +1.0 (Cleo)
|
||||
- Bewertung >4.5 → +ratingValue × 0.5
|
||||
- Gemüse-Vielfalt → +0.3 pro Art
|
||||
Reference in New Issue
Block a user