Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,97 @@
|
||||
# Meal-Planning Cron Setup — Known Pitfalls & Fixes
|
||||
|
||||
Condensed troubleshooting guide for the `nutrition-coach` cronjob stack. Each section provides the symptom → root cause → fix.
|
||||
|
||||
---
|
||||
|
||||
## 1. Interactive Interview Via Cronjob Is Impossible
|
||||
|
||||
**Symptom**
|
||||
```
|
||||
Wed 19:30: Agent sends 10 questions to Telegram group.
|
||||
User replies 19:35–19:42 in Telegram.
|
||||
current_interview.json: all answers are null
|
||||
```
|
||||
|
||||
**Root cause**
|
||||
A cronjob is a single-shot process. It sends, then exits. It never reads Telegram inbound messages. User replies arrive in the gateway but there is no running session to pair them with.
|
||||
|
||||
**Fix options**
|
||||
1. **Replace with a non-interactive reminder** (recommended for automation):
|
||||
```bash
|
||||
hermes cron edit <id> --prompt "🔔 Wochenplan: Schreibe \"interview\" oder \"plan\" um loszulegen." --no-agent
|
||||
```
|
||||
User types a keyword in chat → triggers the interview in a live session.
|
||||
2. **Use a live session for the interview.** Load the skill (`/skill meal-planning`) and run the interview interactively. Store the JSON for Friday.
|
||||
|
||||
---
|
||||
|
||||
## 2. Delivery Target Fails on Named Groups
|
||||
|
||||
**Symptom**
|
||||
```
|
||||
delivery error: Telegram send failed:
|
||||
invalid literal for int() with base 10: 'Schoen Inc'
|
||||
```
|
||||
|
||||
**Root cause**
|
||||
Cron `deliver` field expects a numeric `chat_id` or a pre-registered alias. `"telegram:Schoen Inc"` is a display name, not a parseable ID.
|
||||
|
||||
**Fix**
|
||||
Use one of the following verified targets for the nutrition-coach profile:
|
||||
```json
|
||||
"deliver": "telegram:D S" // DM with Dominik
|
||||
"deliver": "telegram:-5265906503" // Schoen Inc group
|
||||
"deliver": "telegram" // home channel
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Override in Cron JSON Is Not Updated by `hermes config set`
|
||||
|
||||
**Symptom**
|
||||
Bulk-switched profile to `vllm/glm-5-2-nvfp4` via `hermes config set`, but cronjob still calls `vllm/moonshotai/kimi-k2.6`.
|
||||
|
||||
**Root cause**
|
||||
Cron jobs hard-code `model` and `provider` inside their own JSON entry in `profiles/nutrition-coach/cron/jobs.json`. The profile `config.yaml` is not consulted at runtime.
|
||||
|
||||
**Fix — Python one-liner**
|
||||
```python
|
||||
import json
|
||||
path = '/home/debian/.hermes/profiles/nutrition-coach/cron/jobs.json'
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
for j in data['jobs']:
|
||||
if 'nutrition' in j['name'].lower():
|
||||
j['model'] = 'vllm/glm-5-2-nvfp4'
|
||||
j['provider'] = 'noris'
|
||||
with open(path, 'w') as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Interview JSON Schema Mismatch
|
||||
|
||||
**Symptom**
|
||||
`nutrition-plan-generator-v4.py` crashes with `KeyError` on fields it expects (e.g. `dominik_ho`, `special_events`).
|
||||
|
||||
**Root cause**
|
||||
The interview data file (`current_interview.json`) was created by a different schema than the generator expects. Cronjobs that produce JSON by LLM free-form output are fragile.
|
||||
|
||||
**Fix**
|
||||
Always use the Interview Schema template (`templates/interview-data-schema.json`) when persisting. Validate before saving:
|
||||
```bash
|
||||
python3 -m json.tool current_interview.json >/dev/null || echo "INVALID JSON"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist After Any Change
|
||||
|
||||
```bash
|
||||
hermes --profile nutrition-coach cron list # jobs running under correct profile?
|
||||
hermes config --profile nutrition-coach | grep Model # profile model correct?
|
||||
python3 -c "import json; d=json.load(open('/home/debian/.hermes/profiles/nutrition-coach/cron/jobs.json')); print([j['model'] for j in d['jobs']])" # cron model correct?
|
||||
cat /home/debian/.hermes/profiles/nutrition-coach/plan_data/current_interview.json | python3 -m json.tool # interview valid?
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
# Generator v4 Scoring Algorithm
|
||||
|
||||
## Data Source (EXCLUSIVE — NO FALLBACK)
|
||||
- **ONLY SOURCE**: `~/.hermes/profiles/nutrition-coach/real_recipes.jsonl` (25k+ Chefkoch recipes, JSONL, one recipe per line)
|
||||
- **NEVER use**: `recipes.json` (old dummy data), live web scraping, or external recipe APIs
|
||||
- **Minimum rating filter**: 4.3 stars (raises quality, reduces noise)
|
||||
- **Excluded categories**: Backen, Kuchen, Dessert (penalty −20)
|
||||
|
||||
## Scoring Dimensions
|
||||
|
||||
| Factor | Weight | Logic |
|
||||
|---|---|---|
|
||||
| Base rating | ×2 | Higher-rated recipes start with advantage |
|
||||
| Weather heat ≥30°C | +6 / −4 | Cold/salad boosted; hot cooking penalized |
|
||||
| Weather heat ≥25°C | +4 / −3 | Same direction, lesser magnitude |
|
||||
| Cold ≤15°C | +4 | Soups/stews/curry boosted |
|
||||
| Perishables (fish, herbs) | +4 early / −5 late | Fresh items forced to Mon–Tue |
|
||||
| Asian boost | +3 | curry, wok, thai, vietnam, ramen tags |
|
||||
| Low-carb / protein | +2 | Dominik weight-goal diet |
|
||||
| Carb boost | +1.5 | Pasta, rice, potato — Sarah preference |
|
||||
| Cleo-friendly | +1.5 | pasta, nudeln, gnocchi, schnell, einfach tags |
|
||||
| Meal-Prep days (Di/Do) | +6 | oven, stew, curry, gratin, lasagne boosted |
|
||||
| Meal-Prep: fast ≤40min | +2 | Quick prep bonus on Di/Do |
|
||||
| Meal-Prep: fry penalty | −10 | Frittiertes / Schnitzel / Tempura bad for meal-prep |
|
||||
| Thursday travel | +10 | ≤30min recipes if special events contain "reise" |
|
||||
| Friday events (picknick) | +15 | salad, wrap, cold, fingerfood boosted |
|
||||
| Monday events (grill) | +15 / +20 | grill tags / barbecue category boosted |
|
||||
| Diversity: new category | +2 | Unseen category gets bonus |
|
||||
| Diversity: name overlap | −8 | Similar names penalized to avoid repetition |
|
||||
| Craving match | +5 | If keyword in name or tags |
|
||||
|
||||
## Cleo Logic
|
||||
- **Joins family meal** if recipe tags contain: pasta, nudeln, gnocchi, kartoffel, reis
|
||||
- **Gets extra dish** otherwise: "Nudeln mit Butter & Käse (frisch)"
|
||||
- Hard rule: NO leftovers for Cleo, ever
|
||||
|
||||
## Output
|
||||
- Compact Telegram summary: 7 lines (one per day) + list count
|
||||
- Full details written to `~/Documents/Essensplanung/KW<NN>_plan.md`
|
||||
- Shopping list to `~/Documents/Essensplanung/KW<NN>_einkaufsliste.md`
|
||||
@@ -0,0 +1,203 @@
|
||||
# International Recipe Sources (Researched 2026-06-28, Updated 2026-06-28)
|
||||
|
||||
Focus: Asian cuisine, light/healthy main courses, internationally diverse. English-language sites with JSON-LD for scraping + translation to German.
|
||||
|
||||
## Tier 1 — Ready to Integrate (JSON-LD confirmed + sitemap available + no AI-bot blocks)
|
||||
|
||||
### BBC Good Food (bbcgoodfood.com) — ~5,045 recipes ⭐ TOP PICK
|
||||
- **Sitemap:** Quarterly recipe sitemaps at `https://www.bbcgoodfood.com/sitemaps/YYYY-QN-recipe.xml` (indexed from `sitemap.xml`)
|
||||
- **Counts per quarter:** Q2-2026: 149, Q1-2026: 156, Q4-2025: 210, Q3-2025: 179, Q2-2025: 182, Q1-2025: 214, Q4-2024: 292, Q3-2024: 179, Q2-2024: 160, Q1-2024: 223, Q4-2023: 290, Q3-2023: 262, Q2-2023: 132, Q1-2023: 203, Q4-2022: 296, Q3-2022: 231, Q2-2022: 272, Q1-2022: 207, Q4-2021: 250, Q3-2021: 206, Q2-2021: 154, Q1-2021: 191, Q4-2020: 278, Q3-2020: 129. **Total: ~5,045**
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Huge international coverage, including dedicated Asian/Indian/Thai sections
|
||||
- **Robots.txt:** Permissive — declares sitemaps, no AI-bot blocks
|
||||
- **Quality:** Professional editorial team, nutrition data, tested recipes
|
||||
- **ID prefix:** `bgf_`
|
||||
|
||||
### RecipeTin Eats (recipetineats.com) — ~1,697 recipes
|
||||
- **Sitemap:** `https://www.recipetineats.com/sitemap_index.xml` → 4 post sitemaps (495+500+499+203 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Pan-Asian (Japanese, Thai, Korean, Chinese), Australian, fusion. ~404 URLs match Asian keywords.
|
||||
- **Robots.txt:** Blocks anthropic-ai and Claude-Web specifically, but does NOT block generic crawlers. Standard UA with polite delays works.
|
||||
- **Quality:** High — detailed instructions, ratings, step photos. Many authentic Asian recipes.
|
||||
- **ID prefix:** `rt_`
|
||||
|
||||
### Woks of Life (thewoksoflife.com) — ~1,555 recipes ⭐ BEST CHINESE
|
||||
- **Sitemap:** `https://thewoksoflife.com/sitemap_index.xml` → post-sitemap.xml (997) + post-sitemap2.xml (558)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Chinese (Cantonese, Sichuan, regional), some Asian-American
|
||||
- **Robots.txt:** Permissive (blocks wp-admin, cdn-cgi, wp-json only)
|
||||
- **Quality:** Family-authored, authentic Chinese techniques, detailed step photos
|
||||
- **ID prefix:** `wol_`
|
||||
|
||||
### SkinnyTaste (skinnytaste.com) — ~2,354 recipes ⭐ BEST LIGHT/HEALTHY
|
||||
- **Sitemap:** `https://www.skinnytaste.com/sitemap_index.xml` → 3 post sitemaps (685+862+807 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Healthy, low-calorie, light mains, WW-friendly. Strong international coverage including Asian-inspired light dishes.
|
||||
- **Robots.txt:** Blocks Google-Extended, GPTBot, ChatGPT-User, PiplBot. Does NOT block generic crawlers.
|
||||
- **Quality:** Excellent — Gina Homolka provides nutrition data (calories, macros) for every recipe. Professionally tested.
|
||||
- **ID prefix:** `st_`
|
||||
|
||||
### Rasa Malaysia (rasamalaysia.com) — ~1,390 recipes ⭐ BEST SE-ASIAN
|
||||
- **Sitemap:** `https://rasamalaysia.com/sitemap_index.xml` → post-sitemap.xml (984) + post-sitemap2.xml (406)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Malaysian, Singaporean, Thai, Chinese, pan-Southeast-Asian
|
||||
- **Robots.txt:** Permissive (blocks cdn-cgi, wp-admin only)
|
||||
- **Quality:** High — authentic Southeast Asian recipes with detailed cultural context
|
||||
- **ID prefix:** `rm_`
|
||||
|
||||
### Pinch of Yum (pinchofyum.com) — ~1,483 recipes
|
||||
- **Sitemap:** `https://pinchofyum.com/sitemap_index.xml` → post-sitemap.xml (912) + post-sitemap2.xml (571)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Comfort food, international, some Asian-inspired
|
||||
- **Robots.txt:** Permissive (blocks wp-admin, search only)
|
||||
- **Quality:** Good — standardized format, nutrition data sometimes included
|
||||
- **ID prefix:** `poy_`
|
||||
|
||||
### Budget Bytes (budgetbytes.com) — ~1,981 recipes
|
||||
- **Sitemap:** `https://www.budgetbytes.com/post-sitemap.xml` (976) + post-sitemap2.xml (974) + post-sitemap3.xml (31)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** American comfort, international fusion, budget-friendly mains
|
||||
- **Robots.txt:** Permissive
|
||||
- **ID prefix:** `bb_`
|
||||
|
||||
### Omnivore's Cookbook (omnivorescookbook.com) — ~772 recipes
|
||||
- **Sitemap:** `https://omnivorescookbook.com/sitemap_index.xml` → post-sitemap.xml (772 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Modern Chinese, some Asian fusion
|
||||
- **Robots.txt:** Has Raptive ad-network block list but does NOT block generic crawlers
|
||||
- **Quality:** Good — authentic Chinese with modern adaptations
|
||||
- **ID prefix:** `oc_`
|
||||
|
||||
### Hot Thai Kitchen (hot-thai-kitchen.com) — ~464 recipes ⭐ MOST AUTHENTIC THAI
|
||||
- **Sitemap:** `https://hot-thai-kitchen.com/sitemap_index.xml` → post-sitemap.xml (464 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Authentic Thai (not Westernized)
|
||||
- **Robots.txt:** Permissive (Yoast standard)
|
||||
- **Quality:** Exceptional — Pai (author) explains Thai cooking theory, ingredient substitution within Thai context
|
||||
- **ID prefix:** `htk_`
|
||||
|
||||
### Spoon Fork Bacon (spoonforkbacon.com) — ~911 recipes
|
||||
- **Sitemap:** `https://www.spoonforkbacon.com/sitemap_index.xml` → post-sitemap.xml (911 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Asian fusion, modern international
|
||||
- **Robots.txt:** Permissive (wp-admin, search only)
|
||||
- **ID prefix:** `sfb_`
|
||||
|
||||
### Korean Bapsang (koreanbapsang.com) — ~275 recipes ⭐ BEST KOREAN
|
||||
- **Sitemap:** `https://www.koreanbapsang.com/sitemap_index.xml` → post-sitemap.xml (275 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Authentic Korean home cooking
|
||||
- **Robots.txt:** Permissive (Yoast standard)
|
||||
- **Quality:** Excellent — Susan (author) provides authentic Korean recipes with cultural context
|
||||
- **ID prefix:** `kb_`
|
||||
|
||||
### Sanjeev Kapoor (sanjeevkapoor.com) — ~1,965 recipes ⭐ BEST INDIAN
|
||||
- **Sitemap:** `https://www.sanjeevkapoor.com/webcontent-sitemap.xml` (1,965 URLs)
|
||||
- **JSON-LD:** ⚠️ Non-standard — uses `@type: Recipe` but in a different JSON structure (not standard schema.org/Recipe JSON-LD block). Has `recipeInstructions` and `recipeIngredient` fields. Needs custom parser.
|
||||
- **Cuisine focus:** Indian (all regions, vegetarian + non-veg)
|
||||
- **Robots.txt:** Declares sitemaps, no AI-bot blocks
|
||||
- **Quality:** Celebrity chef, authentic Indian recipes
|
||||
- **ID prefix:** `sk_`
|
||||
- **Pitfall:** JSON-LD is present but structured differently from WP recipe plugins. The `@type` appears as `'Recipe` (note quote style) rather than `"Recipe"`. May need regex extraction instead of standard JSON-LD parsing.
|
||||
|
||||
### Minimalist Baker (minimalistbaker.com) — ~650 recipes
|
||||
- **Sitemap:** `https://minimalistbaker.com/wp-sitemap.xml` → `post-sitemap.xml` (0 URLs — appears empty) + `post-sitemap2.xml` (650 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Simple plant-based, ≤10 ingredients, GF options
|
||||
- **Robots.txt:** Disallows `/r/`, `/ecourse/`, `/university/` paths. Blocks anthropic-ai.
|
||||
- **Quality:** High — Dana Schultz (author) provides consistently formatted recipes
|
||||
- **ID prefix:** `mb_`
|
||||
- **Pitfall:** `post-sitemap.xml` returns 0 URLs — use `post-sitemap2.xml` instead. This caused a misdiagnosis in the first research pass.
|
||||
|
||||
## Tier 2 — JSON-LD Confirmed but Blocks AI Bots (use generic UA)
|
||||
|
||||
### Cookie and Kate (cookieandkate.com) — ~920 recipes
|
||||
- **Sitemap:** `https://cookieandkate.com/post-sitemap.xml` (920 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Vegetarian, healthy, Mediterranean-inspired, some Asian fusion
|
||||
- **Robots.txt:** Blocks anthropic-ai, Claude-Web, bot "008". Does NOT block generic Mozilla UA.
|
||||
- **ID prefix:** `ckate_`
|
||||
|
||||
### Healthy Nibbles (healthynibblesandbits.com) — ~640 recipes
|
||||
- **Sitemap:** `https://healthynibblesandbits.com/post-sitemap.xml` (640 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Asian, healthy, light mains
|
||||
- **Robots.txt:** Blocks ChatGPT-User, GPTBot, anthropic-ai via Raptive. Generic UA works.
|
||||
- **ID prefix:** `hn_`
|
||||
|
||||
## Tier 3 — Blocked or Not Feasible
|
||||
|
||||
### Just One Cookbook (justonecookbook.com) — ~1,200+ recipes
|
||||
- **Status:** ❌ Cloudflare blocks ALL automated access. Returns Cloudflare challenge page for curl requests. Sitemap returns HTML error page, not XML.
|
||||
- **JSON-LD:** ✅ Exists (confirmed via browser in earlier session)
|
||||
- **Note:** Previously listed as "no sitemap" — incorrect. The site HAS a sitemap (`sitemap_index.xml` declared in robots.txt) but Cloudflare blocks programmatic access to it. Would need Playwright/browser automation to scrape.
|
||||
- **Correction from earlier doc:** Original reference said "No sitemap index or post-sitemap found" — this was wrong. The sitemap exists but is behind Cloudflare protection.
|
||||
|
||||
### Maangchi (maangchi.com) — ~500+ recipes
|
||||
- **Status:** ❌ Returns HTTP 403 for all programmatic requests (even with browser-like UA)
|
||||
- **JSON-LD:** Could not verify (blocked)
|
||||
- **Cuisine focus:** Korean (most popular English-language Korean site)
|
||||
- **Note:** robots.txt is permissive (only blocks /post-comments/) but the server itself rejects non-browser requests.
|
||||
|
||||
### AllRecipes (allrecipes.com)
|
||||
- **Status:** ❌ Sitemaps exist (`sitemap_1.xml` through `sitemap_4.xml`) but contain 0 URLs — appear to be empty or dynamically generated.
|
||||
- **JSON-LD:** Could not confirm on sampled recipe URLs
|
||||
- **Note:** Largest US recipe site but not scrapable via sitemap approach.
|
||||
|
||||
### Archana's Kitchen (archanaskitchen.com) — 8,428 URLs
|
||||
- **Status:** ❌ Sitemap has 8,428 URLs but NO JSON-LD on recipe pages (0 blocks found). Would need custom HTML parsing.
|
||||
- **Cuisine focus:** Indian
|
||||
- **Note:** Large volume but no structured data — not worth the effort vs. Sanjeev Kapoor which has JSON-LD.
|
||||
|
||||
### Bon Appétit (bonappetit.com) / Epicurious (epicurious.com)
|
||||
- **Status:** ❌ No JSON-LD on recipe pages. Both use Condé Nast's proprietary structured data format.
|
||||
- **Note:** High quality but not scrapable via JSON-LD approach.
|
||||
|
||||
### Serious Eats (seriouseats.com)
|
||||
- **Status:** ❌ Sitemap exists but returned 0 recipe URLs. JSON-LD not confirmed on sampled URLs.
|
||||
- **Note:** May use WPRM (WordPress Recipe Maker) structured data instead of JSON-LD. Would need custom parser.
|
||||
|
||||
### Pick Up Limes (pickuplimes.com) — ~240 recipes
|
||||
- **Sitemap:** `https://www.pickuplimes.com/sitemap.xml` — only 1 recipe URL found in sitemap (very small)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Robots.txt:** Returns HTML page, not a proper robots.txt
|
||||
- **Note:** Very small volume (~240). High quality vegan but not worth integration effort for the volume.
|
||||
|
||||
## Summary: Recommended Integration Priority
|
||||
|
||||
| Priority | Source | Recipes | Why |
|
||||
|----------|--------|---------|-----|
|
||||
| 1 | BBC Good Food | ~5,045 | Largest, international, permissive |
|
||||
| 2 | SkinnyTaste | ~2,354 | Best for light/healthy gap |
|
||||
| 3 | Woks of Life | ~1,555 | Best authentic Chinese |
|
||||
| 4 | RecipeTin Eats | ~1,697 | Pan-Asian, high quality |
|
||||
| 5 | Rasa Malaysia | ~1,390 | Best SE-Asian |
|
||||
| 6 | Sanjeev Kapoor | ~1,965 | Best Indian (needs custom parser) |
|
||||
| 7 | Budget Bytes | ~1,981 | Broad international, budget |
|
||||
| 8 | Pinch of Yum | ~1,483 | Comfort food, international |
|
||||
| 9 | Omnivore's Cookbook | ~772 | Modern Chinese |
|
||||
| 10 | Hot Thai Kitchen | ~464 | Most authentic Thai |
|
||||
| 11 | Spoon Fork Bacon | ~911 | Asian fusion |
|
||||
| 12 | Korean Bapsang | ~275 | Authentic Korean |
|
||||
| 13 | Minimalist Baker | ~650 | Simple vegan/GF |
|
||||
| 14 | Cookie and Kate | ~920 | Vegetarian (use generic UA) |
|
||||
| 15 | Healthy Nibbles | ~640 | Asian healthy (use generic UA) |
|
||||
|
||||
**Total potential: ~20,000+ new recipes**, majority Asian or light/healthy.
|
||||
|
||||
## Translation Strategy
|
||||
|
||||
Since these are English-language sources, recipes need translation before or after insertion:
|
||||
|
||||
1. **Pre-insertion translation (recommended):** Translate title, description, ingredients, and instructions to German before DB insert. Use LLM batch translation (GLM-5.2 via noris, or a dedicated translation model).
|
||||
2. **Store both languages:** Keep original English text in a separate column (`title_en`, `description_en`) for reference.
|
||||
3. **Ingredient mapping:** English ingredients should map to the SAME canonical names in the `ingredients` table (e.g., `garlic→knoblauch`, `chicken breast→hähnchenbrust`). The `CANONICAL_OVERRIDES` dict in `ingredient_normalizer.py` already includes English→German mappings.
|
||||
4. **Source prefix:** Use the ID prefixes listed above for each source.
|
||||
|
||||
## Scraping Pitfalls (International Sources)
|
||||
|
||||
- **Cloudflare blocks ≠ "no sitemap".** Just One Cookbook was initially diagnosed as "no sitemap" — actually the sitemap exists but Cloudflare returns an HTML challenge page instead of XML. Always check HTTP status codes and content-type, not just whether XML parse succeeds.
|
||||
- **AI-bot blocks in robots.txt are selective.** Many sites (RecipeTin Eats, Cookie and Kate, Healthy Nibbles, SkinnyTaste) block `anthropic-ai`, `Claude-Web`, `GPTBot`, or `ChatGPT-User` specifically. They do NOT block generic `Mozilla/5.0` user agents. Use a standard browser-like UA, not an AI-identifying one.
|
||||
- **Empty first sitemap ≠ empty site.** Minimalist Baker's `post-sitemap.xml` returns 0 URLs but `post-sitemap2.xml` has 650. Always check ALL sitemap files in the index.
|
||||
- **Non-standard JSON-LD.** Sanjeev Kapoor has recipe data but in a non-standard JSON structure (`@type: 'Recipe` with single quotes instead of `"Recipe"`). Standard JSON-LD parsers will miss it. May need regex-based extraction or custom parser.
|
||||
- **Quarterly sitemap pattern.** BBC Good Food organizes recipe sitemaps by quarter (`YYYY-QN-recipe.xml`). Must iterate through all quarters to get full coverage. Older quarters may not exist — check availability before assuming.
|
||||
+304
@@ -0,0 +1,304 @@
|
||||
# Chefkoch.de Internal API v2 — Discovery & Detail
|
||||
|
||||
Discovered during v5.2 scraper development. This endpoint is **public and unauthenticated** (as of 2026-06) and returns structured JSON for 384,252+ recipes.
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
https://api.chefkoch.de/v2/recipes
|
||||
```
|
||||
|
||||
No API key, no auth headers, no cookies required.
|
||||
|
||||
## Endpoints
|
||||
|
||||
### 1. Listing (Discovery)
|
||||
|
||||
```
|
||||
GET /v2/recipes?offset={N}&limit={limit}
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `offset` — integer, start index (pagination)
|
||||
- `limit` — integer, max 100 per request (default 30)
|
||||
|
||||
**Response shape**
|
||||
|
||||
```json
|
||||
{
|
||||
"count": 384252,
|
||||
"queryId": "",
|
||||
"results": [
|
||||
{
|
||||
"recipe": {
|
||||
"id": "4073161636112189",
|
||||
"title": "Flammkuchen mit Kartoffeln und Schwarzwälder Schinken",
|
||||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||||
"siteUrl": "https://www.chefkoch.de/rezepte/4073161636112189/Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken.html",
|
||||
"difficulty": 1,
|
||||
"preparationTime": 15,
|
||||
"rating": {"rating": 4.75, "numVotes": 20},
|
||||
"hasImage": true,
|
||||
"isPremium": true,
|
||||
"createdAt": "2021-11-05T13:36:51+01:00"
|
||||
},
|
||||
"score": 384252
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Discovery-only fields** (useful for filtering before detail fetch):
|
||||
- `id` — recipe ID (string)
|
||||
- `title` / `slug` / `siteUrl` — human metadata
|
||||
- `preparationTime` — minutes (int)
|
||||
- `difficulty` — 1=easy, 3=hard
|
||||
- `rating.rating` / `rating.numVotes`
|
||||
- `isPremium` / `isPlus` — premium flag
|
||||
|
||||
### 2. Detail (Full Recipe)
|
||||
|
||||
```
|
||||
GET /v2/recipes/{id}
|
||||
```
|
||||
|
||||
**Response shape** (key fields)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "4073161636112189",
|
||||
"title": "Flammkuchen…",
|
||||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||||
"siteUrl": "https://www.chefkoch.de/rezepte/...",
|
||||
"subtitle": "Originalrezept von Viki Fuchs",
|
||||
"additionalDescription": "",
|
||||
"instructions": "Den Ofen auf 220 °C Umluft vorheizen…",
|
||||
"ingredientGroups": [
|
||||
{
|
||||
"header": "Für den Teig: ",
|
||||
"ingredients": [
|
||||
{
|
||||
"id": "8891376",
|
||||
"name": "Öl",
|
||||
"unit": "EL",
|
||||
"unitId": "13",
|
||||
"amount": 4.0,
|
||||
"isBasic": false,
|
||||
"usageInfo": "",
|
||||
"foodId": "184"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"tags": ["Schnell", "einfach", "Backen", "Frankreich"],
|
||||
"rating": {"rating": 4.75, "numVotes": 20},
|
||||
"preparationTime": 15,
|
||||
"kCalories": 0,
|
||||
"servings": 2,
|
||||
"nutrition": null,
|
||||
"difficulty": 1
|
||||
}
|
||||
```
|
||||
|
||||
## Structured Ingredients (API vs JSON-LD)
|
||||
|
||||
| Aspect | JSON-LD (`recipeIngredient`) | API (`ingredientGroups`) |
|
||||
|--------|------------------------------|--------------------------|
|
||||
| Format | raw strings: `"4 EL Öl"` | objects: `{amount: 4, unit: "EL", name: "Öl"}` |
|
||||
| Groups | flat list | grouped by `header` (e.g. "Für den Teig:") |
|
||||
| Parsing | requires regex heuristics | already parsed |
|
||||
| Accuracy | ~95% (edge cases in parentheses) | ~99% |
|
||||
|
||||
**Normalization recipe** (from API objects to flat strings):
|
||||
|
||||
```python
|
||||
ingredients = []
|
||||
for group in data.get("ingredientGroups", []):
|
||||
header = group.get("header", "").strip()
|
||||
if header:
|
||||
ingredients.append(f"--- {header} ---")
|
||||
for ing in group.get("ingredients", []):
|
||||
parts = [str(p) for p in [ing.get("amount"), ing.get("unit"), ing.get("name")] if p]
|
||||
if parts:
|
||||
ingredients.append(" ".join(parts))
|
||||
```
|
||||
|
||||
## Rate Limits & Politeness
|
||||
|
||||
**Observed behavior:**
|
||||
- No explicit rate limiting (no 429 responses observed at 16 concurrent requests)
|
||||
- Server responds reliably with `application/json`
|
||||
- Total count fluctuates slightly (cache propagation)
|
||||
|
||||
**Recommended politeness:**
|
||||
- 0.3s between listing batches
|
||||
- 16 concurrent detail fetches (ThreadPoolExecutor)
|
||||
- No backoff needed for normal operation
|
||||
|
||||
## Comparison: API vs Playwright
|
||||
|
||||
| Metric | Playwright v4/v5 | API v5.2 |
|
||||
|--------|------------------|----------|
|
||||
| Discovery | Playwright scroll + DOM parse | 1 HTTP GET per 100 recipes |
|
||||
| Detail fetch | urllib.requests + JSON-LD regex | 1 HTTP GET per recipe |
|
||||
| Avg per recipe | ~3–5s | ~0.2s |
|
||||
| Browser dependency | Yes (chromium binary) | No |
|
||||
| Headless issues | Cookie banners, JS timing | None |
|
||||
| Max recipes reachable | ~25k (search exhaustion) | ~384k (full catalog) |
|
||||
| Bot detection risk | Moderate (requires stealth) | Very low (same API as site frontend) |
|
||||
|
||||
## Migration from Playwright to API
|
||||
|
||||
```python
|
||||
# old (v5.1)
|
||||
urls = await _fetch_links(page, seed_url) # Playwright
|
||||
recipe = await _scrape_one(page, url) # requests + JSON-LD
|
||||
|
||||
# new (v5.2)
|
||||
recipes_meta = _fetch_recipe_list(offset, limit=100) # API listing
|
||||
recipe = _fetch_recipe_detail(meta["id"]) # API detail
|
||||
```
|
||||
|
||||
## Listing API: Offset Ceiling Reality
|
||||
|
||||
The unfiltered listing endpoint (`/v2/recipes?offset=N&limit=100`) appears to return **the same global list** regardless of any `sort=`, `order=`, `with=`, or `difficulty=` parameters tested (2026-06-22). The only way to get diversity is to advance `offset`.
|
||||
|
||||
**CRITICAL:** Offset has a practical ceiling. In testing:
|
||||
- `offset=0` returns 100 results ✅
|
||||
- `offset=2000` returns 0 results ❌ (server silently drops)
|
||||
|
||||
This means the unfiltered listing CANNOT be used to reach the full 384k catalog by offset alone. For deep scraping, use the **search query strategy** below.
|
||||
|
||||
## Search Queries via `?query=`
|
||||
|
||||
The endpoint supports text search with real filtering:
|
||||
|
||||
```
|
||||
GET /v2/recipes?query={term}&offset=0&limit=100
|
||||
```
|
||||
|
||||
Verified search queries (with approximate counts as of 2026-06):
|
||||
|
||||
| Query | Count | Usable offsets |
|
||||
|-------|-------|----------------|
|
||||
| Hauptspeise | ~158,000 | 0–~1000 |
|
||||
| Kuchen | ~58,000 | 0–~1000 |
|
||||
| Vegan | ~28,000 | 0–~1000 |
|
||||
| Italienisch | ~7,000 | 0–~1000 |
|
||||
| (blank) | ~384,000 | 0–~2000 (barely) |
|
||||
|
||||
**Search offset ceiling:** ~1000 results (10 pages of 100). Beyond that, the search returns empty.
|
||||
|
||||
## Deep Scraper Strategy for 25k+ Recipes (v5.2)
|
||||
|
||||
Because no single query reaches beyond ~1000–2000 offsets, the scraper must **rotate queries**:
|
||||
|
||||
1. Maintain a list of 50+ search terms (categories, ingredients, cuisines)
|
||||
2. Each run: pick a rotation window (e.g., 20 terms)
|
||||
3. For each term: random offset between 0 and ~100, fetch listing
|
||||
4. Filter known IDs, fetch details in parallel
|
||||
5. Rotate to next term
|
||||
|
||||
This prevents seed exhaustion while staying inside each query's usable window.
|
||||
|
||||
```python
|
||||
QUERIES = [
|
||||
"Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
|
||||
"Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
|
||||
"Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
|
||||
"Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
|
||||
"Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
|
||||
"Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
|
||||
"Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
|
||||
"Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
|
||||
]
|
||||
|
||||
def _fetch_list(query, offset, limit=100):
|
||||
url = f"https://api.chefkoch.de/v2/recipes?query={query}&offset={offset}&limit={limit}"
|
||||
...
|
||||
```
|
||||
|
||||
## Performance observed (2026-06-22):
|
||||
|
||||
**Initial projection under-estimated actual throughput significantly.**
|
||||
|
||||
With 12 workers: ~10–15 recipes/sec (throughput ceiling is detail-fetch latency, not listing)
|
||||
With 16 workers: ~800–900 recipes/minute (~13–15 recipes/sec)
|
||||
With 32 workers: ~750 recipes/minute sustained (diminishing returns from context switching)
|
||||
|
||||
**Key insight:** The listing fetch is negligible latency (~100ms). Detail fetch dominates (~200ms per recipe at 32 concurrent). 16 concurrent detail fetches saturates the network + server response pipe. More workers add no value.
|
||||
|
||||
**Recommended configuration:**
|
||||
- 16 ThreadPool workers (sweet spot)
|
||||
- 0.3s delay between listing batches
|
||||
- limit=100 per listing request
|
||||
- Each run rotates to next query when hitting the offset ceiling
|
||||
|
||||
## State Management for API Scraper
|
||||
|
||||
### State file format
|
||||
|
||||
```json
|
||||
{
|
||||
"query_index": 0,
|
||||
"api_offset": 0,
|
||||
"source_counts": {"chefkoch": 25020},
|
||||
"new_this_run": 0,
|
||||
"failed_listings": 0
|
||||
}
|
||||
```
|
||||
|
||||
**Field semantics:**
|
||||
- `query_index` — position in `QUERIES` list (rotates when exhausted)
|
||||
- `api_offset` — current offset within the active query's pagination window
|
||||
- `source_counts` — total recipes scraped per source
|
||||
- `new_this_run` — deduplicated count since last state write
|
||||
- `failed_listings` — consecutive failed listing requests (used for fast-exit)
|
||||
|
||||
### CRITICAL: Offset increment must be `+= limit`, not `+= 1`
|
||||
|
||||
The scraper fetches **batches of `limit` recipes per listing call**. State must advance `api_offset` by `limit` (e.g., 100), not by 1. Incrementing by 1 causes the same 100-recipe batch to be re-fetched on every subsequent run, producing **zero new recipes** while appearing to "run successfully".
|
||||
|
||||
```python
|
||||
# CORRECT — advances to next batch
|
||||
def main():
|
||||
state = _load_state()
|
||||
offset = state.get("api_offset", 0)
|
||||
limit = 100
|
||||
# ... fetch listing at offset ...
|
||||
state["api_offset"] = offset + limit # 0 → 100 → 200
|
||||
_save_state(state)
|
||||
|
||||
# WRONG — re-fetches same batch every time
|
||||
def main():
|
||||
state = _load_state()
|
||||
offset = state.get("api_offset", 0)
|
||||
# ... fetch listing at offset ...
|
||||
state["api_offset"] = offset + 1 # 0 → 1; next run: fetches offset 1 (same as offset 0 in effect)
|
||||
```
|
||||
|
||||
**The same batch always returns the same 100 recipe IDs.** Deduplication drops everything, `new_this_run` stays at 0, but the state increments by 1 each run, crawling nowhere at ~1 recipe offset per launch.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **Do not use** `api.chefkoch.de/v2/recipes/{id}/ingredients` — returns `resource_not_found` (404)
|
||||
2. **Ingredients live inside the detail response** under `ingredientGroups` — there is no separate endpoint
|
||||
3. **Tag list may contain empty strings** — filter with `.strip()` before storage
|
||||
4. **Count is approximate** — the `count` field drifts slightly; do not hardcode it as a loop bound
|
||||
5. **Offset-based pagination with a hard ceiling** — unfiltered listing exhausts at ~2000; search queries at ~1000
|
||||
6. **Query parameter filtering is NOT supported on the listing endpoint** — `sort=`, `difficulty=`, etc. are ignored
|
||||
7. **Do not cache the `count` field as a loop bound** — use query rotation instead
|
||||
|
||||
## Headers Required
|
||||
|
||||
Minimal headers suffice; full mimicry not required:
|
||||
|
||||
```python
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json",
|
||||
"Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
|
||||
"Referer": "https://www.chefkoch.de/",
|
||||
}
|
||||
```
|
||||
+178
@@ -0,0 +1,178 @@
|
||||
# Chefkoch.de: Hybrid Scraper (Playwright + Requests + JSON-LD)
|
||||
|
||||
Chefkoch.de (380k+ Rezepte) blockiert **direkte `requests` auf Suchseiten** (liefert JS-Shell statt Inhalt), **aber** einzelne Rezept-Seiten können über **Requests + JSON-LD** geparst werden. Die optimale Architektur ist ein **Zwei-Phasen-Ansatz**.
|
||||
|
||||
## Warum der alte Ansatz scheitert
|
||||
|
||||
**Falsch:** `recipe_scrapers.scrape_html(page.content(), url)` auf Chefkoch-Suchseiten — Chefkoch erkennt nicht-JS-Clients und liefert leere DOM-Shells.
|
||||
|
||||
**Falsch:** Playwright pro Rezept — zu langsam (5s+ pro Rezept) und überfordert bei 100+ Rezepten.
|
||||
|
||||
## Korrekte Architektur: Zwei Phasen
|
||||
|
||||
### Phase 1: Link-Sammlung via Playwright (Browser nötig)
|
||||
|
||||
Nur für die **Suchseite** — Chefkoch rendernt Ergebnisse client-side.
|
||||
|
||||
```python
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
async with async_playwright() as p:
|
||||
browser = await p.chromium.launch(headless=True)
|
||||
page = await browser.new_page(
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
)
|
||||
await page.goto("https://www.chefkoch.de/rs/s0/{query}/Rezepte.html",
|
||||
wait_until="networkidle", timeout=30000)
|
||||
|
||||
# Scroll to load more results
|
||||
for _ in range(15):
|
||||
links = await page.query_selector_all("a[href*='rezepte/']")
|
||||
for el in links:
|
||||
href = await el.get_attribute("href")
|
||||
if href and "/rezepte/" in href:
|
||||
if not any(skip in href for skip in ["was-koche", "was-backe", "alle-rezepte", "/rs/s"]):
|
||||
recipe_links.add(href.split("?")[0])
|
||||
await page.evaluate("window.scrollBy(0, 500)")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
await browser.close()
|
||||
```
|
||||
|
||||
**Pitfall:** Keine `.recipe-list` oder `.bi-recipe-item` Selektoren verwenden — diese Klassenamen ändern sich. Stattdessen: `a[href*='rezepte/']` mit URL-Pattern-Filtern.
|
||||
|
||||
### Phase 2: Rezept-Daten via Requests + JSON-LD (kein Browser)
|
||||
|
||||
**Schnell** (~300ms statt 5s pro Rezept) und **zuverlässiger** als DOM-Scraping.
|
||||
|
||||
```python
|
||||
import urllib.request, re, json
|
||||
|
||||
def fetch_recipe_jsonld(url: str) -> dict:
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
# Extract JSON-LD blocks
|
||||
blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
|
||||
|
||||
for block in blocks:
|
||||
try:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return item
|
||||
elif data.get("@type") == "Recipe":
|
||||
return data
|
||||
except:
|
||||
continue
|
||||
return None
|
||||
|
||||
# Usage
|
||||
recipe = fetch_recipe_jsonld("https://www.chefkoch.de/rezepte/...")
|
||||
name = recipe["name"]
|
||||
ingredients = recipe.get("recipeIngredient", []) # List of strings like "4 EL Sojasauce"
|
||||
prep_time = recipe.get("prepTime", "") # ISO 8601: "PT25M"
|
||||
rating = recipe.get("aggregateRating", {}).get("ratingValue", "n/a")
|
||||
```
|
||||
|
||||
## Wichtige JSON-LD Felder
|
||||
|
||||
| Feld | Typ | Beispiel |
|
||||
|------|-----|----------|
|
||||
| `name` | str | `"Chicken Teriyaki - der japanische Klassiker"` |
|
||||
| `recipeIngredient` | list[str] | `["4 EL Sojasauce", "250 g Hähnchenbrust"]` |
|
||||
| `recipeInstructions` | list[dict/text] | `[{"@type": "HowToStep", "text": "..."}]` |
|
||||
| `prepTime` / `cookTime` / `totalTime` | str (ISO 8601) | `"PT25M"`, `"PT1H30M"` |
|
||||
| `aggregateRating.ratingValue` | float | `4.79` |
|
||||
| `aggregateRating.ratingCount` | int | `251` |
|
||||
|
||||
## Zutaten-String Parsen
|
||||
|
||||
`recipeIngredient` liefert rote Strings. Parser-Heuristik:
|
||||
|
||||
```python
|
||||
def parse_ingredient(s: str) -> dict:
|
||||
s = re.sub(r"[\t\n]+", " ", s.strip())
|
||||
parts = s.split(None, 2) # maxsplit=2
|
||||
|
||||
# Default
|
||||
amount, unit, name = 1.0, "Stk", s
|
||||
|
||||
# Try: <number> [unit] <name>
|
||||
m = re.match(r"([\d.,/]+(?:\s*-\s*[\d.,/]+)?)", parts[0])
|
||||
if m:
|
||||
num_str = m.group(1).replace(",", ".")
|
||||
# Parse number (handles fractions, ranges)
|
||||
if "-" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("-")]
|
||||
amount = (a + b) / 2
|
||||
elif "/" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("/")]
|
||||
amount = a / b
|
||||
else:
|
||||
amount = float(num_str.strip())
|
||||
|
||||
# Check if second part is known unit
|
||||
KNOWN_UNITS = {"el", "tl", "g", "kg", "ml", "l", "cl", "dl",
|
||||
"stk", "stück", "packung", "pck", "bund", "zweig",
|
||||
"dose", "glas", "tasse", "prise", "scheibe"}
|
||||
if len(parts) >= 2 and parts[1].lower().rstrip(".") in KNOWN_UNITS:
|
||||
unit = parts[1]
|
||||
name = parts[2] if len(parts) > 2 else ""
|
||||
else:
|
||||
name = " ".join(parts[1:])
|
||||
|
||||
return {"item": name.strip(), "amount": amount, "unit": unit}
|
||||
```
|
||||
|
||||
**Pitfall:** regex mit `re.match(r"...", first)` und `split(None, 2)` statt komplexem Gesamt-regex — vermeidet "unbalanced parenthesis" Fehler bei verschachtelten Klammern in Zutaten-Namen wie "Reiswein (Mirin oder Sake)".
|
||||
|
||||
## Playwright Install
|
||||
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```python
|
||||
python3 -c "from playwright.sync_api import sync_playwright; \
|
||||
p = sync_playwright().start(); \
|
||||
b = p.chromium.launch(); b.close(); p.stop(); \
|
||||
print('Chromium OK')"
|
||||
```
|
||||
|
||||
**Fehler ohne Installation:**
|
||||
```
|
||||
Error: Executable doesn't exist at ~/.cache/ms-playwright/chromium-1071/...
|
||||
Please run: playwright install
|
||||
```
|
||||
|
||||
**Im Hintergrund-Cronjob:** Immer `async_playwright` verwenden, nie `sync_playwright` — blockiert den Event Loop.
|
||||
|
||||
## Performance-Vergleich
|
||||
|
||||
| Ansatz | Zeit pro Rezept | 100 Rezepte |
|
||||
|--------|----------------|-------------|
|
||||
| Playwright pro Rezept | 5–8s | 8–13 Min |
|
||||
| **Hybrid (recommended)** | **0.3s (HTTP) + 3s Suchseite** | **~3 Min** |
|
||||
|
||||
## Dynamische Queries aus Interview-Daten
|
||||
|
||||
| Interview-Input | Generierte Query |
|
||||
|-----------------|-----------------|
|
||||
| 27°C + asiatisch | "asiatisch leicht huhn" |
|
||||
| Sarah + low-carb | "low carb spargel fisch" |
|
||||
| Regen + Familie | "einfach familie warm" |
|
||||
|
||||
## Familien-Fit-Score
|
||||
|
||||
- Asiatisch → +2.0 (Dominik)
|
||||
- Low-Carb/Salat/Fisch → +1.5 (Sarah)
|
||||
- Nudeln/Huhn/Eier → +1.0 (Cleo)
|
||||
- Bewertung >4.5 → +ratingValue × 0.5
|
||||
- Gemüse-Vielfalt → +0.3 pro Art
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
# Chefkoch Scraper Evolution: v4 Playwright → v5.2 API
|
||||
|
||||
Session learning from 2026-06-22: migrating the Chefkoch scraper from Playwright-based discovery to internal API discovery. This is a **10× speed increase** and removes all headless-browser fragility.
|
||||
|
||||
## Why v4 Stalled at ~25k
|
||||
|
||||
| Problem | Symptom | Root cause |
|
||||
|---------|---------|-----------|
|
||||
| Search exhaustion | Same seeds return same recipes | 2-letter alphabet combos cover finite space |
|
||||
| Bot detection | Empty search pages, no links | Chefkoch serves JS shell to non-browser clients |
|
||||
| Playwright fragility | `Executable doesn't exist`, EPIPE | Browser binary mismatches, env propagation issues |
|
||||
| Seed cycle reset | `seed_cycle` stuck at 0 | Missing `global SEED_CYCLE` in `main()` |
|
||||
| Cookie walls | Content blocked | GDPR consent banners block link extraction |
|
||||
| Rate ~0.4/s | 100 recipes takes ~4 min | Browser overhead per search page |
|
||||
|
||||
The fundamental issue: **Chefkoch search pages require a real browser**, and the browser-based discovery approach has natural throughput limits and maintenance overhead.
|
||||
|
||||
## The API Discovery Path (v5.2)
|
||||
|
||||
### How we found it
|
||||
|
||||
During debugging, a test request to `api.chefkoch.de/v2/recipes?offset=0&limit=30` returned structured JSON with **384,252 recipes** listed. The endpoint is the same one Chefkoch's own frontend uses — no auth, no rate limiting observed.
|
||||
|
||||
### Architecture comparison
|
||||
|
||||
| Aspect | v4/v5.1 (Playwright) | v5.2 (API) |
|
||||
|--------|----------------------|------------|
|
||||
| Discovery | Browser-rendered search pages | HTTP listing API (100 recipes/page) |
|
||||
| Detail fetch | JSON-LD regex from HTML | Structured JSON from API |
|
||||
| Browser needed | Yes (Chromium binary) | No |
|
||||
| Max reachable | ~25k (search exhaustion) | ~384k (full catalog) |
|
||||
| Ingredients | Raw strings from JSON-LD | Objects with amount/unit/name |
|
||||
| Avg speed | ~0.4 recipes/sec | ~5 recipes/sec |
|
||||
| Headless issues | Yes (cookies, JS timing, EPIPE) | None |
|
||||
| Maintenance | Browser updates, path management | Minimal (HTTP only) |
|
||||
|
||||
## v5.2 Architecture (query-rotation based)
|
||||
|
||||
Because the listing API has an offset ceiling (~2000 for unfiltered, ~1000 for search queries), v5.2 uses **query rotation** instead of simple offset pagination:
|
||||
|
||||
```python
|
||||
QUERIES = [
|
||||
"Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
|
||||
"Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
|
||||
"Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
|
||||
"Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
|
||||
"Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
|
||||
"Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
|
||||
"Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
|
||||
"Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
|
||||
]
|
||||
|
||||
def _fetch_recipe_list(query, offset, limit=100):
|
||||
# Uses ?query=... not bare offset
|
||||
url = f"{API_BASE}?query={query}&offset={offset}&limit={limit}"
|
||||
...
|
||||
```
|
||||
|
||||
State tracks `query_cycle` (position in QUERIES list) not just `api_offset`.
|
||||
|
||||
## Migration notes
|
||||
|
||||
**State field change:** v4/v5.1 uses `seed_cycle` (integer, position in infinite sequence). v5.2 uses `api_offset` (integer, offset in API listing). The state file can carry both:
|
||||
|
||||
```json
|
||||
{
|
||||
"seed_cycle": 9050,
|
||||
"api_offset": 25000,
|
||||
"source_counts": {"chefkoch": 25000, "hellofresh": 0},
|
||||
"migrated_from_v4": true
|
||||
}
|
||||
```
|
||||
|
||||
**Existing recipe IDs:** v4 IDs were `ck_347aaae514e25800` (16-char hex from URL). v5.2 IDs are `ck_4073161636112189` (numeric string from API). Both coexist in `real_recipes.jsonl` without conflict since dedup is per-ID.
|
||||
|
||||
**When to migrate from v4/v5.1 to v5.2:**
|
||||
- Scraper is stuck at same count for multiple runs → migrate immediately
|
||||
- Target > 25,000 → v5.2 is the only viable path (search exhaustion)
|
||||
- Playwright/browser issues recurring → v5.2 eliminates browser dependency
|
||||
- Building from scratch → start with v5.2
|
||||
|
||||
## Performance benchmark (observed)
|
||||
|
||||
| Metric | v5.1 (Playwright) | v5.2 (API) |
|
||||
|--------|-------------------|------------|
|
||||
| Recipes/hour | ~1,500 | ~5,000–9,000 |
|
||||
| Recipes/min | ~25 | ~80–150 |
|
||||
| Failure rate | ~5-10% (timeouts, empty pages) | ~0.1% |
|
||||
| Memory footprint | High (Chromium process) | Low (pure Python + urllib) |
|
||||
| CPU usage | High (browser rendering) | Low (JSON parse only) |
|
||||
| Max reachable | ~25k (search exhaustion) | ~50k+ (query rotation) |
|
||||
|
||||
## Remaining work for 50k target
|
||||
|
||||
At 25,020 recipes (current), v5.2 needs to fetch ~25,000 more. At ~800 recipes/minute with 16 workers:
|
||||
- **Time estimate:** ~35 minutes of continuous runtime
|
||||
- **API call count:** ~250 listing fetches + 25,000 detail fetches
|
||||
- **Query rotation:** ~10 unique queries required (each yields ~2,500 unique IDs)
|
||||
- **File size estimate:** ~25 MB of JSONL (assuming ~1 KB per recipe)
|
||||
|
||||
**Note on offsets:** Do NOT attempt to reach offset 25,000 on a single query. The unfiltered listing exhausts at ~2,000 and search queries at ~1,000. Use query rotation to distribute the target across many small offset windows instead.
|
||||
|
||||
**v5.3 refinements learned during deep scraping:**
|
||||
- `api_offset` must advance by `limit` (e.g. 100) per batch, not by 1. Advancing by 1 re-fetches the same 100-recipe window with a 1-recipe offset shift, producing zero new recipes while appearing to "progress"
|
||||
- 16 ThreadPool workers is the practical maximum; 32 workers showed no throughput improvement (~750 vs ~850 recipes/min) due to server-side latency becoming the dominant factor
|
||||
|
||||
## Pitfalls learned during migration
|
||||
|
||||
1. **Do not try `api.chefkoch.de/v2/recipes/{id}/ingredients`** — returns `resource_not_found`. Ingredients are inside the detail response under `ingredientGroups`.
|
||||
2. **Tag filtering required** — API returns empty strings in `tags` array; always `strip()` and filter.
|
||||
3. **Offset, not page** — The API uses `offset`, not `page`. Paginate with `offset += limit`, not `page += 1`.
|
||||
4. **Count is approximate** — `count` field drifts; never use it as a hard loop bound.
|
||||
5. **No `with=` query parameter works** — `with=ingredients,steps` is ignored; always fetch detail endpoint.
|
||||
6. **Concurrent detail fetches** — ThreadPoolExecutor with 16 workers gives good throughput without hitting any observed limits.
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
# HelloFresh.de — Recipe Scraping via SSR Hydration
|
||||
|
||||
HelloFresh.de serves recipe content as Next.js SSR pages with recipe data embedded in `__NEXT_DATA__` hydration scripts. No public API is exposed; extraction requires HTML parsing of the dehydrated React Query state.
|
||||
|
||||
## Architecture
|
||||
|
||||
Next.js + TanStack Query. Recipe data fetched server-side and serialized into `__NEXT_DATA__.props.pageProps.ssrPayload.dehydratedState.queries[]`.
|
||||
|
||||
### Key query keys
|
||||
|
||||
| Query key | Data |
|
||||
|-----------|------|
|
||||
| `['recipe.search', {...}]` | Recipe list for category page |
|
||||
| `['foodContentHubRecipe.byId', {'id': '...'}]` | Full recipe detail |
|
||||
|
||||
## Category Page (Discovery)
|
||||
|
||||
```python
|
||||
import urllib.request, json, re
|
||||
|
||||
def hf_fetch_category(category: str, skip=0, take=50) -> tuple[list[dict], int]:
|
||||
url = f"https://www.hellofresh.de/recipes/{category}?skip={skip}&take={take}"
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0...", "Accept": "text/html...",
|
||||
"Accept-Language": "de-DE,de;q=0.9", "Referer": "https://www.hellofresh.de/",
|
||||
}
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
|
||||
data = json.loads(match.group(1))
|
||||
ssr = data['props']['pageProps']['ssrPayload']
|
||||
for q in ssr.get('dehydratedState', {}).get('queries', []):
|
||||
key = q.get('queryKey', [])
|
||||
if len(key) > 0 and key[0] == 'recipe.search':
|
||||
d = q['state']['data']
|
||||
return d.get('items', []), d.get('total', 0)
|
||||
return [], 0
|
||||
```
|
||||
|
||||
**Category slugs** (62 total): amerikanische-rezepte, asiatische-rezepte, beliebteste-rezepte, burger, chinesische-rezepte, deutsche-rezepte, einfache-rezepte, eintopf-rezepte, familien-rezepte, flammkuchen-rezepte, fleisch-rezepte, franzosische-rezepte, fusions-rezepte, gesunde-rezepte, glutenfreie-rezepte, griechische-rezepte, grillrezepte, indische-rezepte, italienische-rezepte, japanische-rezepte, kalorienreduzierte-rezepte, karibische-rezepte, koreanische-rezepte, low-carb-rezepte, marokkanische-rezepte, mediterrane-rezepte, meeresfruechte-rezepte, mexikanische-rezepte, orientalische-rezepte, pasta-rezepte, pizza, proteinreiche-rezepte, rezepte-naher-osten, salat-rezepte, sandwich-rezepte, schnelle-gerichte, schnitzel, skandinavische-rezepte, spanische-rezepte, suppen-rezepte, thailandische-rezepte, tofu, tuerkische-rezepte, vegane-rezepte, vegetarische-rezepte, vietnamesische-rezepte, weihnachtsessen, zitrone, etc.
|
||||
|
||||
## Recipe Detail
|
||||
|
||||
```python
|
||||
def hf_fetch_detail(recipe_id: str, slug: str) -> dict | None:
|
||||
url = f"https://www.hellofresh.de/recipes/{slug}-{recipe_id}"
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
|
||||
data = json.loads(match.group(1))
|
||||
ssr = data['props']['pageProps']['ssrPayload']
|
||||
for q in ssr.get('dehydratedState', {}).get('queries', []):
|
||||
key = q.get('queryKey', [])
|
||||
if len(key) > 0 and key[0] == 'foodContentHubRecipe.byId':
|
||||
return q['state']['data'].get('recipe')
|
||||
return None
|
||||
```
|
||||
|
||||
## Normalization
|
||||
|
||||
| Field | Source | Notes |
|
||||
|-------|--------|-------|
|
||||
| id | `recipe.id` | Strip `-de-DE` suffix |
|
||||
| title | `recipe.name` | |
|
||||
| description | `recipe.description` or `recipe.headline` | |
|
||||
| ingredients | `yields[0].ingredients` + `ingredients` lookup | Two parallel arrays |
|
||||
| instructions | `steps[].instructionsMarkdown` -> strip HTML | |
|
||||
| prep_time | `recipe.prepTime` ISO 8601 | Parse PT30M -> 30 |
|
||||
| servings | `yields[0].yields` | Default 2 |
|
||||
| calories | `nutrition[]` where name contains "kcal" | |
|
||||
| tags | `recipe.tags[].name` | Skip "SEO" |
|
||||
| category | `recipe.cuisines` | May be str or dict with `name` |
|
||||
|
||||
### Ingredient assembly
|
||||
|
||||
```python
|
||||
def assemble_ingredients(recipe):
|
||||
ing_lookup = {ing['id']: ing['name'] for ing in recipe.get('ingredients', [])}
|
||||
ingredients = []
|
||||
for ying in recipe.get('yields', [{}])[0].get('ingredients', []):
|
||||
name = ing_lookup.get(ying.get('id', ''), '')
|
||||
amount = ying.get('amount', 0)
|
||||
unit = ying.get('unit', '')
|
||||
if name and amount is not None:
|
||||
if unit and unit != 'nach Geschmack':
|
||||
ingredients.append(f"{amount} {unit} {name}")
|
||||
elif unit == 'nach Geschmack':
|
||||
ingredients.append(f"{name} (nach Geschmack)")
|
||||
else:
|
||||
ingredients.append(f"{amount} {name}")
|
||||
elif name:
|
||||
ingredients.append(name)
|
||||
return ingredients
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- Category page: ~0.5-1.0s
|
||||
- Detail page: ~0.5-1.5s
|
||||
- Throughput: ~10-20 recipes/min with 8 workers
|
||||
- Total public recipes: ~5,000-10,000
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **No public API** - HTML hydration only; may break on Next.js upgrade
|
||||
2. **`cuisines` polymorphic** - `list[str]` or `list[dict]`; check `isinstance`
|
||||
3. **Amount of 0** means "nach Geschmank" or optional
|
||||
4. **Deduplicate by id** - recipes appear across categories
|
||||
5. **Keep polite** - 8 workers max, 0.5s between batches
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
# Infinite Refill Scraper v4 (Pattern B)
|
||||
|
||||
## When to Use
|
||||
|
||||
- Target is >10,000 recipes (batch seeds run out)
|
||||
- Continuous background operation without daily cronjob intervention
|
||||
- Resilient against search exhaustion (alphabet/categories eventually cover everything)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Infinite Seed Generator URL Discovery (Playwright) Recipe Extraction (Requests)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
[Alphabet combos] ──► Search pages ──► new URLs ──► JSON-LD parse ──► real_recipes.jsonl
|
||||
[Categories+rand] ──► (5 workers) ──► (queue) ──► (fast, no JS) ──► append-only
|
||||
[Random 3-letter] ──► ──► dedup ──► ──► crash-safe
|
||||
```
|
||||
|
||||
## Seed Generator (Python)
|
||||
|
||||
Cycles through 4 strategies infinitely, never running out:
|
||||
|
||||
```python
|
||||
ALPHABET = "abcdefghijklmnopqrstuvwxyzäöü"
|
||||
CATEGORIES = [
|
||||
"Rezepte", "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage",
|
||||
"Fruehstueck", "Backen", "Grillen", "Salat", "Suppe", "Eintopf",
|
||||
"Pasta", "Dessert", "Kuchen", "Brot", "Getraenk", "Fingerfood",
|
||||
]
|
||||
|
||||
def _next_seeds(batch_size=200, cycle_counter):
|
||||
seeds = []
|
||||
while len(seeds) < batch_size:
|
||||
cycle = cycle_counter % 4
|
||||
if cycle == 0:
|
||||
# Alphabetic 2-letter combos (e.g. "ab", "ac" ...):
|
||||
a = ALPHABET[cycle_counter % len(ALPHABET)]
|
||||
b = ALPHABET[(cycle_counter // len(ALPHABET)) % len(ALPHABET)]
|
||||
term = a + b
|
||||
elif cycle == 1:
|
||||
# Category + random letter (e.g. "Pasta+a")
|
||||
cat = CATEGORIES[cycle_counter % len(CATEGORIES)]
|
||||
letter = random.choice(ALPHABET)
|
||||
term = cat + "+" + letter
|
||||
elif cycle == 2:
|
||||
# Just category (e.g. "Pasta")
|
||||
term = CATEGORIES[cycle_counter % len(CATEGORIES)]
|
||||
else:
|
||||
# Random 3-letter nonsense (e.g. "xyz")
|
||||
term = "".join(random.choices(ALPHABET, k=3))
|
||||
seeds.append(f"https://www.chefkoch.de/rs/s0/{term}/Rezepte.html")
|
||||
cycle_counter += 1
|
||||
return seeds, cycle_counter
|
||||
```
|
||||
|
||||
**Coverage estimate:** 29 letters × 29 letters = 841 combos for 2-letter alone. Plus 17 categories × 29 = 493. Plus pure category = 17. Plus random 3-letter = infinite. Total: effectively unlimited.
|
||||
|
||||
## Two-Phase Extraction
|
||||
|
||||
### Phase 1: Link Discovery
|
||||
|
||||
```python
|
||||
async def _fetch_links(page, url, timeout=30):
|
||||
await page.goto(url, wait_until="domcontentloaded", timeout=timeout*1000)
|
||||
await asyncio.sleep(1.5) # Let JS render
|
||||
links = await page.eval_on_selector_all(
|
||||
"a[href*='/rezepte/']",
|
||||
"els => els.map(e => e.href)"
|
||||
)
|
||||
# Deduplicate and filter
|
||||
seen = set()
|
||||
result = []
|
||||
for l in links:
|
||||
if "chefkoch.de/rezepte/" in l and l not in seen:
|
||||
seen.add(l)
|
||||
result.append(l)
|
||||
return result
|
||||
```
|
||||
|
||||
### Phase 2: Recipe Extraction (NO Playwright)
|
||||
|
||||
```python
|
||||
import urllib.request, re, json
|
||||
|
||||
async def _scrape_one(url, timeout=15):
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
blocks = re.findall(
|
||||
r'<script type="application/ld\+json">(.*?)</script>',
|
||||
html, re.DOTALL | re.IGNORECASE
|
||||
)
|
||||
for block in blocks:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict) and item.get("@type") == "Recipe":
|
||||
return _normalize_recipe(item, url)
|
||||
elif isinstance(data, dict) and data.get("@type") == "Recipe":
|
||||
return _normalize_recipe(data, url)
|
||||
# Fallback: look for mainEntity
|
||||
for block in blocks:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, dict) and "mainEntity" in data:
|
||||
me = data["mainEntity"]
|
||||
if isinstance(me, dict) and me.get("@type") == "Recipe":
|
||||
return _normalize_recipe(me, url)
|
||||
return None
|
||||
```
|
||||
|
||||
**Why not use Playwright for recipe pages?** JSON-LD is embedded in raw HTML and extracts faster (no browser overhead). Playwright is only needed for search pages because Chefkoch serves a JS shell there.
|
||||
|
||||
## State Management
|
||||
|
||||
```python
|
||||
STATE = {
|
||||
"seed_cycle": 0, # Position in infinite sequence
|
||||
"started_at": "", # ISO timestamp
|
||||
"recipes_target": 25000
|
||||
}
|
||||
```
|
||||
|
||||
Save every batch. On restart, the script resumes from the last seed_cycle.
|
||||
|
||||
## Normalization Function
|
||||
|
||||
```python
|
||||
def _normalize_recipe(data, url):
|
||||
rid = (data.get("identifier", {}).get("value", "")
|
||||
or url.split("/")[-1].split("-")[0])
|
||||
title = data.get("name", "")
|
||||
if not title:
|
||||
return None
|
||||
|
||||
keywords = data.get("keywords", "")
|
||||
tags = [t.strip().lower() for t in keywords.split(",") if t.strip()]
|
||||
recipe_cat = data.get("recipeCategory", "")
|
||||
cuisine = data.get("recipeCuisine", "")
|
||||
if recipe_cat:
|
||||
tags.append(recipe_cat.lower())
|
||||
if cuisine:
|
||||
tags.append(cuisine.lower())
|
||||
tags = list(set(tags))
|
||||
|
||||
rating = data.get("aggregateRating", {})
|
||||
nutrition = data.get("nutrition", {})
|
||||
|
||||
return {
|
||||
"id": str(rid),
|
||||
"title": title,
|
||||
"description": data.get("description", ""),
|
||||
"source_url": url,
|
||||
"ingredients": data.get("recipeIngredient", []),
|
||||
"instructions": data.get("recipeInstructions", ""),
|
||||
"tags": tags,
|
||||
"category": recipe_cat or (tags[0] if tags else "Sonstiges"),
|
||||
"rating_value": rating.get("ratingValue"),
|
||||
"rating_count": rating.get("ratingCount"),
|
||||
"prep_time": data.get("prepTime", ""),
|
||||
"cook_time": data.get("cookTime", ""),
|
||||
"total_time": data.get("totalTime", ""),
|
||||
"calories": str(nutrition.get("calories", "")),
|
||||
"scraped_at": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
```
|
||||
|
||||
## Execution Pattern
|
||||
|
||||
### DO NOT use LLM-based cronjob
|
||||
|
||||
❌ **WRONG:** A cronjob that uses `hermes` LLM agent to run Python. Fails with:
|
||||
- `Error 400: model is required` (if no model configured)
|
||||
- `No Anthropic credentials` (if wrong provider configured)
|
||||
- Unnecessary cost: paying LLM tokens just to run `wc -l` and `python script.py`
|
||||
|
||||
✅ **CORRECT:** `terminal(background=True)` with `notify_on_complete`
|
||||
|
||||
```python
|
||||
# Start directly
|
||||
cd /profile/nutrition-coach && \
|
||||
source .venv/bin/activate && \
|
||||
python3 /path/to/chefkoch_scraper_v4.py
|
||||
|
||||
# In Hermes Agent:
|
||||
# terminal(background=True, notify_on_complete=True)
|
||||
```
|
||||
|
||||
### Why background process beats cron-checked scraper
|
||||
|
||||
| Aspect | Cronjob (daily check + start) | Background process |
|
||||
|--------|-------------------------------|-------------------|
|
||||
| Failure mode | LLM errors, credential mismatches | None (pure Python) |
|
||||
| Latency | Up to 24h until next check | Continuous |
|
||||
| State tracking | External (cron state DB) | Internal (scraper_state.json) |
|
||||
| Backpressure | Manual | Automatic (sleep on queue overflow) |
|
||||
| Monitoring | Cron output files | Log tail + notify_on_complete |
|
||||
| Cost | LLM tokens per check | Zero |
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
| Configuration | Recipes/min | 5,000 recipes | 25,000 recipes |
|
||||
|--------------|-------------|---------------|----------------|
|
||||
| 5 workers (batch, fixed seeds) | 60–80 | ~60–90 min | N/A (seeds exhausted) |
|
||||
| Infinite v4 (single context) | 40–60 | ~80–120 min | ~7–10 hours |
|
||||
| Infinite v4 (5 contexts, remote) | 80–120 | ~40–60 min | ~4–6 hours |
|
||||
|
||||
**Note:** v4 uses a single Playwright page context for discovery (lighter than 5 full workers). The extraction is HTTP requests which are much faster.*
|
||||
|
||||
## Known v4 Failure Modes
|
||||
|
||||
If the scraper is stuck at the exact same recipe count for multiple cron runs (check `scraper.log` — repeated `START run | current: N/N` with identical `N`), one of these three root causes is almost certainly present.
|
||||
|
||||
### Cause 1: `SEED_CYCLE` not declared global in `main()`
|
||||
|
||||
**Symptom:** `seed_cycle` in state JSON is `0` even after dozens of runs. The same exhausted 2-letter seeds are visited forever.
|
||||
|
||||
**Root cause:** In Python, a bare assignment `SEED_CYCLE = state["seed_cycle"]` inside `main()` creates a *local* variable if `global SEED_CYCLE` is not declared. The module-level global stays at `0`, so `_next_seeds()` always starts from `0`.
|
||||
|
||||
**Fix:** Add in `main()`:
|
||||
```python
|
||||
state.setdefault("seed_cycle", 0)
|
||||
global SEED_CYCLE # ← required
|
||||
SEED_CYCLE = state["seed_cycle"]
|
||||
```
|
||||
|
||||
**Reproduce:** Open Python, import the module, inspect `chefkoch_scraper_v4.SEED_CYCLE` after `main()` runs — it remains `0` without the `global` line.
|
||||
|
||||
### Cause 2: Playwright EPIPE / `Executable doesn't exist`
|
||||
|
||||
**Symptom:** Log shows `Executable doesn't exist at ...chrome-linux/chrome` or Node `EPIPE` errors. The process exits immediately.
|
||||
|
||||
**Root cause:** Environment variable `PLAYWRIGHT_BROWSERS_PATH` is set in the caller's shell but not propagated to the background process spawned by `terminal(background=True)`. The agent's background launcher creates a new non-login bash that inherits `PATH` but not the caller's env overrides.
|
||||
|
||||
**Fix:** Hardcode the browser path inside the script itself:
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/.../.cache/ms-playwright")
|
||||
```
|
||||
|
||||
Do **not** rely on shell `export` when launching via `terminal(background=True)`.
|
||||
|
||||
### Cause 3: Stale `completed_seeds` blockers in state
|
||||
|
||||
**Symptom:** Scraper runs but repeatedly visits the same search pages with no new URLs, or many search pages return zero links.
|
||||
|
||||
**Root cause:** `scraper_state.json` contains a `completed_seeds` dict that prevents revisiting certain terms. If these terms were exhausted naturally, the seeds never advance.
|
||||
|
||||
**Fix:** Wipe or prune the dict before a fresh run:
|
||||
```python
|
||||
import json
|
||||
s = json.load(open("scraper_state.json"))
|
||||
s.pop("completed_seeds", None)
|
||||
s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
|
||||
json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False)
|
||||
```
|
||||
|
||||
### Quick diagnostic flow
|
||||
|
||||
```bash
|
||||
# 1. Check recipe count isn't moving
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
|
||||
# 2. Check state
|
||||
python3 -c "import json; s=json.load(open('scraper_state.json')); print('seed_cycle:', s.get('seed_cycle')); print('completed_seeds keys:', list(s.get('completed_seeds',{}).keys())[:5])"
|
||||
|
||||
# 3. Check Playwright browser
|
||||
find /home/debian/.cache/ms-playwright -name chrome | head -1
|
||||
|
||||
# 4. Check if process is actually scraping (not just discovering)
|
||||
ps auxf | grep -E 'chefkoch|chrome' | grep -v grep | wc -l
|
||||
```
|
||||
|
||||
See `references/v4-known-issues-and-fixes.md` for the full patch snippets and state-reset script.
|
||||
|
||||
## File Locations (standard)
|
||||
|
||||
```
|
||||
~/.hermes/profiles/nutrition-coach/
|
||||
├── real_recipes.jsonl # Append-only output
|
||||
├── scraper_state.json # Resume state
|
||||
├── scraper.log # Human-readable log
|
||||
└── scripts/
|
||||
└── chefkoch_scraper_v4.py # Standalone script
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check progress
|
||||
tail -n 5 ~/.hermes/profiles/nutrition-coach/scraper.log
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
|
||||
# Check if running
|
||||
ps aux | grep chefkoch_scraper
|
||||
|
||||
# Restart if needed
|
||||
cd ~/.hermes/profiles/nutrition-coach && \
|
||||
source .venv/bin/activate && \
|
||||
nohup python3 scripts/chefkoch_scraper_v4.py > scraper.log 2>&1 &
|
||||
```
|
||||
|
||||
## Error Recovery
|
||||
|
||||
```
|
||||
Error streak >= 100 → Script aborts automatically
|
||||
→ Check Chefkoch availability: curl -I https://www.chefkoch.de
|
||||
→ Verify Playwright install: playwright install chromium
|
||||
→ Check disk space: df -h
|
||||
→ Restart with fresh state (or resume from last saved seed_cycle)
|
||||
```
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Browser path detection (common error)
|
||||
|
||||
If Playwright fails with `Executable doesn't exist at ...chrome-headless-shell`:
|
||||
|
||||
```bash
|
||||
# Find existing browser installations
|
||||
find /home/debian -name "chrome" -o -name "chromium" 2>/dev/null
|
||||
|
||||
# Usually found under /home/debian/.cache/ms-playwright/
|
||||
# Set the environment variable before starting:
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
```
|
||||
|
||||
If the browser is not installed at all:
|
||||
```bash
|
||||
cd /profile/nutrition-coach && source .venv/bin/activate
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Do NOT rely on `playwright install` in the crontab — install once, reference via env var.**
|
||||
|
||||
### nohup wrapper for daemon mode
|
||||
|
||||
Create `~/.hermes/profiles/nutrition-coach/scripts/start_scraper_v4.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
LOG="/home/debian/.hermes/profiles/nutrition-coach/scraper_v4.out"
|
||||
PIDFILE="/tmp/chefkoch_scraper_v4.pid"
|
||||
if [ -f "$PIDFILE" ] && kill -0 "$(cat $PIDFILE)" 2>/dev/null; then
|
||||
echo "Scraper already running (PID $(cat $PIDFILE))"; exit 0
|
||||
fi
|
||||
cd /home/debian/.hermes/profiles/nutrition-coach || exit 1
|
||||
export PYTHONUNBUFFERED=1
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
nohup python3 scripts/chefkoch_scraper_v4.py --no-agent > "$LOG" 2>&1 &
|
||||
echo $! > "$PIDFILE"
|
||||
echo "Scraper started (PID $!)"
|
||||
```
|
||||
|
||||
Then: `chmod +x scripts/start_scraper_v4.sh`
|
||||
|
||||
**Path note:** In this environment the script lives under `scripts/chefkoch_scraper_v4.py` inside the profile directory, not `~/.hermes/scripts/`. Always check `ls scripts/` before constructing the invocation path.
|
||||
|
||||
### Watchdog cronjob (no_agent)
|
||||
|
||||
```bash
|
||||
cronjob action=create \
|
||||
name=chefkoch-scraper-watchdog \
|
||||
script=start_scraper_v4.sh \
|
||||
schedule="*/10 * * * *" \
|
||||
no_agent=true \
|
||||
deliver=local
|
||||
```
|
||||
|
||||
This restarts the scraper if it crashes, without involving any LLM.
|
||||
|
||||
### DO NOT use `terminal(background=true)` with `tee` or complex pipelines
|
||||
|
||||
❌ `terminal(background=true, command="... | tee log.out")` — Output buffering causes `tee` to swallow data, and the `terminal` pseudo-tty layer conflicts with Playwright's subprocess management. Result: log file remains empty or stale, process output invisible.
|
||||
|
||||
✅ Start via explicit file redirect with `PYTHONUNBUFFERED=1`:
|
||||
|
||||
```python
|
||||
# In Hermes Agent — background process with file redirect
|
||||
terminal(
|
||||
background=True,
|
||||
command="cd /profile && PYTHONUNBUFFERED=1 python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1"
|
||||
)
|
||||
```
|
||||
|
||||
Or use the `start_scraper_v4.sh` wrapper script shown above. Always verify progress with `wc -l real_recipes.jsonl` and `tail scraper_v4.out` rather than relying solely on process output capture.
|
||||
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
# Mass Scraping: Chefkoch-Harvesting (>1000 Rezepte)
|
||||
|
||||
## Überblick
|
||||
|
||||
Bei Zielen >1000 Rezepten reicht der Single-URL-Fall-back-Modus nicht aus. Chefkoch bietet
|
||||
380.000+ Rezepte; die verlässliche Technik ist ein **Two-Phase Discovery + Harvest**
|
||||
mit Playwright-Multi-Worker-Architektur.
|
||||
|
||||
## Architektur-Blueprint
|
||||
|
||||
```
|
||||
Discovery-Phase Harvest-Phase
|
||||
──────────────────────── ──────────────────
|
||||
Search-Seite (Playwright) → Worker-Pool (5–10 Threads)
|
||||
└─ Scroll-Trigger → │- Browser-Context pro Worker
|
||||
└─ a[href*='rezepte/'] → │- JSON-LD/DOM-Fallback
|
||||
│ │- Append-Only JSONL
|
||||
▼ ↓
|
||||
[URL-Queue] real_recipes.jsonl
|
||||
│ scraper_state.json
|
||||
└─ Deduplication (URL-Hash-Set)
|
||||
```
|
||||
|
||||
## Phase 1: Link-Discovery
|
||||
|
||||
```python
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
def discover_urls(seed: str, page_num: int) -> list[str]:
|
||||
search_url = f"https://www.chefkoch.de/rs/s{page_num}/{seed}/Rezepte.html"
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 900})
|
||||
page.goto(search_url, wait_until="networkidle", timeout=25000)
|
||||
# Lazy-load trigger
|
||||
for _ in range(4):
|
||||
page.evaluate("window.scrollBy(0, 1000)")
|
||||
time.sleep(0.4)
|
||||
links = page.evaluate('''() => {
|
||||
const cards = document.querySelectorAll('article[class*="recipe"], .ds-recipe-card');
|
||||
const found = [];
|
||||
for (const card of cards) {
|
||||
const a = card.querySelector('a[href*="/rezepte/"]');
|
||||
if (a && a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
}
|
||||
if (found.length === 0) {
|
||||
document.querySelectorAll('a[href*="/rezepte/"]').forEach(a => {
|
||||
if (a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
});
|
||||
}
|
||||
return [...new Set(found)];
|
||||
}''')
|
||||
browser.close()
|
||||
# Filter to real recipe URLs only
|
||||
return [u for u in links if re.search(r'/rezepte/\d+/', u)]
|
||||
```
|
||||
|
||||
### Seed-Strategie
|
||||
|
||||
30+ breite Suchbegriffe (einfach, asiatisch, hähnchen, dessert, ...) erzeugen
|
||||
~150 Suchseiten = Tausende eindeutige Rezept-URLs. Der Queue wird gemischt, um
|
||||
keine Quelle zu überlasten.
|
||||
|
||||
## Phase 2: Rezept-Harvest
|
||||
|
||||
### A) JSON-LD-Extraktion (primär)
|
||||
|
||||
```python
|
||||
def scrape_jsonld(page, url: str) -> dict | None:
|
||||
page.goto(url, wait_until="networkidle", timeout=20000)
|
||||
scripts = page.locator('script[type="application/ld+json"]').all()
|
||||
for script in scripts:
|
||||
try:
|
||||
data = json.loads(script.inner_text())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return _normalize(item, url)
|
||||
elif data.get("@type") == "Recipe":
|
||||
return _normalize(data, url)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
```
|
||||
|
||||
### B) DOM-Fallback (selektiv)
|
||||
|
||||
Wenn JSON-LD fehlt oder leer:
|
||||
|
||||
```python
|
||||
# Zutaten-Tabelle
|
||||
rows = page.locator("table.ingredients tbody tr").all()
|
||||
# Anleitung
|
||||
steps = page.locator("article p").all()
|
||||
```
|
||||
|
||||
## Speicherformat: JSON Lines (JSONL)
|
||||
|
||||
**Niemals ein großes JSON-Dump verwenden** — bei Absturz ist alles verloren.
|
||||
|
||||
```
|
||||
real_recipes.jsonl # append-only
|
||||
scraper_state.json # resume-Informationen
|
||||
scraper.log # menschenlesbarer Fortschritt
|
||||
```
|
||||
|
||||
### JSONL-Struktur pro Zeile
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "347aaae514e25800",
|
||||
"name": "Erdbeerkuchen - einfaches Rezept",
|
||||
"description": "...",
|
||||
"category": "Backen",
|
||||
"tags": ["Sommer", "Kuchen", "Frühling"],
|
||||
"difficulty": "einfach",
|
||||
"base_servings": "12 Port.",
|
||||
"prep_time": "P0DT0H40M",
|
||||
"cook_time": "P0DT0H25M",
|
||||
"total_time": "P0DT1H5M",
|
||||
"ingredients": [{"raw": "125 g Zucker"}, ...],
|
||||
"instructions": ["Schritt 1 ...", "Schritt 2 ..."],
|
||||
"nutrition": {},
|
||||
"rating": "4.77",
|
||||
"review_count": "724",
|
||||
"author": "Loeckchen87",
|
||||
"images": ["https://img.chefkoch-cdn.de/..."],
|
||||
"source_url": "https://www.chefkoch.de/rezepte/...",
|
||||
"sources": [{"site": "chefkoch.de", "url": "...", "type": "de"}],
|
||||
"_scraped_at": "2026-06-18T05:13:31"
|
||||
}
|
||||
```
|
||||
|
||||
## Worker-Pool & Concurrency
|
||||
|
||||
```python
|
||||
WORKERS = 5
|
||||
DELAY_RANGE = (1.0, 2.5) # pro Worker individuell
|
||||
```
|
||||
|
||||
**Niemals 20+ Worker** — Chefkoch merkt sich IPs und verzögert massiv. 5 Worker
|
||||
mit 1–2 s Delay liefern
|
||||
60–80 Rezepte/Minute, bei 5.000 Rezepten ≈ 60–90 Min.
|
||||
|
||||
## State-Management
|
||||
|
||||
```json
|
||||
{
|
||||
"started_at": "2026-06-18T05:13:20",
|
||||
"last_update": "2026-06-18T05:22:26",
|
||||
"recipes_count": 5000,
|
||||
"completed_seeds": {"asiatisch": 4, "pasta": 2, ...},
|
||||
"failed_urls": [],
|
||||
"worker_stats": {
|
||||
"0": {"scraped": 1020, "errors": 3},
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Patience & Monitoring
|
||||
|
||||
- **Starten**: Hintergrundprozess, kein Blocking der CLI
|
||||
- **Log**: `tail -f scraper.log`
|
||||
- **Zwischenspeichern**: Nach jeweils 10 Rezepten die DB flushen
|
||||
- **Auto-Refill-Cronjob**: Täglich 06:00 — prüft Count, startet nur wenn < Ziel
|
||||
|
||||
## Bekannte Fehler & Workarounds
|
||||
|
||||
| Fehler | Ursache | Workaround |
|
||||
|--------|---------|------------|
|
||||
| `links = 0` auf Search-Seite | Cookie-Banner blockiert Scroll | Klick auf Zustimmen vor Scroll |
|
||||
| `JSONDecodeError` bei JSON-LD | Verunreinigte Blocks mit HTML | `try/except` pro Block |
|
||||
| `TimeoutError` bei Recipe-Seite | Langsamer Server | Erhöhe auf 30 s, 1 Retry |
|
||||
| `Duplicate URLs` | `/rezepte/123/` vs `/rezepte/123/Name.html` | Immer `split('#')[0].split('?')[0]` |
|
||||
| Worker-Crash | Memory-Overflow bei Langzeitsession | Browser pro Worker alle 500 Seiten restarten |
|
||||
|
||||
## Ergebnis-Kosten pro 5000 Rezepte
|
||||
|
||||
| Metrik | Wert |
|
||||
|--------|------|
|
||||
| Scraping-Zeit | ~60–90 min |
|
||||
| JSONL-Dateigröße | ~30–50 MB |
|
||||
| Einzigartige Rezepte | ~95% der gescrapeten URLs (Deduplication) |
|
||||
| Fehlerrate | <1% (Timeouts / 404) |
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
# v4 Known Issues & Fixes
|
||||
|
||||
Session-record of bugs found and fixed during production Chefkoch scraping runs.
|
||||
This file is referenced from the main `infinite-refill-scraper-v4.md` reference.
|
||||
|
||||
## Environment: Context for these fixes
|
||||
|
||||
- Host: Debian container, Python 3.11
|
||||
- Playwright 1.60.0 installed in profile venv
|
||||
- Browser cache: `/home/debian/.cache/ms-playwright/chromium-1223/`
|
||||
- Background process launcher: Hermes Agent `terminal(background=True)`
|
||||
- Cron: `chefkoch-scraper-watchdog` every 10 minutes (`no_agent=true`)
|
||||
|
||||
---
|
||||
|
||||
## Fix 1 — Playwright browser path not inherited
|
||||
|
||||
### Failure signature
|
||||
|
||||
```
|
||||
playwright._impl._api_types.Error: Executable doesn't exist at
|
||||
/home/debian/.cache/ms-playwright/chromium-1071/chrome-linux/chrome
|
||||
```
|
||||
|
||||
Note: The error references a **different** chromium version (`-1071`) than what is actually installed (`-1223`). This mismatch happens because Playwright falls back to its own lookup logic when `PLAYWRIGHT_BROWSERS_PATH` is absent.
|
||||
|
||||
### Why it happens
|
||||
|
||||
Setting the env var in the Hermes Agent shell workspace does **not** propagate to the background bash launched by `terminal(background=True)`. The background process inherits `PATH` and `VIRTUAL_ENV` but not custom `export FOO=bar` values.
|
||||
|
||||
### Fix options (ranked)
|
||||
|
||||
1. **Best:** Hardcode inside `chefkoch_scraper_v4.py` before any Playwright import:
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/debian/.cache/ms-playwright")
|
||||
```
|
||||
|
||||
2. **Acceptable:** Shell wrapper with `exec` and explicit `export`:
|
||||
```bash
|
||||
#! /bin/bash
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
exec python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1
|
||||
```
|
||||
|
||||
3. **Do not:** Rely on cron env or parent-shell export alone.
|
||||
|
||||
---
|
||||
|
||||
## Fix 2 — `global SEED_CYCLE` missing
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper runs every 10 minutes (cron restart) but recipe count never increases. Log shows:
|
||||
|
||||
```
|
||||
START run | current: 12736/25000
|
||||
```
|
||||
|
||||
State file `seed_cycle` is always `0`.
|
||||
|
||||
### Why it happens
|
||||
|
||||
Simplified pseudocode:
|
||||
|
||||
```python
|
||||
SEED_CYCLE = 0 # module-level global
|
||||
|
||||
def _next_seeds(batch_size=50):
|
||||
global SEED_CYCLE
|
||||
for i in range(batch_size):
|
||||
# ... compute seed based on SEED_CYCLE ...
|
||||
SEED_CYCLE += 1
|
||||
|
||||
async def main():
|
||||
state = _load_state()
|
||||
SEED_CYCLE = state["seed_cycle"] # ← BUG: creates a local variable in main()
|
||||
while ...:
|
||||
seeds = _next_seeds(50) # reads/writes the GLOBAL SEED_CYCLE
|
||||
# but the global is still 0 because main()'s assignment was local!
|
||||
```
|
||||
|
||||
Because `SEED_CYCLE` is assigned inside `main()` without `global SEED_CYCLE`, Python creates a local variable shadowing the module global. `_next_seeds()` then reads the global (still 0), increments it, but on the next cron restart `main()` sets the local back to whatever is in state (`0`). The seeds are the same every single run.
|
||||
|
||||
### Fix
|
||||
|
||||
```python
|
||||
async def main():
|
||||
state = _load_state()
|
||||
state.setdefault("seed_cycle", 0)
|
||||
global SEED_CYCLE # ← required
|
||||
SEED_CYCLE = state["seed_cycle"]
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import chefkoch_scraper_v4 as m
|
||||
m.main() # or simulate the assignment
|
||||
print('global SEED_CYCLE after main:', m.SEED_CYCLE)
|
||||
"
|
||||
```
|
||||
Without `global` it prints `0`. With `global` it prints whatever was in state.
|
||||
|
||||
---
|
||||
|
||||
## Fix 3 — Stale `completed_seeds` locking the scraper
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper starts, opens browser, visits search pages, but `new_urls` is always empty because all returned URLs are already in `existing`. No new recipes are appended.
|
||||
|
||||
### Why it happens
|
||||
|
||||
`scraper_state.json` accumulated a `completed_seeds` dict during a multi-worker batch run. The v4 infinite-refill script does not use `completed_seeds`, but the state file still carries it from an older script version. Certain seed terms are effectively blacklisted.
|
||||
|
||||
### Fix
|
||||
|
||||
```python
|
||||
import json
|
||||
s = json.load(open("scraper_state.json"))
|
||||
s.pop("completed_seeds", None)
|
||||
# Advance seed_cycle past known-exhausted 2-letter alphabet seeds
|
||||
s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
|
||||
json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False, indent=2)
|
||||
```
|
||||
|
||||
Setting `seed_cycle >= 1000` pushes the generator into the category+letter and category-only strategies, which yield far more fresh URLs than the `aa`, `ab`, `ac` alphabet exhaustion zone.
|
||||
|
||||
---
|
||||
|
||||
## Fix 4 — Dual script / dual Python trap (watchdog silently fails)
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper cron log shows `Scraper started (PID NNN)` every 10 minutes. No errors ever appear. `scraper_v4.out` only contains `Starting: X/Y` line (the `print()` from `main()` before Playwright is used). Recipe count stays flat for days or weeks.
|
||||
|
||||
### Why it happens
|
||||
|
||||
The watchdog script (`~/.hermes/scripts/start_scraper_v4.sh`) uses:
|
||||
1. **Global `python3`** → Playwright 1.59.0 (looks for `chromium-1071`, which does not exist in this environment).
|
||||
2. **Global script** (`~/.hermes/scripts/chefkoch_scraper_v4.py`) → missing `global SEED_CYCLE` in `main()`.
|
||||
|
||||
Both of these individually produce crashes (Fix 1 + Fix 2 above). Combined, the global python immediately crashes on browser launch, outputting nothing useful to the log. The watchdog sees the PID exit, and restarts it again 10 minutes later. This loop repeats forever with zero recipes.
|
||||
|
||||
Meanwhile, a **working** copy exists at:
|
||||
- `~/.hermes/profiles/nutrition-coach/.venv/bin/python3` (Playwright 1.60.0+, matches installed browsers)
|
||||
- `~/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py` (has `global SEED_CYCLE`)
|
||||
|
||||
Both copies look very similar from a distance, making it easy to invoke the wrong pair.
|
||||
|
||||
### Fix
|
||||
|
||||
Replace the watchdog shell script with the version that explicitly points to the profile-local python and script:
|
||||
|
||||
```bash
|
||||
nohup /home/debian/.hermes/profiles/nutrition-coach/.venv/bin/python3 \
|
||||
/home/debian/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py \
|
||||
--no-agent >> "$LOG" 2>&1 &
|
||||
```
|
||||
|
||||
Also verify there is no stale PID file (otherwise the wrapper exits early assuming it's already running):
|
||||
|
||||
```bash
|
||||
cat /tmp/chefkoch_scraper_v4.pid 2>/dev/null && (kill -0 "$(cat /tmp/chefkoch_scraper_v4.pid)" 2>/dev/null || rm -f /tmp/chefkoch_scraper_v4.pid)
|
||||
```
|
||||
|
||||
### Detection
|
||||
|
||||
```bash
|
||||
# 1. Check if log ever got past the first line
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/scraper_v4.out
|
||||
# Unhealthy: 1 line forever
|
||||
|
||||
# 2. Check which python the startup script actually calls
|
||||
grep 'python3' ~/.hermes/scripts/start_scraper_v4.sh
|
||||
# Unhealthy: just "python3 ..." (no venv path)
|
||||
|
||||
# 3. Check which script path it points to
|
||||
grep 'chefkoch_scraper' ~/.hermes/scripts/start_scraper_v4.sh
|
||||
# Unhealthy: ~/.hermes/scripts/chefkoch_scraper_v4.py
|
||||
```
|
||||
|
||||
### Recovery after fix
|
||||
|
||||
After updating the watchdog script:
|
||||
|
||||
```bash
|
||||
# 1. Kill any stale scraper
|
||||
pkill -f chefkoch_scraper_v4.py || true
|
||||
rm -f /tmp/chefkoch_scraper_v4.pid
|
||||
|
||||
# 2. Advance state past exhaustion
|
||||
python3 -c "
|
||||
import json, os
|
||||
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
|
||||
s = json.load(open(p))
|
||||
s.pop('completed_seeds', None)
|
||||
s['seed_cycle'] = max(s.get('seed_cycle', 0), 2000)
|
||||
json.dump(s, open(p, 'w'), ensure_ascii=False, indent=2)
|
||||
print('State reset — seed_cycle now', s['seed_cycle'])
|
||||
"
|
||||
|
||||
# 3. Start with fixed script
|
||||
bash ~/.hermes/scripts/start_scraper_v4.sh
|
||||
|
||||
# 4. Verify within 2 minutes
|
||||
sleep 120 && wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## One-liner state reset (copy-paste)
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import json, os
|
||||
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
|
||||
s = json.load(open(p))
|
||||
s.pop('completed_seeds', None)
|
||||
s['seed_cycle'] = max(s.get('seed_cycle',0), 1000)
|
||||
json.dump(s, open(p,'w'), ensure_ascii=False, indent=2)
|
||||
print('Reset OK — seed_cycle now', s['seed_cycle'])
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring checklist (what to watch)
|
||||
|
||||
| Check | Command | Healthy / Unhealthy |
|
||||
|-------|---------|---------------------|
|
||||
| Recipe count moving | `wc -l real_recipes.jsonl` | Increases between checks |
|
||||
| Process alive | `ps auxf \| grep chefkoch_scraper` | chromium + python present |
|
||||
| State advancing | `python3 -c "... seed_cycle ..."` | value increases over time |
|
||||
| Log not looping | `tail -n 5 scraper_v4.out` | NOT just "Starting: X/Y" only |
|
||||
| Disk space | `df -h .` | > 1 GB free |
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
# v5.2 Scraper: Wo liegen die Daten?
|
||||
|
||||
## Sitzung 2026-06-22 — Problem: Datenbank nicht auffindbar
|
||||
|
||||
### Kontext
|
||||
User wollte einen konkreten Link zu einem neuen Chefkoch-Rezept (v5.2). Ich fand keine `real_recipes.jsonl`, keinen laufenden Container, keine DuckDB. Die 25k+ Rezepte waren in einer früheren Sitzung entstanden, deren Deployment-Kontext nicht mehr greifbar war.
|
||||
|
||||
### Gesuchte Orte (kein Treffer)
|
||||
|
||||
| Ort | Ergebnis | Bemerkung |
|
||||
|-----|----------|-----------|
|
||||
| `/tmp/*.duckdb` | — | Keine DuckDB-Datei |
|
||||
| `/tmp/chefkoch-api/` | Nur `chefkoch-api` npm-Paket | Keine Daten |
|
||||
| Local `~/.hermes/profiles/nutrition-coach/` | Keine `real_recipes.jsonl` | v4-Legacy-Dateien |
|
||||
| Docker-Host 10.0.30.99 | Container `scraper` existiert nicht | Kein Recipe-Container läuft |
|
||||
| Docker-Host 10.0.30.99 `/opt/` `/mnt/` `/srv/` | Nur `data/seafile/seadoc` | Keine Recipe-Daten |
|
||||
| `~/.hermes/seafile_inventory.md` | Kein Recipe-Eintrag | Keine Seafile-Struktur |
|
||||
|
||||
### Schlussfolgerung
|
||||
|
||||
**v5.2-Daten liegen nicht auf dem aktuellen Host.** Mögliche Erklärungen:
|
||||
1. Scraper lief auf einem anderen Host / in einer anderen Sitzung
|
||||
2. Container wurde gelöscht / Volume war nicht persistent
|
||||
3. Daten sind in Seafile / Gitea / anderem Storage abgelegt (aber nicht indexiert)
|
||||
4. Testlauf war temporär (z.B. `/tmp/` ohne Persistenz)
|
||||
|
||||
### Erkennungs-Skript (zur Verwendung bei zukünftigen Suchen)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# v5.2-where-is-my-data.sh
|
||||
set -e
|
||||
|
||||
echo "=== v5.2 Daten-Suche ==="
|
||||
|
||||
# 1. Lokale Suche
|
||||
echo "1. Lokale JSONL-Dateien..."
|
||||
find / -name "real_recipes.jsonl" -type f 2>/dev/null | head -5
|
||||
|
||||
# 2. Container-Suche
|
||||
echo "2. Docker-Container..."
|
||||
docker ps -a | grep -i -E "scrap|recipe|food|chefkoch" || echo "Kein Container"
|
||||
|
||||
# 3. Seafile-Remote-Suche
|
||||
echo "3. Seafile..."
|
||||
# Via API: List library 'Casa' or 'Favorites' for recipe subdirs
|
||||
|
||||
# 4. JSONL-Größe check
|
||||
echo "4. Falls gefunden, Größe:"
|
||||
ls -lh $(find / -name "*.jsonl" -type f 2>/dev/null | head -5)
|
||||
```
|
||||
|
||||
### Pflicht-Checkliste: v5.2 nach Deployment
|
||||
|
||||
Bevor man sagt „v5.2 läuft und hat N Rezepte“, muss verifiziert sein:
|
||||
|
||||
1. **Wo ist `real_recipes.jsonl`?** Absolute Pfad notieren.
|
||||
2. **Ist der Pfad persistent?** (nicht `/tmp/`, nicht Ephemeral-Container)
|
||||
3. **Ist ein Container definiert?** `docker ps -a` zeigt ihn.
|
||||
4. **Cronjob / Watchdog** greift auf denselben Pfad zu wie der Scraper.
|
||||
5. **Seafile-Backup** abgleichen: Liegen Rezepte auch in der Cloud?
|
||||
|
||||
### Übergabe-Wissen für nächste Sitzung
|
||||
|
||||
Wenn ein Agent gefragt wird „zeig mir ein Rezept-Link“, und `real_recipes.jsonl` nicht lokal ist:
|
||||
1. NICHT raten (keine Beispiel-URLs aus der API-Doku generieren)
|
||||
2. Prüfen: Container? Lokales File? Seafile?
|
||||
3. Wenn nirgends: User informieren „Datenbank nicht gefunden, Scraper muss neu gestartet werden“
|
||||
|
||||
**Hinweis:** Die API-Doku (`chefkoch-api-v2-recipes.md`) gibt real existierende IDs, aber ohne JSONL kann ich nicht ohne Request validieren, dass ein bestimmtes Rezept wirklich gescraped wurde.
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Generator Scoring Algorithm Reference
|
||||
|
||||
Session: 2026-06-17
|
||||
|
||||
## Weather Scoring
|
||||
|
||||
```python
|
||||
if temp >= 35:
|
||||
if is_cold: score += 5.0
|
||||
if hot_cooking: score -= 4.0
|
||||
if not is_cold: score -= 2.0 # blanket
|
||||
if fish and day_index >= 4: score -= 4.0
|
||||
|
||||
elif temp >= 30:
|
||||
if is_cold: score += 4.0
|
||||
if hot_cooking: score -= 2.5
|
||||
if soup: score -= 3.0
|
||||
if fish and day_index >= 4: score -= 3.0
|
||||
```
|
||||
|
||||
**Critical pitfall learned:** Without blanket `-2.0` on non-cold meals at >=35°C, family-preference scores (+1.5 Asian) can overpower weather penalties. Result: Carbonara at 37°C.
|
||||
|
||||
## Perishability Scoring
|
||||
|
||||
```python
|
||||
if perishable or (fresh and 'salat' in tags):
|
||||
if day_index <= 2: score += 2.5
|
||||
elif day_index >= 4: score -= 3.0
|
||||
elif day_index == 3: score -= 1.0
|
||||
```
|
||||
|
||||
User example: "Lachs vom Samstag ist am Donnerstag schlecht."
|
||||
|
||||
## Tag Taxonomy
|
||||
|
||||
- **Cold:** salat, bowl, frisch, roh, griechisch, mediterran
|
||||
- **Hot cooking:** pasta, nudeln, ofen, eintopf, curry, schmor, gratin, hackfleisch
|
||||
- **Perishable:** fisch, lachs, thunfisch, kräuter, minze, koriander, basilikum, rucola, spinat
|
||||
- **Fresh:** salat, tomate, gurke, avocado
|
||||
|
||||
## Complete Code
|
||||
|
||||
See `scripts/nutrition-plan-generator-v3.py` — full implementation with all scoring dimensions.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Recipe Data Sources (Tested 2026-06-28)
|
||||
|
||||
## Active Sources (v7 Scraper — Direct MySQL)
|
||||
|
||||
### Chefkoch.de — ~300k+ total, ~25k scraped
|
||||
- **Access:** REST API v2 (`https://api.chefkoch.de/v2/recipes`)
|
||||
- **Method:** Query rotation across 100+ German cooking terms, pagination via offset
|
||||
- **Structured data:** API returns JSON with title, rating, ingredients, tags, times
|
||||
- **Anti-scraping:** Rate-limited but no Cloudflare; 0.3-0.5s delay sufficient
|
||||
- **Field mapping:** `id`, `title`, `category`, `tags`, `rating`, `prepTime`, `cookTime`, `totalTime`, `ingredients[].raw`, `siteUrl`, `nutrition.calories`
|
||||
|
||||
### Kuechengoetter.de — 30,000 recipes
|
||||
- **Access:** Sitemap (`https://www.kuechengoetter.de/sitemaps/recipes-1.xml`)
|
||||
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
|
||||
- **Structured data:** Schema.org Recipe in `<script type="application/ld+json">`
|
||||
- **Fields extracted:** name, aggregateRating.ratingValue, prepTime, cookTime, totalTime, recipeIngredient[], recipeCategory, recipeInstructions[], keywords→tags
|
||||
- **Anti-scraping:** None observed; polite delays (0.3-1.0s) work fine
|
||||
- **Quality:** Ratings present but sparse (many show rating=1, meaning single vote). Ingredient lists well-structured.
|
||||
|
||||
### Kitchen Stories (kitchenstories.com/de) — 3,390 German recipes
|
||||
- **Access:** Sitemap (`https://www.kitchenstories.com/sitemap.xml`)
|
||||
- **Filter:** `/de/rezepte/` (German recipe pages only; site has EN/DE bilingual)
|
||||
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
|
||||
- **Structured data:** Schema.org Recipe, 3 JSON-LD blocks per page (Recipe + WebSite + BreadcrumbList)
|
||||
- **Fields:** name, totalTime (ISO 8601 like PT20M), recipeIngredient[], recipeCategory, recipeInstructions[]
|
||||
- **Anti-scraping:** None observed
|
||||
- **Quality:** High-quality editorial recipes. No ratings in JSON-LD. Good metadata (times, categories).
|
||||
|
||||
### HelloFresh.de — ~16,831 total, ~10,000 estimated unique, 555 already scraped
|
||||
- **Access:** Category pages via `__NEXT_DATA__` (Next.js SSR hydration blob)
|
||||
- **Listing URL pattern:** `https://www.hellofresh.de/recipes/{category}?skip={N}&take=50`
|
||||
- **Detail URL pattern:** `https://www.hellofresh.de/recipes/{slug}-{recipeId}`
|
||||
- **Method:** Enumerate categories → paginate each → fetch detail per recipe → parse `__NEXT_DATA__`
|
||||
- **Structured data:** NOT JSON-LD. Next.js `__NEXT_DATA__` script tag containing `ssrPayload.dehydratedState.queries[]`
|
||||
- Listing: find query with `queryKey[0] == 'recipe.search'` → `state.data.items[]` (each has `id`, `slug`, `name`)
|
||||
- Detail: find query with `queryKey[0] == 'foodContentHubRecipe.byId'` → `state.data.recipe`
|
||||
- **Normalization:** Custom `_hf_normalize()` — builds ingredients from `yields[0].ingredients[]` joined with `ingredients[]` lookup, instructions from `steps[].instructionsMarkdown`, calories from `nutrition[]` where name contains 'kcal'
|
||||
- **Working categories (19/28):** familien-rezepte (6,208), vegetarische-rezepte (4,625), einfache-rezepte (1,444), gesunde-rezepte (1,395), eintopf-rezepte (934), italienische-rezepte (481), deutsche-rezepte (463), asiatische-rezepte (185), amerikanische-rezepte (172), orientalische-rezepte (167), kartoffel-rezepte, mediterrane-rezepte, mexikanische-rezepte, nudel-rezepte, suppen, salate, burger, chinesische-rezepte, flammkuchen-rezepte
|
||||
- **Dead categories (404):** belibteste-rezepte, pancakes, pasta, pizza, reis-rezepte, sandwich, fleisch-rezepte, franzosische-rezepte, griechische-rezepte
|
||||
- **Anti-scraping:** None observed; Referer header recommended
|
||||
- **Quality:** HIGHEST of all sources. 99% have calories, 93% have ratings, all have `instructions`, `description`, `difficulty`, `servings`. Professionally curated with exact quantities.
|
||||
- **ID prefix:** `hf_` (NOT `hellofresh_`) — must match v5.3 legacy IDs for deduplication
|
||||
- **Headers:** Requires `Referer: https://www.hellofresh.de/` and standard browser UA
|
||||
|
||||
### GuteKueche.de — 29,392 recipes (INTEGRATED in v6)
|
||||
- **Access:** Gzipped sitemap (`https://cdn.gutekueche.de/sitemaps/recipe.xml.gz`)
|
||||
- **Discovery:** robots.txt → `Sitemap:` directive led to CDN-hosted gzipped sitemap index → 4 sub-sitemaps, `recipe.xml.gz` has 29,392 recipe URLs
|
||||
- **Method:** Same as Kuechengoetter/Kitchen Stories — sitemap discovery → JSON-LD extraction. Scraper auto-detects gzip via magic bytes `\x1f\x8b` or config flag `"gzipped": True`.
|
||||
- **Config:** Added to `SITEMAP_SOURCES` dict with `"gzipped": True` flag. `discover_sitemap_urls()` enhanced with gzip support (urllib + `gzip.decompress()`).
|
||||
- **URL pattern:** `https://www.gutekueche.de/<name>-rezept-<id>`
|
||||
- **Structured data:** Schema.org Recipe in JSON-LD, **9/11 quality fields** (highest of all sitemap sources)
|
||||
- **Fields present:** nutrition (calories ✅), recipeInstructions ✅, aggregateRating ✅ (e.g. 4.5★), totalTime, recipeIngredient, recipeCategory, recipeYield, prepTime, cookTime
|
||||
- **Missing:** recipeCuisine, suitableForDiet
|
||||
- **Anti-scraping:** None; robots.txt allows all (only disallows /user/, /suche, /api/)
|
||||
- **Quality:** Strong — professional editorial recipes with calorie data and ratings. Second-best overall quality after HelloFresh.
|
||||
|
||||
## Tested but Rejected
|
||||
|
||||
| Source | Reason |
|
||||
|--------|--------|
|
||||
| AllRecipes.com | HTTP 403 Forbidden (Cloudflare) |
|
||||
| Yummly.com | HTTP 403 Forbidden |
|
||||
| Tasty.co | HTTP 404 on sitemap |
|
||||
| Marmiton.org | HTTP 404 on sitemap (French, URL structure changed) |
|
||||
| EatThis.org | SSL certificate verification failed |
|
||||
| Lecker.de | Sitemap XML files return 0 URLs (likely gzipped or JS-rendered) |
|
||||
| Essen-und-trinken.de | **Has Recipe JSON-LD on recipe pages** (6/11 fields, calories ✅, ratings ✅), BUT robots.txt blocks ALL AI bots (ClaudeBot, ChatGPT, Claude-Web, CCBot, etc.). Respect robots.txt — do not scrape. |
|
||||
| Maggi.de | HTTP 403 Forbidden |
|
||||
| Knorr.de | HTTP 403 Forbidden |
|
||||
| Serious Eats | HTTP 403 Forbidden |
|
||||
| Dr. Oetker | No JSON-LD on recipe pages |
|
||||
| Kochbar.de | No JSON-LD |
|
||||
| Marley Spoon | No JSON-LD (JS-rendered) |
|
||||
| Frauenzimmer.de | JSON-LD present but no Recipe type on recipe pages |
|
||||
| Bunte.de | 404 on recipe page URLs (structure unclear) |
|
||||
|
||||
## Storage: MariaDB Galera Cluster
|
||||
|
||||
Recipe data migrated from JSONL to the existing Galera cluster (2026-06-28).
|
||||
|
||||
- **Why Galera over SQLite:** User preference — reuse existing HA infrastructure (3-node cluster already running for HA Recorder) instead of adding new dependencies. Concurrent read/write via MaxScale readwritesplit.
|
||||
- **Connection:** `10.0.30.70:3306` (MaxScale VIP) → Galera cluster (db1=.71, db2=.72, db3=.73)
|
||||
- **Credentials:** DB=`recipes`, User=`recipe_app@%`, password in migration script
|
||||
- **Table schema:** 20 columns, InnoDB, utf8mb4_unicode_ci
|
||||
- **Indexes:** FULLTEXT ft_title_desc (title, description), FULLTEXT ft_ingredients (ingredients), BTREE idx_source, idx_category, idx_rating, idx_difficulty
|
||||
- **Migration script:** `scripts/migrate_to_galera.py` — INSERT IGNORE with batch=200, handles dict-valued `category` and `nutrition` fields, skips existing IDs
|
||||
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. Migration script must coerce these to strings/extract values before INSERT.
|
||||
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
|
||||
- **Architecture:** Scraper writes directly to Galera via `INSERT IGNORE` (no JSONL intermediate). v7 eliminated the JSONL → migration → DB pipeline. JSONL (`real_recipes.jsonl`) is kept as historical archive only.
|
||||
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. `recipe_to_row()` in v7 coerces these to strings/extracts values before INSERT.
|
||||
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
|
||||
- **Pitfall:** Per-row `db_has_id()` dedup is catastrophically slow through MaxScale — each check is a network roundtrip. v7 loads all IDs into a Python `set()` at startup via `db_all_ids()` (~0.3s for 28k), then all dedup is in-memory. Additionally, derive the likely recipe ID from the URL path and skip the HTTP fetch entirely if already known.
|
||||
|
||||
## Potential Future Sources (not yet integrated)
|
||||
|
||||
| Source | Recipes | Access | Notes |
|
||||
|--------|---------|--------|-------|
|
||||
| Food.com | Large | JSON-LD confirmed on recipe pages | English; sitemap index exists |
|
||||
| BBC Good Food | ~300/quarter | Sitemap recipe files, JSON-LD confirmed | English; premium content mixed in |
|
||||
| HelloFresh.de | ~~548 scraped~~ → **now active in v6** (see above) | Was thought exhausted; actually has ~16,831 recipes |
|
||||
|
||||
## Recipe Source Scanning Methodology
|
||||
|
||||
When evaluating new recipe sources for the scraper:
|
||||
|
||||
1. **Don't judge by landing page alone.** Landing/listing pages rarely embed Recipe JSON-LD — they have WebSite or ItemList schemas. Always fetch an **individual recipe page** and check for `@type: "Recipe"` in JSON-LD blocks.
|
||||
2. **Check `robots.txt` first.** Many sites declare their sitemap location there (`Sitemap: <url>`). Standard paths like `/sitemap.xml` frequently 404. Also check for AI-bot blocks — respect them.
|
||||
3. **Sitemaps may be gzipped.** Look for `.gz` extensions or `Content-Encoding: gzip`. Use `gzip.decompress()` in Python.
|
||||
4. **Volume estimation from sitemap index.** Fetch the index, count sub-sitemaps, sample 5-10 sub-sitemaps, extrapolate. Filter URLs by path patterns (`/rezept-`, `/rezepte/`, `/recipes/`).
|
||||
5. **Quality scoring (11-field checklist):** Count presence of: `nutrition`, `recipeInstructions`, `aggregateRating`, `totalTime`, `recipeIngredient`, `recipeCategory`, `recipeYield`, `prepTime`, `cookTime`, `recipeCuisine`, `suitableForDiet`. Score = present/11. HelloFresh ≈ 11/11 (custom parsing), GuteKueche ≈ 9/11, Kuechengoetter ≈ 6/11.
|
||||
6. **Quality ≠ Volume.** A source with 548 high-quality recipes (calories, instructions, ratings, difficulty) may be more valuable than a 30k source with sparse metadata. Evaluate both dimensions. **Never dismiss a source solely for low volume without assessing data quality.**
|
||||
|
||||
## v7 Scraper Architecture (Direct MySQL — replaces v6)
|
||||
|
||||
v7 eliminates JSONL entirely. Writes directly to MariaDB Galera via PyMySQL.
|
||||
|
||||
```
|
||||
recipe_scraper_v7.py
|
||||
├── Phase 1: Sitemap sources (Kuechengoetter → GuteKueche → Kitchen Stories)
|
||||
│ ├── discover_sitemap_urls() — fetch sitemap (auto-gzip), extract <loc> URLs
|
||||
│ ├── scrape_sitemap_source() — visit each URL, extract JSON-LD
|
||||
│ └── normalize_jsonld_recipe() — Schema.org → our format
|
||||
├── Phase 2: HelloFresh (__NEXT_DATA__ extraction)
|
||||
│ ├── _hf_fetch_category_page() — paginate categories via skip/take
|
||||
│ ├── _hf_fetch_detail() — fetch recipe page, parse __NEXT_DATA__
|
||||
│ ├── _hf_normalize() — convert HF nested structure to our format
|
||||
│ └── Per-category skip tracking via state["hf_cat_offset"]
|
||||
├── Phase 3: Chefkoch API (fills remaining gap to 100k)
|
||||
│ ├── fetch_ck_page() — API query with rotation
|
||||
│ └── fetch_ck_detail() — full recipe by ID
|
||||
├── State: scraper_state.json (tracks tried URLs, query cycles, source counts, hf_cat_offset)
|
||||
├── Dedup: db_all_ids() loads ALL IDs into Python set() at startup (~0.3s for 28k)
|
||||
│ └── Pre-check: derives likely ID from URL path, skips HTTP fetch if already in set
|
||||
├── Insert: INSERT IGNORE per recipe (idempotent, safe for re-runs)
|
||||
├── Output: Direct to MariaDB Galera (10.0.30.70:3306, DB=recipes)
|
||||
├── Cronjob: recipe-refill-v7 (daily 06:00, deliver: origin)
|
||||
└── Target: 100,000 recipes
|
||||
```
|
||||
|
||||
**Key differences from v6:**
|
||||
- No JSONL file, no migration step — single pipeline
|
||||
- In-memory dedup via `db_all_ids()` instead of per-row `db_has_id()` (avoids catastrophic Galera roundtrip latency)
|
||||
- URL pre-check skips HTTP fetch entirely for known recipes
|
||||
- Progress logging every 200 URLs
|
||||
- `recipe_to_row()` sanitizes dict/list fields (category, calories) before INSERT
|
||||
|
||||
**Separate step — Embeddings:**
|
||||
After scraping, run `generate_embeddings.py` to populate the `embedding` JSON column with 1024-dim harrier vectors. This is NOT done by the scraper.
|
||||
|
||||
**Output format** (unified across all sources, stored in `recipes` table):
|
||||
```json
|
||||
{
|
||||
"id": "kuechengoetter_seelachs-mit-rucola-1",
|
||||
"name": "Seelachs mit Rucola",
|
||||
"title": "Seelachs mit Rucola",
|
||||
"category": "fish",
|
||||
"tags": ["fisch", "seelachs", "rucola"],
|
||||
"rating": 4.5,
|
||||
"rating_value": 4.5,
|
||||
"prep_time": "PT15M",
|
||||
"cook_time": "PT20M",
|
||||
"total_time": "PT35M",
|
||||
"ingredients": [{"raw": "300g Seelachsfilet"}, ...],
|
||||
"source_url": "https://...",
|
||||
"source": "kuechengoetter",
|
||||
"calories": "450 kcal"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,98 @@
|
||||
# Semantic Ontology Classification — Implementation
|
||||
|
||||
Date: 2026-06-28
|
||||
Status: **Implemented, running in background**
|
||||
|
||||
## Overview
|
||||
|
||||
Rule-based semantic classifier that annotates every recipe across 11 ontology dimensions. Pure Python keyword matching — no LLM/API calls needed. Processes ~500 recipes/batch at ~50-500/s throughput.
|
||||
|
||||
## Script
|
||||
|
||||
`profiles/nutrition-coach/scripts/semantic_classifier.py`
|
||||
|
||||
- Incremental: `LEFT JOIN recipe_semantics rs ON r.id = rs.recipe_id ... WHERE rs.recipe_id IS NULL`
|
||||
- Batch insert: 500 recipes per `executemany()`
|
||||
- Fallback to one-by-one insert on batch error
|
||||
- Confidence: 0.85 (rule-based, not LLM-derived)
|
||||
- Runtime: ~10 min for 54k recipes
|
||||
|
||||
## Database Table: `recipe_semantics`
|
||||
|
||||
```sql
|
||||
CREATE TABLE recipe_semantics (
|
||||
recipe_id VARCHAR(255) PRIMARY KEY, -- ⚠️ 255, not 128!
|
||||
diet_type ENUM('vegan','vegetarian','pescatarian','flexitarian','omnivore') NOT NULL,
|
||||
meal_course JSON NOT NULL, -- breakfast,starter,soup,salad,main,side,dessert,snack,beverage,bread
|
||||
cuisine JSON NOT NULL, -- italian,french,german,asian,mexican,thai,japanese,chinese,greek,spanish,turkish,middle_eastern,american,nordic,african,caribbean,austrian,swiss
|
||||
cooking_method JSON NOT NULL, -- baking,frying,boiling,grilling,steaming,stewing,roasting,braising,raw,microwaving
|
||||
protein_source JSON NOT NULL, -- beef,pork,poultry,fish,seafood,plant,dairy,egg,none
|
||||
allergens JSON NOT NULL, -- gluten,eggs,dairy,nuts,soy,fish,shellfish,peanuts,sesame,celery,mustard,sulphites,lupin,molluscs
|
||||
flavor_profile JSON NOT NULL, -- sweet,savory,salty,sour,bitter,umami,spicy,smoky,creamy,fresh
|
||||
complexity ENUM('quick','simple','moderate','elaborate') NOT NULL,
|
||||
season JSON NOT NULL, -- spring,summer,autumn,winter,year_round
|
||||
caloric_density ENUM('light','balanced','hearty','indulgent') NOT NULL,
|
||||
preparation_style JSON NOT NULL, -- one_pot,sheet_pan,pan_fried,oven_baked,grilled,no_cook,slow_cooked,meal_prep
|
||||
confidence FLOAT DEFAULT 0.85,
|
||||
analyzed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
## Ontology Dimensions
|
||||
|
||||
### 1. Diet Type (ENUM, single value)
|
||||
Priority: explicit tags (vegan/vegetarisch) → meat detection → fish/seafood → dairy/egg → default omnivore.
|
||||
|
||||
### 2. Meal Course (JSON array)
|
||||
Keyword matching on title+category+tags+ingredients+description. Fallback to category field if no match. Default: `main`.
|
||||
|
||||
### 3. Cuisine (JSON array)
|
||||
18 cuisine types with German+English keywords. Also checks tags for country names. Default: `["german"]` if `deutsch` in tags.
|
||||
|
||||
### 4. Cooking Method (JSON array)
|
||||
10 methods. Default: `["boiling"]` if no match.
|
||||
|
||||
### 5. Protein Source (JSON array)
|
||||
Ordered priority: beef → pork → poultry → fish → seafood → egg → dairy → plant → none. Multiple can match.
|
||||
|
||||
### 6. Allergens (JSON array)
|
||||
14 EU-regulated allergens. Keyword-based, broad coverage (German+English).
|
||||
|
||||
### 7. Flavor Profile (JSON array)
|
||||
10 flavor dimensions. Desserts get `sweet` added if not already detected. Default: `["savory"]`.
|
||||
|
||||
### 8. Complexity (ENUM)
|
||||
Based on max(prep_time, cook_time, total_time): ≤15min=quick, ≤30min=simple, ≤60min=moderate, >60min=elaborate. Falls back to tags (schnell/einfach).
|
||||
|
||||
### 9. Season (JSON array)
|
||||
Keyword-based. Default: `["year_round"]`.
|
||||
|
||||
### 10. Caloric Density (ENUM)
|
||||
From calories: <300=light, <500=balanced, <800=hearty, ≥800=indulgent. Falls back to keyword estimation (kalorienarm, deftig, dessert→indulgent).
|
||||
|
||||
### 11. Preparation Style (JSON array)
|
||||
8 styles. Falls back to cooking method if no direct match.
|
||||
|
||||
## Cronjob
|
||||
|
||||
`semantic-classifier` — daily at 07:00, delivers to `telegram:223926918`. Runs after scraper (06:00) and embeddings (06:30).
|
||||
|
||||
## Pitfalls (Galera-Specific)
|
||||
|
||||
### 1. Foreign Key creation fails on collation mismatch
|
||||
The `recipes` table uses `utf8mb4_unicode_ci` but new tables default to `utf8mb4_general_ci`. FK creation fails with `errno 150`. Fix: either omit the FK (preferred — no cascade needed) or specify `COLLATE utf8mb4_unicode_ci` on the new table.
|
||||
|
||||
### 2. JOINs between tables with different collations fail
|
||||
`LEFT JOIN recipe_semantics rs ON r.id = rs.recipe_id` fails with `Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT)`. Fix: add `COLLATE utf8mb4_unicode_ci` to the JOIN condition.
|
||||
|
||||
### 3. VARCHAR(128) is too short for some recipe IDs
|
||||
Chefkoch IDs with encoded state fragments (e.g., `ck_Kinderueberraschungsbrot.html#eyJvYm...`) can be up to 248 chars. Use `VARCHAR(255)` for any column storing recipe IDs.
|
||||
|
||||
### 4. Time and calorie columns contain string values
|
||||
Some `prep_time`, `cook_time`, `total_time`, and `calories` values are stored as strings (e.g., `"PT20M"` or `"450 kcal"`), not integers. Always wrap with a `safe_int()` helper that catches `ValueError`/`TypeError` before using in comparisons or arithmetic.
|
||||
|
||||
### 5. Concurrent writes cause Galera deadlocks
|
||||
Running the semantic classifier and embedding generator simultaneously causes `Deadlock found when trying to get lock (1213)`. Fix: add retry logic with rollback+backoff (`safe_db_execute()`), or run sequentially. The daily cronjob schedule (06:00 scraper → 06:30 embeddings → 07:00 classifier) ensures sequential execution.
|
||||
|
||||
### 6. Galera FK constraints are fragile
|
||||
Even without collation issues, Galera's certification-based replication can reject DDL that works on standalone MySQL. Prefer logical relationships (application-level integrity checks) over physical FKs when working with Galera.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Semantic Search + Preference Learning — Implementation
|
||||
|
||||
Date: 2026-06-28
|
||||
Status: **Implemented and tested**
|
||||
|
||||
## Embedding Model
|
||||
|
||||
**Model:** `vllm/harrier-oss-v1-0.6b` (noris)
|
||||
- 1024 dimensions, multilingual (German-capable)
|
||||
- Dedicated embedding model (NOT an LLM), 0.6B params
|
||||
- Endpoint: `POST https://ai.noris.de/v1/embeddings`
|
||||
- Auth: Same noris API key from `config.yaml`
|
||||
- Batch: Up to 64 inputs per call, ~31 recipes/sec throughput
|
||||
|
||||
### Discovery
|
||||
|
||||
Initial investigation found that norris does NOT serve embeddings via standard LLM models (glm, gemma, qwen all return 404). The model list included `vllm/harrier-oss-v1-0.6b` which was the only embedding model that actually works. Other embedding names (bge-m3, e5-large, gte-large) are recognized by the Bifrost gateway but not activated on any API key.
|
||||
|
||||
### Quality Validation
|
||||
|
||||
Cosine similarity matrix (higher = more similar):
|
||||
|
||||
| | Curry | Salat | Eintopf | Hähnchen | Wintergericht |
|
||||
|-------------|-------|-------|---------|----------|---------------|
|
||||
| Curry | 1.000 | 0.647 | 0.677 | **0.812** | 0.660 |
|
||||
| Salat | 0.647 | 1.000 | 0.715 | 0.667 | 0.661 |
|
||||
| Eintopf | 0.677 | 0.715 | 1.000 | 0.696 | **0.829** |
|
||||
| Hähnchen | 0.812 | 0.667 | 0.696 | 1.000 | 0.694 |
|
||||
| Wintergericht| 0.660 | 0.661 | 0.829 | 0.694 | 1.000 |
|
||||
|
||||
Curry↔Hähnchen highest (both chicken dishes), Eintopf↔Wintergericht high (both warm/hearty), Salat↔Eintopf lower (different concepts). Working as expected.
|
||||
|
||||
## Database Schema
|
||||
|
||||
### `recipes.embedding` (JSON column)
|
||||
|
||||
Added via `ALTER TABLE recipes ADD COLUMN embedding JSON NULL`.
|
||||
Stores 1024-dim embedding as JSON array: `[0.023, -0.054, ...]`.
|
||||
|
||||
### `meal_decisions` table
|
||||
|
||||
```sql
|
||||
CREATE TABLE meal_decisions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
decision_date DATE NOT NULL,
|
||||
recipe_id VARCHAR(255) NOT NULL,
|
||||
recipe_title VARCHAR(500),
|
||||
recipe_category VARCHAR(200),
|
||||
accepted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
context_temp FLOAT NULL, -- outdoor temp °C
|
||||
context_season VARCHAR(20) NULL, -- spring/summer/autumn/winter
|
||||
context_weekday VARCHAR(20) NULL,
|
||||
context_last_meals TEXT NULL, -- JSON array of recent meal categories
|
||||
reason TEXT NULL, -- free-text rejection reason
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### `preference_rules` table
|
||||
|
||||
```sql
|
||||
CREATE TABLE preference_rules (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
rule_type ENUM('context_category','context_tag','variety','semantic') NOT NULL,
|
||||
condition_json TEXT NOT NULL, -- {"temp_min":20, "temp_max":99}
|
||||
effect_json TEXT NOT NULL, -- {"penalty_category":"Eintopf"}
|
||||
weight FLOAT DEFAULT 1.0, -- grows with confirmations (capped at 10)
|
||||
confirmations INT DEFAULT 1,
|
||||
rejections INT DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
All in `profiles/nutrition-coach/scripts/`:
|
||||
|
||||
### `generate_embeddings.py`
|
||||
- Batch-generates embeddings for recipes where `embedding IS NULL`
|
||||
- Builds embedding text from: `title + category + tags[:10] + ingredients[:20] + description[:200]` (capped at 1000 chars)
|
||||
- Batch size: 64 recipes per API call
|
||||
- Rate limit: 0.2s between calls
|
||||
- Run: `python3 scripts/generate_embeddings.py`
|
||||
- Should be run after scraper fills new recipes (could be chained in cronjob)
|
||||
|
||||
### `semantic_search.py`
|
||||
Core module with these functions:
|
||||
|
||||
#### `semantic_search(query, filters, top_k, min_rating)`
|
||||
- Gets query embedding from norris
|
||||
- Loads up to 5000 candidate recipes from DB (ordered by rating DESC)
|
||||
- Computes cosine similarity in Python
|
||||
- Blends similarity with rating: `score = similarity * (1 + rating/10)`
|
||||
- Supports filters: `category`, `source`, `exclude_categories`, `max_calories`
|
||||
|
||||
#### `record_decision(recipe_id, accepted, context)`
|
||||
- Logs to `meal_decisions` table
|
||||
- Auto-creates preference rule if rejection reason mentions weather/temperature
|
||||
- Example: `record_decision("hf_123", False, {"temp": 28, "season": "summer", "reason": "zu heiß für Eintopf"})`
|
||||
- Auto-rule: temp>20°C → penalize category from the rejected recipe
|
||||
|
||||
#### `get_preference_adjustments(context)`
|
||||
- Returns `{boost_categories, penalty_categories, boost_tags, penalty_tags}` based on matching rules
|
||||
- Each entry includes the rule's weight
|
||||
|
||||
#### `rank_with_preferences(recipes, context)`
|
||||
- Re-ranks a list of recipe dicts by applying:
|
||||
- Category penalties/boosts from learned rules (-0.1 × weight per match)
|
||||
- Tag penalties/boosts (-0.05 × weight per match)
|
||||
- Variety penalty: -0.15 if same category as recent meals
|
||||
- Checks `category + title + tags` concatenated (not just category field — see pitfall below)
|
||||
|
||||
#### `mine_rules(min_confidence=2)`
|
||||
- Analyzes `meal_decisions` for statistical patterns
|
||||
- Groups by `(category, temperature_bucket)` and `(category, season)`
|
||||
- If rejection rate > 50% with enough samples → creates/upgrades rule
|
||||
- Should run periodically as decisions accumulate
|
||||
|
||||
## Tested Examples
|
||||
|
||||
### Semantic Search
|
||||
```
|
||||
Query: "schnelles Hähnchengericht"
|
||||
→ Hähnchenbrust in Joghurt-Parmesan-Salbei-Panade (sim: 0.622, ★5.0)
|
||||
→ Hähnchenschnitzel auf Muttis Art (sim: 0.626, ★4.9)
|
||||
→ Hühnchenschnitzel (sim: 0.614, ★5.0)
|
||||
```
|
||||
|
||||
### Preference Learning
|
||||
```
|
||||
1. record_decision("ck_Erbsensuppe", False, {"temp": 28, "reason": "zu heiß für Eintopf"})
|
||||
→ Auto-rule created: temp>20°C → penalize "Eintopf" (weight: 1.0)
|
||||
|
||||
2. semantic_search("Eintopf mit Kartoffeln", top_k=10)
|
||||
→ rank_with_preferences(results, {"temp": 28, "season": "summer"})
|
||||
|
||||
Result: All Eintopf recipes get Δ=-0.2000 (two matching rules × 0.1 weight each)
|
||||
0.7365 (Δ-0.2000) | Möhren-Kartoffel-Eintopf (Kochen)
|
||||
0.7304 (Δ-0.2000) | Eintopf mit Spitzkohl, Hackfleisch (Kartoffeln)
|
||||
0.7226 (Δ-0.2000) | Möhreneintopf alla Isa (Eintopf)
|
||||
```
|
||||
|
||||
## Key Pitfalls
|
||||
|
||||
1. **Category field mismatch:** Most "Eintopf" recipes have category "Kochen" or "Kartoffeln", not "Eintopf". Rule matching must check `title + tags + category` as a concatenated lowercase string.
|
||||
|
||||
2. **Decimal type from MySQL:** `rating` comes back as `decimal.Decimal`, not `float`. Must cast with `float(rating or 0)` before arithmetic.
|
||||
|
||||
3. **Auto-rule with non-existent recipe IDs:** `record_decision("test_reject_1", ...)` creates a rule with empty category because the recipe doesn't exist in the DB. Always use real recipe IDs from the DB.
|
||||
|
||||
4. **Duplicate rules from auto-creation + mining:** Both `_auto_create_rule()` and `mine_rules()` can create the same rule. `_upsert_rule()` handles this with matching on `(rule_type, condition_json, effect_json)` and increments weight/confirmations.
|
||||
|
||||
## Integration with Meal Planner
|
||||
|
||||
The meal planner (`nutrition-plan-generator-v4.py`) should be updated to:
|
||||
1. Call `semantic_search()` instead of linear JSONL scan
|
||||
2. Call `rank_with_preferences()` with weather context
|
||||
3. Call `record_decision()` when user accepts/rejects proposals in Phase 2
|
||||
4. Periodically call `mine_rules()` to discover new patterns
|
||||
|
||||
Context for decisions:
|
||||
- Temperature: From HA weather sensor or Tibber API
|
||||
- Season: Derived from date
|
||||
- Weekday: From date
|
||||
- Last meals: From previous week's finalized plan
|
||||
@@ -0,0 +1,108 @@
|
||||
# Semantic Search + Preference Learning Investigation
|
||||
|
||||
Date: 2026-06-28
|
||||
|
||||
## Goal
|
||||
|
||||
Add semantic recipe search ("etwas Leichtes für heiße Tage") and preference learning ("wenn warm dann kein Eintopf") to the meal-planning pipeline.
|
||||
|
||||
## Embedding Model Requirements
|
||||
|
||||
- **Dimensions:** 1024 (user requirement)
|
||||
- **Language:** Multilingual / German-capable
|
||||
- **Hosting:** Preferred on norris (same provider as LLM models)
|
||||
|
||||
## Norris Embedding Endpoint Investigation
|
||||
|
||||
Tested all 20 models listed at `https://ai.noris.de/v1/models` against the `/v1/embeddings` endpoint:
|
||||
|
||||
### Models tested via `/v1/embeddings`
|
||||
| Model | Result |
|
||||
|-------|--------|
|
||||
| vllm/bge-reranker-v2-m3 | 404 — provider API error |
|
||||
| vllm/glm-5-2-nvfp4 | 404 — provider API error |
|
||||
| vllm/gemma-4-31b-it | 404 — provider API error |
|
||||
| vllm/qwen3.6-27b-nvfp4 | 404 — provider API error |
|
||||
|
||||
### Additional embedding models tested by name
|
||||
| Model | Result |
|
||||
|-------|--------|
|
||||
| vllm/bge-m3 | "no keys found that support model: bge-m3" |
|
||||
| vllm/multilingual-e5-large | "no keys found that support model" |
|
||||
| vllm/e5-mistral-7b-instruct | "no keys found that support model" |
|
||||
| vllm/gte-large | "no keys found that support model" |
|
||||
| vllm/NV-Embed-v2 | "no keys found that support model" |
|
||||
| vllm/snowflake-arctic-embed-l-v2.0 | "no keys found that support model" |
|
||||
|
||||
### Conclusion
|
||||
Norris recognizes embedding model names (gateway returns "no keys found" not "model not found"), meaning the Bifrost gateway knows about them but no API key has them activated. The `/v1/embeddings` endpoint returns 404 for all currently-available models (LLM models don't serve embeddings).
|
||||
|
||||
**Action needed:** User must request norris to activate an embedding model (bge-m3 recommended — 1024 dims, multilingual, strong German performance).
|
||||
|
||||
## Local vLLM Check
|
||||
|
||||
Checked for local vLLM instances that might serve embeddings:
|
||||
- `10.0.30.97:8000` (ComfyUI CT) — No route to host (SSH and HTTP both unreachable)
|
||||
- `10.0.30.102:8000` (OpenWebUI CT) — No response
|
||||
- `10.0.30.99:8000` (Docker host) — No response
|
||||
- `localhost:8000/8001` — No response
|
||||
|
||||
No local vLLM embedding server found. ComfyUI CT (10.0.30.97) has GPU but was unreachable during investigation.
|
||||
|
||||
## Proposed Architecture (pending embedding model availability)
|
||||
|
||||
### Database Schema Additions
|
||||
|
||||
```sql
|
||||
-- Add embedding column to recipes table
|
||||
ALTER TABLE recipes ADD COLUMN embedding JSON NULL COMMENT '1024-dim vector as JSON array';
|
||||
|
||||
-- Decision logging
|
||||
CREATE TABLE meal_decisions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
decided_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
recipe_id VARCHAR(255) NOT NULL,
|
||||
decision ENUM('accepted', 'rejected') NOT NULL,
|
||||
context JSON COMMENT 'temperature, season, weekday, recent_meals',
|
||||
INDEX idx_recipe (recipe_id),
|
||||
INDEX idx_decided (decided_at)
|
||||
);
|
||||
|
||||
-- Learned preference rules
|
||||
CREATE TABLE preference_rules (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
rule_pattern VARCHAR(255) NOT NULL COMMENT 'e.g. temp>20=>avoid:eintopf',
|
||||
weight FLOAT DEFAULT 1.0 COMMENT 'confidence, grows with confirmations',
|
||||
confirmed_count INT DEFAULT 0,
|
||||
rejected_count INT DEFAULT 0,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uniq_pattern (rule_pattern)
|
||||
);
|
||||
```
|
||||
|
||||
### Pipeline Flow
|
||||
|
||||
```
|
||||
Meal Planner selects recipes:
|
||||
1. SQL filter (rating > 4.0, not used this week)
|
||||
2. Preference rules apply (boost/penalize by context)
|
||||
3. Semantic search for "mood" query (light, hearty, asian...)
|
||||
4. Top-K presented as proposal
|
||||
|
||||
User feedback ("nein, kein Eintopf bei dem Wetter"):
|
||||
→ INSERT INTO meal_decisions (rejected, temp=28°C, category=eintopf)
|
||||
→ Rule mining detects pattern → INSERT/UPDATE preference_rules
|
||||
→ Next run: Eintopf auto-downweighted when warm
|
||||
```
|
||||
|
||||
### Embedding Generation Strategy
|
||||
|
||||
- Batch-generate embeddings for existing 28k+ recipes (one-time)
|
||||
- Incrementally embed new recipes as v7 scraper inserts them
|
||||
- Store as JSON array in `embedding` column (simple, no pgvector/Qdrant dependency)
|
||||
- Cosine similarity computed in Python at query time (28k × 1024 floats ≈ 115MB RAM, <1s)
|
||||
- At 100k recipes: ~400MB RAM for full similarity matrix — still feasible in-process
|
||||
|
||||
### Alternative: MariaDB VECTOR Type
|
||||
|
||||
MariaDB 11.6+ supports native `VECTOR` type with `VEC_DISTANCE` functions. If the Galera cluster upgrades, switch from JSON-stored arrays to native vectors for indexed similarity search. Current MariaDB version on the cluster should be checked before relying on this.
|
||||
Reference in New Issue
Block a user