Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,316 @@
|
||||
---
|
||||
name: meal-planning
|
||||
description: "Class-level skill for weekly family meal planning. 3-phase workflow: automated proposal (Wed cronjob) → interactive discussion (Telegram chat) → finalization (Fri cronjob). Weather adaptation, diet constraints, grocery list production."
|
||||
version: 2.0.0
|
||||
tags: [meal-planning, proposal, discussion, generator, weather, family]
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This skill manages the weekly meal planning for the Schön family (Dominik, Sarah, Cleo 4).
|
||||
|
||||
The combined skill provides:
|
||||
1. **Automated proposal generation** (Wednesday cronjob via `nutrition-plan-generator-v4.py`)
|
||||
2. **Interactive discussion** (Wednesday/Thursday in Telegram DM with nutrition-coach bot)
|
||||
3. **Finalization + shopping list** (Friday cronjob)
|
||||
4. **Scoring algorithm** (see `references/generator-v4-scoring.md`)
|
||||
|
||||
### Usage
|
||||
|
||||
- Load this skill before any meal‑planning request.
|
||||
- **3-Phase Workflow:** Vorschlag (Mi 19:00 cron) → Diskussion (Mi/Do interaktiv) → Finale (Fr 18:00 cron)
|
||||
- **Phase 1 (Vorschlag):** Cronjob `essensplan-vorschlag` generiert Plan via `nutrition-plan-generator-v4.py`, speichert `plan_data/current_proposal.md`, liefert Summary in Telegram DM
|
||||
- **Phase 2 (Diskussion):** User reagiert im Telegram-Chat → Coach lädt `current_proposal.md`, tauscht Gerichte, aktualisiert die Datei
|
||||
- **Phase 3 (Finale):** Cronjob `essensplan-finale` liest `current_proposal.md`, schreibt `KW<NN>_<YYYY>_plan.md` + `KW<NN>_<YYYY>_einkaufsliste.md`, liefert finale Summary
|
||||
- **v4 uses real recipes**: Reads from `real_recipes.jsonl` (25k+ Chefkoch entries), NOT the old `recipes.json` which contained 5000 dummy placeholders.
|
||||
- **Compact Telegram output**: Cronjob delivers a 7-line summary; full details remain in `~/Documents/Essensplanung/`.
|
||||
- **Delivery**: All cronjobs deliver to `origin` (Telegram DM), NOT to group chats.
|
||||
- **Model**: `vllm/glm-5-2-nvfp4` via `noris`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Cronjobs cannot conduct interactive interviews.** A cronjob fires once, sends output, and exits. User replies arrive in Telegram but are never read back. The old Wednesday interview cronjob produced `current_interview.json` with all answers `null`.
|
||||
- *Fixed:* Replaced with 3-phase workflow. Wednesday cronjob generates proposal (no interview needed). Discussion happens in interactive Telegram chat.
|
||||
- **Cron delivery target must be `origin` or numeric chat ID.** `telegram:Schoen Inc` fails with `invalid literal for int() with base 10: 'Schoen Inc'`. Use `origin` for DM delivery.
|
||||
- **Per-job model overrides in cron JSON.** Each job stores its own `model`/`provider` fields in `profiles/nutrition-coach/cron/jobs.json`. Bulk-switching the profile model does **not** update existing cron jobs. Patch the JSON or recreate the jobs after any model migration.
|
||||
- **Scraper ID prefixes must stay consistent across versions.** v5.3 used `hf_` for HelloFresh recipe IDs. v6 initially generated `hellofresh_` — this silently broke deduplication and would have re-scraped all 555 existing recipes. Always check existing ID formats in `real_recipes.jsonl` before changing a normalizer.
|
||||
- **Scraper state migration across versions.** Legacy state files from older scraper versions use different field names (`ck_cycle` vs `ck_query_cycle`, `hf_cycle: 8763` from v5.3). When upgrading the scraper, explicitly initialize/reset state fields. Don't assume the new code handles old state gracefully.
|
||||
- **Python variable ordering in config sections.** When adding a new source config block (e.g., `HF_HEADERS` referencing `UA`), ensure shared variables like `UA` are defined **before** any config dict that references them. Module-level execution is top-down; a `NameError` at import time kills the entire scraper.
|
||||
- **HelloFresh is NOT exhausted.** Previous assessment said "~548 scraped, exhausted" — wrong. HelloFresh has ~16,831 recipes across 19 working categories (9 categories 404). Estimated ~10,000 unique after cross-category dedup. Highest data quality of all sources (99% have calories, 93% ratings, all have instructions+descriptions).
|
||||
- **Quality ≠ Volume — don't dismiss low-volume sources without assessing data quality.** HelloFresh was dropped from v6 because "only 548 recipes, marginal ROI." User corrected: HelloFresh recipes are professionally curated with exact quantities, step-by-step instructions, calorie data, and difficulty ratings — qualitatively superior to community-submitted recipes. Always evaluate a source's data completeness (11-field checklist) alongside its volume before deciding to include/exclude it.
|
||||
- **GuteKueche.de integrated in v6.** 29,392 recipes with 9/11 JSON-LD quality fields (calories ✅, ratings ✅, instructions ✅, prep/cook time ✅). Gzipped sitemap at `cdn.gutekueche.de/sitemaps/recipe.xml.gz` (found via robots.txt). Added to `SITEMAP_SOURCES` with `"gzipped": True` flag. Scraper `discover_sitemap_urls()` now auto-detects gzip via magic bytes `\x1f\x8b`. Combined potential: Chefkoch 25k + Kuechengoetter 30k + GuteKueche 29k + HelloFresh 10k + Kitchen Stories 3.3k = ~97k — 100k target reachable without international sources.
|
||||
|
||||
## References
|
||||
|
||||
- `references/recipe-sources.md` — Tested recipe data sources (Chefkoch API, Kuechengoetter, Kitchen Stories, GuteKueche, HelloFresh), JSON-LD availability, sitemap URLs, v7 scraper architecture
|
||||
- `references/international-recipe-sources.md` — English-language sources for Asian/light/healthy cuisine (RecipeTin Eats, Budget Bytes, Cookie and Kate, Pick Up Limes, Just One Cookbook, Serious Eats, Minimalist Baker) with JSON-LD/sitemap status, translation strategy, and gap analysis
|
||||
- `references/semantic-search-implementation.md` — Implemented semantic search + preference learning system: harrier embeddings, DB schema, API usage, tested examples
|
||||
- `references/semantic-ontology-classification.md` — Rule-based ontology classifier: 11 dimensions, recipe_semantics table schema, Galera pitfalls (collation, FK, deadlocks, VARCHAR sizes)
|
||||
- `references/semantic-search-investigation.md` — Historical investigation notes (pre-implementation, kept for context on what was tested and ruled out)
|
||||
- `references/generator-v4-scoring.md` — Current v4 scoring dimensions, event overrides, Cleo logic
|
||||
- `references/nutrition-interview-soest/generator-scoring-algorithm.md` — Legacy v3 algorithm (superseded)
|
||||
- `references/multi-source-recipe-scraper/` — Scraper integration docs (mass-scraping, API v2, Playwright bypass, v5.2 migration)
|
||||
- `templates/interview-data-schema.json` — JSON schema for `current_interview.json` (legacy, kept for reference)
|
||||
|
||||
## Scraper (v7 — Direct MySQL)
|
||||
|
||||
The v7 scraper (`profiles/nutrition-coach/scripts/recipe_scraper_v7.py`) writes recipes **directly** to MariaDB Galera — no JSONL intermediate.
|
||||
- **Sources:** Chefkoch API + Kuechengoetter sitemap (30k) + GuteKueche sitemap (29k, gzipped) + Kitchen Stories sitemap (3.3k) + HelloFresh categories (~10k unique)
|
||||
- **Target:** 100,000 recipes
|
||||
- **Cronjob:** `recipe-refill-v7` runs daily at 06:00, delivers to `origin`
|
||||
- **State:** `scraper_state.json` tracks tried URLs, query cycles, source counts, `hf_cat_offset` (per-category skip)
|
||||
- **Dedup:** Loads all existing IDs into RAM at startup (`db_all_ids()`), checks in-memory — NEVER per-row DB roundtrips
|
||||
- **URL pre-check:** Derives likely recipe ID from URL path and skips HTTP request entirely if ID already in DB
|
||||
- **Insert:** `INSERT IGNORE` per recipe (idempotent, safe for re-runs)
|
||||
- **Scrape order:** Sitemap sources first (finite URL sets) → HelloFresh (high quality) → Chefkoch (fill remainder)
|
||||
- **Legacy JSONL:** `real_recipes.jsonl` is kept as historical archive only. v7 does NOT read or write it. The `migrate_to_galera.py` script was a one-time migration tool.
|
||||
- See `references/recipe-sources.md` for full source documentation including HelloFresh `__NEXT_DATA__` technique, GuteKueche discovery, and source scanning methodology
|
||||
- **Embeddings must be regenerated after scraping new recipes.** v7 does NOT generate embeddings. The `embedding-refill` cronjob (daily 06:30) runs `generate_embeddings.py` incrementally — only embeds recipes where `embedding IS NULL`. As of 2026-06-28, all 54,613 recipes have embeddings (100% coverage). The script includes `safe_db_execute()` with Galera deadlock retry (rollback + backoff) for robustness during concurrent writes.
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
**Active: MariaDB Galera Cluster** (migrated from JSONL 2026-06-28, v7 writes directly — no JSONL intermediate).
|
||||
|
||||
User chose Galera over SQLite — reuses existing HA infrastructure, no new dependencies. v7 scraper writes directly via `INSERT IGNORE`; `real_recipes.jsonl` is a historical archive only.
|
||||
|
||||
- **Connection:** `10.0.30.70:3306` (MaxScale VIP) → 3-node Galera cluster (10.0.30.71-73)
|
||||
- **Database:** `recipes`, **User:** `recipe_app@%`
|
||||
- **Table:** `recipes` with InnoDB engine, utf8mb4
|
||||
- **Indexes:** FULLTEXT on (title, description) and (ingredients); BTREE on source, category, rating, difficulty
|
||||
- **Schema fields (verified 2026-06-28 via DESCRIBE):** `id` (PK), `title`, `name`, `description`, `category`, `tags`, `ingredients` (JSON text, FULLTEXT indexed), `instructions`, `source`, `source_url`, `rating`, `rating_value`, `rating_count`, `prep_time`, `cook_time`, `total_time`, `servings`, `calories`, `difficulty`, `scraped_at`, `embedding` (JSON, nullable)
|
||||
- **⚠️ Column name reality vs older docs:** There is NO `image_url` column (use `source_url`). There is NO `ingredients_json` column (use `ingredients`). There is NO `url` column (use `source_url`). There is NO `created_at` column (use `scraped_at`). Columns that exist but were previously undocumented: `name`, `rating_value`, `total_time`, `difficulty`.
|
||||
- **Migration script:** `profiles/nutrition-coach/scripts/migrate_to_galera.py` — one-time migration tool (INSERT IGNORE with batch=200, skips existing IDs). No longer needed for ongoing operation — v7 writes directly to Galera.
|
||||
- **Concurrent access:** MaxScale readwritesplit → scraper writes while planner reads (no locking)
|
||||
- **Query examples:** `SELECT * FROM recipes WHERE rating >= 4.3 AND MATCH(title, description) AGAINST('hähnchen curry' IN BOOLEAN MODE)`
|
||||
- **Legacy JSONL:** `real_recipes.jsonl` is kept as historical archive only. v7 does NOT read or write it. The `migrate_to_galera.py` script was a one-time migration tool.
|
||||
|
||||
## Semantic Search + Preference Learning (Implemented 2026-06-28)
|
||||
|
||||
Fully built and tested. Uses `vllm/harrier-oss-v1-0.6b` (1024-dim) embeddings via norris `/v1/embeddings` endpoint.
|
||||
|
||||
### Scripts (all in `profiles/nutrition-coach/scripts/`)
|
||||
|
||||
- **`semantic_search.py`** — Core module: `semantic_search()`, `record_decision()`, `get_preference_adjustments()`, `rank_with_preferences()`, `mine_rules()`
|
||||
- **`generate_embeddings.py`** — Batch embedding generator (64 recipes/API call, ~31/s, ETA ~15min for 31k recipes). Stores as JSON in `recipes.embedding` column.
|
||||
- **`recipe_scraper_v7.py`** — Direct MySQL scraper (does NOT generate embeddings; run `generate_embeddings.py` separately after scraping)
|
||||
|
||||
### Database Tables
|
||||
|
||||
- **`recipes.embedding`** (JSON) — 1024-dim embedding array. Generated by `generate_embeddings.py`.
|
||||
- **`meal_decisions`** — Every accept/reject with context. Columns (verified 2026-06-28): `id`, `decision_date`, `recipe_id`, `recipe_title`, `recipe_category`, `accepted` (bool), `context_temp`, `context_season`, `context_weekday`, `context_last_meals`, `reason`, `created_at`. NOTE: context fields are separate columns, NOT a single JSON blob.
|
||||
- **`preference_rules`** — Learned rules. Columns (verified 2026-06-28): `id`, `rule_type`, `condition_json`, `effect_json`, `weight` (FLOAT), `confirmations` (INT), `rejections` (INT), `created_at`, `updated_at`. NOTE: condition/effect are JSON strings, not separate columns.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Semantic search:** Query text → harrier embedding → cosine similarity against all embedded recipes → ranked results blended with rating score
|
||||
2. **Decision recording:** User accepts/rejects a recipe → `record_decision()` logs context + reason → auto-creates rule if reason matches weather/temperature patterns
|
||||
3. **Preference ranking:** `rank_with_preferences()` applies learned rules (boost/penalty) on top of semantic scores. Also enforces variety (penalizes categories from recent meals).
|
||||
4. **Rule mining:** `mine_rules()` analyzes accumulated decisions for statistical patterns (e.g., "Eintopf rejected 80% of the time when temp>20°C" → creates/upgrades rule)
|
||||
|
||||
### Embedding Model: vllm/harrier-oss-v1-0.6b
|
||||
|
||||
- **NOT** an LLM — it's a dedicated embedding model on norris (0.6B params)
|
||||
- 1024 dimensions, multilingual (German works well)
|
||||
- Available via `POST https://ai.noris.de/v1/embeddings` with `{"model": "vllm/harrier-oss-v1-0.6b", "input": [...]}`
|
||||
- Batch API supports up to 64 inputs per call
|
||||
- Other noris models (bge-m3, e5-large, etc.) are recognized by the gateway but NOT activated — only harrier works
|
||||
- API key: same noris key from `config.yaml` providers.noris.api_key
|
||||
|
||||
See `references/semantic-search-implementation.md` for full architecture details.
|
||||
|
||||
## Semantic Ontology Classification (Added 2026-06-28)
|
||||
|
||||
Rule-based classifier annotates every recipe across 11 ontology dimensions: diet type, meal course, cuisine, cooking method, protein source, allergens, flavor profile, complexity, season, caloric density, and preparation style. Pure keyword matching — no LLM/API calls. Stores results in `recipe_semantics` table.
|
||||
|
||||
- **Script:** `profiles/nutrition-coach/scripts/semantic_classifier.py` (~500/batch, ~50-500/s, ~10min for 54k recipes)
|
||||
- **Table:** `recipe_semantics` (recipe_id PK, 11 dimension columns as ENUM/JSON, confidence, analyzed_at)
|
||||
- **Cronjob:** `semantic-classifier` daily at 07:00 (after scraper 06:00 + embeddings 06:30)
|
||||
- **Incremental:** LEFT JOIN finds unclassified recipes, only processes new ones. Uses `INSERT IGNORE` to skip duplicate recipe IDs (some Chefkoch IDs appear multiple times in the DB due to URL-state-encoded duplicates).
|
||||
- See `references/semantic-ontology-classification.md` for full ontology dimensions, table schema, and Galera-specific pitfalls
|
||||
|
||||
## Ingredient Normalization + Substitution Graph (Added 2026-06-28)
|
||||
|
||||
Relational ingredient system enabling exact filtering, allergen cross-checking, and "what can I substitute?" queries. Three tables + a parser/normalizer script.
|
||||
|
||||
### Tables
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `ingredients` | Master ingredient table (normalized) | `id`, `canonical_name`, `slug`, `category` (ENUM 21 values), `allergen_flags` (JSON), `is_vegan`, `is_vegetarian` |
|
||||
| `recipe_ingredient_links` | Junction table (many-to-many) | `recipe_id`, `ingredient_id`, `quantity` (DECIMAL), `unit`, `raw_text`, `modifier`, `position` |
|
||||
| `ingredient_substitutes` | Substitution graph (bidirectional) | `ingredient_id`, `substitute_id`, `similarity_score` (0-1), `context`, `reason` |
|
||||
|
||||
### Script: `scripts/ingredient_normalizer.py`
|
||||
|
||||
- **Parsing:** Regex extracts (quantity, unit, name, modifier) from raw strings like `"400 g Lachsfilet(s) (mit Haut)"` → `(400, "g", "Lachsfilet", "mit Haut")`
|
||||
- **Canonicalization:** Plural→singular, alias overrides (~250 manual mappings, e.g. `Mohrrüben→karotte`, `hühnchen→hähnchen`), descriptor stripping (`frisch`, `getrocknet`, `Bio`, etc.), English→German mapping (`garlic→knoblauch`)
|
||||
- **Categorization:** 21 food categories (meat, poultry, fish, seafood, vegetable, fruit, grain, legume, nut_seed, dairy, egg, fat_oil, spice_herb, condiment_sauce, sweetener, flour_starch, liquid, baking, beverage, other) via keyword matching
|
||||
- **Allergen detection:** Checks 14 EU allergen categories against canonical name
|
||||
- **Substitution graph:** Seeded with ~60 culinary substitution pairs, bidirectional. Each pair has similarity_score (0-1), context (when the substitution works), and reason (why they're similar). Examples: koriander↔petersilie (0.75, "European/Western dishes"), schmand↔crème fraîche (0.95, "General"), milch↔hafermilch (0.90, "General/vegan")
|
||||
|
||||
### Query Examples
|
||||
|
||||
```sql
|
||||
-- Recipes WITH zucchini, WITHOUT paprika, <500 kcal
|
||||
SELECT DISTINCT r.* FROM recipes r
|
||||
JOIN recipe_ingredient_links rl ON r.id = rl.recipe_id
|
||||
JOIN ingredients i ON rl.ingredient_id = i.id AND i.canonical_name = 'zucchini'
|
||||
WHERE r.calories < 500
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM recipe_ingredient_links rl2
|
||||
JOIN ingredients i2 ON rl2.ingredient_id = i2.id
|
||||
WHERE rl2.recipe_id = r.id AND i2.canonical_name = 'paprika'
|
||||
)
|
||||
|
||||
-- What can I substitute for coriander?
|
||||
SELECT i2.canonical_name, s.similarity_score, s.context, s.reason
|
||||
FROM ingredient_substitutes s
|
||||
JOIN ingredients i1 ON s.ingredient_id = i1.id AND i1.canonical_name = 'koriander'
|
||||
JOIN ingredients i2 ON s.substitute_id = i2.id
|
||||
ORDER BY s.similarity_score DESC
|
||||
```
|
||||
|
||||
### Pitfalls (Ingredient Normalization)
|
||||
|
||||
- **Galera FK creation fails on collation mismatch.** Same issue as `recipe_semantics` — `recipes` uses `utf8mb4_unicode_ci`, new tables default to `utf8mb4_general_ci`. Omit FKs or specify matching COLLATE.
|
||||
- **Galera deadlocks from concurrent writes.** Running `ingredient_normalizer.py` and `semantic_classifier.py` simultaneously causes Deadlock (1213). Run sequentially or add retry logic.
|
||||
- **Some recipe IDs exceed VARCHAR(128).** Chefkoch IDs with URL-encoded state fragments can be 248 chars. Use VARCHAR(255) for `recipe_id` columns.
|
||||
- **Compound ingredients like "Salz und Pfeffer"** are kept as-is (not split). The parser only splits when a quantity prefix is present.
|
||||
- **Ingredient string diversity is massive.** ~250k unique raw strings across 54k recipes, normalizing to ~2-5k canonical ingredients. The `CANONICAL_OVERRIDES` dict needs expansion over time for better coverage.
|
||||
|
||||
## Scraper v8 — International Sources + Auto-Translation (Added 2026-06-28)
|
||||
|
||||
The v8 scraper (`profiles/nutrition-coach/scripts/recipe_scraper_v8_intl.py`) extends v7 with 13 international English-language sources and integrated LLM translation to German.
|
||||
|
||||
### Sources (13 international, ~20k recipes total)
|
||||
|
||||
All sources use sitemap + JSON-LD (same infrastructure as v7). See `references/international-recipe-sources.md` for full details including sitemap URLs, recipe counts, and ID prefixes.
|
||||
|
||||
BBC Good Food (~5k), SkinnyTaste (~2.3k), Woks of Life (~1.5k), RecipeTin Eats (~1.7k), Rasa Malaysia (~1.4k), Sanjeev Kapoor (~2k, custom parser), Budget Bytes (~2k), Pinch of Yum (~1.5k), Omnivore's Cookbook (~772), Hot Thai Kitchen (~464), Spoon Fork Bacon (~911), Korean Bapsang (~275), Minimalist Baker (~650).
|
||||
|
||||
### Translation Pipeline
|
||||
|
||||
1. **Scrape** recipe page → extract JSON-LD → normalize to recipe dict
|
||||
2. **Batch** 3 recipes per LLM call
|
||||
3. **Translate** via `vllm/gemma-4-31b-it-dynamo` (non-reasoning model, noris API)
|
||||
4. **Unit conversion** baked into translation prompt: cups→g/ml, tbsp→EL, tsp→TL, oz→30g, lb→450g, °F→°C, stick butter→125g
|
||||
5. **Insert** translated recipe into Galera (German title, ingredients, instructions)
|
||||
|
||||
### Translation Model Choice — CRITICAL
|
||||
|
||||
- **USE `vllm/gemma-4-31b-it-dynamo`** (non-reasoning, fast, 4s/batch, 169 completion tokens)
|
||||
- **DO NOT USE `vllm/glm-5-2-nvfp4`** for translation — it's a reasoning model that wastes tokens on chain-of-thought (1710 completion tokens for same task, 10x cost) and sometimes returns EMPTY `content` field when reasoning exhausts max_tokens before generating the answer.
|
||||
- User explicitly directed: "nutze für übersetzungen ein nicht-reasoning modell wie gemma4-dynamo"
|
||||
|
||||
### Unit Conversion Rules (embedded in LLM system prompt)
|
||||
|
||||
| US Unit | Metric Equivalent |
|
||||
|---------|-------------------|
|
||||
| 1 cup (liquid) | 240 ml |
|
||||
| 1 cup (solid: flour/sugar) | 240 g |
|
||||
| 1 tablespoon (tbsp) | 1 EL |
|
||||
| 1 teaspoon (tsp) | 1 TL |
|
||||
| 1 ounce (oz) | 30 g |
|
||||
| 1 pound (lb) | 450 g |
|
||||
| 1 pint | 475 ml |
|
||||
| 1 quart | 950 ml |
|
||||
| 1 stick butter | 125 g Butter |
|
||||
| °F | °C ((F-32) × 5/9) |
|
||||
|
||||
User requirement: "Keine cups und spoons" — ALL recipes must use metric units.
|
||||
|
||||
### Pitfalls (v8 Scraper + Translation)
|
||||
|
||||
- **Reasoning models return empty `content` for translation tasks.** GLM-5.2 spends tokens on internal reasoning before generating the answer. With max_tokens=4000, reasoning consumed all tokens and `content` was empty string. Fix: use non-reasoning model (gemma-dynamo) OR increase max_tokens to 6000+. The non-reasoning model is the correct fix — translation is mechanical, not reasoning-intensive.
|
||||
- **Translation batch size matters for reliability.** Batch of 5 recipes sometimes exceeded the model's reliable output capacity. Reduced to 3 recipes per batch for consistent results.
|
||||
- **Sanjeev Kapoor needs custom parser.** Non-standard JSON-LD with single-quote `@type: 'Recipe` instead of `"Recipe"`. The v8 scraper includes `extract_sanjeev_recipe()` with regex fallback for this case.
|
||||
- **BBC Good Food quarterly sitemaps.** Must iterate through all quarterly sitemap files (`YYYY-QN-recipe.xml`) to get full coverage. 24 quarters yielded ~5,045 recipes.
|
||||
- **Translation adds significant latency.** Each batch of 3 recipes takes ~4-10s for the LLM call plus scraping time. Total throughput: ~1 recipe/2s. For 20k recipes, expect ~11 hours. Run as background process overnight.
|
||||
- **Post-scrape pipeline runs automatically.** The v8 scraper's `main()` function calls `generate_embeddings.py` → `semantic_classifier.py` → `ingredient_normalizer.py` via `subprocess.run()` after scraping completes (only if `grand_added > 0`). No separate cronjobs needed. See "Daily Pipeline" section above.
|
||||
|
||||
## Daily Pipeline (Integrated — Updated 2026-06-28)
|
||||
|
||||
**Single cronjob** `recipe-scraper-v8-pipeline` (daily 06:00) runs the full pipeline sequentially:
|
||||
|
||||
```
|
||||
recipe_scraper_v8_intl.py
|
||||
├── 1. Scrape international sources + translate to German
|
||||
├── 2. generate_embeddings.py (embeddings for new recipes)
|
||||
├── 3. semantic_classifier.py (ontology classification)
|
||||
└── 4. ingredient_normalizer.py (relational ingredient normalization)
|
||||
```
|
||||
|
||||
Pipeline steps 2–4 only run if `grand_added > 0` (no wasted work when nothing new scraped). Each step has a 2h timeout. Logs go to `scraper_v8_log.txt`.
|
||||
|
||||
**Why integrated (user correction):** User asked "Warum nicht embedding refill, classifier und ingredient normalizer direkt am Ende des scraper laufs?" — separate cronjobs with fixed schedules waste time waiting (06:30 embeddings waits 30min after scraper finishes) and risk overlapping if the scraper runs long. Integrated pipeline runs immediately after scraping, sequentially, no deadlocks.
|
||||
|
||||
**Previous setup (REMOVED):** Three separate cronjobs — `recipe-refill-v7` (06:00), `embedding-refill` (06:30), `semantic-classifier` (07:00). The 06:30 and 07:00 jobs were deleted; the 06:00 job was renamed to `recipe-scraper-v8-pipeline` and its prompt updated to run the v8 scraper (which includes the pipeline internally).
|
||||
|
||||
## Recipe MCP Server (Added 2026-06-28)
|
||||
|
||||
A stdio MCP server (`profiles/nutrition-coach/scripts/recipe_mcp_server.py`) exposes the Galera recipe DB as 7 first-class Hermes tools. Registered in `config.yaml` under `mcp_servers.recipes`. Tools auto-discover at gateway startup and appear as `mcp_recipes_*` in all profiles.
|
||||
|
||||
### Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `search_recipes` | FULLTEXT search on title+description and ingredients, with filters (category, calories, prep_time, rating, source) |
|
||||
| `semantic_recipe_search` | Vector similarity search using harrier embeddings (cosine sim over 500-candidate pool, re-ranked by similarity) |
|
||||
| `get_recipe` | Full recipe by ID (includes ingredients, instructions, all fields) |
|
||||
| `random_recipes` | Random recipes for meal-planning inspiration (with category/source/rating filters) |
|
||||
| `recipe_stats` | DB statistics: total count, per-source breakdown, embedding coverage |
|
||||
| `record_meal_decision` | Log accept/reject for preference learning (denormalizes recipe title+category into meal_decisions) |
|
||||
| `get_preferences` | Show learned preference rules + decision stats |
|
||||
|
||||
### Config (in `~/.hermes/config.yaml`)
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
recipes:
|
||||
command: "/home/debian/hermes-agent/venv/bin/python3"
|
||||
args: ["/home/debian/.hermes/profiles/nutrition-coach/scripts/recipe_mcp_server.py"]
|
||||
env:
|
||||
RECIPES_DB_PASS: "<galera-password>"
|
||||
NORIS_API_KEY: "<noris-api-key>"
|
||||
timeout: 60
|
||||
```
|
||||
|
||||
### Pitfalls
|
||||
|
||||
- **Column names differ from older skill docs.** The MCP server initially referenced `image_url` and `url` columns that don't exist. Always `DESCRIBE recipes` before writing SQL. See the verified schema above.
|
||||
- **FULLTEXT search requires German umlauts.** Searching "Haehnchen" returns 0 results; "Hähnchen" works. The FULLTEXT index is case-insensitive but NOT umlaut-normalized.
|
||||
- **MCP env vars are filtered.** Hermes only passes `PATH`, `HOME`, etc. to stdio subprocesses. DB password and API key MUST be explicitly set in the `env` block of the MCP server config — they are NOT inherited from `.env`.
|
||||
- **Gateway restart required after adding MCP servers.** MCP tools are discovered at startup. Adding a server to `config.yaml` does not hot-reload. Use `/restart` (gateway) or restart the systemd service.
|
||||
|
||||
- **Scraper max_new limits must be high for background operation.** v7 defaulted to `max_new=5000` per sitemap source and `max_new=2000` for Chefkoch per run. This was designed for daily cronjob safety, but in background operation it needlessly caps throughput — Kuechengoetter (30k URLs) and GuteKueche (29k URLs) stopped at 5k each per run, requiring 6+ runs to fully scrape. Fix (2026-06-28): raised to 50k (sitemap), 50k (HelloFresh), 10k (Chefkoch). Result: Kuechengoetter jumped from 5k to 20,442 in one run. For cronjob mode, the daily run naturally bounds total work; for background runs, the higher limits allow completing a source in one pass.
|
||||
- **Per-row dedup checks are catastrophically slow on Galera.** v7 initial design called db_has_id(rid) for every candidate URL — each a network roundtrip through MaxScale. With 28k existing recipes and 30k sitemap URLs mostly already in DB, the scraper appeared hung for minutes doing nothing but dedup queries. Fix: db_all_ids() loads all IDs into a Python set() once at startup (~0.3s for 28k), then all dedup checks are in-memory. Additionally, derive the likely recipe ID from the URL path and skip the HTTP fetch entirely if the ID is already known.
|
||||
- **Preference rule category matching must check title and tags, not just the category field.** Most Eintopf recipes have category Kochen, Kartoffeln, or Hauptspeise — not Eintopf. A rule penalizing category Eintopf would miss Moehren-Kartoffel-Eintopf (category: Kochen). Fix in rank_with_preferences(): concatenate category + title + tags into one lowercase string and check if the penalty/boost term appears anywhere in it.
|
||||
- **recipe_to_row() must sanitize dict/list fields before MySQL INSERT.** Some recipes have category as a dict (e.g., {"name": "..."}) or calories as a nested dict. PyMySQL raises TypeError: dict can not be used as parameter. Always coerce to string/primitive before passing to executemany().
|
||||
- **HelloFresh recipes may have EMPTY ingredients arrays in the DB.** Some HelloFresh recipes show ingredients as an empty JSON array despite being successfully scraped. The _hf_normalize() function builds ingredients from yields[0].ingredients[] joined with the ingredients[] lookup — if the yield structure differs or ingredient IDs do not match, the result is empty. This breaks ingredient-based SQL searches (LIKE finds nothing). Chefkoch recipes reliably have populated ingredient lists.
|
||||
- **Searching JSON columns with LIKE requires CAST.** MariaDB stores ingredients as LONGTEXT (JSON-encoded). Direct LIKE may fail depending on MariaDB version. Use LOWER(CAST(ingredients AS CHAR)) LIKE for reliable JSON field searching. For production queries, prefer the FULLTEXT index: MATCH(ingredients) AGAINST in BOOLEAN MODE.
|
||||
- **Norris embedding endpoint only works with vllm/harrier-oss-v1-0.6b.** All other norris models (glm, gemma, qwen, bge-reranker) return 404 on /v1/embeddings. Standard embedding model names (bge-m3, e5-large, gte-large) are recognized by the Bifrost gateway but NOT activated on any API key. Only harrier-oss-v1-0.6b (1024 dim) is available for embeddings.
|
||||
- **Galera FK creation fails on collation mismatch.** The `recipes` table uses `utf8mb4_unicode_ci` but new tables default to `utf8mb4_general_ci`. FK creation fails with `errno 150`. Fix: omit the FK (preferred) or specify matching COLLATE on the new table.
|
||||
- **Galera JOINs between tables with different collations fail.** `LEFT JOIN ... ON r.id = rs.recipe_id` fails with `Illegal mix of collations`. Fix: add `COLLATE utf8mb4_unicode_ci` to the JOIN condition.
|
||||
- **VARCHAR(128) is too short for some recipe IDs.** Chefkoch IDs with encoded state fragments (e.g., `ck_Kinderueberraschungsbrot.html#eyJvYm...`) can be up to 248 chars. Use `VARCHAR(255)` for any column storing recipe IDs.
|
||||
- **Time/calorie columns contain string values.** Some `prep_time`, `cook_time`, `total_time`, `calories` values are stored as strings (e.g., `"PT20M"`, `"450 kcal"`), not integers. Always use a `safe_int()` helper before comparisons/arithmetic.
|
||||
- **Concurrent writes cause Galera deadlocks.** Running the semantic classifier and embedding generator simultaneously causes `Deadlock (1213)`. Fix: add retry logic with rollback+backoff, or run sequentially (daily cronjob schedule ensures this).
|
||||
- **Classifier batch inserts hit duplicate-key errors.** Some Chefkoch recipe IDs appear multiple times in the `recipes` table (URL-encoded state fragments create variant IDs that normalize to the same key). Using `INSERT IGNORE` in the `executemany()` call silently skips duplicates instead of failing the entire batch. Without `INSERT IGNORE`, every batch containing a duplicate fails and falls through to one-by-one insertion, cutting throughput by ~10×.
|
||||
- **Cloudflare blocks ≠ "no sitemap".** Just One Cookbook was initially diagnosed as "no sitemap" — actually the sitemap exists but Cloudflare returns an HTML challenge page instead of XML. Always check HTTP status codes and content-type, not just whether XML parse succeeds. Maangchi similarly returns HTTP 403 for all programmatic requests despite a permissive robots.txt.
|
||||
- **AI-bot blocks in robots.txt are selective.** Many sites (RecipeTin Eats, Cookie and Kate, Healthy Nibbles, SkinnyTaste) block `anthropic-ai`, `Claude-Web`, `GPTBot`, or `ChatGPT-User` specifically. They do NOT block generic `Mozilla/5.0` user agents. Use a standard browser-like UA, not an AI-identifying one.
|
||||
- **Empty first sitemap ≠ empty site.** Minimalist Baker's `post-sitemap.xml` returns 0 URLs but `post-sitemap2.xml` has 650. Always check ALL sitemap files in the index.
|
||||
- **Non-standard JSON-LD.** Sanjeev Kapoor has recipe data but in a non-standard JSON structure (`@type: 'Recipe` with single quotes). Standard JSON-LD parsers will miss it. May need regex-based extraction or custom parser.
|
||||
- **Quarterly sitemap pattern.** BBC Good Food organizes recipe sitemaps by quarter (`YYYY-QN-recipe.xml`). Must iterate through all quarters to get full coverage (~5,045 recipes across 24+ quarterly files).
|
||||
- **AllRecipes sitemaps are empty shells.** Despite having `sitemap_1.xml` through `sitemap_4.xml`, they contain 0 URLs. Not scrapable via sitemap approach.
|
||||
- **Archana's Kitchen has 8,428 sitemap URLs but NO JSON-LD.** Would need custom HTML parsing — not worth the effort vs. Sanjeev Kapoor which has (non-standard) JSON-LD.
|
||||
- **Bon Appétit and Epicurious use Condé Nast's proprietary structured data format,** not schema.org/Recipe JSON-LD. Not scrapable with standard JSON-LD approach.
|
||||
- **Running semantic_classifier.py and ingredient_normalizer.py simultaneously causes Galera deadlocks.** Both write to different tables but certification conflicts on shared rows cause `Deadlock (1213)`. Run sequentially or add retry logic with rollback+backoff.
|
||||
- **Ingredient normalizer produces ~33k unique ingredient names from 54k recipes.** The `CANONICAL_OVERRIDES` dict (~250 manual mappings) covers the most common cases but many rare ingredients remain un-canonicalized. Expect ~16k ingredients categorized as "other" — expand the categorization keyword lists and canonical overrides over time.
|
||||
- **Substitution graph seeding skips ingredients not in the DB.** 27 of 92 planned substitution pairs were skipped because one ingredient wasn't found in the first run. Re-seed after expanding canonical overrides.
|
||||
- **Galera FK creation fails on collation mismatch.** Same issue as `recipe_semantics` — `recipes` uses `utf8mb4_unicode_ci`, new tables default to `utf8mb4_general_ci`. Omit FKs or specify matching COLLATE.
|
||||
- **Galera deadlocks from concurrent writes.** Running `ingredient_normalizer.py` and `semantic_classifier.py` simultaneously causes Deadlock (1213). Run sequentially or add retry logic.
|
||||
- **Some recipe IDs exceed VARCHAR(128).** Chefkoch IDs with URL-encoded state fragments can be 248 chars. Use VARCHAR(255) for `recipe_id` columns.
|
||||
- **Compound ingredients like "Salz und Pfeffer"** are kept as-is (not split). The parser only splits when a quantity prefix is present.
|
||||
- **Ingredient string diversity is massive.** ~250k unique raw strings across 54k recipes, normalizing to ~33k canonical ingredients. The `CANONICAL_OVERRIDES` dict needs expansion over time for better coverage.
|
||||
- **Ingredients ARE relationally searchable (implemented 2026-06-28).** Three tables power the ingredient subsystem: `ingredients` (master table with canonical names, categories, allergen flags, vegan/vegetarian), `recipe_ingredient_links` (junction with quantity/unit/modifier/position), and `ingredient_substitutes` (substitution graph with similarity scores and context). Script: `scripts/ingredient_normalizer.py`. Parses raw ingredient strings via regex (quantity + unit + name + modifier), canonicalizes names (plural→singular, alias overrides, descriptor stripping), categorizes into 21 food categories, and seeds a culinary substitution graph (~60 pairs, e.g. koriander↔petersilie 0.75, schmand↔crème fraîche 0.95). Enables SQL queries like "recipes WITH zucchini AND WITHOUT paprika AND <500 kcal" and "what can I substitute for coriander?".
|
||||
- **Galera FK constraints are fragile.** Certification-based replication can reject DDL that works on standalone MySQL. Prefer logical relationships (application-level integrity) over physical FKs.
|
||||
- **International recipe sources research (updated 2026-06-28).** 15 sources evaluated across 4 tiers. Tier 1 (ready to integrate, ~20k recipes total): BBC Good Food (~5k), SkinnyTaste (~2.3k, light/healthy), Woks of Life (~1.5k, Chinese), RecipeTin Eats (~1.7k, pan-Asian), Rasa Malaysia (~1.4k, SE-Asian), Sanjeev Kapoor (~2k, Indian, non-standard JSON-LD), Budget Bytes (~2k), Pinch of Yum (~1.5k), Omnivore's Cookbook (~772, Chinese), Hot Thai Kitchen (~464, authentic Thai), Spoon Fork Bacon (~911, Asian fusion), Korean Bapsang (~275, Korean), Minimalist Baker (~650, vegan/GF). Tier 2 (AI-bot blocks, use generic UA): Cookie and Kate (~920, vegetarian), Healthy Nibbles (~640, Asian healthy). Tier 3 (not feasible): Just One Cookbook (Cloudflare), Maangchi (403), AllRecipes (empty sitemaps), Archana's Kitchen (no JSON-LD), Bon Appétit/Epicurious (proprietary format), Serious Eats (no JSON-LD confirmed). See `references/international-recipe-sources.md` for full details including sitemap URLs, recipe counts, and ID prefixes.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Meal-Planning Cron Setup — Known Pitfalls & Fixes
|
||||
|
||||
Condensed troubleshooting guide for the `nutrition-coach` cronjob stack. Each section provides the symptom → root cause → fix.
|
||||
|
||||
---
|
||||
|
||||
## 1. Interactive Interview Via Cronjob Is Impossible
|
||||
|
||||
**Symptom**
|
||||
```
|
||||
Wed 19:30: Agent sends 10 questions to Telegram group.
|
||||
User replies 19:35–19:42 in Telegram.
|
||||
current_interview.json: all answers are null
|
||||
```
|
||||
|
||||
**Root cause**
|
||||
A cronjob is a single-shot process. It sends, then exits. It never reads Telegram inbound messages. User replies arrive in the gateway but there is no running session to pair them with.
|
||||
|
||||
**Fix options**
|
||||
1. **Replace with a non-interactive reminder** (recommended for automation):
|
||||
```bash
|
||||
hermes cron edit <id> --prompt "🔔 Wochenplan: Schreibe \"interview\" oder \"plan\" um loszulegen." --no-agent
|
||||
```
|
||||
User types a keyword in chat → triggers the interview in a live session.
|
||||
2. **Use a live session for the interview.** Load the skill (`/skill meal-planning`) and run the interview interactively. Store the JSON for Friday.
|
||||
|
||||
---
|
||||
|
||||
## 2. Delivery Target Fails on Named Groups
|
||||
|
||||
**Symptom**
|
||||
```
|
||||
delivery error: Telegram send failed:
|
||||
invalid literal for int() with base 10: 'Schoen Inc'
|
||||
```
|
||||
|
||||
**Root cause**
|
||||
Cron `deliver` field expects a numeric `chat_id` or a pre-registered alias. `"telegram:Schoen Inc"` is a display name, not a parseable ID.
|
||||
|
||||
**Fix**
|
||||
Use one of the following verified targets for the nutrition-coach profile:
|
||||
```json
|
||||
"deliver": "telegram:D S" // DM with Dominik
|
||||
"deliver": "telegram:-5265906503" // Schoen Inc group
|
||||
"deliver": "telegram" // home channel
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Override in Cron JSON Is Not Updated by `hermes config set`
|
||||
|
||||
**Symptom**
|
||||
Bulk-switched profile to `vllm/glm-5-2-nvfp4` via `hermes config set`, but cronjob still calls `vllm/moonshotai/kimi-k2.6`.
|
||||
|
||||
**Root cause**
|
||||
Cron jobs hard-code `model` and `provider` inside their own JSON entry in `profiles/nutrition-coach/cron/jobs.json`. The profile `config.yaml` is not consulted at runtime.
|
||||
|
||||
**Fix — Python one-liner**
|
||||
```python
|
||||
import json
|
||||
path = '/home/debian/.hermes/profiles/nutrition-coach/cron/jobs.json'
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
for j in data['jobs']:
|
||||
if 'nutrition' in j['name'].lower():
|
||||
j['model'] = 'vllm/glm-5-2-nvfp4'
|
||||
j['provider'] = 'noris'
|
||||
with open(path, 'w') as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Interview JSON Schema Mismatch
|
||||
|
||||
**Symptom**
|
||||
`nutrition-plan-generator-v4.py` crashes with `KeyError` on fields it expects (e.g. `dominik_ho`, `special_events`).
|
||||
|
||||
**Root cause**
|
||||
The interview data file (`current_interview.json`) was created by a different schema than the generator expects. Cronjobs that produce JSON by LLM free-form output are fragile.
|
||||
|
||||
**Fix**
|
||||
Always use the Interview Schema template (`templates/interview-data-schema.json`) when persisting. Validate before saving:
|
||||
```bash
|
||||
python3 -m json.tool current_interview.json >/dev/null || echo "INVALID JSON"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist After Any Change
|
||||
|
||||
```bash
|
||||
hermes --profile nutrition-coach cron list # jobs running under correct profile?
|
||||
hermes config --profile nutrition-coach | grep Model # profile model correct?
|
||||
python3 -c "import json; d=json.load(open('/home/debian/.hermes/profiles/nutrition-coach/cron/jobs.json')); print([j['model'] for j in d['jobs']])" # cron model correct?
|
||||
cat /home/debian/.hermes/profiles/nutrition-coach/plan_data/current_interview.json | python3 -m json.tool # interview valid?
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
# Generator v4 Scoring Algorithm
|
||||
|
||||
## Data Source (EXCLUSIVE — NO FALLBACK)
|
||||
- **ONLY SOURCE**: `~/.hermes/profiles/nutrition-coach/real_recipes.jsonl` (25k+ Chefkoch recipes, JSONL, one recipe per line)
|
||||
- **NEVER use**: `recipes.json` (old dummy data), live web scraping, or external recipe APIs
|
||||
- **Minimum rating filter**: 4.3 stars (raises quality, reduces noise)
|
||||
- **Excluded categories**: Backen, Kuchen, Dessert (penalty −20)
|
||||
|
||||
## Scoring Dimensions
|
||||
|
||||
| Factor | Weight | Logic |
|
||||
|---|---|---|
|
||||
| Base rating | ×2 | Higher-rated recipes start with advantage |
|
||||
| Weather heat ≥30°C | +6 / −4 | Cold/salad boosted; hot cooking penalized |
|
||||
| Weather heat ≥25°C | +4 / −3 | Same direction, lesser magnitude |
|
||||
| Cold ≤15°C | +4 | Soups/stews/curry boosted |
|
||||
| Perishables (fish, herbs) | +4 early / −5 late | Fresh items forced to Mon–Tue |
|
||||
| Asian boost | +3 | curry, wok, thai, vietnam, ramen tags |
|
||||
| Low-carb / protein | +2 | Dominik weight-goal diet |
|
||||
| Carb boost | +1.5 | Pasta, rice, potato — Sarah preference |
|
||||
| Cleo-friendly | +1.5 | pasta, nudeln, gnocchi, schnell, einfach tags |
|
||||
| Meal-Prep days (Di/Do) | +6 | oven, stew, curry, gratin, lasagne boosted |
|
||||
| Meal-Prep: fast ≤40min | +2 | Quick prep bonus on Di/Do |
|
||||
| Meal-Prep: fry penalty | −10 | Frittiertes / Schnitzel / Tempura bad for meal-prep |
|
||||
| Thursday travel | +10 | ≤30min recipes if special events contain "reise" |
|
||||
| Friday events (picknick) | +15 | salad, wrap, cold, fingerfood boosted |
|
||||
| Monday events (grill) | +15 / +20 | grill tags / barbecue category boosted |
|
||||
| Diversity: new category | +2 | Unseen category gets bonus |
|
||||
| Diversity: name overlap | −8 | Similar names penalized to avoid repetition |
|
||||
| Craving match | +5 | If keyword in name or tags |
|
||||
|
||||
## Cleo Logic
|
||||
- **Joins family meal** if recipe tags contain: pasta, nudeln, gnocchi, kartoffel, reis
|
||||
- **Gets extra dish** otherwise: "Nudeln mit Butter & Käse (frisch)"
|
||||
- Hard rule: NO leftovers for Cleo, ever
|
||||
|
||||
## Output
|
||||
- Compact Telegram summary: 7 lines (one per day) + list count
|
||||
- Full details written to `~/Documents/Essensplanung/KW<NN>_plan.md`
|
||||
- Shopping list to `~/Documents/Essensplanung/KW<NN>_einkaufsliste.md`
|
||||
@@ -0,0 +1,203 @@
|
||||
# International Recipe Sources (Researched 2026-06-28, Updated 2026-06-28)
|
||||
|
||||
Focus: Asian cuisine, light/healthy main courses, internationally diverse. English-language sites with JSON-LD for scraping + translation to German.
|
||||
|
||||
## Tier 1 — Ready to Integrate (JSON-LD confirmed + sitemap available + no AI-bot blocks)
|
||||
|
||||
### BBC Good Food (bbcgoodfood.com) — ~5,045 recipes ⭐ TOP PICK
|
||||
- **Sitemap:** Quarterly recipe sitemaps at `https://www.bbcgoodfood.com/sitemaps/YYYY-QN-recipe.xml` (indexed from `sitemap.xml`)
|
||||
- **Counts per quarter:** Q2-2026: 149, Q1-2026: 156, Q4-2025: 210, Q3-2025: 179, Q2-2025: 182, Q1-2025: 214, Q4-2024: 292, Q3-2024: 179, Q2-2024: 160, Q1-2024: 223, Q4-2023: 290, Q3-2023: 262, Q2-2023: 132, Q1-2023: 203, Q4-2022: 296, Q3-2022: 231, Q2-2022: 272, Q1-2022: 207, Q4-2021: 250, Q3-2021: 206, Q2-2021: 154, Q1-2021: 191, Q4-2020: 278, Q3-2020: 129. **Total: ~5,045**
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Huge international coverage, including dedicated Asian/Indian/Thai sections
|
||||
- **Robots.txt:** Permissive — declares sitemaps, no AI-bot blocks
|
||||
- **Quality:** Professional editorial team, nutrition data, tested recipes
|
||||
- **ID prefix:** `bgf_`
|
||||
|
||||
### RecipeTin Eats (recipetineats.com) — ~1,697 recipes
|
||||
- **Sitemap:** `https://www.recipetineats.com/sitemap_index.xml` → 4 post sitemaps (495+500+499+203 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Pan-Asian (Japanese, Thai, Korean, Chinese), Australian, fusion. ~404 URLs match Asian keywords.
|
||||
- **Robots.txt:** Blocks anthropic-ai and Claude-Web specifically, but does NOT block generic crawlers. Standard UA with polite delays works.
|
||||
- **Quality:** High — detailed instructions, ratings, step photos. Many authentic Asian recipes.
|
||||
- **ID prefix:** `rt_`
|
||||
|
||||
### Woks of Life (thewoksoflife.com) — ~1,555 recipes ⭐ BEST CHINESE
|
||||
- **Sitemap:** `https://thewoksoflife.com/sitemap_index.xml` → post-sitemap.xml (997) + post-sitemap2.xml (558)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Chinese (Cantonese, Sichuan, regional), some Asian-American
|
||||
- **Robots.txt:** Permissive (blocks wp-admin, cdn-cgi, wp-json only)
|
||||
- **Quality:** Family-authored, authentic Chinese techniques, detailed step photos
|
||||
- **ID prefix:** `wol_`
|
||||
|
||||
### SkinnyTaste (skinnytaste.com) — ~2,354 recipes ⭐ BEST LIGHT/HEALTHY
|
||||
- **Sitemap:** `https://www.skinnytaste.com/sitemap_index.xml` → 3 post sitemaps (685+862+807 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Healthy, low-calorie, light mains, WW-friendly. Strong international coverage including Asian-inspired light dishes.
|
||||
- **Robots.txt:** Blocks Google-Extended, GPTBot, ChatGPT-User, PiplBot. Does NOT block generic crawlers.
|
||||
- **Quality:** Excellent — Gina Homolka provides nutrition data (calories, macros) for every recipe. Professionally tested.
|
||||
- **ID prefix:** `st_`
|
||||
|
||||
### Rasa Malaysia (rasamalaysia.com) — ~1,390 recipes ⭐ BEST SE-ASIAN
|
||||
- **Sitemap:** `https://rasamalaysia.com/sitemap_index.xml` → post-sitemap.xml (984) + post-sitemap2.xml (406)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Malaysian, Singaporean, Thai, Chinese, pan-Southeast-Asian
|
||||
- **Robots.txt:** Permissive (blocks cdn-cgi, wp-admin only)
|
||||
- **Quality:** High — authentic Southeast Asian recipes with detailed cultural context
|
||||
- **ID prefix:** `rm_`
|
||||
|
||||
### Pinch of Yum (pinchofyum.com) — ~1,483 recipes
|
||||
- **Sitemap:** `https://pinchofyum.com/sitemap_index.xml` → post-sitemap.xml (912) + post-sitemap2.xml (571)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Comfort food, international, some Asian-inspired
|
||||
- **Robots.txt:** Permissive (blocks wp-admin, search only)
|
||||
- **Quality:** Good — standardized format, nutrition data sometimes included
|
||||
- **ID prefix:** `poy_`
|
||||
|
||||
### Budget Bytes (budgetbytes.com) — ~1,981 recipes
|
||||
- **Sitemap:** `https://www.budgetbytes.com/post-sitemap.xml` (976) + post-sitemap2.xml (974) + post-sitemap3.xml (31)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** American comfort, international fusion, budget-friendly mains
|
||||
- **Robots.txt:** Permissive
|
||||
- **ID prefix:** `bb_`
|
||||
|
||||
### Omnivore's Cookbook (omnivorescookbook.com) — ~772 recipes
|
||||
- **Sitemap:** `https://omnivorescookbook.com/sitemap_index.xml` → post-sitemap.xml (772 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Modern Chinese, some Asian fusion
|
||||
- **Robots.txt:** Has Raptive ad-network block list but does NOT block generic crawlers
|
||||
- **Quality:** Good — authentic Chinese with modern adaptations
|
||||
- **ID prefix:** `oc_`
|
||||
|
||||
### Hot Thai Kitchen (hot-thai-kitchen.com) — ~464 recipes ⭐ MOST AUTHENTIC THAI
|
||||
- **Sitemap:** `https://hot-thai-kitchen.com/sitemap_index.xml` → post-sitemap.xml (464 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Authentic Thai (not Westernized)
|
||||
- **Robots.txt:** Permissive (Yoast standard)
|
||||
- **Quality:** Exceptional — Pai (author) explains Thai cooking theory, ingredient substitution within Thai context
|
||||
- **ID prefix:** `htk_`
|
||||
|
||||
### Spoon Fork Bacon (spoonforkbacon.com) — ~911 recipes
|
||||
- **Sitemap:** `https://www.spoonforkbacon.com/sitemap_index.xml` → post-sitemap.xml (911 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Asian fusion, modern international
|
||||
- **Robots.txt:** Permissive (wp-admin, search only)
|
||||
- **ID prefix:** `sfb_`
|
||||
|
||||
### Korean Bapsang (koreanbapsang.com) — ~275 recipes ⭐ BEST KOREAN
|
||||
- **Sitemap:** `https://www.koreanbapsang.com/sitemap_index.xml` → post-sitemap.xml (275 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Authentic Korean home cooking
|
||||
- **Robots.txt:** Permissive (Yoast standard)
|
||||
- **Quality:** Excellent — Susan (author) provides authentic Korean recipes with cultural context
|
||||
- **ID prefix:** `kb_`
|
||||
|
||||
### Sanjeev Kapoor (sanjeevkapoor.com) — ~1,965 recipes ⭐ BEST INDIAN
|
||||
- **Sitemap:** `https://www.sanjeevkapoor.com/webcontent-sitemap.xml` (1,965 URLs)
|
||||
- **JSON-LD:** ⚠️ Non-standard — uses `@type: Recipe` but in a different JSON structure (not standard schema.org/Recipe JSON-LD block). Has `recipeInstructions` and `recipeIngredient` fields. Needs custom parser.
|
||||
- **Cuisine focus:** Indian (all regions, vegetarian + non-veg)
|
||||
- **Robots.txt:** Declares sitemaps, no AI-bot blocks
|
||||
- **Quality:** Celebrity chef, authentic Indian recipes
|
||||
- **ID prefix:** `sk_`
|
||||
- **Pitfall:** JSON-LD is present but structured differently from WP recipe plugins. The `@type` appears as `'Recipe` (note quote style) rather than `"Recipe"`. May need regex extraction instead of standard JSON-LD parsing.
|
||||
|
||||
### Minimalist Baker (minimalistbaker.com) — ~650 recipes
|
||||
- **Sitemap:** `https://minimalistbaker.com/wp-sitemap.xml` → `post-sitemap.xml` (0 URLs — appears empty) + `post-sitemap2.xml` (650 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Simple plant-based, ≤10 ingredients, GF options
|
||||
- **Robots.txt:** Disallows `/r/`, `/ecourse/`, `/university/` paths. Blocks anthropic-ai.
|
||||
- **Quality:** High — Dana Schultz (author) provides consistently formatted recipes
|
||||
- **ID prefix:** `mb_`
|
||||
- **Pitfall:** `post-sitemap.xml` returns 0 URLs — use `post-sitemap2.xml` instead. This caused a misdiagnosis in the first research pass.
|
||||
|
||||
## Tier 2 — JSON-LD Confirmed but Blocks AI Bots (use generic UA)
|
||||
|
||||
### Cookie and Kate (cookieandkate.com) — ~920 recipes
|
||||
- **Sitemap:** `https://cookieandkate.com/post-sitemap.xml` (920 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Vegetarian, healthy, Mediterranean-inspired, some Asian fusion
|
||||
- **Robots.txt:** Blocks anthropic-ai, Claude-Web, bot "008". Does NOT block generic Mozilla UA.
|
||||
- **ID prefix:** `ckate_`
|
||||
|
||||
### Healthy Nibbles (healthynibblesandbits.com) — ~640 recipes
|
||||
- **Sitemap:** `https://healthynibblesandbits.com/post-sitemap.xml` (640 URLs)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Cuisine focus:** Asian, healthy, light mains
|
||||
- **Robots.txt:** Blocks ChatGPT-User, GPTBot, anthropic-ai via Raptive. Generic UA works.
|
||||
- **ID prefix:** `hn_`
|
||||
|
||||
## Tier 3 — Blocked or Not Feasible
|
||||
|
||||
### Just One Cookbook (justonecookbook.com) — ~1,200+ recipes
|
||||
- **Status:** ❌ Cloudflare blocks ALL automated access. Returns Cloudflare challenge page for curl requests. Sitemap returns HTML error page, not XML.
|
||||
- **JSON-LD:** ✅ Exists (confirmed via browser in earlier session)
|
||||
- **Note:** Previously listed as "no sitemap" — incorrect. The site HAS a sitemap (`sitemap_index.xml` declared in robots.txt) but Cloudflare blocks programmatic access to it. Would need Playwright/browser automation to scrape.
|
||||
- **Correction from earlier doc:** Original reference said "No sitemap index or post-sitemap found" — this was wrong. The sitemap exists but is behind Cloudflare protection.
|
||||
|
||||
### Maangchi (maangchi.com) — ~500+ recipes
|
||||
- **Status:** ❌ Returns HTTP 403 for all programmatic requests (even with browser-like UA)
|
||||
- **JSON-LD:** Could not verify (blocked)
|
||||
- **Cuisine focus:** Korean (most popular English-language Korean site)
|
||||
- **Note:** robots.txt is permissive (only blocks /post-comments/) but the server itself rejects non-browser requests.
|
||||
|
||||
### AllRecipes (allrecipes.com)
|
||||
- **Status:** ❌ Sitemaps exist (`sitemap_1.xml` through `sitemap_4.xml`) but contain 0 URLs — appear to be empty or dynamically generated.
|
||||
- **JSON-LD:** Could not confirm on sampled recipe URLs
|
||||
- **Note:** Largest US recipe site but not scrapable via sitemap approach.
|
||||
|
||||
### Archana's Kitchen (archanaskitchen.com) — 8,428 URLs
|
||||
- **Status:** ❌ Sitemap has 8,428 URLs but NO JSON-LD on recipe pages (0 blocks found). Would need custom HTML parsing.
|
||||
- **Cuisine focus:** Indian
|
||||
- **Note:** Large volume but no structured data — not worth the effort vs. Sanjeev Kapoor which has JSON-LD.
|
||||
|
||||
### Bon Appétit (bonappetit.com) / Epicurious (epicurious.com)
|
||||
- **Status:** ❌ No JSON-LD on recipe pages. Both use Condé Nast's proprietary structured data format.
|
||||
- **Note:** High quality but not scrapable via JSON-LD approach.
|
||||
|
||||
### Serious Eats (seriouseats.com)
|
||||
- **Status:** ❌ Sitemap exists but returned 0 recipe URLs. JSON-LD not confirmed on sampled URLs.
|
||||
- **Note:** May use WPRM (WordPress Recipe Maker) structured data instead of JSON-LD. Would need custom parser.
|
||||
|
||||
### Pick Up Limes (pickuplimes.com) — ~240 recipes
|
||||
- **Sitemap:** `https://www.pickuplimes.com/sitemap.xml` — only 1 recipe URL found in sitemap (very small)
|
||||
- **JSON-LD:** ✅ Confirmed
|
||||
- **Robots.txt:** Returns HTML page, not a proper robots.txt
|
||||
- **Note:** Very small volume (~240). High quality vegan but not worth integration effort for the volume.
|
||||
|
||||
## Summary: Recommended Integration Priority
|
||||
|
||||
| Priority | Source | Recipes | Why |
|
||||
|----------|--------|---------|-----|
|
||||
| 1 | BBC Good Food | ~5,045 | Largest, international, permissive |
|
||||
| 2 | SkinnyTaste | ~2,354 | Best for light/healthy gap |
|
||||
| 3 | Woks of Life | ~1,555 | Best authentic Chinese |
|
||||
| 4 | RecipeTin Eats | ~1,697 | Pan-Asian, high quality |
|
||||
| 5 | Rasa Malaysia | ~1,390 | Best SE-Asian |
|
||||
| 6 | Sanjeev Kapoor | ~1,965 | Best Indian (needs custom parser) |
|
||||
| 7 | Budget Bytes | ~1,981 | Broad international, budget |
|
||||
| 8 | Pinch of Yum | ~1,483 | Comfort food, international |
|
||||
| 9 | Omnivore's Cookbook | ~772 | Modern Chinese |
|
||||
| 10 | Hot Thai Kitchen | ~464 | Most authentic Thai |
|
||||
| 11 | Spoon Fork Bacon | ~911 | Asian fusion |
|
||||
| 12 | Korean Bapsang | ~275 | Authentic Korean |
|
||||
| 13 | Minimalist Baker | ~650 | Simple vegan/GF |
|
||||
| 14 | Cookie and Kate | ~920 | Vegetarian (use generic UA) |
|
||||
| 15 | Healthy Nibbles | ~640 | Asian healthy (use generic UA) |
|
||||
|
||||
**Total potential: ~20,000+ new recipes**, majority Asian or light/healthy.
|
||||
|
||||
## Translation Strategy
|
||||
|
||||
Since these are English-language sources, recipes need translation before or after insertion:
|
||||
|
||||
1. **Pre-insertion translation (recommended):** Translate title, description, ingredients, and instructions to German before DB insert. Use LLM batch translation (GLM-5.2 via noris, or a dedicated translation model).
|
||||
2. **Store both languages:** Keep original English text in a separate column (`title_en`, `description_en`) for reference.
|
||||
3. **Ingredient mapping:** English ingredients should map to the SAME canonical names in the `ingredients` table (e.g., `garlic→knoblauch`, `chicken breast→hähnchenbrust`). The `CANONICAL_OVERRIDES` dict in `ingredient_normalizer.py` already includes English→German mappings.
|
||||
4. **Source prefix:** Use the ID prefixes listed above for each source.
|
||||
|
||||
## Scraping Pitfalls (International Sources)
|
||||
|
||||
- **Cloudflare blocks ≠ "no sitemap".** Just One Cookbook was initially diagnosed as "no sitemap" — actually the sitemap exists but Cloudflare returns an HTML challenge page instead of XML. Always check HTTP status codes and content-type, not just whether XML parse succeeds.
|
||||
- **AI-bot blocks in robots.txt are selective.** Many sites (RecipeTin Eats, Cookie and Kate, Healthy Nibbles, SkinnyTaste) block `anthropic-ai`, `Claude-Web`, `GPTBot`, or `ChatGPT-User` specifically. They do NOT block generic `Mozilla/5.0` user agents. Use a standard browser-like UA, not an AI-identifying one.
|
||||
- **Empty first sitemap ≠ empty site.** Minimalist Baker's `post-sitemap.xml` returns 0 URLs but `post-sitemap2.xml` has 650. Always check ALL sitemap files in the index.
|
||||
- **Non-standard JSON-LD.** Sanjeev Kapoor has recipe data but in a non-standard JSON structure (`@type: 'Recipe` with single quotes instead of `"Recipe"`). Standard JSON-LD parsers will miss it. May need regex-based extraction or custom parser.
|
||||
- **Quarterly sitemap pattern.** BBC Good Food organizes recipe sitemaps by quarter (`YYYY-QN-recipe.xml`). Must iterate through all quarters to get full coverage. Older quarters may not exist — check availability before assuming.
|
||||
+304
@@ -0,0 +1,304 @@
|
||||
# Chefkoch.de Internal API v2 — Discovery & Detail
|
||||
|
||||
Discovered during v5.2 scraper development. This endpoint is **public and unauthenticated** (as of 2026-06) and returns structured JSON for 384,252+ recipes.
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
https://api.chefkoch.de/v2/recipes
|
||||
```
|
||||
|
||||
No API key, no auth headers, no cookies required.
|
||||
|
||||
## Endpoints
|
||||
|
||||
### 1. Listing (Discovery)
|
||||
|
||||
```
|
||||
GET /v2/recipes?offset={N}&limit={limit}
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `offset` — integer, start index (pagination)
|
||||
- `limit` — integer, max 100 per request (default 30)
|
||||
|
||||
**Response shape**
|
||||
|
||||
```json
|
||||
{
|
||||
"count": 384252,
|
||||
"queryId": "",
|
||||
"results": [
|
||||
{
|
||||
"recipe": {
|
||||
"id": "4073161636112189",
|
||||
"title": "Flammkuchen mit Kartoffeln und Schwarzwälder Schinken",
|
||||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||||
"siteUrl": "https://www.chefkoch.de/rezepte/4073161636112189/Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken.html",
|
||||
"difficulty": 1,
|
||||
"preparationTime": 15,
|
||||
"rating": {"rating": 4.75, "numVotes": 20},
|
||||
"hasImage": true,
|
||||
"isPremium": true,
|
||||
"createdAt": "2021-11-05T13:36:51+01:00"
|
||||
},
|
||||
"score": 384252
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Discovery-only fields** (useful for filtering before detail fetch):
|
||||
- `id` — recipe ID (string)
|
||||
- `title` / `slug` / `siteUrl` — human metadata
|
||||
- `preparationTime` — minutes (int)
|
||||
- `difficulty` — 1=easy, 3=hard
|
||||
- `rating.rating` / `rating.numVotes`
|
||||
- `isPremium` / `isPlus` — premium flag
|
||||
|
||||
### 2. Detail (Full Recipe)
|
||||
|
||||
```
|
||||
GET /v2/recipes/{id}
|
||||
```
|
||||
|
||||
**Response shape** (key fields)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "4073161636112189",
|
||||
"title": "Flammkuchen…",
|
||||
"slug": "Flammkuchen-mit-Kartoffeln-und-Schwarzwaelder-Schinken",
|
||||
"siteUrl": "https://www.chefkoch.de/rezepte/...",
|
||||
"subtitle": "Originalrezept von Viki Fuchs",
|
||||
"additionalDescription": "",
|
||||
"instructions": "Den Ofen auf 220 °C Umluft vorheizen…",
|
||||
"ingredientGroups": [
|
||||
{
|
||||
"header": "Für den Teig: ",
|
||||
"ingredients": [
|
||||
{
|
||||
"id": "8891376",
|
||||
"name": "Öl",
|
||||
"unit": "EL",
|
||||
"unitId": "13",
|
||||
"amount": 4.0,
|
||||
"isBasic": false,
|
||||
"usageInfo": "",
|
||||
"foodId": "184"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"tags": ["Schnell", "einfach", "Backen", "Frankreich"],
|
||||
"rating": {"rating": 4.75, "numVotes": 20},
|
||||
"preparationTime": 15,
|
||||
"kCalories": 0,
|
||||
"servings": 2,
|
||||
"nutrition": null,
|
||||
"difficulty": 1
|
||||
}
|
||||
```
|
||||
|
||||
## Structured Ingredients (API vs JSON-LD)
|
||||
|
||||
| Aspect | JSON-LD (`recipeIngredient`) | API (`ingredientGroups`) |
|
||||
|--------|------------------------------|--------------------------|
|
||||
| Format | raw strings: `"4 EL Öl"` | objects: `{amount: 4, unit: "EL", name: "Öl"}` |
|
||||
| Groups | flat list | grouped by `header` (e.g. "Für den Teig:") |
|
||||
| Parsing | requires regex heuristics | already parsed |
|
||||
| Accuracy | ~95% (edge cases in parentheses) | ~99% |
|
||||
|
||||
**Normalization recipe** (from API objects to flat strings):
|
||||
|
||||
```python
|
||||
ingredients = []
|
||||
for group in data.get("ingredientGroups", []):
|
||||
header = group.get("header", "").strip()
|
||||
if header:
|
||||
ingredients.append(f"--- {header} ---")
|
||||
for ing in group.get("ingredients", []):
|
||||
parts = [str(p) for p in [ing.get("amount"), ing.get("unit"), ing.get("name")] if p]
|
||||
if parts:
|
||||
ingredients.append(" ".join(parts))
|
||||
```
|
||||
|
||||
## Rate Limits & Politeness
|
||||
|
||||
**Observed behavior:**
|
||||
- No explicit rate limiting (no 429 responses observed at 16 concurrent requests)
|
||||
- Server responds reliably with `application/json`
|
||||
- Total count fluctuates slightly (cache propagation)
|
||||
|
||||
**Recommended politeness:**
|
||||
- 0.3s between listing batches
|
||||
- 16 concurrent detail fetches (ThreadPoolExecutor)
|
||||
- No backoff needed for normal operation
|
||||
|
||||
## Comparison: API vs Playwright
|
||||
|
||||
| Metric | Playwright v4/v5 | API v5.2 |
|
||||
|--------|------------------|----------|
|
||||
| Discovery | Playwright scroll + DOM parse | 1 HTTP GET per 100 recipes |
|
||||
| Detail fetch | urllib.requests + JSON-LD regex | 1 HTTP GET per recipe |
|
||||
| Avg per recipe | ~3–5s | ~0.2s |
|
||||
| Browser dependency | Yes (chromium binary) | No |
|
||||
| Headless issues | Cookie banners, JS timing | None |
|
||||
| Max recipes reachable | ~25k (search exhaustion) | ~384k (full catalog) |
|
||||
| Bot detection risk | Moderate (requires stealth) | Very low (same API as site frontend) |
|
||||
|
||||
## Migration from Playwright to API
|
||||
|
||||
```python
|
||||
# old (v5.1)
|
||||
urls = await _fetch_links(page, seed_url) # Playwright
|
||||
recipe = await _scrape_one(page, url) # requests + JSON-LD
|
||||
|
||||
# new (v5.2)
|
||||
recipes_meta = _fetch_recipe_list(offset, limit=100) # API listing
|
||||
recipe = _fetch_recipe_detail(meta["id"]) # API detail
|
||||
```
|
||||
|
||||
## Listing API: Offset Ceiling Reality
|
||||
|
||||
The unfiltered listing endpoint (`/v2/recipes?offset=N&limit=100`) appears to return **the same global list** regardless of any `sort=`, `order=`, `with=`, or `difficulty=` parameters tested (2026-06-22). The only way to get diversity is to advance `offset`.
|
||||
|
||||
**CRITICAL:** Offset has a practical ceiling. In testing:
|
||||
- `offset=0` returns 100 results ✅
|
||||
- `offset=2000` returns 0 results ❌ (server silently drops)
|
||||
|
||||
This means the unfiltered listing CANNOT be used to reach the full 384k catalog by offset alone. For deep scraping, use the **search query strategy** below.
|
||||
|
||||
## Search Queries via `?query=`
|
||||
|
||||
The endpoint supports text search with real filtering:
|
||||
|
||||
```
|
||||
GET /v2/recipes?query={term}&offset=0&limit=100
|
||||
```
|
||||
|
||||
Verified search queries (with approximate counts as of 2026-06):
|
||||
|
||||
| Query | Count | Usable offsets |
|
||||
|-------|-------|----------------|
|
||||
| Hauptspeise | ~158,000 | 0–~1000 |
|
||||
| Kuchen | ~58,000 | 0–~1000 |
|
||||
| Vegan | ~28,000 | 0–~1000 |
|
||||
| Italienisch | ~7,000 | 0–~1000 |
|
||||
| (blank) | ~384,000 | 0–~2000 (barely) |
|
||||
|
||||
**Search offset ceiling:** ~1000 results (10 pages of 100). Beyond that, the search returns empty.
|
||||
|
||||
## Deep Scraper Strategy for 25k+ Recipes (v5.2)
|
||||
|
||||
Because no single query reaches beyond ~1000–2000 offsets, the scraper must **rotate queries**:
|
||||
|
||||
1. Maintain a list of 50+ search terms (categories, ingredients, cuisines)
|
||||
2. Each run: pick a rotation window (e.g., 20 terms)
|
||||
3. For each term: random offset between 0 and ~100, fetch listing
|
||||
4. Filter known IDs, fetch details in parallel
|
||||
5. Rotate to next term
|
||||
|
||||
This prevents seed exhaustion while staying inside each query's usable window.
|
||||
|
||||
```python
|
||||
QUERIES = [
|
||||
"Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
|
||||
"Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
|
||||
"Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
|
||||
"Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
|
||||
"Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
|
||||
"Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
|
||||
"Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
|
||||
"Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
|
||||
]
|
||||
|
||||
def _fetch_list(query, offset, limit=100):
|
||||
url = f"https://api.chefkoch.de/v2/recipes?query={query}&offset={offset}&limit={limit}"
|
||||
...
|
||||
```
|
||||
|
||||
## Performance observed (2026-06-22):
|
||||
|
||||
**Initial projection under-estimated actual throughput significantly.**
|
||||
|
||||
With 12 workers: ~10–15 recipes/sec (throughput ceiling is detail-fetch latency, not listing)
|
||||
With 16 workers: ~800–900 recipes/minute (~13–15 recipes/sec)
|
||||
With 32 workers: ~750 recipes/minute sustained (diminishing returns from context switching)
|
||||
|
||||
**Key insight:** The listing fetch is negligible latency (~100ms). Detail fetch dominates (~200ms per recipe at 32 concurrent). 16 concurrent detail fetches saturates the network + server response pipe. More workers add no value.
|
||||
|
||||
**Recommended configuration:**
|
||||
- 16 ThreadPool workers (sweet spot)
|
||||
- 0.3s delay between listing batches
|
||||
- limit=100 per listing request
|
||||
- Each run rotates to next query when hitting the offset ceiling
|
||||
|
||||
## State Management for API Scraper
|
||||
|
||||
### State file format
|
||||
|
||||
```json
|
||||
{
|
||||
"query_index": 0,
|
||||
"api_offset": 0,
|
||||
"source_counts": {"chefkoch": 25020},
|
||||
"new_this_run": 0,
|
||||
"failed_listings": 0
|
||||
}
|
||||
```
|
||||
|
||||
**Field semantics:**
|
||||
- `query_index` — position in `QUERIES` list (rotates when exhausted)
|
||||
- `api_offset` — current offset within the active query's pagination window
|
||||
- `source_counts` — total recipes scraped per source
|
||||
- `new_this_run` — deduplicated count since last state write
|
||||
- `failed_listings` — consecutive failed listing requests (used for fast-exit)
|
||||
|
||||
### CRITICAL: Offset increment must be `+= limit`, not `+= 1`
|
||||
|
||||
The scraper fetches **batches of `limit` recipes per listing call**. State must advance `api_offset` by `limit` (e.g., 100), not by 1. Incrementing by 1 causes the same 100-recipe batch to be re-fetched on every subsequent run, producing **zero new recipes** while appearing to "run successfully".
|
||||
|
||||
```python
|
||||
# CORRECT — advances to next batch
|
||||
def main():
|
||||
state = _load_state()
|
||||
offset = state.get("api_offset", 0)
|
||||
limit = 100
|
||||
# ... fetch listing at offset ...
|
||||
state["api_offset"] = offset + limit # 0 → 100 → 200
|
||||
_save_state(state)
|
||||
|
||||
# WRONG — re-fetches same batch every time
|
||||
def main():
|
||||
state = _load_state()
|
||||
offset = state.get("api_offset", 0)
|
||||
# ... fetch listing at offset ...
|
||||
state["api_offset"] = offset + 1 # 0 → 1; next run: fetches offset 1 (same as offset 0 in effect)
|
||||
```
|
||||
|
||||
**The same batch always returns the same 100 recipe IDs.** Deduplication drops everything, `new_this_run` stays at 0, but the state increments by 1 each run, crawling nowhere at ~1 recipe offset per launch.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **Do not use** `api.chefkoch.de/v2/recipes/{id}/ingredients` — returns `resource_not_found` (404)
|
||||
2. **Ingredients live inside the detail response** under `ingredientGroups` — there is no separate endpoint
|
||||
3. **Tag list may contain empty strings** — filter with `.strip()` before storage
|
||||
4. **Count is approximate** — the `count` field drifts slightly; do not hardcode it as a loop bound
|
||||
5. **Offset-based pagination with a hard ceiling** — unfiltered listing exhausts at ~2000; search queries at ~1000
|
||||
6. **Query parameter filtering is NOT supported on the listing endpoint** — `sort=`, `difficulty=`, etc. are ignored
|
||||
7. **Do not cache the `count` field as a loop bound** — use query rotation instead
|
||||
|
||||
## Headers Required
|
||||
|
||||
Minimal headers suffice; full mimicry not required:
|
||||
|
||||
```python
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json",
|
||||
"Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
|
||||
"Referer": "https://www.chefkoch.de/",
|
||||
}
|
||||
```
|
||||
+178
@@ -0,0 +1,178 @@
|
||||
# Chefkoch.de: Hybrid Scraper (Playwright + Requests + JSON-LD)
|
||||
|
||||
Chefkoch.de (380k+ Rezepte) blockiert **direkte `requests` auf Suchseiten** (liefert JS-Shell statt Inhalt), **aber** einzelne Rezept-Seiten können über **Requests + JSON-LD** geparst werden. Die optimale Architektur ist ein **Zwei-Phasen-Ansatz**.
|
||||
|
||||
## Warum der alte Ansatz scheitert
|
||||
|
||||
**Falsch:** `recipe_scrapers.scrape_html(page.content(), url)` auf Chefkoch-Suchseiten — Chefkoch erkennt nicht-JS-Clients und liefert leere DOM-Shells.
|
||||
|
||||
**Falsch:** Playwright pro Rezept — zu langsam (5s+ pro Rezept) und überfordert bei 100+ Rezepten.
|
||||
|
||||
## Korrekte Architektur: Zwei Phasen
|
||||
|
||||
### Phase 1: Link-Sammlung via Playwright (Browser nötig)
|
||||
|
||||
Nur für die **Suchseite** — Chefkoch rendernt Ergebnisse client-side.
|
||||
|
||||
```python
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
async with async_playwright() as p:
|
||||
browser = await p.chromium.launch(headless=True)
|
||||
page = await browser.new_page(
|
||||
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
)
|
||||
await page.goto("https://www.chefkoch.de/rs/s0/{query}/Rezepte.html",
|
||||
wait_until="networkidle", timeout=30000)
|
||||
|
||||
# Scroll to load more results
|
||||
for _ in range(15):
|
||||
links = await page.query_selector_all("a[href*='rezepte/']")
|
||||
for el in links:
|
||||
href = await el.get_attribute("href")
|
||||
if href and "/rezepte/" in href:
|
||||
if not any(skip in href for skip in ["was-koche", "was-backe", "alle-rezepte", "/rs/s"]):
|
||||
recipe_links.add(href.split("?")[0])
|
||||
await page.evaluate("window.scrollBy(0, 500)")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
await browser.close()
|
||||
```
|
||||
|
||||
**Pitfall:** Keine `.recipe-list` oder `.bi-recipe-item` Selektoren verwenden — diese Klassenamen ändern sich. Stattdessen: `a[href*='rezepte/']` mit URL-Pattern-Filtern.
|
||||
|
||||
### Phase 2: Rezept-Daten via Requests + JSON-LD (kein Browser)
|
||||
|
||||
**Schnell** (~300ms statt 5s pro Rezept) und **zuverlässiger** als DOM-Scraping.
|
||||
|
||||
```python
|
||||
import urllib.request, re, json
|
||||
|
||||
def fetch_recipe_jsonld(url: str) -> dict:
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
# Extract JSON-LD blocks
|
||||
blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
|
||||
|
||||
for block in blocks:
|
||||
try:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return item
|
||||
elif data.get("@type") == "Recipe":
|
||||
return data
|
||||
except:
|
||||
continue
|
||||
return None
|
||||
|
||||
# Usage
|
||||
recipe = fetch_recipe_jsonld("https://www.chefkoch.de/rezepte/...")
|
||||
name = recipe["name"]
|
||||
ingredients = recipe.get("recipeIngredient", []) # List of strings like "4 EL Sojasauce"
|
||||
prep_time = recipe.get("prepTime", "") # ISO 8601: "PT25M"
|
||||
rating = recipe.get("aggregateRating", {}).get("ratingValue", "n/a")
|
||||
```
|
||||
|
||||
## Wichtige JSON-LD Felder
|
||||
|
||||
| Feld | Typ | Beispiel |
|
||||
|------|-----|----------|
|
||||
| `name` | str | `"Chicken Teriyaki - der japanische Klassiker"` |
|
||||
| `recipeIngredient` | list[str] | `["4 EL Sojasauce", "250 g Hähnchenbrust"]` |
|
||||
| `recipeInstructions` | list[dict/text] | `[{"@type": "HowToStep", "text": "..."}]` |
|
||||
| `prepTime` / `cookTime` / `totalTime` | str (ISO 8601) | `"PT25M"`, `"PT1H30M"` |
|
||||
| `aggregateRating.ratingValue` | float | `4.79` |
|
||||
| `aggregateRating.ratingCount` | int | `251` |
|
||||
|
||||
## Zutaten-String Parsen
|
||||
|
||||
`recipeIngredient` liefert rote Strings. Parser-Heuristik:
|
||||
|
||||
```python
|
||||
def parse_ingredient(s: str) -> dict:
|
||||
s = re.sub(r"[\t\n]+", " ", s.strip())
|
||||
parts = s.split(None, 2) # maxsplit=2
|
||||
|
||||
# Default
|
||||
amount, unit, name = 1.0, "Stk", s
|
||||
|
||||
# Try: <number> [unit] <name>
|
||||
m = re.match(r"([\d.,/]+(?:\s*-\s*[\d.,/]+)?)", parts[0])
|
||||
if m:
|
||||
num_str = m.group(1).replace(",", ".")
|
||||
# Parse number (handles fractions, ranges)
|
||||
if "-" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("-")]
|
||||
amount = (a + b) / 2
|
||||
elif "/" in num_str:
|
||||
a, b = [float(x.strip()) for x in num_str.split("/")]
|
||||
amount = a / b
|
||||
else:
|
||||
amount = float(num_str.strip())
|
||||
|
||||
# Check if second part is known unit
|
||||
KNOWN_UNITS = {"el", "tl", "g", "kg", "ml", "l", "cl", "dl",
|
||||
"stk", "stück", "packung", "pck", "bund", "zweig",
|
||||
"dose", "glas", "tasse", "prise", "scheibe"}
|
||||
if len(parts) >= 2 and parts[1].lower().rstrip(".") in KNOWN_UNITS:
|
||||
unit = parts[1]
|
||||
name = parts[2] if len(parts) > 2 else ""
|
||||
else:
|
||||
name = " ".join(parts[1:])
|
||||
|
||||
return {"item": name.strip(), "amount": amount, "unit": unit}
|
||||
```
|
||||
|
||||
**Pitfall:** regex mit `re.match(r"...", first)` und `split(None, 2)` statt komplexem Gesamt-regex — vermeidet "unbalanced parenthesis" Fehler bei verschachtelten Klammern in Zutaten-Namen wie "Reiswein (Mirin oder Sake)".
|
||||
|
||||
## Playwright Install
|
||||
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```python
|
||||
python3 -c "from playwright.sync_api import sync_playwright; \
|
||||
p = sync_playwright().start(); \
|
||||
b = p.chromium.launch(); b.close(); p.stop(); \
|
||||
print('Chromium OK')"
|
||||
```
|
||||
|
||||
**Fehler ohne Installation:**
|
||||
```
|
||||
Error: Executable doesn't exist at ~/.cache/ms-playwright/chromium-1071/...
|
||||
Please run: playwright install
|
||||
```
|
||||
|
||||
**Im Hintergrund-Cronjob:** Immer `async_playwright` verwenden, nie `sync_playwright` — blockiert den Event Loop.
|
||||
|
||||
## Performance-Vergleich
|
||||
|
||||
| Ansatz | Zeit pro Rezept | 100 Rezepte |
|
||||
|--------|----------------|-------------|
|
||||
| Playwright pro Rezept | 5–8s | 8–13 Min |
|
||||
| **Hybrid (recommended)** | **0.3s (HTTP) + 3s Suchseite** | **~3 Min** |
|
||||
|
||||
## Dynamische Queries aus Interview-Daten
|
||||
|
||||
| Interview-Input | Generierte Query |
|
||||
|-----------------|-----------------|
|
||||
| 27°C + asiatisch | "asiatisch leicht huhn" |
|
||||
| Sarah + low-carb | "low carb spargel fisch" |
|
||||
| Regen + Familie | "einfach familie warm" |
|
||||
|
||||
## Familien-Fit-Score
|
||||
|
||||
- Asiatisch → +2.0 (Dominik)
|
||||
- Low-Carb/Salat/Fisch → +1.5 (Sarah)
|
||||
- Nudeln/Huhn/Eier → +1.0 (Cleo)
|
||||
- Bewertung >4.5 → +ratingValue × 0.5
|
||||
- Gemüse-Vielfalt → +0.3 pro Art
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
# Chefkoch Scraper Evolution: v4 Playwright → v5.2 API
|
||||
|
||||
Session learning from 2026-06-22: migrating the Chefkoch scraper from Playwright-based discovery to internal API discovery. This is a **10× speed increase** and removes all headless-browser fragility.
|
||||
|
||||
## Why v4 Stalled at ~25k
|
||||
|
||||
| Problem | Symptom | Root cause |
|
||||
|---------|---------|-----------|
|
||||
| Search exhaustion | Same seeds return same recipes | 2-letter alphabet combos cover finite space |
|
||||
| Bot detection | Empty search pages, no links | Chefkoch serves JS shell to non-browser clients |
|
||||
| Playwright fragility | `Executable doesn't exist`, EPIPE | Browser binary mismatches, env propagation issues |
|
||||
| Seed cycle reset | `seed_cycle` stuck at 0 | Missing `global SEED_CYCLE` in `main()` |
|
||||
| Cookie walls | Content blocked | GDPR consent banners block link extraction |
|
||||
| Rate ~0.4/s | 100 recipes takes ~4 min | Browser overhead per search page |
|
||||
|
||||
The fundamental issue: **Chefkoch search pages require a real browser**, and the browser-based discovery approach has natural throughput limits and maintenance overhead.
|
||||
|
||||
## The API Discovery Path (v5.2)
|
||||
|
||||
### How we found it
|
||||
|
||||
During debugging, a test request to `api.chefkoch.de/v2/recipes?offset=0&limit=30` returned structured JSON with **384,252 recipes** listed. The endpoint is the same one Chefkoch's own frontend uses — no auth, no rate limiting observed.
|
||||
|
||||
### Architecture comparison
|
||||
|
||||
| Aspect | v4/v5.1 (Playwright) | v5.2 (API) |
|
||||
|--------|----------------------|------------|
|
||||
| Discovery | Browser-rendered search pages | HTTP listing API (100 recipes/page) |
|
||||
| Detail fetch | JSON-LD regex from HTML | Structured JSON from API |
|
||||
| Browser needed | Yes (Chromium binary) | No |
|
||||
| Max reachable | ~25k (search exhaustion) | ~384k (full catalog) |
|
||||
| Ingredients | Raw strings from JSON-LD | Objects with amount/unit/name |
|
||||
| Avg speed | ~0.4 recipes/sec | ~5 recipes/sec |
|
||||
| Headless issues | Yes (cookies, JS timing, EPIPE) | None |
|
||||
| Maintenance | Browser updates, path management | Minimal (HTTP only) |
|
||||
|
||||
## v5.2 Architecture (query-rotation based)
|
||||
|
||||
Because the listing API has an offset ceiling (~2000 for unfiltered, ~1000 for search queries), v5.2 uses **query rotation** instead of simple offset pagination:
|
||||
|
||||
```python
|
||||
QUERIES = [
|
||||
"Hauptspeise", "Vorspeise", "Nachtisch", "Beilage", "Fruehstueck",
|
||||
"Backen", "Grillen", "Salat", "Suppe", "Eintopf", "Pasta", "Dessert",
|
||||
"Kuchen", "Brot", "Getraenk", "Fingerfood", "Curry", "Wok", "Auflauf",
|
||||
"Gratin", "Risotto", "Lasagne", "Lachs", "Haehnchen", "Rind", "Schwein",
|
||||
"Fisch", "Gemuese", "Kartoffeln", "Reis", "Linsen", "Vegan", "Vegetarisch",
|
||||
"Low-Carb", "Schnell", "Einfach", "Italienisch", "Asiatisch", "Tacos",
|
||||
"Bowl", "Wrap", "Nudeln", "Haehnchenbrust", "Hackfleisch", "Burger",
|
||||
"Pizza", "Pfannkuchen", "Quiche", "Muffins", "Smoothie",
|
||||
]
|
||||
|
||||
def _fetch_recipe_list(query, offset, limit=100):
|
||||
# Uses ?query=... not bare offset
|
||||
url = f"{API_BASE}?query={query}&offset={offset}&limit={limit}"
|
||||
...
|
||||
```
|
||||
|
||||
State tracks `query_cycle` (position in QUERIES list) not just `api_offset`.
|
||||
|
||||
## Migration notes
|
||||
|
||||
**State field change:** v4/v5.1 uses `seed_cycle` (integer, position in infinite sequence). v5.2 uses `api_offset` (integer, offset in API listing). The state file can carry both:
|
||||
|
||||
```json
|
||||
{
|
||||
"seed_cycle": 9050,
|
||||
"api_offset": 25000,
|
||||
"source_counts": {"chefkoch": 25000, "hellofresh": 0},
|
||||
"migrated_from_v4": true
|
||||
}
|
||||
```
|
||||
|
||||
**Existing recipe IDs:** v4 IDs were `ck_347aaae514e25800` (16-char hex from URL). v5.2 IDs are `ck_4073161636112189` (numeric string from API). Both coexist in `real_recipes.jsonl` without conflict since dedup is per-ID.
|
||||
|
||||
**When to migrate from v4/v5.1 to v5.2:**
|
||||
- Scraper is stuck at same count for multiple runs → migrate immediately
|
||||
- Target > 25,000 → v5.2 is the only viable path (search exhaustion)
|
||||
- Playwright/browser issues recurring → v5.2 eliminates browser dependency
|
||||
- Building from scratch → start with v5.2
|
||||
|
||||
## Performance benchmark (observed)
|
||||
|
||||
| Metric | v5.1 (Playwright) | v5.2 (API) |
|
||||
|--------|-------------------|------------|
|
||||
| Recipes/hour | ~1,500 | ~5,000–9,000 |
|
||||
| Recipes/min | ~25 | ~80–150 |
|
||||
| Failure rate | ~5-10% (timeouts, empty pages) | ~0.1% |
|
||||
| Memory footprint | High (Chromium process) | Low (pure Python + urllib) |
|
||||
| CPU usage | High (browser rendering) | Low (JSON parse only) |
|
||||
| Max reachable | ~25k (search exhaustion) | ~50k+ (query rotation) |
|
||||
|
||||
## Remaining work for 50k target
|
||||
|
||||
At 25,020 recipes (current), v5.2 needs to fetch ~25,000 more. At ~800 recipes/minute with 16 workers:
|
||||
- **Time estimate:** ~35 minutes of continuous runtime
|
||||
- **API call count:** ~250 listing fetches + 25,000 detail fetches
|
||||
- **Query rotation:** ~10 unique queries required (each yields ~2,500 unique IDs)
|
||||
- **File size estimate:** ~25 MB of JSONL (assuming ~1 KB per recipe)
|
||||
|
||||
**Note on offsets:** Do NOT attempt to reach offset 25,000 on a single query. The unfiltered listing exhausts at ~2,000 and search queries at ~1,000. Use query rotation to distribute the target across many small offset windows instead.
|
||||
|
||||
**v5.3 refinements learned during deep scraping:**
|
||||
- `api_offset` must advance by `limit` (e.g. 100) per batch, not by 1. Advancing by 1 re-fetches the same 100-recipe window with a 1-recipe offset shift, producing zero new recipes while appearing to "progress"
|
||||
- 16 ThreadPool workers is the practical maximum; 32 workers showed no throughput improvement (~750 vs ~850 recipes/min) due to server-side latency becoming the dominant factor
|
||||
|
||||
## Pitfalls learned during migration
|
||||
|
||||
1. **Do not try `api.chefkoch.de/v2/recipes/{id}/ingredients`** — returns `resource_not_found`. Ingredients are inside the detail response under `ingredientGroups`.
|
||||
2. **Tag filtering required** — API returns empty strings in `tags` array; always `strip()` and filter.
|
||||
3. **Offset, not page** — The API uses `offset`, not `page`. Paginate with `offset += limit`, not `page += 1`.
|
||||
4. **Count is approximate** — `count` field drifts; never use it as a hard loop bound.
|
||||
5. **No `with=` query parameter works** — `with=ingredients,steps` is ignored; always fetch detail endpoint.
|
||||
6. **Concurrent detail fetches** — ThreadPoolExecutor with 16 workers gives good throughput without hitting any observed limits.
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
# HelloFresh.de — Recipe Scraping via SSR Hydration
|
||||
|
||||
HelloFresh.de serves recipe content as Next.js SSR pages with recipe data embedded in `__NEXT_DATA__` hydration scripts. No public API is exposed; extraction requires HTML parsing of the dehydrated React Query state.
|
||||
|
||||
## Architecture
|
||||
|
||||
Next.js + TanStack Query. Recipe data fetched server-side and serialized into `__NEXT_DATA__.props.pageProps.ssrPayload.dehydratedState.queries[]`.
|
||||
|
||||
### Key query keys
|
||||
|
||||
| Query key | Data |
|
||||
|-----------|------|
|
||||
| `['recipe.search', {...}]` | Recipe list for category page |
|
||||
| `['foodContentHubRecipe.byId', {'id': '...'}]` | Full recipe detail |
|
||||
|
||||
## Category Page (Discovery)
|
||||
|
||||
```python
|
||||
import urllib.request, json, re
|
||||
|
||||
def hf_fetch_category(category: str, skip=0, take=50) -> tuple[list[dict], int]:
|
||||
url = f"https://www.hellofresh.de/recipes/{category}?skip={skip}&take={take}"
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0...", "Accept": "text/html...",
|
||||
"Accept-Language": "de-DE,de;q=0.9", "Referer": "https://www.hellofresh.de/",
|
||||
}
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
|
||||
data = json.loads(match.group(1))
|
||||
ssr = data['props']['pageProps']['ssrPayload']
|
||||
for q in ssr.get('dehydratedState', {}).get('queries', []):
|
||||
key = q.get('queryKey', [])
|
||||
if len(key) > 0 and key[0] == 'recipe.search':
|
||||
d = q['state']['data']
|
||||
return d.get('items', []), d.get('total', 0)
|
||||
return [], 0
|
||||
```
|
||||
|
||||
**Category slugs** (62 total): amerikanische-rezepte, asiatische-rezepte, beliebteste-rezepte, burger, chinesische-rezepte, deutsche-rezepte, einfache-rezepte, eintopf-rezepte, familien-rezepte, flammkuchen-rezepte, fleisch-rezepte, franzosische-rezepte, fusions-rezepte, gesunde-rezepte, glutenfreie-rezepte, griechische-rezepte, grillrezepte, indische-rezepte, italienische-rezepte, japanische-rezepte, kalorienreduzierte-rezepte, karibische-rezepte, koreanische-rezepte, low-carb-rezepte, marokkanische-rezepte, mediterrane-rezepte, meeresfruechte-rezepte, mexikanische-rezepte, orientalische-rezepte, pasta-rezepte, pizza, proteinreiche-rezepte, rezepte-naher-osten, salat-rezepte, sandwich-rezepte, schnelle-gerichte, schnitzel, skandinavische-rezepte, spanische-rezepte, suppen-rezepte, thailandische-rezepte, tofu, tuerkische-rezepte, vegane-rezepte, vegetarische-rezepte, vietnamesische-rezepte, weihnachtsessen, zitrone, etc.
|
||||
|
||||
## Recipe Detail
|
||||
|
||||
```python
|
||||
def hf_fetch_detail(recipe_id: str, slug: str) -> dict | None:
|
||||
url = f"https://www.hellofresh.de/recipes/{slug}-{recipe_id}"
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
|
||||
data = json.loads(match.group(1))
|
||||
ssr = data['props']['pageProps']['ssrPayload']
|
||||
for q in ssr.get('dehydratedState', {}).get('queries', []):
|
||||
key = q.get('queryKey', [])
|
||||
if len(key) > 0 and key[0] == 'foodContentHubRecipe.byId':
|
||||
return q['state']['data'].get('recipe')
|
||||
return None
|
||||
```
|
||||
|
||||
## Normalization
|
||||
|
||||
| Field | Source | Notes |
|
||||
|-------|--------|-------|
|
||||
| id | `recipe.id` | Strip `-de-DE` suffix |
|
||||
| title | `recipe.name` | |
|
||||
| description | `recipe.description` or `recipe.headline` | |
|
||||
| ingredients | `yields[0].ingredients` + `ingredients` lookup | Two parallel arrays |
|
||||
| instructions | `steps[].instructionsMarkdown` -> strip HTML | |
|
||||
| prep_time | `recipe.prepTime` ISO 8601 | Parse PT30M -> 30 |
|
||||
| servings | `yields[0].yields` | Default 2 |
|
||||
| calories | `nutrition[]` where name contains "kcal" | |
|
||||
| tags | `recipe.tags[].name` | Skip "SEO" |
|
||||
| category | `recipe.cuisines` | May be str or dict with `name` |
|
||||
|
||||
### Ingredient assembly
|
||||
|
||||
```python
|
||||
def assemble_ingredients(recipe):
|
||||
ing_lookup = {ing['id']: ing['name'] for ing in recipe.get('ingredients', [])}
|
||||
ingredients = []
|
||||
for ying in recipe.get('yields', [{}])[0].get('ingredients', []):
|
||||
name = ing_lookup.get(ying.get('id', ''), '')
|
||||
amount = ying.get('amount', 0)
|
||||
unit = ying.get('unit', '')
|
||||
if name and amount is not None:
|
||||
if unit and unit != 'nach Geschmack':
|
||||
ingredients.append(f"{amount} {unit} {name}")
|
||||
elif unit == 'nach Geschmack':
|
||||
ingredients.append(f"{name} (nach Geschmack)")
|
||||
else:
|
||||
ingredients.append(f"{amount} {name}")
|
||||
elif name:
|
||||
ingredients.append(name)
|
||||
return ingredients
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- Category page: ~0.5-1.0s
|
||||
- Detail page: ~0.5-1.5s
|
||||
- Throughput: ~10-20 recipes/min with 8 workers
|
||||
- Total public recipes: ~5,000-10,000
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **No public API** - HTML hydration only; may break on Next.js upgrade
|
||||
2. **`cuisines` polymorphic** - `list[str]` or `list[dict]`; check `isinstance`
|
||||
3. **Amount of 0** means "nach Geschmank" or optional
|
||||
4. **Deduplicate by id** - recipes appear across categories
|
||||
5. **Keep polite** - 8 workers max, 0.5s between batches
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
# Infinite Refill Scraper v4 (Pattern B)
|
||||
|
||||
## When to Use
|
||||
|
||||
- Target is >10,000 recipes (batch seeds run out)
|
||||
- Continuous background operation without daily cronjob intervention
|
||||
- Resilient against search exhaustion (alphabet/categories eventually cover everything)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Infinite Seed Generator URL Discovery (Playwright) Recipe Extraction (Requests)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
[Alphabet combos] ──► Search pages ──► new URLs ──► JSON-LD parse ──► real_recipes.jsonl
|
||||
[Categories+rand] ──► (5 workers) ──► (queue) ──► (fast, no JS) ──► append-only
|
||||
[Random 3-letter] ──► ──► dedup ──► ──► crash-safe
|
||||
```
|
||||
|
||||
## Seed Generator (Python)
|
||||
|
||||
Cycles through 4 strategies infinitely, never running out:
|
||||
|
||||
```python
|
||||
ALPHABET = "abcdefghijklmnopqrstuvwxyzäöü"
|
||||
CATEGORIES = [
|
||||
"Rezepte", "Hauptspeise", "Vorspeise", "Nachtisch", "Beilage",
|
||||
"Fruehstueck", "Backen", "Grillen", "Salat", "Suppe", "Eintopf",
|
||||
"Pasta", "Dessert", "Kuchen", "Brot", "Getraenk", "Fingerfood",
|
||||
]
|
||||
|
||||
def _next_seeds(batch_size=200, cycle_counter):
|
||||
seeds = []
|
||||
while len(seeds) < batch_size:
|
||||
cycle = cycle_counter % 4
|
||||
if cycle == 0:
|
||||
# Alphabetic 2-letter combos (e.g. "ab", "ac" ...):
|
||||
a = ALPHABET[cycle_counter % len(ALPHABET)]
|
||||
b = ALPHABET[(cycle_counter // len(ALPHABET)) % len(ALPHABET)]
|
||||
term = a + b
|
||||
elif cycle == 1:
|
||||
# Category + random letter (e.g. "Pasta+a")
|
||||
cat = CATEGORIES[cycle_counter % len(CATEGORIES)]
|
||||
letter = random.choice(ALPHABET)
|
||||
term = cat + "+" + letter
|
||||
elif cycle == 2:
|
||||
# Just category (e.g. "Pasta")
|
||||
term = CATEGORIES[cycle_counter % len(CATEGORIES)]
|
||||
else:
|
||||
# Random 3-letter nonsense (e.g. "xyz")
|
||||
term = "".join(random.choices(ALPHABET, k=3))
|
||||
seeds.append(f"https://www.chefkoch.de/rs/s0/{term}/Rezepte.html")
|
||||
cycle_counter += 1
|
||||
return seeds, cycle_counter
|
||||
```
|
||||
|
||||
**Coverage estimate:** 29 letters × 29 letters = 841 combos for 2-letter alone. Plus 17 categories × 29 = 493. Plus pure category = 17. Plus random 3-letter = infinite. Total: effectively unlimited.
|
||||
|
||||
## Two-Phase Extraction
|
||||
|
||||
### Phase 1: Link Discovery
|
||||
|
||||
```python
|
||||
async def _fetch_links(page, url, timeout=30):
|
||||
await page.goto(url, wait_until="domcontentloaded", timeout=timeout*1000)
|
||||
await asyncio.sleep(1.5) # Let JS render
|
||||
links = await page.eval_on_selector_all(
|
||||
"a[href*='/rezepte/']",
|
||||
"els => els.map(e => e.href)"
|
||||
)
|
||||
# Deduplicate and filter
|
||||
seen = set()
|
||||
result = []
|
||||
for l in links:
|
||||
if "chefkoch.de/rezepte/" in l and l not in seen:
|
||||
seen.add(l)
|
||||
result.append(l)
|
||||
return result
|
||||
```
|
||||
|
||||
### Phase 2: Recipe Extraction (NO Playwright)
|
||||
|
||||
```python
|
||||
import urllib.request, re, json
|
||||
|
||||
async def _scrape_one(url, timeout=15):
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "de-DE,de;q=0.9,en;q=0.8",
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
blocks = re.findall(
|
||||
r'<script type="application/ld\+json">(.*?)</script>',
|
||||
html, re.DOTALL | re.IGNORECASE
|
||||
)
|
||||
for block in blocks:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict) and item.get("@type") == "Recipe":
|
||||
return _normalize_recipe(item, url)
|
||||
elif isinstance(data, dict) and data.get("@type") == "Recipe":
|
||||
return _normalize_recipe(data, url)
|
||||
# Fallback: look for mainEntity
|
||||
for block in blocks:
|
||||
data = json.loads(block.strip())
|
||||
if isinstance(data, dict) and "mainEntity" in data:
|
||||
me = data["mainEntity"]
|
||||
if isinstance(me, dict) and me.get("@type") == "Recipe":
|
||||
return _normalize_recipe(me, url)
|
||||
return None
|
||||
```
|
||||
|
||||
**Why not use Playwright for recipe pages?** JSON-LD is embedded in raw HTML and extracts faster (no browser overhead). Playwright is only needed for search pages because Chefkoch serves a JS shell there.
|
||||
|
||||
## State Management
|
||||
|
||||
```python
|
||||
STATE = {
|
||||
"seed_cycle": 0, # Position in infinite sequence
|
||||
"started_at": "", # ISO timestamp
|
||||
"recipes_target": 25000
|
||||
}
|
||||
```
|
||||
|
||||
Save every batch. On restart, the script resumes from the last seed_cycle.
|
||||
|
||||
## Normalization Function
|
||||
|
||||
```python
|
||||
def _normalize_recipe(data, url):
|
||||
rid = (data.get("identifier", {}).get("value", "")
|
||||
or url.split("/")[-1].split("-")[0])
|
||||
title = data.get("name", "")
|
||||
if not title:
|
||||
return None
|
||||
|
||||
keywords = data.get("keywords", "")
|
||||
tags = [t.strip().lower() for t in keywords.split(",") if t.strip()]
|
||||
recipe_cat = data.get("recipeCategory", "")
|
||||
cuisine = data.get("recipeCuisine", "")
|
||||
if recipe_cat:
|
||||
tags.append(recipe_cat.lower())
|
||||
if cuisine:
|
||||
tags.append(cuisine.lower())
|
||||
tags = list(set(tags))
|
||||
|
||||
rating = data.get("aggregateRating", {})
|
||||
nutrition = data.get("nutrition", {})
|
||||
|
||||
return {
|
||||
"id": str(rid),
|
||||
"title": title,
|
||||
"description": data.get("description", ""),
|
||||
"source_url": url,
|
||||
"ingredients": data.get("recipeIngredient", []),
|
||||
"instructions": data.get("recipeInstructions", ""),
|
||||
"tags": tags,
|
||||
"category": recipe_cat or (tags[0] if tags else "Sonstiges"),
|
||||
"rating_value": rating.get("ratingValue"),
|
||||
"rating_count": rating.get("ratingCount"),
|
||||
"prep_time": data.get("prepTime", ""),
|
||||
"cook_time": data.get("cookTime", ""),
|
||||
"total_time": data.get("totalTime", ""),
|
||||
"calories": str(nutrition.get("calories", "")),
|
||||
"scraped_at": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
```
|
||||
|
||||
## Execution Pattern
|
||||
|
||||
### DO NOT use LLM-based cronjob
|
||||
|
||||
❌ **WRONG:** A cronjob that uses `hermes` LLM agent to run Python. Fails with:
|
||||
- `Error 400: model is required` (if no model configured)
|
||||
- `No Anthropic credentials` (if wrong provider configured)
|
||||
- Unnecessary cost: paying LLM tokens just to run `wc -l` and `python script.py`
|
||||
|
||||
✅ **CORRECT:** `terminal(background=True)` with `notify_on_complete`
|
||||
|
||||
```python
|
||||
# Start directly
|
||||
cd /profile/nutrition-coach && \
|
||||
source .venv/bin/activate && \
|
||||
python3 /path/to/chefkoch_scraper_v4.py
|
||||
|
||||
# In Hermes Agent:
|
||||
# terminal(background=True, notify_on_complete=True)
|
||||
```
|
||||
|
||||
### Why background process beats cron-checked scraper
|
||||
|
||||
| Aspect | Cronjob (daily check + start) | Background process |
|
||||
|--------|-------------------------------|-------------------|
|
||||
| Failure mode | LLM errors, credential mismatches | None (pure Python) |
|
||||
| Latency | Up to 24h until next check | Continuous |
|
||||
| State tracking | External (cron state DB) | Internal (scraper_state.json) |
|
||||
| Backpressure | Manual | Automatic (sleep on queue overflow) |
|
||||
| Monitoring | Cron output files | Log tail + notify_on_complete |
|
||||
| Cost | LLM tokens per check | Zero |
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
| Configuration | Recipes/min | 5,000 recipes | 25,000 recipes |
|
||||
|--------------|-------------|---------------|----------------|
|
||||
| 5 workers (batch, fixed seeds) | 60–80 | ~60–90 min | N/A (seeds exhausted) |
|
||||
| Infinite v4 (single context) | 40–60 | ~80–120 min | ~7–10 hours |
|
||||
| Infinite v4 (5 contexts, remote) | 80–120 | ~40–60 min | ~4–6 hours |
|
||||
|
||||
**Note:** v4 uses a single Playwright page context for discovery (lighter than 5 full workers). The extraction is HTTP requests which are much faster.*
|
||||
|
||||
## Known v4 Failure Modes
|
||||
|
||||
If the scraper is stuck at the exact same recipe count for multiple cron runs (check `scraper.log` — repeated `START run | current: N/N` with identical `N`), one of these three root causes is almost certainly present.
|
||||
|
||||
### Cause 1: `SEED_CYCLE` not declared global in `main()`
|
||||
|
||||
**Symptom:** `seed_cycle` in state JSON is `0` even after dozens of runs. The same exhausted 2-letter seeds are visited forever.
|
||||
|
||||
**Root cause:** In Python, a bare assignment `SEED_CYCLE = state["seed_cycle"]` inside `main()` creates a *local* variable if `global SEED_CYCLE` is not declared. The module-level global stays at `0`, so `_next_seeds()` always starts from `0`.
|
||||
|
||||
**Fix:** Add in `main()`:
|
||||
```python
|
||||
state.setdefault("seed_cycle", 0)
|
||||
global SEED_CYCLE # ← required
|
||||
SEED_CYCLE = state["seed_cycle"]
|
||||
```
|
||||
|
||||
**Reproduce:** Open Python, import the module, inspect `chefkoch_scraper_v4.SEED_CYCLE` after `main()` runs — it remains `0` without the `global` line.
|
||||
|
||||
### Cause 2: Playwright EPIPE / `Executable doesn't exist`
|
||||
|
||||
**Symptom:** Log shows `Executable doesn't exist at ...chrome-linux/chrome` or Node `EPIPE` errors. The process exits immediately.
|
||||
|
||||
**Root cause:** Environment variable `PLAYWRIGHT_BROWSERS_PATH` is set in the caller's shell but not propagated to the background process spawned by `terminal(background=True)`. The agent's background launcher creates a new non-login bash that inherits `PATH` but not the caller's env overrides.
|
||||
|
||||
**Fix:** Hardcode the browser path inside the script itself:
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/.../.cache/ms-playwright")
|
||||
```
|
||||
|
||||
Do **not** rely on shell `export` when launching via `terminal(background=True)`.
|
||||
|
||||
### Cause 3: Stale `completed_seeds` blockers in state
|
||||
|
||||
**Symptom:** Scraper runs but repeatedly visits the same search pages with no new URLs, or many search pages return zero links.
|
||||
|
||||
**Root cause:** `scraper_state.json` contains a `completed_seeds` dict that prevents revisiting certain terms. If these terms were exhausted naturally, the seeds never advance.
|
||||
|
||||
**Fix:** Wipe or prune the dict before a fresh run:
|
||||
```python
|
||||
import json
|
||||
s = json.load(open("scraper_state.json"))
|
||||
s.pop("completed_seeds", None)
|
||||
s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
|
||||
json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False)
|
||||
```
|
||||
|
||||
### Quick diagnostic flow
|
||||
|
||||
```bash
|
||||
# 1. Check recipe count isn't moving
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
|
||||
# 2. Check state
|
||||
python3 -c "import json; s=json.load(open('scraper_state.json')); print('seed_cycle:', s.get('seed_cycle')); print('completed_seeds keys:', list(s.get('completed_seeds',{}).keys())[:5])"
|
||||
|
||||
# 3. Check Playwright browser
|
||||
find /home/debian/.cache/ms-playwright -name chrome | head -1
|
||||
|
||||
# 4. Check if process is actually scraping (not just discovering)
|
||||
ps auxf | grep -E 'chefkoch|chrome' | grep -v grep | wc -l
|
||||
```
|
||||
|
||||
See `references/v4-known-issues-and-fixes.md` for the full patch snippets and state-reset script.
|
||||
|
||||
## File Locations (standard)
|
||||
|
||||
```
|
||||
~/.hermes/profiles/nutrition-coach/
|
||||
├── real_recipes.jsonl # Append-only output
|
||||
├── scraper_state.json # Resume state
|
||||
├── scraper.log # Human-readable log
|
||||
└── scripts/
|
||||
└── chefkoch_scraper_v4.py # Standalone script
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check progress
|
||||
tail -n 5 ~/.hermes/profiles/nutrition-coach/scraper.log
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
|
||||
# Check if running
|
||||
ps aux | grep chefkoch_scraper
|
||||
|
||||
# Restart if needed
|
||||
cd ~/.hermes/profiles/nutrition-coach && \
|
||||
source .venv/bin/activate && \
|
||||
nohup python3 scripts/chefkoch_scraper_v4.py > scraper.log 2>&1 &
|
||||
```
|
||||
|
||||
## Error Recovery
|
||||
|
||||
```
|
||||
Error streak >= 100 → Script aborts automatically
|
||||
→ Check Chefkoch availability: curl -I https://www.chefkoch.de
|
||||
→ Verify Playwright install: playwright install chromium
|
||||
→ Check disk space: df -h
|
||||
→ Restart with fresh state (or resume from last saved seed_cycle)
|
||||
```
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Browser path detection (common error)
|
||||
|
||||
If Playwright fails with `Executable doesn't exist at ...chrome-headless-shell`:
|
||||
|
||||
```bash
|
||||
# Find existing browser installations
|
||||
find /home/debian -name "chrome" -o -name "chromium" 2>/dev/null
|
||||
|
||||
# Usually found under /home/debian/.cache/ms-playwright/
|
||||
# Set the environment variable before starting:
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
```
|
||||
|
||||
If the browser is not installed at all:
|
||||
```bash
|
||||
cd /profile/nutrition-coach && source .venv/bin/activate
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Do NOT rely on `playwright install` in the crontab — install once, reference via env var.**
|
||||
|
||||
### nohup wrapper for daemon mode
|
||||
|
||||
Create `~/.hermes/profiles/nutrition-coach/scripts/start_scraper_v4.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
LOG="/home/debian/.hermes/profiles/nutrition-coach/scraper_v4.out"
|
||||
PIDFILE="/tmp/chefkoch_scraper_v4.pid"
|
||||
if [ -f "$PIDFILE" ] && kill -0 "$(cat $PIDFILE)" 2>/dev/null; then
|
||||
echo "Scraper already running (PID $(cat $PIDFILE))"; exit 0
|
||||
fi
|
||||
cd /home/debian/.hermes/profiles/nutrition-coach || exit 1
|
||||
export PYTHONUNBUFFERED=1
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
nohup python3 scripts/chefkoch_scraper_v4.py --no-agent > "$LOG" 2>&1 &
|
||||
echo $! > "$PIDFILE"
|
||||
echo "Scraper started (PID $!)"
|
||||
```
|
||||
|
||||
Then: `chmod +x scripts/start_scraper_v4.sh`
|
||||
|
||||
**Path note:** In this environment the script lives under `scripts/chefkoch_scraper_v4.py` inside the profile directory, not `~/.hermes/scripts/`. Always check `ls scripts/` before constructing the invocation path.
|
||||
|
||||
### Watchdog cronjob (no_agent)
|
||||
|
||||
```bash
|
||||
cronjob action=create \
|
||||
name=chefkoch-scraper-watchdog \
|
||||
script=start_scraper_v4.sh \
|
||||
schedule="*/10 * * * *" \
|
||||
no_agent=true \
|
||||
deliver=local
|
||||
```
|
||||
|
||||
This restarts the scraper if it crashes, without involving any LLM.
|
||||
|
||||
### DO NOT use `terminal(background=true)` with `tee` or complex pipelines
|
||||
|
||||
❌ `terminal(background=true, command="... | tee log.out")` — Output buffering causes `tee` to swallow data, and the `terminal` pseudo-tty layer conflicts with Playwright's subprocess management. Result: log file remains empty or stale, process output invisible.
|
||||
|
||||
✅ Start via explicit file redirect with `PYTHONUNBUFFERED=1`:
|
||||
|
||||
```python
|
||||
# In Hermes Agent — background process with file redirect
|
||||
terminal(
|
||||
background=True,
|
||||
command="cd /profile && PYTHONUNBUFFERED=1 python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1"
|
||||
)
|
||||
```
|
||||
|
||||
Or use the `start_scraper_v4.sh` wrapper script shown above. Always verify progress with `wc -l real_recipes.jsonl` and `tail scraper_v4.out` rather than relying solely on process output capture.
|
||||
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
# Mass Scraping: Chefkoch-Harvesting (>1000 Rezepte)
|
||||
|
||||
## Überblick
|
||||
|
||||
Bei Zielen >1000 Rezepten reicht der Single-URL-Fall-back-Modus nicht aus. Chefkoch bietet
|
||||
380.000+ Rezepte; die verlässliche Technik ist ein **Two-Phase Discovery + Harvest**
|
||||
mit Playwright-Multi-Worker-Architektur.
|
||||
|
||||
## Architektur-Blueprint
|
||||
|
||||
```
|
||||
Discovery-Phase Harvest-Phase
|
||||
──────────────────────── ──────────────────
|
||||
Search-Seite (Playwright) → Worker-Pool (5–10 Threads)
|
||||
└─ Scroll-Trigger → │- Browser-Context pro Worker
|
||||
└─ a[href*='rezepte/'] → │- JSON-LD/DOM-Fallback
|
||||
│ │- Append-Only JSONL
|
||||
▼ ↓
|
||||
[URL-Queue] real_recipes.jsonl
|
||||
│ scraper_state.json
|
||||
└─ Deduplication (URL-Hash-Set)
|
||||
```
|
||||
|
||||
## Phase 1: Link-Discovery
|
||||
|
||||
```python
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
def discover_urls(seed: str, page_num: int) -> list[str]:
|
||||
search_url = f"https://www.chefkoch.de/rs/s{page_num}/{seed}/Rezepte.html"
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 900})
|
||||
page.goto(search_url, wait_until="networkidle", timeout=25000)
|
||||
# Lazy-load trigger
|
||||
for _ in range(4):
|
||||
page.evaluate("window.scrollBy(0, 1000)")
|
||||
time.sleep(0.4)
|
||||
links = page.evaluate('''() => {
|
||||
const cards = document.querySelectorAll('article[class*="recipe"], .ds-recipe-card');
|
||||
const found = [];
|
||||
for (const card of cards) {
|
||||
const a = card.querySelector('a[href*="/rezepte/"]');
|
||||
if (a && a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
}
|
||||
if (found.length === 0) {
|
||||
document.querySelectorAll('a[href*="/rezepte/"]').forEach(a => {
|
||||
if (a.href) found.push(a.href.split('#')[0].split('?')[0]);
|
||||
});
|
||||
}
|
||||
return [...new Set(found)];
|
||||
}''')
|
||||
browser.close()
|
||||
# Filter to real recipe URLs only
|
||||
return [u for u in links if re.search(r'/rezepte/\d+/', u)]
|
||||
```
|
||||
|
||||
### Seed-Strategie
|
||||
|
||||
30+ breite Suchbegriffe (einfach, asiatisch, hähnchen, dessert, ...) erzeugen
|
||||
~150 Suchseiten = Tausende eindeutige Rezept-URLs. Der Queue wird gemischt, um
|
||||
keine Quelle zu überlasten.
|
||||
|
||||
## Phase 2: Rezept-Harvest
|
||||
|
||||
### A) JSON-LD-Extraktion (primär)
|
||||
|
||||
```python
|
||||
def scrape_jsonld(page, url: str) -> dict | None:
|
||||
page.goto(url, wait_until="networkidle", timeout=20000)
|
||||
scripts = page.locator('script[type="application/ld+json"]').all()
|
||||
for script in scripts:
|
||||
try:
|
||||
data = json.loads(script.inner_text())
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if item.get("@type") == "Recipe":
|
||||
return _normalize(item, url)
|
||||
elif data.get("@type") == "Recipe":
|
||||
return _normalize(data, url)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
```
|
||||
|
||||
### B) DOM-Fallback (selektiv)
|
||||
|
||||
Wenn JSON-LD fehlt oder leer:
|
||||
|
||||
```python
|
||||
# Zutaten-Tabelle
|
||||
rows = page.locator("table.ingredients tbody tr").all()
|
||||
# Anleitung
|
||||
steps = page.locator("article p").all()
|
||||
```
|
||||
|
||||
## Speicherformat: JSON Lines (JSONL)
|
||||
|
||||
**Niemals ein großes JSON-Dump verwenden** — bei Absturz ist alles verloren.
|
||||
|
||||
```
|
||||
real_recipes.jsonl # append-only
|
||||
scraper_state.json # resume-Informationen
|
||||
scraper.log # menschenlesbarer Fortschritt
|
||||
```
|
||||
|
||||
### JSONL-Struktur pro Zeile
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "347aaae514e25800",
|
||||
"name": "Erdbeerkuchen - einfaches Rezept",
|
||||
"description": "...",
|
||||
"category": "Backen",
|
||||
"tags": ["Sommer", "Kuchen", "Frühling"],
|
||||
"difficulty": "einfach",
|
||||
"base_servings": "12 Port.",
|
||||
"prep_time": "P0DT0H40M",
|
||||
"cook_time": "P0DT0H25M",
|
||||
"total_time": "P0DT1H5M",
|
||||
"ingredients": [{"raw": "125 g Zucker"}, ...],
|
||||
"instructions": ["Schritt 1 ...", "Schritt 2 ..."],
|
||||
"nutrition": {},
|
||||
"rating": "4.77",
|
||||
"review_count": "724",
|
||||
"author": "Loeckchen87",
|
||||
"images": ["https://img.chefkoch-cdn.de/..."],
|
||||
"source_url": "https://www.chefkoch.de/rezepte/...",
|
||||
"sources": [{"site": "chefkoch.de", "url": "...", "type": "de"}],
|
||||
"_scraped_at": "2026-06-18T05:13:31"
|
||||
}
|
||||
```
|
||||
|
||||
## Worker-Pool & Concurrency
|
||||
|
||||
```python
|
||||
WORKERS = 5
|
||||
DELAY_RANGE = (1.0, 2.5) # pro Worker individuell
|
||||
```
|
||||
|
||||
**Niemals 20+ Worker** — Chefkoch merkt sich IPs und verzögert massiv. 5 Worker
|
||||
mit 1–2 s Delay liefern
|
||||
60–80 Rezepte/Minute, bei 5.000 Rezepten ≈ 60–90 Min.
|
||||
|
||||
## State-Management
|
||||
|
||||
```json
|
||||
{
|
||||
"started_at": "2026-06-18T05:13:20",
|
||||
"last_update": "2026-06-18T05:22:26",
|
||||
"recipes_count": 5000,
|
||||
"completed_seeds": {"asiatisch": 4, "pasta": 2, ...},
|
||||
"failed_urls": [],
|
||||
"worker_stats": {
|
||||
"0": {"scraped": 1020, "errors": 3},
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Patience & Monitoring
|
||||
|
||||
- **Starten**: Hintergrundprozess, kein Blocking der CLI
|
||||
- **Log**: `tail -f scraper.log`
|
||||
- **Zwischenspeichern**: Nach jeweils 10 Rezepten die DB flushen
|
||||
- **Auto-Refill-Cronjob**: Täglich 06:00 — prüft Count, startet nur wenn < Ziel
|
||||
|
||||
## Bekannte Fehler & Workarounds
|
||||
|
||||
| Fehler | Ursache | Workaround |
|
||||
|--------|---------|------------|
|
||||
| `links = 0` auf Search-Seite | Cookie-Banner blockiert Scroll | Klick auf Zustimmen vor Scroll |
|
||||
| `JSONDecodeError` bei JSON-LD | Verunreinigte Blocks mit HTML | `try/except` pro Block |
|
||||
| `TimeoutError` bei Recipe-Seite | Langsamer Server | Erhöhe auf 30 s, 1 Retry |
|
||||
| `Duplicate URLs` | `/rezepte/123/` vs `/rezepte/123/Name.html` | Immer `split('#')[0].split('?')[0]` |
|
||||
| Worker-Crash | Memory-Overflow bei Langzeitsession | Browser pro Worker alle 500 Seiten restarten |
|
||||
|
||||
## Ergebnis-Kosten pro 5000 Rezepte
|
||||
|
||||
| Metrik | Wert |
|
||||
|--------|------|
|
||||
| Scraping-Zeit | ~60–90 min |
|
||||
| JSONL-Dateigröße | ~30–50 MB |
|
||||
| Einzigartige Rezepte | ~95% der gescrapeten URLs (Deduplication) |
|
||||
| Fehlerrate | <1% (Timeouts / 404) |
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
# v4 Known Issues & Fixes
|
||||
|
||||
Session-record of bugs found and fixed during production Chefkoch scraping runs.
|
||||
This file is referenced from the main `infinite-refill-scraper-v4.md` reference.
|
||||
|
||||
## Environment: Context for these fixes
|
||||
|
||||
- Host: Debian container, Python 3.11
|
||||
- Playwright 1.60.0 installed in profile venv
|
||||
- Browser cache: `/home/debian/.cache/ms-playwright/chromium-1223/`
|
||||
- Background process launcher: Hermes Agent `terminal(background=True)`
|
||||
- Cron: `chefkoch-scraper-watchdog` every 10 minutes (`no_agent=true`)
|
||||
|
||||
---
|
||||
|
||||
## Fix 1 — Playwright browser path not inherited
|
||||
|
||||
### Failure signature
|
||||
|
||||
```
|
||||
playwright._impl._api_types.Error: Executable doesn't exist at
|
||||
/home/debian/.cache/ms-playwright/chromium-1071/chrome-linux/chrome
|
||||
```
|
||||
|
||||
Note: The error references a **different** chromium version (`-1071`) than what is actually installed (`-1223`). This mismatch happens because Playwright falls back to its own lookup logic when `PLAYWRIGHT_BROWSERS_PATH` is absent.
|
||||
|
||||
### Why it happens
|
||||
|
||||
Setting the env var in the Hermes Agent shell workspace does **not** propagate to the background bash launched by `terminal(background=True)`. The background process inherits `PATH` and `VIRTUAL_ENV` but not custom `export FOO=bar` values.
|
||||
|
||||
### Fix options (ranked)
|
||||
|
||||
1. **Best:** Hardcode inside `chefkoch_scraper_v4.py` before any Playwright import:
|
||||
```python
|
||||
import os
|
||||
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/home/debian/.cache/ms-playwright")
|
||||
```
|
||||
|
||||
2. **Acceptable:** Shell wrapper with `exec` and explicit `export`:
|
||||
```bash
|
||||
#! /bin/bash
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
exec python3 scripts/chefkoch_scraper_v4.py --no-agent >> scraper_v4.out 2>&1
|
||||
```
|
||||
|
||||
3. **Do not:** Rely on cron env or parent-shell export alone.
|
||||
|
||||
---
|
||||
|
||||
## Fix 2 — `global SEED_CYCLE` missing
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper runs every 10 minutes (cron restart) but recipe count never increases. Log shows:
|
||||
|
||||
```
|
||||
START run | current: 12736/25000
|
||||
```
|
||||
|
||||
State file `seed_cycle` is always `0`.
|
||||
|
||||
### Why it happens
|
||||
|
||||
Simplified pseudocode:
|
||||
|
||||
```python
|
||||
SEED_CYCLE = 0 # module-level global
|
||||
|
||||
def _next_seeds(batch_size=50):
|
||||
global SEED_CYCLE
|
||||
for i in range(batch_size):
|
||||
# ... compute seed based on SEED_CYCLE ...
|
||||
SEED_CYCLE += 1
|
||||
|
||||
async def main():
|
||||
state = _load_state()
|
||||
SEED_CYCLE = state["seed_cycle"] # ← BUG: creates a local variable in main()
|
||||
while ...:
|
||||
seeds = _next_seeds(50) # reads/writes the GLOBAL SEED_CYCLE
|
||||
# but the global is still 0 because main()'s assignment was local!
|
||||
```
|
||||
|
||||
Because `SEED_CYCLE` is assigned inside `main()` without `global SEED_CYCLE`, Python creates a local variable shadowing the module global. `_next_seeds()` then reads the global (still 0), increments it, but on the next cron restart `main()` sets the local back to whatever is in state (`0`). The seeds are the same every single run.
|
||||
|
||||
### Fix
|
||||
|
||||
```python
|
||||
async def main():
|
||||
state = _load_state()
|
||||
state.setdefault("seed_cycle", 0)
|
||||
global SEED_CYCLE # ← required
|
||||
SEED_CYCLE = state["seed_cycle"]
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import chefkoch_scraper_v4 as m
|
||||
m.main() # or simulate the assignment
|
||||
print('global SEED_CYCLE after main:', m.SEED_CYCLE)
|
||||
"
|
||||
```
|
||||
Without `global` it prints `0`. With `global` it prints whatever was in state.
|
||||
|
||||
---
|
||||
|
||||
## Fix 3 — Stale `completed_seeds` locking the scraper
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper starts, opens browser, visits search pages, but `new_urls` is always empty because all returned URLs are already in `existing`. No new recipes are appended.
|
||||
|
||||
### Why it happens
|
||||
|
||||
`scraper_state.json` accumulated a `completed_seeds` dict during a multi-worker batch run. The v4 infinite-refill script does not use `completed_seeds`, but the state file still carries it from an older script version. Certain seed terms are effectively blacklisted.
|
||||
|
||||
### Fix
|
||||
|
||||
```python
|
||||
import json
|
||||
s = json.load(open("scraper_state.json"))
|
||||
s.pop("completed_seeds", None)
|
||||
# Advance seed_cycle past known-exhausted 2-letter alphabet seeds
|
||||
s["seed_cycle"] = max(s.get("seed_cycle", 0), 1000)
|
||||
json.dump(s, open("scraper_state.json", "w"), ensure_ascii=False, indent=2)
|
||||
```
|
||||
|
||||
Setting `seed_cycle >= 1000` pushes the generator into the category+letter and category-only strategies, which yield far more fresh URLs than the `aa`, `ab`, `ac` alphabet exhaustion zone.
|
||||
|
||||
---
|
||||
|
||||
## Fix 4 — Dual script / dual Python trap (watchdog silently fails)
|
||||
|
||||
### Failure signature
|
||||
|
||||
Scraper cron log shows `Scraper started (PID NNN)` every 10 minutes. No errors ever appear. `scraper_v4.out` only contains `Starting: X/Y` line (the `print()` from `main()` before Playwright is used). Recipe count stays flat for days or weeks.
|
||||
|
||||
### Why it happens
|
||||
|
||||
The watchdog script (`~/.hermes/scripts/start_scraper_v4.sh`) uses:
|
||||
1. **Global `python3`** → Playwright 1.59.0 (looks for `chromium-1071`, which does not exist in this environment).
|
||||
2. **Global script** (`~/.hermes/scripts/chefkoch_scraper_v4.py`) → missing `global SEED_CYCLE` in `main()`.
|
||||
|
||||
Both of these individually produce crashes (Fix 1 + Fix 2 above). Combined, the global python immediately crashes on browser launch, outputting nothing useful to the log. The watchdog sees the PID exit, and restarts it again 10 minutes later. This loop repeats forever with zero recipes.
|
||||
|
||||
Meanwhile, a **working** copy exists at:
|
||||
- `~/.hermes/profiles/nutrition-coach/.venv/bin/python3` (Playwright 1.60.0+, matches installed browsers)
|
||||
- `~/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py` (has `global SEED_CYCLE`)
|
||||
|
||||
Both copies look very similar from a distance, making it easy to invoke the wrong pair.
|
||||
|
||||
### Fix
|
||||
|
||||
Replace the watchdog shell script with the version that explicitly points to the profile-local python and script:
|
||||
|
||||
```bash
|
||||
nohup /home/debian/.hermes/profiles/nutrition-coach/.venv/bin/python3 \
|
||||
/home/debian/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py \
|
||||
--no-agent >> "$LOG" 2>&1 &
|
||||
```
|
||||
|
||||
Also verify there is no stale PID file (otherwise the wrapper exits early assuming it's already running):
|
||||
|
||||
```bash
|
||||
cat /tmp/chefkoch_scraper_v4.pid 2>/dev/null && (kill -0 "$(cat /tmp/chefkoch_scraper_v4.pid)" 2>/dev/null || rm -f /tmp/chefkoch_scraper_v4.pid)
|
||||
```
|
||||
|
||||
### Detection
|
||||
|
||||
```bash
|
||||
# 1. Check if log ever got past the first line
|
||||
wc -l ~/.hermes/profiles/nutrition-coach/scraper_v4.out
|
||||
# Unhealthy: 1 line forever
|
||||
|
||||
# 2. Check which python the startup script actually calls
|
||||
grep 'python3' ~/.hermes/scripts/start_scraper_v4.sh
|
||||
# Unhealthy: just "python3 ..." (no venv path)
|
||||
|
||||
# 3. Check which script path it points to
|
||||
grep 'chefkoch_scraper' ~/.hermes/scripts/start_scraper_v4.sh
|
||||
# Unhealthy: ~/.hermes/scripts/chefkoch_scraper_v4.py
|
||||
```
|
||||
|
||||
### Recovery after fix
|
||||
|
||||
After updating the watchdog script:
|
||||
|
||||
```bash
|
||||
# 1. Kill any stale scraper
|
||||
pkill -f chefkoch_scraper_v4.py || true
|
||||
rm -f /tmp/chefkoch_scraper_v4.pid
|
||||
|
||||
# 2. Advance state past exhaustion
|
||||
python3 -c "
|
||||
import json, os
|
||||
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
|
||||
s = json.load(open(p))
|
||||
s.pop('completed_seeds', None)
|
||||
s['seed_cycle'] = max(s.get('seed_cycle', 0), 2000)
|
||||
json.dump(s, open(p, 'w'), ensure_ascii=False, indent=2)
|
||||
print('State reset — seed_cycle now', s['seed_cycle'])
|
||||
"
|
||||
|
||||
# 3. Start with fixed script
|
||||
bash ~/.hermes/scripts/start_scraper_v4.sh
|
||||
|
||||
# 4. Verify within 2 minutes
|
||||
sleep 120 && wc -l ~/.hermes/profiles/nutrition-coach/real_recipes.jsonl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## One-liner state reset (copy-paste)
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import json, os
|
||||
p = os.path.expanduser('~/.hermes/profiles/nutrition-coach/scraper_state.json')
|
||||
s = json.load(open(p))
|
||||
s.pop('completed_seeds', None)
|
||||
s['seed_cycle'] = max(s.get('seed_cycle',0), 1000)
|
||||
json.dump(s, open(p,'w'), ensure_ascii=False, indent=2)
|
||||
print('Reset OK — seed_cycle now', s['seed_cycle'])
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring checklist (what to watch)
|
||||
|
||||
| Check | Command | Healthy / Unhealthy |
|
||||
|-------|---------|---------------------|
|
||||
| Recipe count moving | `wc -l real_recipes.jsonl` | Increases between checks |
|
||||
| Process alive | `ps auxf \| grep chefkoch_scraper` | chromium + python present |
|
||||
| State advancing | `python3 -c "... seed_cycle ..."` | value increases over time |
|
||||
| Log not looping | `tail -n 5 scraper_v4.out` | NOT just "Starting: X/Y" only |
|
||||
| Disk space | `df -h .` | > 1 GB free |
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
# v5.2 Scraper: Wo liegen die Daten?
|
||||
|
||||
## Sitzung 2026-06-22 — Problem: Datenbank nicht auffindbar
|
||||
|
||||
### Kontext
|
||||
User wollte einen konkreten Link zu einem neuen Chefkoch-Rezept (v5.2). Ich fand keine `real_recipes.jsonl`, keinen laufenden Container, keine DuckDB. Die 25k+ Rezepte waren in einer früheren Sitzung entstanden, deren Deployment-Kontext nicht mehr greifbar war.
|
||||
|
||||
### Gesuchte Orte (kein Treffer)
|
||||
|
||||
| Ort | Ergebnis | Bemerkung |
|
||||
|-----|----------|-----------|
|
||||
| `/tmp/*.duckdb` | — | Keine DuckDB-Datei |
|
||||
| `/tmp/chefkoch-api/` | Nur `chefkoch-api` npm-Paket | Keine Daten |
|
||||
| Local `~/.hermes/profiles/nutrition-coach/` | Keine `real_recipes.jsonl` | v4-Legacy-Dateien |
|
||||
| Docker-Host 10.0.30.99 | Container `scraper` existiert nicht | Kein Recipe-Container läuft |
|
||||
| Docker-Host 10.0.30.99 `/opt/` `/mnt/` `/srv/` | Nur `data/seafile/seadoc` | Keine Recipe-Daten |
|
||||
| `~/.hermes/seafile_inventory.md` | Kein Recipe-Eintrag | Keine Seafile-Struktur |
|
||||
|
||||
### Schlussfolgerung
|
||||
|
||||
**v5.2-Daten liegen nicht auf dem aktuellen Host.** Mögliche Erklärungen:
|
||||
1. Scraper lief auf einem anderen Host / in einer anderen Sitzung
|
||||
2. Container wurde gelöscht / Volume war nicht persistent
|
||||
3. Daten sind in Seafile / Gitea / anderem Storage abgelegt (aber nicht indexiert)
|
||||
4. Testlauf war temporär (z.B. `/tmp/` ohne Persistenz)
|
||||
|
||||
### Erkennungs-Skript (zur Verwendung bei zukünftigen Suchen)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# v5.2-where-is-my-data.sh
|
||||
set -e
|
||||
|
||||
echo "=== v5.2 Daten-Suche ==="
|
||||
|
||||
# 1. Lokale Suche
|
||||
echo "1. Lokale JSONL-Dateien..."
|
||||
find / -name "real_recipes.jsonl" -type f 2>/dev/null | head -5
|
||||
|
||||
# 2. Container-Suche
|
||||
echo "2. Docker-Container..."
|
||||
docker ps -a | grep -i -E "scrap|recipe|food|chefkoch" || echo "Kein Container"
|
||||
|
||||
# 3. Seafile-Remote-Suche
|
||||
echo "3. Seafile..."
|
||||
# Via API: List library 'Casa' or 'Favorites' for recipe subdirs
|
||||
|
||||
# 4. JSONL-Größe check
|
||||
echo "4. Falls gefunden, Größe:"
|
||||
ls -lh $(find / -name "*.jsonl" -type f 2>/dev/null | head -5)
|
||||
```
|
||||
|
||||
### Pflicht-Checkliste: v5.2 nach Deployment
|
||||
|
||||
Bevor man sagt „v5.2 läuft und hat N Rezepte“, muss verifiziert sein:
|
||||
|
||||
1. **Wo ist `real_recipes.jsonl`?** Absolute Pfad notieren.
|
||||
2. **Ist der Pfad persistent?** (nicht `/tmp/`, nicht Ephemeral-Container)
|
||||
3. **Ist ein Container definiert?** `docker ps -a` zeigt ihn.
|
||||
4. **Cronjob / Watchdog** greift auf denselben Pfad zu wie der Scraper.
|
||||
5. **Seafile-Backup** abgleichen: Liegen Rezepte auch in der Cloud?
|
||||
|
||||
### Übergabe-Wissen für nächste Sitzung
|
||||
|
||||
Wenn ein Agent gefragt wird „zeig mir ein Rezept-Link“, und `real_recipes.jsonl` nicht lokal ist:
|
||||
1. NICHT raten (keine Beispiel-URLs aus der API-Doku generieren)
|
||||
2. Prüfen: Container? Lokales File? Seafile?
|
||||
3. Wenn nirgends: User informieren „Datenbank nicht gefunden, Scraper muss neu gestartet werden“
|
||||
|
||||
**Hinweis:** Die API-Doku (`chefkoch-api-v2-recipes.md`) gibt real existierende IDs, aber ohne JSONL kann ich nicht ohne Request validieren, dass ein bestimmtes Rezept wirklich gescraped wurde.
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Generator Scoring Algorithm Reference
|
||||
|
||||
Session: 2026-06-17
|
||||
|
||||
## Weather Scoring
|
||||
|
||||
```python
|
||||
if temp >= 35:
|
||||
if is_cold: score += 5.0
|
||||
if hot_cooking: score -= 4.0
|
||||
if not is_cold: score -= 2.0 # blanket
|
||||
if fish and day_index >= 4: score -= 4.0
|
||||
|
||||
elif temp >= 30:
|
||||
if is_cold: score += 4.0
|
||||
if hot_cooking: score -= 2.5
|
||||
if soup: score -= 3.0
|
||||
if fish and day_index >= 4: score -= 3.0
|
||||
```
|
||||
|
||||
**Critical pitfall learned:** Without blanket `-2.0` on non-cold meals at >=35°C, family-preference scores (+1.5 Asian) can overpower weather penalties. Result: Carbonara at 37°C.
|
||||
|
||||
## Perishability Scoring
|
||||
|
||||
```python
|
||||
if perishable or (fresh and 'salat' in tags):
|
||||
if day_index <= 2: score += 2.5
|
||||
elif day_index >= 4: score -= 3.0
|
||||
elif day_index == 3: score -= 1.0
|
||||
```
|
||||
|
||||
User example: "Lachs vom Samstag ist am Donnerstag schlecht."
|
||||
|
||||
## Tag Taxonomy
|
||||
|
||||
- **Cold:** salat, bowl, frisch, roh, griechisch, mediterran
|
||||
- **Hot cooking:** pasta, nudeln, ofen, eintopf, curry, schmor, gratin, hackfleisch
|
||||
- **Perishable:** fisch, lachs, thunfisch, kräuter, minze, koriander, basilikum, rucola, spinat
|
||||
- **Fresh:** salat, tomate, gurke, avocado
|
||||
|
||||
## Complete Code
|
||||
|
||||
See `scripts/nutrition-plan-generator-v3.py` — full implementation with all scoring dimensions.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Recipe Data Sources (Tested 2026-06-28)
|
||||
|
||||
## Active Sources (v7 Scraper — Direct MySQL)
|
||||
|
||||
### Chefkoch.de — ~300k+ total, ~25k scraped
|
||||
- **Access:** REST API v2 (`https://api.chefkoch.de/v2/recipes`)
|
||||
- **Method:** Query rotation across 100+ German cooking terms, pagination via offset
|
||||
- **Structured data:** API returns JSON with title, rating, ingredients, tags, times
|
||||
- **Anti-scraping:** Rate-limited but no Cloudflare; 0.3-0.5s delay sufficient
|
||||
- **Field mapping:** `id`, `title`, `category`, `tags`, `rating`, `prepTime`, `cookTime`, `totalTime`, `ingredients[].raw`, `siteUrl`, `nutrition.calories`
|
||||
|
||||
### Kuechengoetter.de — 30,000 recipes
|
||||
- **Access:** Sitemap (`https://www.kuechengoetter.de/sitemaps/recipes-1.xml`)
|
||||
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
|
||||
- **Structured data:** Schema.org Recipe in `<script type="application/ld+json">`
|
||||
- **Fields extracted:** name, aggregateRating.ratingValue, prepTime, cookTime, totalTime, recipeIngredient[], recipeCategory, recipeInstructions[], keywords→tags
|
||||
- **Anti-scraping:** None observed; polite delays (0.3-1.0s) work fine
|
||||
- **Quality:** Ratings present but sparse (many show rating=1, meaning single vote). Ingredient lists well-structured.
|
||||
|
||||
### Kitchen Stories (kitchenstories.com/de) — 3,390 German recipes
|
||||
- **Access:** Sitemap (`https://www.kitchenstories.com/sitemap.xml`)
|
||||
- **Filter:** `/de/rezepte/` (German recipe pages only; site has EN/DE bilingual)
|
||||
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
|
||||
- **Structured data:** Schema.org Recipe, 3 JSON-LD blocks per page (Recipe + WebSite + BreadcrumbList)
|
||||
- **Fields:** name, totalTime (ISO 8601 like PT20M), recipeIngredient[], recipeCategory, recipeInstructions[]
|
||||
- **Anti-scraping:** None observed
|
||||
- **Quality:** High-quality editorial recipes. No ratings in JSON-LD. Good metadata (times, categories).
|
||||
|
||||
### HelloFresh.de — ~16,831 total, ~10,000 estimated unique, 555 already scraped
|
||||
- **Access:** Category pages via `__NEXT_DATA__` (Next.js SSR hydration blob)
|
||||
- **Listing URL pattern:** `https://www.hellofresh.de/recipes/{category}?skip={N}&take=50`
|
||||
- **Detail URL pattern:** `https://www.hellofresh.de/recipes/{slug}-{recipeId}`
|
||||
- **Method:** Enumerate categories → paginate each → fetch detail per recipe → parse `__NEXT_DATA__`
|
||||
- **Structured data:** NOT JSON-LD. Next.js `__NEXT_DATA__` script tag containing `ssrPayload.dehydratedState.queries[]`
|
||||
- Listing: find query with `queryKey[0] == 'recipe.search'` → `state.data.items[]` (each has `id`, `slug`, `name`)
|
||||
- Detail: find query with `queryKey[0] == 'foodContentHubRecipe.byId'` → `state.data.recipe`
|
||||
- **Normalization:** Custom `_hf_normalize()` — builds ingredients from `yields[0].ingredients[]` joined with `ingredients[]` lookup, instructions from `steps[].instructionsMarkdown`, calories from `nutrition[]` where name contains 'kcal'
|
||||
- **Working categories (19/28):** familien-rezepte (6,208), vegetarische-rezepte (4,625), einfache-rezepte (1,444), gesunde-rezepte (1,395), eintopf-rezepte (934), italienische-rezepte (481), deutsche-rezepte (463), asiatische-rezepte (185), amerikanische-rezepte (172), orientalische-rezepte (167), kartoffel-rezepte, mediterrane-rezepte, mexikanische-rezepte, nudel-rezepte, suppen, salate, burger, chinesische-rezepte, flammkuchen-rezepte
|
||||
- **Dead categories (404):** belibteste-rezepte, pancakes, pasta, pizza, reis-rezepte, sandwich, fleisch-rezepte, franzosische-rezepte, griechische-rezepte
|
||||
- **Anti-scraping:** None observed; Referer header recommended
|
||||
- **Quality:** HIGHEST of all sources. 99% have calories, 93% have ratings, all have `instructions`, `description`, `difficulty`, `servings`. Professionally curated with exact quantities.
|
||||
- **ID prefix:** `hf_` (NOT `hellofresh_`) — must match v5.3 legacy IDs for deduplication
|
||||
- **Headers:** Requires `Referer: https://www.hellofresh.de/` and standard browser UA
|
||||
|
||||
### GuteKueche.de — 29,392 recipes (INTEGRATED in v6)
|
||||
- **Access:** Gzipped sitemap (`https://cdn.gutekueche.de/sitemaps/recipe.xml.gz`)
|
||||
- **Discovery:** robots.txt → `Sitemap:` directive led to CDN-hosted gzipped sitemap index → 4 sub-sitemaps, `recipe.xml.gz` has 29,392 recipe URLs
|
||||
- **Method:** Same as Kuechengoetter/Kitchen Stories — sitemap discovery → JSON-LD extraction. Scraper auto-detects gzip via magic bytes `\x1f\x8b` or config flag `"gzipped": True`.
|
||||
- **Config:** Added to `SITEMAP_SOURCES` dict with `"gzipped": True` flag. `discover_sitemap_urls()` enhanced with gzip support (urllib + `gzip.decompress()`).
|
||||
- **URL pattern:** `https://www.gutekueche.de/<name>-rezept-<id>`
|
||||
- **Structured data:** Schema.org Recipe in JSON-LD, **9/11 quality fields** (highest of all sitemap sources)
|
||||
- **Fields present:** nutrition (calories ✅), recipeInstructions ✅, aggregateRating ✅ (e.g. 4.5★), totalTime, recipeIngredient, recipeCategory, recipeYield, prepTime, cookTime
|
||||
- **Missing:** recipeCuisine, suitableForDiet
|
||||
- **Anti-scraping:** None; robots.txt allows all (only disallows /user/, /suche, /api/)
|
||||
- **Quality:** Strong — professional editorial recipes with calorie data and ratings. Second-best overall quality after HelloFresh.
|
||||
|
||||
## Tested but Rejected
|
||||
|
||||
| Source | Reason |
|
||||
|--------|--------|
|
||||
| AllRecipes.com | HTTP 403 Forbidden (Cloudflare) |
|
||||
| Yummly.com | HTTP 403 Forbidden |
|
||||
| Tasty.co | HTTP 404 on sitemap |
|
||||
| Marmiton.org | HTTP 404 on sitemap (French, URL structure changed) |
|
||||
| EatThis.org | SSL certificate verification failed |
|
||||
| Lecker.de | Sitemap XML files return 0 URLs (likely gzipped or JS-rendered) |
|
||||
| Essen-und-trinken.de | **Has Recipe JSON-LD on recipe pages** (6/11 fields, calories ✅, ratings ✅), BUT robots.txt blocks ALL AI bots (ClaudeBot, ChatGPT, Claude-Web, CCBot, etc.). Respect robots.txt — do not scrape. |
|
||||
| Maggi.de | HTTP 403 Forbidden |
|
||||
| Knorr.de | HTTP 403 Forbidden |
|
||||
| Serious Eats | HTTP 403 Forbidden |
|
||||
| Dr. Oetker | No JSON-LD on recipe pages |
|
||||
| Kochbar.de | No JSON-LD |
|
||||
| Marley Spoon | No JSON-LD (JS-rendered) |
|
||||
| Frauenzimmer.de | JSON-LD present but no Recipe type on recipe pages |
|
||||
| Bunte.de | 404 on recipe page URLs (structure unclear) |
|
||||
|
||||
## Storage: MariaDB Galera Cluster
|
||||
|
||||
Recipe data migrated from JSONL to the existing Galera cluster (2026-06-28).
|
||||
|
||||
- **Why Galera over SQLite:** User preference — reuse existing HA infrastructure (3-node cluster already running for HA Recorder) instead of adding new dependencies. Concurrent read/write via MaxScale readwritesplit.
|
||||
- **Connection:** `10.0.30.70:3306` (MaxScale VIP) → Galera cluster (db1=.71, db2=.72, db3=.73)
|
||||
- **Credentials:** DB=`recipes`, User=`recipe_app@%`, password in migration script
|
||||
- **Table schema:** 20 columns, InnoDB, utf8mb4_unicode_ci
|
||||
- **Indexes:** FULLTEXT ft_title_desc (title, description), FULLTEXT ft_ingredients (ingredients), BTREE idx_source, idx_category, idx_rating, idx_difficulty
|
||||
- **Migration script:** `scripts/migrate_to_galera.py` — INSERT IGNORE with batch=200, handles dict-valued `category` and `nutrition` fields, skips existing IDs
|
||||
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. Migration script must coerce these to strings/extract values before INSERT.
|
||||
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
|
||||
- **Architecture:** Scraper writes directly to Galera via `INSERT IGNORE` (no JSONL intermediate). v7 eliminated the JSONL → migration → DB pipeline. JSONL (`real_recipes.jsonl`) is kept as historical archive only.
|
||||
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. `recipe_to_row()` in v7 coerces these to strings/extracts values before INSERT.
|
||||
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
|
||||
- **Pitfall:** Per-row `db_has_id()` dedup is catastrophically slow through MaxScale — each check is a network roundtrip. v7 loads all IDs into a Python `set()` at startup via `db_all_ids()` (~0.3s for 28k), then all dedup is in-memory. Additionally, derive the likely recipe ID from the URL path and skip the HTTP fetch entirely if already known.
|
||||
|
||||
## Potential Future Sources (not yet integrated)
|
||||
|
||||
| Source | Recipes | Access | Notes |
|
||||
|--------|---------|--------|-------|
|
||||
| Food.com | Large | JSON-LD confirmed on recipe pages | English; sitemap index exists |
|
||||
| BBC Good Food | ~300/quarter | Sitemap recipe files, JSON-LD confirmed | English; premium content mixed in |
|
||||
| HelloFresh.de | ~~548 scraped~~ → **now active in v6** (see above) | Was thought exhausted; actually has ~16,831 recipes |
|
||||
|
||||
## Recipe Source Scanning Methodology
|
||||
|
||||
When evaluating new recipe sources for the scraper:
|
||||
|
||||
1. **Don't judge by landing page alone.** Landing/listing pages rarely embed Recipe JSON-LD — they have WebSite or ItemList schemas. Always fetch an **individual recipe page** and check for `@type: "Recipe"` in JSON-LD blocks.
|
||||
2. **Check `robots.txt` first.** Many sites declare their sitemap location there (`Sitemap: <url>`). Standard paths like `/sitemap.xml` frequently 404. Also check for AI-bot blocks — respect them.
|
||||
3. **Sitemaps may be gzipped.** Look for `.gz` extensions or `Content-Encoding: gzip`. Use `gzip.decompress()` in Python.
|
||||
4. **Volume estimation from sitemap index.** Fetch the index, count sub-sitemaps, sample 5-10 sub-sitemaps, extrapolate. Filter URLs by path patterns (`/rezept-`, `/rezepte/`, `/recipes/`).
|
||||
5. **Quality scoring (11-field checklist):** Count presence of: `nutrition`, `recipeInstructions`, `aggregateRating`, `totalTime`, `recipeIngredient`, `recipeCategory`, `recipeYield`, `prepTime`, `cookTime`, `recipeCuisine`, `suitableForDiet`. Score = present/11. HelloFresh ≈ 11/11 (custom parsing), GuteKueche ≈ 9/11, Kuechengoetter ≈ 6/11.
|
||||
6. **Quality ≠ Volume.** A source with 548 high-quality recipes (calories, instructions, ratings, difficulty) may be more valuable than a 30k source with sparse metadata. Evaluate both dimensions. **Never dismiss a source solely for low volume without assessing data quality.**
|
||||
|
||||
## v7 Scraper Architecture (Direct MySQL — replaces v6)
|
||||
|
||||
v7 eliminates JSONL entirely. Writes directly to MariaDB Galera via PyMySQL.
|
||||
|
||||
```
|
||||
recipe_scraper_v7.py
|
||||
├── Phase 1: Sitemap sources (Kuechengoetter → GuteKueche → Kitchen Stories)
|
||||
│ ├── discover_sitemap_urls() — fetch sitemap (auto-gzip), extract <loc> URLs
|
||||
│ ├── scrape_sitemap_source() — visit each URL, extract JSON-LD
|
||||
│ └── normalize_jsonld_recipe() — Schema.org → our format
|
||||
├── Phase 2: HelloFresh (__NEXT_DATA__ extraction)
|
||||
│ ├── _hf_fetch_category_page() — paginate categories via skip/take
|
||||
│ ├── _hf_fetch_detail() — fetch recipe page, parse __NEXT_DATA__
|
||||
│ ├── _hf_normalize() — convert HF nested structure to our format
|
||||
│ └── Per-category skip tracking via state["hf_cat_offset"]
|
||||
├── Phase 3: Chefkoch API (fills remaining gap to 100k)
|
||||
│ ├── fetch_ck_page() — API query with rotation
|
||||
│ └── fetch_ck_detail() — full recipe by ID
|
||||
├── State: scraper_state.json (tracks tried URLs, query cycles, source counts, hf_cat_offset)
|
||||
├── Dedup: db_all_ids() loads ALL IDs into Python set() at startup (~0.3s for 28k)
|
||||
│ └── Pre-check: derives likely ID from URL path, skips HTTP fetch if already in set
|
||||
├── Insert: INSERT IGNORE per recipe (idempotent, safe for re-runs)
|
||||
├── Output: Direct to MariaDB Galera (10.0.30.70:3306, DB=recipes)
|
||||
├── Cronjob: recipe-refill-v7 (daily 06:00, deliver: origin)
|
||||
└── Target: 100,000 recipes
|
||||
```
|
||||
|
||||
**Key differences from v6:**
|
||||
- No JSONL file, no migration step — single pipeline
|
||||
- In-memory dedup via `db_all_ids()` instead of per-row `db_has_id()` (avoids catastrophic Galera roundtrip latency)
|
||||
- URL pre-check skips HTTP fetch entirely for known recipes
|
||||
- Progress logging every 200 URLs
|
||||
- `recipe_to_row()` sanitizes dict/list fields (category, calories) before INSERT
|
||||
|
||||
**Separate step — Embeddings:**
|
||||
After scraping, run `generate_embeddings.py` to populate the `embedding` JSON column with 1024-dim harrier vectors. This is NOT done by the scraper.
|
||||
|
||||
**Output format** (unified across all sources, stored in `recipes` table):
|
||||
```json
|
||||
{
|
||||
"id": "kuechengoetter_seelachs-mit-rucola-1",
|
||||
"name": "Seelachs mit Rucola",
|
||||
"title": "Seelachs mit Rucola",
|
||||
"category": "fish",
|
||||
"tags": ["fisch", "seelachs", "rucola"],
|
||||
"rating": 4.5,
|
||||
"rating_value": 4.5,
|
||||
"prep_time": "PT15M",
|
||||
"cook_time": "PT20M",
|
||||
"total_time": "PT35M",
|
||||
"ingredients": [{"raw": "300g Seelachsfilet"}, ...],
|
||||
"source_url": "https://...",
|
||||
"source": "kuechengoetter",
|
||||
"calories": "450 kcal"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,98 @@
|
||||
# Semantic Ontology Classification — Implementation
|
||||
|
||||
Date: 2026-06-28
|
||||
Status: **Implemented, running in background**
|
||||
|
||||
## Overview
|
||||
|
||||
Rule-based semantic classifier that annotates every recipe across 11 ontology dimensions. Pure Python keyword matching — no LLM/API calls needed. Processes ~500 recipes/batch at ~50-500/s throughput.
|
||||
|
||||
## Script
|
||||
|
||||
`profiles/nutrition-coach/scripts/semantic_classifier.py`
|
||||
|
||||
- Incremental: `LEFT JOIN recipe_semantics rs ON r.id = rs.recipe_id ... WHERE rs.recipe_id IS NULL`
|
||||
- Batch insert: 500 recipes per `executemany()`
|
||||
- Fallback to one-by-one insert on batch error
|
||||
- Confidence: 0.85 (rule-based, not LLM-derived)
|
||||
- Runtime: ~10 min for 54k recipes
|
||||
|
||||
## Database Table: `recipe_semantics`
|
||||
|
||||
```sql
|
||||
CREATE TABLE recipe_semantics (
|
||||
recipe_id VARCHAR(255) PRIMARY KEY, -- ⚠️ 255, not 128!
|
||||
diet_type ENUM('vegan','vegetarian','pescatarian','flexitarian','omnivore') NOT NULL,
|
||||
meal_course JSON NOT NULL, -- breakfast,starter,soup,salad,main,side,dessert,snack,beverage,bread
|
||||
cuisine JSON NOT NULL, -- italian,french,german,asian,mexican,thai,japanese,chinese,greek,spanish,turkish,middle_eastern,american,nordic,african,caribbean,austrian,swiss
|
||||
cooking_method JSON NOT NULL, -- baking,frying,boiling,grilling,steaming,stewing,roasting,braising,raw,microwaving
|
||||
protein_source JSON NOT NULL, -- beef,pork,poultry,fish,seafood,plant,dairy,egg,none
|
||||
allergens JSON NOT NULL, -- gluten,eggs,dairy,nuts,soy,fish,shellfish,peanuts,sesame,celery,mustard,sulphites,lupin,molluscs
|
||||
flavor_profile JSON NOT NULL, -- sweet,savory,salty,sour,bitter,umami,spicy,smoky,creamy,fresh
|
||||
complexity ENUM('quick','simple','moderate','elaborate') NOT NULL,
|
||||
season JSON NOT NULL, -- spring,summer,autumn,winter,year_round
|
||||
caloric_density ENUM('light','balanced','hearty','indulgent') NOT NULL,
|
||||
preparation_style JSON NOT NULL, -- one_pot,sheet_pan,pan_fried,oven_baked,grilled,no_cook,slow_cooked,meal_prep
|
||||
confidence FLOAT DEFAULT 0.85,
|
||||
analyzed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
## Ontology Dimensions
|
||||
|
||||
### 1. Diet Type (ENUM, single value)
|
||||
Priority: explicit tags (vegan/vegetarisch) → meat detection → fish/seafood → dairy/egg → default omnivore.
|
||||
|
||||
### 2. Meal Course (JSON array)
|
||||
Keyword matching on title+category+tags+ingredients+description. Fallback to category field if no match. Default: `main`.
|
||||
|
||||
### 3. Cuisine (JSON array)
|
||||
18 cuisine types with German+English keywords. Also checks tags for country names. Default: `["german"]` if `deutsch` in tags.
|
||||
|
||||
### 4. Cooking Method (JSON array)
|
||||
10 methods. Default: `["boiling"]` if no match.
|
||||
|
||||
### 5. Protein Source (JSON array)
|
||||
Ordered priority: beef → pork → poultry → fish → seafood → egg → dairy → plant → none. Multiple can match.
|
||||
|
||||
### 6. Allergens (JSON array)
|
||||
14 EU-regulated allergens. Keyword-based, broad coverage (German+English).
|
||||
|
||||
### 7. Flavor Profile (JSON array)
|
||||
10 flavor dimensions. Desserts get `sweet` added if not already detected. Default: `["savory"]`.
|
||||
|
||||
### 8. Complexity (ENUM)
|
||||
Based on max(prep_time, cook_time, total_time): ≤15min=quick, ≤30min=simple, ≤60min=moderate, >60min=elaborate. Falls back to tags (schnell/einfach).
|
||||
|
||||
### 9. Season (JSON array)
|
||||
Keyword-based. Default: `["year_round"]`.
|
||||
|
||||
### 10. Caloric Density (ENUM)
|
||||
From calories: <300=light, <500=balanced, <800=hearty, ≥800=indulgent. Falls back to keyword estimation (kalorienarm, deftig, dessert→indulgent).
|
||||
|
||||
### 11. Preparation Style (JSON array)
|
||||
8 styles. Falls back to cooking method if no direct match.
|
||||
|
||||
## Cronjob
|
||||
|
||||
`semantic-classifier` — daily at 07:00, delivers to `telegram:223926918`. Runs after scraper (06:00) and embeddings (06:30).
|
||||
|
||||
## Pitfalls (Galera-Specific)
|
||||
|
||||
### 1. Foreign Key creation fails on collation mismatch
|
||||
The `recipes` table uses `utf8mb4_unicode_ci` but new tables default to `utf8mb4_general_ci`. FK creation fails with `errno 150`. Fix: either omit the FK (preferred — no cascade needed) or specify `COLLATE utf8mb4_unicode_ci` on the new table.
|
||||
|
||||
### 2. JOINs between tables with different collations fail
|
||||
`LEFT JOIN recipe_semantics rs ON r.id = rs.recipe_id` fails with `Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT)`. Fix: add `COLLATE utf8mb4_unicode_ci` to the JOIN condition.
|
||||
|
||||
### 3. VARCHAR(128) is too short for some recipe IDs
|
||||
Chefkoch IDs with encoded state fragments (e.g., `ck_Kinderueberraschungsbrot.html#eyJvYm...`) can be up to 248 chars. Use `VARCHAR(255)` for any column storing recipe IDs.
|
||||
|
||||
### 4. Time and calorie columns contain string values
|
||||
Some `prep_time`, `cook_time`, `total_time`, and `calories` values are stored as strings (e.g., `"PT20M"` or `"450 kcal"`), not integers. Always wrap with a `safe_int()` helper that catches `ValueError`/`TypeError` before using in comparisons or arithmetic.
|
||||
|
||||
### 5. Concurrent writes cause Galera deadlocks
|
||||
Running the semantic classifier and embedding generator simultaneously causes `Deadlock found when trying to get lock (1213)`. Fix: add retry logic with rollback+backoff (`safe_db_execute()`), or run sequentially. The daily cronjob schedule (06:00 scraper → 06:30 embeddings → 07:00 classifier) ensures sequential execution.
|
||||
|
||||
### 6. Galera FK constraints are fragile
|
||||
Even without collation issues, Galera's certification-based replication can reject DDL that works on standalone MySQL. Prefer logical relationships (application-level integrity checks) over physical FKs when working with Galera.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Semantic Search + Preference Learning — Implementation
|
||||
|
||||
Date: 2026-06-28
|
||||
Status: **Implemented and tested**
|
||||
|
||||
## Embedding Model
|
||||
|
||||
**Model:** `vllm/harrier-oss-v1-0.6b` (noris)
|
||||
- 1024 dimensions, multilingual (German-capable)
|
||||
- Dedicated embedding model (NOT an LLM), 0.6B params
|
||||
- Endpoint: `POST https://ai.noris.de/v1/embeddings`
|
||||
- Auth: Same noris API key from `config.yaml`
|
||||
- Batch: Up to 64 inputs per call, ~31 recipes/sec throughput
|
||||
|
||||
### Discovery
|
||||
|
||||
Initial investigation found that norris does NOT serve embeddings via standard LLM models (glm, gemma, qwen all return 404). The model list included `vllm/harrier-oss-v1-0.6b` which was the only embedding model that actually works. Other embedding names (bge-m3, e5-large, gte-large) are recognized by the Bifrost gateway but not activated on any API key.
|
||||
|
||||
### Quality Validation
|
||||
|
||||
Cosine similarity matrix (higher = more similar):
|
||||
|
||||
| | Curry | Salat | Eintopf | Hähnchen | Wintergericht |
|
||||
|-------------|-------|-------|---------|----------|---------------|
|
||||
| Curry | 1.000 | 0.647 | 0.677 | **0.812** | 0.660 |
|
||||
| Salat | 0.647 | 1.000 | 0.715 | 0.667 | 0.661 |
|
||||
| Eintopf | 0.677 | 0.715 | 1.000 | 0.696 | **0.829** |
|
||||
| Hähnchen | 0.812 | 0.667 | 0.696 | 1.000 | 0.694 |
|
||||
| Wintergericht| 0.660 | 0.661 | 0.829 | 0.694 | 1.000 |
|
||||
|
||||
Curry↔Hähnchen highest (both chicken dishes), Eintopf↔Wintergericht high (both warm/hearty), Salat↔Eintopf lower (different concepts). Working as expected.
|
||||
|
||||
## Database Schema
|
||||
|
||||
### `recipes.embedding` (JSON column)
|
||||
|
||||
Added via `ALTER TABLE recipes ADD COLUMN embedding JSON NULL`.
|
||||
Stores 1024-dim embedding as JSON array: `[0.023, -0.054, ...]`.
|
||||
|
||||
### `meal_decisions` table
|
||||
|
||||
```sql
|
||||
CREATE TABLE meal_decisions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
decision_date DATE NOT NULL,
|
||||
recipe_id VARCHAR(255) NOT NULL,
|
||||
recipe_title VARCHAR(500),
|
||||
recipe_category VARCHAR(200),
|
||||
accepted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
context_temp FLOAT NULL, -- outdoor temp °C
|
||||
context_season VARCHAR(20) NULL, -- spring/summer/autumn/winter
|
||||
context_weekday VARCHAR(20) NULL,
|
||||
context_last_meals TEXT NULL, -- JSON array of recent meal categories
|
||||
reason TEXT NULL, -- free-text rejection reason
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### `preference_rules` table
|
||||
|
||||
```sql
|
||||
CREATE TABLE preference_rules (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
rule_type ENUM('context_category','context_tag','variety','semantic') NOT NULL,
|
||||
condition_json TEXT NOT NULL, -- {"temp_min":20, "temp_max":99}
|
||||
effect_json TEXT NOT NULL, -- {"penalty_category":"Eintopf"}
|
||||
weight FLOAT DEFAULT 1.0, -- grows with confirmations (capped at 10)
|
||||
confirmations INT DEFAULT 1,
|
||||
rejections INT DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
All in `profiles/nutrition-coach/scripts/`:
|
||||
|
||||
### `generate_embeddings.py`
|
||||
- Batch-generates embeddings for recipes where `embedding IS NULL`
|
||||
- Builds embedding text from: `title + category + tags[:10] + ingredients[:20] + description[:200]` (capped at 1000 chars)
|
||||
- Batch size: 64 recipes per API call
|
||||
- Rate limit: 0.2s between calls
|
||||
- Run: `python3 scripts/generate_embeddings.py`
|
||||
- Should be run after scraper fills new recipes (could be chained in cronjob)
|
||||
|
||||
### `semantic_search.py`
|
||||
Core module with these functions:
|
||||
|
||||
#### `semantic_search(query, filters, top_k, min_rating)`
|
||||
- Gets query embedding from norris
|
||||
- Loads up to 5000 candidate recipes from DB (ordered by rating DESC)
|
||||
- Computes cosine similarity in Python
|
||||
- Blends similarity with rating: `score = similarity * (1 + rating/10)`
|
||||
- Supports filters: `category`, `source`, `exclude_categories`, `max_calories`
|
||||
|
||||
#### `record_decision(recipe_id, accepted, context)`
|
||||
- Logs to `meal_decisions` table
|
||||
- Auto-creates preference rule if rejection reason mentions weather/temperature
|
||||
- Example: `record_decision("hf_123", False, {"temp": 28, "season": "summer", "reason": "zu heiß für Eintopf"})`
|
||||
- Auto-rule: temp>20°C → penalize category from the rejected recipe
|
||||
|
||||
#### `get_preference_adjustments(context)`
|
||||
- Returns `{boost_categories, penalty_categories, boost_tags, penalty_tags}` based on matching rules
|
||||
- Each entry includes the rule's weight
|
||||
|
||||
#### `rank_with_preferences(recipes, context)`
|
||||
- Re-ranks a list of recipe dicts by applying:
|
||||
- Category penalties/boosts from learned rules (-0.1 × weight per match)
|
||||
- Tag penalties/boosts (-0.05 × weight per match)
|
||||
- Variety penalty: -0.15 if same category as recent meals
|
||||
- Checks `category + title + tags` concatenated (not just category field — see pitfall below)
|
||||
|
||||
#### `mine_rules(min_confidence=2)`
|
||||
- Analyzes `meal_decisions` for statistical patterns
|
||||
- Groups by `(category, temperature_bucket)` and `(category, season)`
|
||||
- If rejection rate > 50% with enough samples → creates/upgrades rule
|
||||
- Should run periodically as decisions accumulate
|
||||
|
||||
## Tested Examples
|
||||
|
||||
### Semantic Search
|
||||
```
|
||||
Query: "schnelles Hähnchengericht"
|
||||
→ Hähnchenbrust in Joghurt-Parmesan-Salbei-Panade (sim: 0.622, ★5.0)
|
||||
→ Hähnchenschnitzel auf Muttis Art (sim: 0.626, ★4.9)
|
||||
→ Hühnchenschnitzel (sim: 0.614, ★5.0)
|
||||
```
|
||||
|
||||
### Preference Learning
|
||||
```
|
||||
1. record_decision("ck_Erbsensuppe", False, {"temp": 28, "reason": "zu heiß für Eintopf"})
|
||||
→ Auto-rule created: temp>20°C → penalize "Eintopf" (weight: 1.0)
|
||||
|
||||
2. semantic_search("Eintopf mit Kartoffeln", top_k=10)
|
||||
→ rank_with_preferences(results, {"temp": 28, "season": "summer"})
|
||||
|
||||
Result: All Eintopf recipes get Δ=-0.2000 (two matching rules × 0.1 weight each)
|
||||
0.7365 (Δ-0.2000) | Möhren-Kartoffel-Eintopf (Kochen)
|
||||
0.7304 (Δ-0.2000) | Eintopf mit Spitzkohl, Hackfleisch (Kartoffeln)
|
||||
0.7226 (Δ-0.2000) | Möhreneintopf alla Isa (Eintopf)
|
||||
```
|
||||
|
||||
## Key Pitfalls
|
||||
|
||||
1. **Category field mismatch:** Most "Eintopf" recipes have category "Kochen" or "Kartoffeln", not "Eintopf". Rule matching must check `title + tags + category` as a concatenated lowercase string.
|
||||
|
||||
2. **Decimal type from MySQL:** `rating` comes back as `decimal.Decimal`, not `float`. Must cast with `float(rating or 0)` before arithmetic.
|
||||
|
||||
3. **Auto-rule with non-existent recipe IDs:** `record_decision("test_reject_1", ...)` creates a rule with empty category because the recipe doesn't exist in the DB. Always use real recipe IDs from the DB.
|
||||
|
||||
4. **Duplicate rules from auto-creation + mining:** Both `_auto_create_rule()` and `mine_rules()` can create the same rule. `_upsert_rule()` handles this with matching on `(rule_type, condition_json, effect_json)` and increments weight/confirmations.
|
||||
|
||||
## Integration with Meal Planner
|
||||
|
||||
The meal planner (`nutrition-plan-generator-v4.py`) should be updated to:
|
||||
1. Call `semantic_search()` instead of linear JSONL scan
|
||||
2. Call `rank_with_preferences()` with weather context
|
||||
3. Call `record_decision()` when user accepts/rejects proposals in Phase 2
|
||||
4. Periodically call `mine_rules()` to discover new patterns
|
||||
|
||||
Context for decisions:
|
||||
- Temperature: From HA weather sensor or Tibber API
|
||||
- Season: Derived from date
|
||||
- Weekday: From date
|
||||
- Last meals: From previous week's finalized plan
|
||||
@@ -0,0 +1,108 @@
|
||||
# Semantic Search + Preference Learning Investigation
|
||||
|
||||
Date: 2026-06-28
|
||||
|
||||
## Goal
|
||||
|
||||
Add semantic recipe search ("etwas Leichtes für heiße Tage") and preference learning ("wenn warm dann kein Eintopf") to the meal-planning pipeline.
|
||||
|
||||
## Embedding Model Requirements
|
||||
|
||||
- **Dimensions:** 1024 (user requirement)
|
||||
- **Language:** Multilingual / German-capable
|
||||
- **Hosting:** Preferred on norris (same provider as LLM models)
|
||||
|
||||
## Norris Embedding Endpoint Investigation
|
||||
|
||||
Tested all 20 models listed at `https://ai.noris.de/v1/models` against the `/v1/embeddings` endpoint:
|
||||
|
||||
### Models tested via `/v1/embeddings`
|
||||
| Model | Result |
|
||||
|-------|--------|
|
||||
| vllm/bge-reranker-v2-m3 | 404 — provider API error |
|
||||
| vllm/glm-5-2-nvfp4 | 404 — provider API error |
|
||||
| vllm/gemma-4-31b-it | 404 — provider API error |
|
||||
| vllm/qwen3.6-27b-nvfp4 | 404 — provider API error |
|
||||
|
||||
### Additional embedding models tested by name
|
||||
| Model | Result |
|
||||
|-------|--------|
|
||||
| vllm/bge-m3 | "no keys found that support model: bge-m3" |
|
||||
| vllm/multilingual-e5-large | "no keys found that support model" |
|
||||
| vllm/e5-mistral-7b-instruct | "no keys found that support model" |
|
||||
| vllm/gte-large | "no keys found that support model" |
|
||||
| vllm/NV-Embed-v2 | "no keys found that support model" |
|
||||
| vllm/snowflake-arctic-embed-l-v2.0 | "no keys found that support model" |
|
||||
|
||||
### Conclusion
|
||||
Norris recognizes embedding model names (gateway returns "no keys found" not "model not found"), meaning the Bifrost gateway knows about them but no API key has them activated. The `/v1/embeddings` endpoint returns 404 for all currently-available models (LLM models don't serve embeddings).
|
||||
|
||||
**Action needed:** User must request norris to activate an embedding model (bge-m3 recommended — 1024 dims, multilingual, strong German performance).
|
||||
|
||||
## Local vLLM Check
|
||||
|
||||
Checked for local vLLM instances that might serve embeddings:
|
||||
- `10.0.30.97:8000` (ComfyUI CT) — No route to host (SSH and HTTP both unreachable)
|
||||
- `10.0.30.102:8000` (OpenWebUI CT) — No response
|
||||
- `10.0.30.99:8000` (Docker host) — No response
|
||||
- `localhost:8000/8001` — No response
|
||||
|
||||
No local vLLM embedding server found. ComfyUI CT (10.0.30.97) has GPU but was unreachable during investigation.
|
||||
|
||||
## Proposed Architecture (pending embedding model availability)
|
||||
|
||||
### Database Schema Additions
|
||||
|
||||
```sql
|
||||
-- Add embedding column to recipes table
|
||||
ALTER TABLE recipes ADD COLUMN embedding JSON NULL COMMENT '1024-dim vector as JSON array';
|
||||
|
||||
-- Decision logging
|
||||
CREATE TABLE meal_decisions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
decided_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
recipe_id VARCHAR(255) NOT NULL,
|
||||
decision ENUM('accepted', 'rejected') NOT NULL,
|
||||
context JSON COMMENT 'temperature, season, weekday, recent_meals',
|
||||
INDEX idx_recipe (recipe_id),
|
||||
INDEX idx_decided (decided_at)
|
||||
);
|
||||
|
||||
-- Learned preference rules
|
||||
CREATE TABLE preference_rules (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
rule_pattern VARCHAR(255) NOT NULL COMMENT 'e.g. temp>20=>avoid:eintopf',
|
||||
weight FLOAT DEFAULT 1.0 COMMENT 'confidence, grows with confirmations',
|
||||
confirmed_count INT DEFAULT 0,
|
||||
rejected_count INT DEFAULT 0,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uniq_pattern (rule_pattern)
|
||||
);
|
||||
```
|
||||
|
||||
### Pipeline Flow
|
||||
|
||||
```
|
||||
Meal Planner selects recipes:
|
||||
1. SQL filter (rating > 4.0, not used this week)
|
||||
2. Preference rules apply (boost/penalize by context)
|
||||
3. Semantic search for "mood" query (light, hearty, asian...)
|
||||
4. Top-K presented as proposal
|
||||
|
||||
User feedback ("nein, kein Eintopf bei dem Wetter"):
|
||||
→ INSERT INTO meal_decisions (rejected, temp=28°C, category=eintopf)
|
||||
→ Rule mining detects pattern → INSERT/UPDATE preference_rules
|
||||
→ Next run: Eintopf auto-downweighted when warm
|
||||
```
|
||||
|
||||
### Embedding Generation Strategy
|
||||
|
||||
- Batch-generate embeddings for existing 28k+ recipes (one-time)
|
||||
- Incrementally embed new recipes as v7 scraper inserts them
|
||||
- Store as JSON array in `embedding` column (simple, no pgvector/Qdrant dependency)
|
||||
- Cosine similarity computed in Python at query time (28k × 1024 floats ≈ 115MB RAM, <1s)
|
||||
- At 100k recipes: ~400MB RAM for full similarity matrix — still feasible in-process
|
||||
|
||||
### Alternative: MariaDB VECTOR Type
|
||||
|
||||
MariaDB 11.6+ supports native `VECTOR` type with `VEC_DISTANCE` functions. If the Galera cluster upgrades, switch from JSON-stored arrays to native vectors for indexed similarity search. Current MariaDB version on the cluster should be checked before relying on this.
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
# Check real recipe database status and control the mass scraper
|
||||
# Usage: ./check-real-recipes.sh [status|report|start|stop]
|
||||
# status — quick count + log tail
|
||||
# report — count, delta, target met, and whether scraper is running
|
||||
# start — start v4 scraper if below target and not already running
|
||||
# stop — stop any running scraper process
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
LOG="${HOME}/.hermes/profiles/nutrition-coach/scraper.log"
|
||||
OUT="${HOME}/.hermes/profiles/nutrition-coach/scraper_v4.out"
|
||||
DB="${HOME}/.hermes/profiles/nutrition-coach/real_recipes.jsonl"
|
||||
TARGET=25000
|
||||
|
||||
count=$(wc -l < "$DB" 2>/dev/null || echo 0)
|
||||
delta=$((TARGET - count))
|
||||
if [ "$count" -ge "$TARGET" ]; then
|
||||
met="YES"
|
||||
else
|
||||
met="NO"
|
||||
fi
|
||||
pct=$(( count * 100 / TARGET ))
|
||||
is_running=$(pgrep -f chefkoch_scraper_v4.py >/dev/null && echo "yes" || echo "no")
|
||||
|
||||
case "${1:-status}" in
|
||||
status|report)
|
||||
printf "Real recipe DB: %d / %d recipes (%d%%)\n" "$count" "$TARGET" "$pct"
|
||||
printf "Delta: %d | Target met: %s | Scraper running: %s\n" "$delta" "$met" "$is_running"
|
||||
if [ -f "$OUT" ]; then
|
||||
printf "\n--- Scraper out tail ---\n"
|
||||
tail -n 5 "$OUT"
|
||||
fi
|
||||
if [ -f "$LOG" ]; then
|
||||
printf "\n--- Log tail ---\n"
|
||||
tail -n 5 "$LOG"
|
||||
fi
|
||||
;;
|
||||
count)
|
||||
echo "$count"
|
||||
;;
|
||||
start)
|
||||
if [ "$count" -ge "$TARGET" ]; then
|
||||
echo "Target $TARGET already reached."
|
||||
exit 0
|
||||
fi
|
||||
if [ "$is_running" = "yes" ]; then
|
||||
echo "Scraper already running."
|
||||
exit 0
|
||||
fi
|
||||
echo "Starting v4 mass scraper..."
|
||||
cd "${HOME}/.hermes/profiles/nutrition-coach" || exit 1
|
||||
export PYTHONUNBUFFERED=1
|
||||
export PLAYWRIGHT_BROWSERS_PATH="${HOME}/.cache/ms-playwright"
|
||||
nohup python3 scripts/chefkoch_scraper_v4.py --no-agent >> "$OUT" 2>&1 &
|
||||
pid=$!
|
||||
echo "PID: $pid"
|
||||
;;
|
||||
stop)
|
||||
pkill -f chefkoch_scraper_v4.py && echo "Stopped scraper" || echo "Scraper not running"
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/bin/bash
|
||||
# Start Chefkoch Scraper v4 safely as background daemon
|
||||
# Place this in ~/.hermes/scripts/ and chmod +x
|
||||
LOG="/home/debian/.hermes/profiles/nutrition-coach/scraper_v4.out"
|
||||
PIDFILE="/tmp/chefkoch_scraper_v4.pid"
|
||||
|
||||
# Check if already running
|
||||
if [ -f "$PIDFILE" ] && kill -0 "$(cat $PIDFILE)" 2>/dev/null; then
|
||||
echo "Scraper already running (PID $(cat $PIDFILE))"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cd /home/debian/.hermes/profiles/nutrition-coach || exit 1
|
||||
|
||||
export PYTHONUNBUFFERED=1
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
|
||||
nohup python3 /home/debian/.hermes/scripts/chefkoch_scraper_v4.py > "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo $PID > "$PIDFILE"
|
||||
echo "Scraper started (PID $PID). Log: $LOG"
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
# Start Chefkoch Scraper v4 safely using profile .venv + profile-local fixed script
|
||||
# Place in ~/.hermes/scripts/ and chmod +x
|
||||
LOG="/home/debian/.hermes/profiles/nutrition-coach/scraper_v4.out"
|
||||
PIDFILE="/tmp/chefkoch_scraper_v4.pid"
|
||||
|
||||
# Check if already running
|
||||
if [ -f "$PIDFILE" ] && kill -0 "$(cat $PIDFILE)" 2>/dev/null; then
|
||||
echo "Scraper already running (PID $(cat $PIDFILE))"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cd /home/debian/.hermes/profiles/nutrition-coach || exit 1
|
||||
|
||||
export PYTHONUNBUFFERED=1
|
||||
export PLAYWRIGHT_BROWSERS_PATH=/home/debian/.cache/ms-playwright
|
||||
|
||||
nohup /home/debian/.hermes/profiles/nutrition-coach/.venv/bin/python3 \
|
||||
/home/debian/.hermes/profiles/nutrition-coach/scripts/chefkoch_scraper_v4.py \
|
||||
--no-agent >> "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo $PID > "$PIDFILE"
|
||||
echo "Scraper started (PID $PID). Log: $LOG"
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"dominik_feeling": "string (optional)",
|
||||
"sarah_mood": "string (optional)",
|
||||
"special_events": "string — events trigger plan overrides: 'grillen', 'picknick', 'reise', 'besuch', 'konzert'",
|
||||
"dominik_ho": ["Montag", "Mittwoch", "..."],
|
||||
"sarah_ho": ["Dienstag", "..."],
|
||||
"used_recipes_in_session": [],
|
||||
"kitchen_status": {
|
||||
"fridge": "string",
|
||||
"leftovers": "string",
|
||||
"speed": "normal | schnell | langsam"
|
||||
},
|
||||
"tagesgefuehl": "string — free-text mood, may contain weather/craving hints",
|
||||
"cravings": "comma-separated keywords, e.g. 'asiatisch, leicht, schnell'",
|
||||
"impact": "1-10 or descriptive string",
|
||||
"kitabox_typ": "süß | herzhaft | gemischt"
|
||||
}
|
||||
+118
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Template: Chefkoch Nightly Scraper (async Playwright)
|
||||
|
||||
Scrapes recipe metadata from Chefkoch.de via headless browser.
|
||||
Intended to be customized per search query and run as background job.
|
||||
|
||||
Prerequisite: playwright install chromium
|
||||
"""
|
||||
import asyncio, json, re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
# ─── Config ─────────────────────────────────────────────────────────────────
|
||||
QUERY = "asiatisch einfach familie"
|
||||
MAX_RESULTS = 10
|
||||
OUTPUT_DIR = Path.home() / "Documents/Essensplanung"
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
OUTPUT = OUTPUT_DIR / f"scrape_{re.sub(r'[^a-z0-9]', '_', QUERY.lower())}_{datetime.now().strftime('%Y%m%d')}.json"
|
||||
|
||||
# ─── Playwright ─────────────────────────────────────────────────────────────
|
||||
|
||||
async def accept_cookies(page):
|
||||
try:
|
||||
await page.click("button[title='Zustimmen']", timeout=2000)
|
||||
except:
|
||||
pass
|
||||
|
||||
async def scrape_page(browser, query, max_results):
|
||||
results = []
|
||||
url = f"https://www.chefkoch.de/rs/s0/{query.replace(' ', '%20')}/Rezepte.html"
|
||||
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until="networkidle", timeout=30000)
|
||||
await accept_cookies(page)
|
||||
|
||||
# Scrolle bis genug Karten geladen
|
||||
for _ in range(15):
|
||||
cards = await page.query_selector_all(".ds-recipe-card")
|
||||
if len(cards) >= max_results * 2:
|
||||
break
|
||||
await page.evaluate("window.scrollBy(0, 800)")
|
||||
await asyncio.sleep(1.5)
|
||||
|
||||
# Extrahiere Links
|
||||
links = await page.eval_on_selector_all(
|
||||
"a.ds-recipe-card__title-link",
|
||||
"els => [...new Set(els.map(e => e.href))]"
|
||||
)
|
||||
|
||||
print(f" {len(links)} Links gefunden")
|
||||
|
||||
# Lade Details
|
||||
for link in list(dict.fromkeys(links))[:max_results]:
|
||||
detail = await get_recipe_detail(browser, link)
|
||||
if detail:
|
||||
results.append(detail)
|
||||
print(f" + {detail['name'][:40]}")
|
||||
await asyncio.sleep(0.5)
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
return results
|
||||
|
||||
async def get_recipe_detail(browser, url):
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until="networkidle", timeout=15000)
|
||||
await accept_cookies(page)
|
||||
|
||||
# Name
|
||||
name = await page.eval_on_selector("h1", "e => e?.innerText.trim() || 'Unbekannt'")
|
||||
|
||||
# Zutaten
|
||||
ings = []
|
||||
rows = await page.query_selector_all("table.ingredients tbody tr")
|
||||
for row in rows[:15]:
|
||||
try:
|
||||
amt = await row.eval_on_selector("td.amount", "e => e.innerText.trim()")
|
||||
item = await row.eval_on_selector("td.name", "e => e.innerText.trim()")
|
||||
if item: ings.append({"amount": amt, "item": item})
|
||||
except: pass
|
||||
|
||||
# Zeit
|
||||
time = await page.eval_on_selector(
|
||||
".ds-recipe-meta__icon-text, .bi-recipe-meta-duration",
|
||||
"e => e?.innerText.trim() || '?'"
|
||||
)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"url": url,
|
||||
"ingredients": ings,
|
||||
"prep_time_text": time,
|
||||
}
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Fehler: {e}")
|
||||
return None
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
async def main():
|
||||
print(f"🔍 Scrape: '{QUERY}' → {MAX_RESULTS}x")
|
||||
|
||||
async with async_playwright() as p:
|
||||
browser = await p.chromium.launch(headless=True)
|
||||
results = await scrape_page(browser, QUERY, MAX_RESULTS)
|
||||
await browser.close()
|
||||
|
||||
OUTPUT.write_text(json.dumps(results, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f"💾 {len(results)} Rezepte → {OUTPUT}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user