# Recipe Data Sources (Tested 2026-06-28)

## Active Sources (v7 Scraper — Direct MySQL)

### Chefkoch.de — ~300k+ total, ~25k scraped
- **Access:** REST API v2 (`https://api.chefkoch.de/v2/recipes`)
- **Method:** Query rotation across 100+ German cooking terms, pagination via offset
- **Structured data:** API returns JSON with title, rating, ingredients, tags, times
- **Anti-scraping:** Rate-limited but no Cloudflare; 0.3-0.5s delay sufficient
- **Field mapping:** `id`, `title`, `category`, `tags`, `rating`, `prepTime`, `cookTime`, `totalTime`, `ingredients[].raw`, `siteUrl`, `nutrition.calories`

### Kuechengoetter.de — 30,000 recipes
- **Access:** Sitemap (`https://www.kuechengoetter.de/sitemaps/recipes-1.xml`)
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
- **Structured data:** Schema.org Recipe in `<script type="application/ld+json">`
- **Fields extracted:** name, aggregateRating.ratingValue, prepTime, cookTime, totalTime, recipeIngredient[], recipeCategory, recipeInstructions[], keywords→tags
- **Anti-scraping:** None observed; polite delays (0.3-1.0s) work fine
- **Quality:** Ratings present but sparse (many show rating=1, meaning single vote). Ingredient lists well-structured.

### Kitchen Stories (kitchenstories.com/de) — 3,390 German recipes
- **Access:** Sitemap (`https://www.kitchenstories.com/sitemap.xml`)
- **Filter:** `/de/rezepte/` (German recipe pages only; site has EN/DE bilingual)
- **Method:** Sitemap discovery → individual page fetch → JSON-LD extraction
- **Structured data:** Schema.org Recipe, 3 JSON-LD blocks per page (Recipe + WebSite + BreadcrumbList)
- **Fields:** name, totalTime (ISO 8601 like PT20M), recipeIngredient[], recipeCategory, recipeInstructions[]
- **Anti-scraping:** None observed
- **Quality:** High-quality editorial recipes. No ratings in JSON-LD. Good metadata (times, categories).

### HelloFresh.de — ~16,831 total, ~10,000 estimated unique, 555 already scraped
- **Access:** Category pages via `__NEXT_DATA__` (Next.js SSR hydration blob)
- **Listing URL pattern:** `https://www.hellofresh.de/recipes/{category}?skip={N}&take=50`
- **Detail URL pattern:** `https://www.hellofresh.de/recipes/{slug}-{recipeId}`
- **Method:** Enumerate categories → paginate each → fetch detail per recipe → parse `__NEXT_DATA__`
- **Structured data:** NOT JSON-LD. Next.js `__NEXT_DATA__` script tag containing `ssrPayload.dehydratedState.queries[]`
  - Listing: find query with `queryKey[0] == 'recipe.search'` → `state.data.items[]` (each has `id`, `slug`, `name`)
  - Detail: find query with `queryKey[0] == 'foodContentHubRecipe.byId'` → `state.data.recipe`
- **Normalization:** Custom `_hf_normalize()` — builds ingredients from `yields[0].ingredients[]` joined with `ingredients[]` lookup, instructions from `steps[].instructionsMarkdown`, calories from `nutrition[]` where name contains 'kcal'
- **Working categories (19/28):** familien-rezepte (6,208), vegetarische-rezepte (4,625), einfache-rezepte (1,444), gesunde-rezepte (1,395), eintopf-rezepte (934), italienische-rezepte (481), deutsche-rezepte (463), asiatische-rezepte (185), amerikanische-rezepte (172), orientalische-rezepte (167), kartoffel-rezepte, mediterrane-rezepte, mexikanische-rezepte, nudel-rezepte, suppen, salate, burger, chinesische-rezepte, flammkuchen-rezepte
- **Dead categories (404):** belibteste-rezepte, pancakes, pasta, pizza, reis-rezepte, sandwich, fleisch-rezepte, franzosische-rezepte, griechische-rezepte
- **Anti-scraping:** None observed; Referer header recommended
- **Quality:** HIGHEST of all sources. 99% have calories, 93% have ratings, all have `instructions`, `description`, `difficulty`, `servings`. Professionally curated with exact quantities.
- **ID prefix:** `hf_` (NOT `hellofresh_`) — must match v5.3 legacy IDs for deduplication
- **Headers:** Requires `Referer: https://www.hellofresh.de/` and standard browser UA

### GuteKueche.de — 29,392 recipes (INTEGRATED in v6)
- **Access:** Gzipped sitemap (`https://cdn.gutekueche.de/sitemaps/recipe.xml.gz`)
- **Discovery:** robots.txt → `Sitemap:` directive led to CDN-hosted gzipped sitemap index → 4 sub-sitemaps, `recipe.xml.gz` has 29,392 recipe URLs
- **Method:** Same as Kuechengoetter/Kitchen Stories — sitemap discovery → JSON-LD extraction. Scraper auto-detects gzip via magic bytes `\x1f\x8b` or config flag `"gzipped": True`.
- **Config:** Added to `SITEMAP_SOURCES` dict with `"gzipped": True` flag. `discover_sitemap_urls()` enhanced with gzip support (urllib + `gzip.decompress()`).
- **URL pattern:** `https://www.gutekueche.de/<name>-rezept-<id>`
- **Structured data:** Schema.org Recipe in JSON-LD, **9/11 quality fields** (highest of all sitemap sources)
- **Fields present:** nutrition (calories ✅), recipeInstructions ✅, aggregateRating ✅ (e.g. 4.5★), totalTime, recipeIngredient, recipeCategory, recipeYield, prepTime, cookTime
- **Missing:** recipeCuisine, suitableForDiet
- **Anti-scraping:** None; robots.txt allows all (only disallows /user/, /suche, /api/)
- **Quality:** Strong — professional editorial recipes with calorie data and ratings. Second-best overall quality after HelloFresh.

## Tested but Rejected

| Source | Reason |
|--------|--------|
| AllRecipes.com | HTTP 403 Forbidden (Cloudflare) |
| Yummly.com | HTTP 403 Forbidden |
| Tasty.co | HTTP 404 on sitemap |
| Marmiton.org | HTTP 404 on sitemap (French, URL structure changed) |
| EatThis.org | SSL certificate verification failed |
| Lecker.de | Sitemap XML files return 0 URLs (likely gzipped or JS-rendered) |
| Essen-und-trinken.de | **Has Recipe JSON-LD on recipe pages** (6/11 fields, calories ✅, ratings ✅), BUT robots.txt blocks ALL AI bots (ClaudeBot, ChatGPT, Claude-Web, CCBot, etc.). Respect robots.txt — do not scrape. |
| Maggi.de | HTTP 403 Forbidden |
| Knorr.de | HTTP 403 Forbidden |
| Serious Eats | HTTP 403 Forbidden |
| Dr. Oetker | No JSON-LD on recipe pages |
| Kochbar.de | No JSON-LD |
| Marley Spoon | No JSON-LD (JS-rendered) |
| Frauenzimmer.de | JSON-LD present but no Recipe type on recipe pages |
| Bunte.de | 404 on recipe page URLs (structure unclear) |

## Storage: MariaDB Galera Cluster

Recipe data migrated from JSONL to the existing Galera cluster (2026-06-28).

- **Why Galera over SQLite:** User preference — reuse existing HA infrastructure (3-node cluster already running for HA Recorder) instead of adding new dependencies. Concurrent read/write via MaxScale readwritesplit.
- **Connection:** `10.0.30.70:3306` (MaxScale VIP) → Galera cluster (db1=.71, db2=.72, db3=.73)
- **Credentials:** DB=`recipes`, User=`recipe_app@%`, password in migration script
- **Table schema:** 20 columns, InnoDB, utf8mb4_unicode_ci
- **Indexes:** FULLTEXT ft_title_desc (title, description), FULLTEXT ft_ingredients (ingredients), BTREE idx_source, idx_category, idx_rating, idx_difficulty
- **Migration script:** `scripts/migrate_to_galera.py` — INSERT IGNORE with batch=200, handles dict-valued `category` and `nutrition` fields, skips existing IDs
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. Migration script must coerce these to strings/extract values before INSERT.
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
- **Architecture:** Scraper writes directly to Galera via `INSERT IGNORE` (no JSONL intermediate). v7 eliminated the JSONL → migration → DB pipeline. JSONL (`real_recipes.jsonl`) is kept as historical archive only.
- **Pitfall:** `category` field is a dict (not string) in 184 Chefkoch records, `nutrition` is a dict in 5004 records. `recipe_to_row()` in v7 coerces these to strings/extracts values before INSERT.
- **Pitfall:** Galera replication overhead makes bulk inserts slower than local SQLite. Batch size 200 is the sweet spot — larger batches risk wsrep replication lag timeouts.
- **Pitfall:** Per-row `db_has_id()` dedup is catastrophically slow through MaxScale — each check is a network roundtrip. v7 loads all IDs into a Python `set()` at startup via `db_all_ids()` (~0.3s for 28k), then all dedup is in-memory. Additionally, derive the likely recipe ID from the URL path and skip the HTTP fetch entirely if already known.

## Potential Future Sources (not yet integrated)

| Source | Recipes | Access | Notes |
|--------|---------|--------|-------|
| Food.com | Large | JSON-LD confirmed on recipe pages | English; sitemap index exists |
| BBC Good Food | ~300/quarter | Sitemap recipe files, JSON-LD confirmed | English; premium content mixed in |
| HelloFresh.de | ~~548 scraped~~ → **now active in v6** (see above) | Was thought exhausted; actually has ~16,831 recipes |

## Recipe Source Scanning Methodology

When evaluating new recipe sources for the scraper:

1. **Don't judge by landing page alone.** Landing/listing pages rarely embed Recipe JSON-LD — they have WebSite or ItemList schemas. Always fetch an **individual recipe page** and check for `@type: "Recipe"` in JSON-LD blocks.
2. **Check `robots.txt` first.** Many sites declare their sitemap location there (`Sitemap: <url>`). Standard paths like `/sitemap.xml` frequently 404. Also check for AI-bot blocks — respect them.
3. **Sitemaps may be gzipped.** Look for `.gz` extensions or `Content-Encoding: gzip`. Use `gzip.decompress()` in Python.
4. **Volume estimation from sitemap index.** Fetch the index, count sub-sitemaps, sample 5-10 sub-sitemaps, extrapolate. Filter URLs by path patterns (`/rezept-`, `/rezepte/`, `/recipes/`).
5. **Quality scoring (11-field checklist):** Count presence of: `nutrition`, `recipeInstructions`, `aggregateRating`, `totalTime`, `recipeIngredient`, `recipeCategory`, `recipeYield`, `prepTime`, `cookTime`, `recipeCuisine`, `suitableForDiet`. Score = present/11. HelloFresh ≈ 11/11 (custom parsing), GuteKueche ≈ 9/11, Kuechengoetter ≈ 6/11.
6. **Quality ≠ Volume.** A source with 548 high-quality recipes (calories, instructions, ratings, difficulty) may be more valuable than a 30k source with sparse metadata. Evaluate both dimensions. **Never dismiss a source solely for low volume without assessing data quality.**

## v7 Scraper Architecture (Direct MySQL — replaces v6)

v7 eliminates JSONL entirely. Writes directly to MariaDB Galera via PyMySQL.

```
recipe_scraper_v7.py
├── Phase 1: Sitemap sources (Kuechengoetter → GuteKueche → Kitchen Stories)
│   ├── discover_sitemap_urls() — fetch sitemap (auto-gzip), extract <loc> URLs
│   ├── scrape_sitemap_source() — visit each URL, extract JSON-LD
│   └── normalize_jsonld_recipe() — Schema.org → our format
├── Phase 2: HelloFresh (__NEXT_DATA__ extraction)
│   ├── _hf_fetch_category_page() — paginate categories via skip/take
│   ├── _hf_fetch_detail() — fetch recipe page, parse __NEXT_DATA__
│   ├── _hf_normalize() — convert HF nested structure to our format
│   └── Per-category skip tracking via state["hf_cat_offset"]
├── Phase 3: Chefkoch API (fills remaining gap to 100k)
│   ├── fetch_ck_page() — API query with rotation
│   └── fetch_ck_detail() — full recipe by ID
├── State: scraper_state.json (tracks tried URLs, query cycles, source counts, hf_cat_offset)
├── Dedup: db_all_ids() loads ALL IDs into Python set() at startup (~0.3s for 28k)
│   └── Pre-check: derives likely ID from URL path, skips HTTP fetch if already in set
├── Insert: INSERT IGNORE per recipe (idempotent, safe for re-runs)
├── Output: Direct to MariaDB Galera (10.0.30.70:3306, DB=recipes)
├── Cronjob: recipe-refill-v7 (daily 06:00, deliver: origin)
└── Target: 100,000 recipes
```

**Key differences from v6:**
- No JSONL file, no migration step — single pipeline
- In-memory dedup via `db_all_ids()` instead of per-row `db_has_id()` (avoids catastrophic Galera roundtrip latency)
- URL pre-check skips HTTP fetch entirely for known recipes
- Progress logging every 200 URLs
- `recipe_to_row()` sanitizes dict/list fields (category, calories) before INSERT

**Separate step — Embeddings:**
After scraping, run `generate_embeddings.py` to populate the `embedding` JSON column with 1024-dim harrier vectors. This is NOT done by the scraper.

**Output format** (unified across all sources, stored in `recipes` table):
```json
{
  "id": "kuechengoetter_seelachs-mit-rucola-1",
  "name": "Seelachs mit Rucola",
  "title": "Seelachs mit Rucola",
  "category": "fish",
  "tags": ["fisch", "seelachs", "rucola"],
  "rating": 4.5,
  "rating_value": 4.5,
  "prep_time": "PT15M",
  "cook_time": "PT20M",
  "total_time": "PT35M",
  "ingredients": [{"raw": "300g Seelachsfilet"}, ...],
  "source_url": "https://...",
  "source": "kuechengoetter",
  "calories": "450 kcal"
}
```