# Semantic Search + Preference Learning — Implementation

Date: 2026-06-28
Status: **Implemented and tested**

## Embedding Model

**Model:** `vllm/harrier-oss-v1-0.6b` (noris)
- 1024 dimensions, multilingual (German-capable)
- Dedicated embedding model (NOT an LLM), 0.6B params
- Endpoint: `POST https://ai.noris.de/v1/embeddings`
- Auth: Same noris API key from `config.yaml`
- Batch: Up to 64 inputs per call, ~31 recipes/sec throughput

### Discovery

Initial investigation found that norris does NOT serve embeddings via standard LLM models (glm, gemma, qwen all return 404). The model list included `vllm/harrier-oss-v1-0.6b` which was the only embedding model that actually works. Other embedding names (bge-m3, e5-large, gte-large) are recognized by the Bifrost gateway but not activated on any API key.

### Quality Validation

Cosine similarity matrix (higher = more similar):

|              | Curry | Salat | Eintopf | Hähnchen | Wintergericht |
|-------------|-------|-------|---------|----------|---------------|
| Curry        | 1.000 | 0.647 | 0.677   | **0.812** | 0.660         |
| Salat        | 0.647 | 1.000 | 0.715   | 0.667    | 0.661         |
| Eintopf      | 0.677 | 0.715 | 1.000   | 0.696    | **0.829**     |
| Hähnchen     | 0.812 | 0.667 | 0.696   | 1.000    | 0.694         |
| Wintergericht| 0.660 | 0.661 | 0.829   | 0.694    | 1.000         |

Curry↔Hähnchen highest (both chicken dishes), Eintopf↔Wintergericht high (both warm/hearty), Salat↔Eintopf lower (different concepts). Working as expected.

## Database Schema

### `recipes.embedding` (JSON column)

Added via `ALTER TABLE recipes ADD COLUMN embedding JSON NULL`.
Stores 1024-dim embedding as JSON array: `[0.023, -0.054, ...]`.

### `meal_decisions` table

```sql
CREATE TABLE meal_decisions (
  id INT AUTO_INCREMENT PRIMARY KEY,
  decision_date DATE NOT NULL,
  recipe_id VARCHAR(255) NOT NULL,
  recipe_title VARCHAR(500),
  recipe_category VARCHAR(200),
  accepted BOOLEAN NOT NULL DEFAULT FALSE,
  context_temp FLOAT NULL,           -- outdoor temp °C
  context_season VARCHAR(20) NULL,   -- spring/summer/autumn/winter
  context_weekday VARCHAR(20) NULL,
  context_last_meals TEXT NULL,       -- JSON array of recent meal categories
  reason TEXT NULL,                   -- free-text rejection reason
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### `preference_rules` table

```sql
CREATE TABLE preference_rules (
  id INT AUTO_INCREMENT PRIMARY KEY,
  rule_type ENUM('context_category','context_tag','variety','semantic') NOT NULL,
  condition_json TEXT NOT NULL,  -- {"temp_min":20, "temp_max":99}
  effect_json TEXT NOT NULL,     -- {"penalty_category":"Eintopf"}
  weight FLOAT DEFAULT 1.0,      -- grows with confirmations (capped at 10)
  confirmations INT DEFAULT 1,
  rejections INT DEFAULT 0,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);
```

## Scripts

All in `profiles/nutrition-coach/scripts/`:

### `generate_embeddings.py`
- Batch-generates embeddings for recipes where `embedding IS NULL`
- Builds embedding text from: `title + category + tags[:10] + ingredients[:20] + description[:200]` (capped at 1000 chars)
- Batch size: 64 recipes per API call
- Rate limit: 0.2s between calls
- Run: `python3 scripts/generate_embeddings.py`
- Should be run after scraper fills new recipes (could be chained in cronjob)

### `semantic_search.py`
Core module with these functions:

#### `semantic_search(query, filters, top_k, min_rating)`
- Gets query embedding from norris
- Loads up to 5000 candidate recipes from DB (ordered by rating DESC)
- Computes cosine similarity in Python
- Blends similarity with rating: `score = similarity * (1 + rating/10)`
- Supports filters: `category`, `source`, `exclude_categories`, `max_calories`

#### `record_decision(recipe_id, accepted, context)`
- Logs to `meal_decisions` table
- Auto-creates preference rule if rejection reason mentions weather/temperature
- Example: `record_decision("hf_123", False, {"temp": 28, "season": "summer", "reason": "zu heiß für Eintopf"})`
- Auto-rule: temp>20°C → penalize category from the rejected recipe

#### `get_preference_adjustments(context)`
- Returns `{boost_categories, penalty_categories, boost_tags, penalty_tags}` based on matching rules
- Each entry includes the rule's weight

#### `rank_with_preferences(recipes, context)`
- Re-ranks a list of recipe dicts by applying:
  - Category penalties/boosts from learned rules (-0.1 × weight per match)
  - Tag penalties/boosts (-0.05 × weight per match)
  - Variety penalty: -0.15 if same category as recent meals
- Checks `category + title + tags` concatenated (not just category field — see pitfall below)

#### `mine_rules(min_confidence=2)`
- Analyzes `meal_decisions` for statistical patterns
- Groups by `(category, temperature_bucket)` and `(category, season)`
- If rejection rate > 50% with enough samples → creates/upgrades rule
- Should run periodically as decisions accumulate

## Tested Examples

### Semantic Search
```
Query: "schnelles Hähnchengericht"
→ Hähnchenbrust in Joghurt-Parmesan-Salbei-Panade (sim: 0.622, ★5.0)
→ Hähnchenschnitzel auf Muttis Art (sim: 0.626, ★4.9)
→ Hühnchenschnitzel (sim: 0.614, ★5.0)
```

### Preference Learning
```
1. record_decision("ck_Erbsensuppe", False, {"temp": 28, "reason": "zu heiß für Eintopf"})
   → Auto-rule created: temp>20°C → penalize "Eintopf" (weight: 1.0)

2. semantic_search("Eintopf mit Kartoffeln", top_k=10)
   → rank_with_preferences(results, {"temp": 28, "season": "summer"})
   
Result: All Eintopf recipes get Δ=-0.2000 (two matching rules × 0.1 weight each)
  0.7365 (Δ-0.2000) | Möhren-Kartoffel-Eintopf (Kochen)
  0.7304 (Δ-0.2000) | Eintopf mit Spitzkohl, Hackfleisch (Kartoffeln)
  0.7226 (Δ-0.2000) | Möhreneintopf alla Isa (Eintopf)
```

## Key Pitfalls

1. **Category field mismatch:** Most "Eintopf" recipes have category "Kochen" or "Kartoffeln", not "Eintopf". Rule matching must check `title + tags + category` as a concatenated lowercase string.

2. **Decimal type from MySQL:** `rating` comes back as `decimal.Decimal`, not `float`. Must cast with `float(rating or 0)` before arithmetic.

3. **Auto-rule with non-existent recipe IDs:** `record_decision("test_reject_1", ...)` creates a rule with empty category because the recipe doesn't exist in the DB. Always use real recipe IDs from the DB.

4. **Duplicate rules from auto-creation + mining:** Both `_auto_create_rule()` and `mine_rules()` can create the same rule. `_upsert_rule()` handles this with matching on `(rule_type, condition_json, effect_json)` and increments weight/confirmations.

## Integration with Meal Planner

The meal planner (`nutrition-plan-generator-v4.py`) should be updated to:
1. Call `semantic_search()` instead of linear JSONL scan
2. Call `rank_with_preferences()` with weather context
3. Call `record_decision()` when user accepts/rejects proposals in Phase 2
4. Periodically call `mine_rules()` to discover new patterns

Context for decisions:
- Temperature: From HA weather sensor or Tibber API
- Season: Derived from date
- Weekday: From date
- Last meals: From previous week's finalized plan