hermes-skills/productivity/visual-qa/SKILL.md

name, description

name	description
visual-qa	Automated visual QA of rendered documents (slides, PDFs, diagrams, screenshots). Detect overlap, overflow, contrast issues by sending images to a vision-capable model. Trigger on 'QA', 'review slides', 'check presentation', 'visual inspection', or when converting documents to images for review.

Visual QA Skill

Automated visual quality assurance of rendered documents — presentations, PDFs, diagrams, screenshots. Sends images to a vision-capable LLM and gets structured feedback on layout issues.

When to Use

• After generating or editing a .pptx deck (pair with the powerpoint skill)
• After rendering PDFs, diagrams, or HTML artifacts to images
• Before delivering visual content to a user
• Any time you need "fresh eyes" on rendered output

Workflow

1. Render to images (if not already done):

   # PPTX → PDF → JPEGs
   soffice --headless --convert-to pdf input.pptx
   pdftoppm -jpeg -r 150 input.pdf slide
   # Produces slide-01.jpg, slide-02.jpg, etc.
   

2. Verify a vision-capable model exists on your endpoint (see references/model-capabilities.md):

   curl -s <base_url>/models -H "Authorization: Bearer <key>" | python3 -m json.tool | grep -i model
   

   Not all models accept images — test with one image first.

3. Run batch QA — use the batch script for efficiency:

   NORIS_API_KEY=<key> python scripts/batch_vision_qa.py /path/to/slides/ <num_slides>
   

4. Fix issues found, re-render affected slides, re-verify.

Key Techniques

Direct API Calls for Batch Processing

For QA of many images, calling the vision model API directly is far more efficient than individual vision_analyze tool calls. A 17-slide deck processes in ~22 seconds via batch API vs. 17 sequential tool calls.

Pattern (Python with urllib.request):

import urllib.request, json, base64

with open(slide_path, "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

payload = {
    "model": "<vision_model>",
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "<QA prompt>"},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}
    ]}],
    "max_tokens": 300
}

req = urllib.request.Request(
    "<base_url>/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Authorization": "Bearer <key>", "Content-Type": "application/json"}
)
resp = urllib.request.urlopen(req, timeout=60)
result = json.loads(resp.read())["choices"][0]["message"]["content"]

Structured QA Prompt

Use a prompt that forces structured JSON output for easy parsing. The base prompt checks overlap, overflow, and contrast:

Analyze this slide for: 1) TEXT OVERLAP, 2) TEXT OVERFLOW, 3) CONTRAST issues.
Respond as JSON: {"slide": N, "overlap": "none|<desc>", "overflow": "none|<desc>", "contrast": "ok|<desc>", "overall": "clean|issues found"}

For comprehensive QA, expand the prompt to also check symbols and content consistency:

Analyze this slide for: 1) TEXT OVERLAP, 2) TEXT OVERFLOW, 3) CONTRAST, 4) LAYOUT (misalignment, spacing, margins),
5) CONTENT (unclear messaging, placeholder text, spelling, logical inconsistencies), 6) SYMBOLS (broken/missing
Unicode characters showing as boxes or empty circles).
Respond as JSON: {"slide": N, "overlap": "...", "overflow": "...", "contrast": "...", "layout": "...",
"content": "...", "symbols": "...", "overall": "clean|issues found"}

Two-Tier QA Strategy

A single batch pass with the base prompt catches obvious layout issues but misses rendering defects and content problems that require closer inspection. Use a two-tier approach:

1. Tier 1 — Batch scan (breadth): Send all slides through the batch script with the expanded prompt above. Identifies which slides have potential issues.

2. Tier 2 — Targeted deep-dive (depth): For slides flagged in Tier 1 (or slides with complex elements like diagrams, badges, special characters), use individual vision_analyze calls with element-specific questions:

   • "Do the Unicode symbols ∅, ≡, ⚙ inside the circles render correctly or show as boxes?"
   • "Are the connecting lines between neurons visible? What color are they?"
   • "Does the example value '42' match the description 'Temperatur = 23°C'?"
   • "Is there adequate padding between the text 'API' and the circle edge?"

Targeted prompts catch what generic prompts miss — the vision model focuses on the specific element rather than scanning the whole slide.

Model Capability Detection

Before batch QA, verify the target model accepts image input. Some models return 400: At most 0 image(s) or claim "I cannot see images" even with proper image_url format. Test with a single image first.

See references/model-capabilities.md for tested models.

Config: vision_analyze Tool

The vision_analyze tool reads model config from TWO places in ~/.hermes/config.yaml:

• auxiliary.vision.* — provider, model, base_url, api_key
• Top-level vision.* — model, base_url, api_key (overrides auxiliary)

Both must have the correct model name (including any release/ prefix) and non-empty base_url + api_key. Use hermes config set to update:

hermes config set auxiliary.vision.model "vllm/release/<model>"
hermes config set auxiliary.vision.base_url "https://<endpoint>/v1"
hermes config set auxiliary.vision.api_key "<key>"
hermes config set vision.model "vllm/release/<model>"
hermes config set vision.base_url "https://<endpoint>/v1"
hermes config set vision.api_key "<key>"

Changes require a session restart to take effect in the vision_analyze tool. For immediate use without restart, call the API directly (see above).

Corporate Template Adoption QA

When a user provides a .pptx template and asks to rebuild a deck to match its design, the QA phase must verify design conformance in addition to layout correctness:

1. Template extraction QA — render template slides to JPEGs and inspect with vision_analyze to understand: background color, card style (dark/light, shadows, corners), accent bar style, logo placement, typography, decorative elements.
2. Theme extraction — parse ppt/theme/theme1.xml via zipfile + xml.etree.ElementTree (not python-pptx's theme API, which can raise AttributeError). Extract <a:srgbClr> values and <a:latin typeface="..."> font names.
3. Embedded image inspection — extract images from ppt/media/ via zipfile, inspect each with vision_analyze. Logos may be white-on-transparent (invisible on white backgrounds). If unusable, build a text+shape logo instead.
4. Post-rebuild QA prompt — add design-conformance checks to the standard QA prompt:

   Check this slide for: 1) noris corporate design (white bg, light grey cards #F2F2F2, 
   petrol #234D5B accents, Open Sans font) 2) logo visible top right 3) no text overlaps 
   4) no layout issues 5) readable text. Reply CLEAN or list ISSUES.
   

5. Non-ASCII quote pitfall in source code — German curly quotes („…" / »…«) cause SyntaxError: Unexpected identifier in Node.js string literals. Use \u201E (left double) and \u201C (right double) Unicode escapes instead. The write_file linter catches this but only if the quotes land inside a JS string.
6. Logo extraction — use the REAL logo, not a text approximation — When a user provides a .pptx template and asks to use its design, they expect the actual corporate logo image, not a reconstructed text+shape approximation. Identify the logo by counting image references across slide _rels/*.xml.rels files (logo appears on 20+ slides). Pitfall: the most-referenced image may be an EMF (unusable in LibreOffice/pptxgenjs) — check for PNG/JPG alternatives. Multi-part logos (two color variants in one image) require pixel-alpha scanning to find the split point and selective inversion for dark-background variants. See references/template-adoption.md §6 for the full workflow.

See references/template-adoption.md for the full extraction workflow with code snippets, including §10 (inserting new slides into an existing deck — module organization, renumbering, agenda/summary updates).

Pitfalls

• Model name prefixes matter: vllm/gemma-4-31b-it ≠ vllm/release/gemma-4-31b-it. Wrong prefix → "no keys found that support model" error.
• Empty base_url/api_key in auxiliary.vision: Even if top-level vision.* is correct, empty auxiliary fields can cause failures. Set both sections.
• Not all LLMs are multimodal: A model serving text completions may reject image inputs. Always test one image first.
• vision_analyze caches config: After hermes config set, the running session may still use old values. Direct API calls bypass this.
• Rate limiting: Add 0.5s sleep between batch calls to avoid throttling.
• Unicode symbols silently fail in LibreOffice PDF rendering: Characters like ∅, ≡, ⚙, ★, ✓, Σ, Ω, → may look fine in PowerPoint but render as blank circles or broken boxes when exported to PDF via soffice --headless. Since visual QA relies on PDF→JPEG conversion, these defects are visible in QA images but the generic batch prompt may not flag them — it sees "a circle with something in it" and passes it. Always use targeted Tier 2 prompts that explicitly ask "do the symbols inside the circles render correctly?" for slides with badges or icon placeholders.
• Content consistency requires targeted prompts: A batch prompt won't catch logical inconsistencies like "example shows 42 but description says 23°C" unless the prompt explicitly asks about content consistency. Add content checks to the expanded prompt and follow up with targeted questions.
• PowerPoint rejects negative shape dimensions/positions — LibreOffice doesn't: pptxgenjs silently emits negative cx/cy (extent) or x/y (offset) values in the OOXML when LINE shapes connect nodes bottom-to-top or right-to-left, or when decorative OVALs are placed partially off-slide (x: -2, y: -2). LibreOffice opens these files fine and the PDF→JPEG render looks correct, so visual QA passes — but PowerPoint refuses to open the file. After generating a PPTX, always run scripts/validate_pptx_compat.py <file.pptx> to catch negative extents/offsets before delivery. Fix in source: use Math.abs() for dimensions, Math.min() for positions, and clamp decorative shapes to non-negative coordinates. See references/template-adoption.md §9.

Narrative Structure Review

After creating or substantially expanding a deck, review the narrative flow ("roter Faden") before delivery. A deck can be visually clean but structurally confused.

Consolidation Rules

• Max 8 agenda points — beyond that, the audience loses the thread. Group related slides under one agenda item.
• Max 5 takeaways — distill into the fewest memorable points. Merge related ones (e.g., "ML learns from data" + "parameters store knowledge" → one takeaway).
• One concept per slide — if a slide teaches two things, split it or merge into a progression.

Flow Audit Steps

1. List all slides with titles and the concept they teach.
2. Identify redundancy: slides repeating the same idea at different depths (e.g., "KI ist nicht intelligent" as both slide 3 and slide 17).
3. Identify interruptions: deep-dives that break the narrative arc (e.g., tensor dimensionality between neural networks and LLMs).
4. Check proximity of linked concepts: Attention-Mechanismus and KV-Cache are technically coupled — 7 slides apart breaks the connection. Keep linked concepts adjacent.
5. Propose merges: combine slides teaching the same concept (e.g., MoE architecture + Dense-vs-MoE comparison → one slide).
6. Propose cuts: slides that don't advance the narrative.

Output Format

Present as a table mapping new agenda items to current slides with actions (Merge/Consolidate/Keep/Cut), then list ≤5 proposed takeaways showing which current points they unify.

Pitfall: Expanding Without Consolidating

Adding content slides to an existing deck is common. Each addition shifts slide numbers, grows the agenda, and lengthens the summary. After expansion, always run a flow audit and propose consolidation — a 23-slide deck with 14 agenda points and 12 takeaways is harder to follow than 18 slides with 8 and 5.

Files

File	Purpose
scripts/batch_vision_qa.py	Batch QA script — sends all slides to vision model, reports issues
scripts/validate_pptx_compat.py	PPTX structural validation — catches negative dimensions/positions that LibreOffice tolerates but PowerPoint rejects
references/model-capabilities.md	Tested model vision capabilities (vLLM/Noris)
references/iterative-fix-verify-loop.md	Multi-cycle QA pattern: fix → re-render → re-QA until zero defects
references/template-adoption.md	Corporate template extraction: theme colors, fonts, logo identification & multi-part processing (§6), design-system rebuild

Related Skills

• powerpoint — PPTX creation/editing; this skill handles the visual QA step of its workflow