hermes-skills/productivity/visual-qa/references/model-capabilities.md

Vision Model Capabilities (vLLM / Noris)

Tested against https://ai.noris.de/v1 — OpenAI-compatible vLLM endpoint.

Multimodal (Accepts Images) ✅

Model	Notes
vllm/release/gemma-4-31b-it	Works. Despite not being labeled "vision", accepts image_url content type and describes images accurately. Good for slide QA.

Text-Only (Rejects Images) ❌

Model	Error	Notes
vllm/release/qwen3.6-27b	400: At most 0 image(s) may be provided in one prompt	Explicitly rejects image input
vllm/release/qwen3.6-35b-a3b	400: Input should be a valid string	Rejects multimodal content format
vllm/release/gpt-oss-120b	Returns "I'm not able to view images"	Accepts the request but model itself can't process images
vllm/release/glm-5-2	Not tested for images	Primary chat model — assumed text-only

Testing Procedure

Send a single image with a simple "describe this image" prompt:

import urllib.request, json, base64

with open("test.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

payload = {
    "model": "<model_to_test>",
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "Describe this image in 1 sentence."},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}
    ]}],
    "max_tokens": 80
}

req = urllib.request.Request(
    "https://ai.noris.de/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Authorization": "Bearer <key>", "Content-Type": "application/json"}
)
resp = urllib.request.urlopen(req, timeout=30)
print(json.loads(resp.read())["choices"][0]["message"]["content"])

If the response describes the image content → multimodal. If it apologizes about not seeing images or returns HTTP 400 → text-only.

Config Gotcha

The auxiliary.vision section in ~/.hermes/config.yaml must have:

• Correct model name WITH prefix (e.g. vllm/release/gemma-4-31b-it, not vllm/gemma-4-31b-it)
• Non-empty base_url and api_key

If these are empty or wrong, vision_analyze fails with "no keys found that support model" — even if the top-level vision.* section is correct. Both sections must be consistent.