57 lines
2.2 KiB
Markdown
57 lines
2.2 KiB
Markdown
# Vision Model Capabilities (vLLM / Noris)
|
|
|
|
Tested against `https://ai.noris.de/v1` — OpenAI-compatible vLLM endpoint.
|
|
|
|
## Multimodal (Accepts Images) ✅
|
|
|
|
| Model | Notes |
|
|
|-------|-------|
|
|
| `vllm/release/gemma-4-31b-it` | Works. Despite not being labeled "vision", accepts `image_url` content type and describes images accurately. Good for slide QA. |
|
|
|
|
## Text-Only (Rejects Images) ❌
|
|
|
|
| Model | Error | Notes |
|
|
|-------|-------|-------|
|
|
| `vllm/release/qwen3.6-27b` | `400: At most 0 image(s) may be provided in one prompt` | Explicitly rejects image input |
|
|
| `vllm/release/qwen3.6-35b-a3b` | `400: Input should be a valid string` | Rejects multimodal content format |
|
|
| `vllm/release/gpt-oss-120b` | Returns "I'm not able to view images" | Accepts the request but model itself can't process images |
|
|
| `vllm/release/glm-5-2` | Not tested for images | Primary chat model — assumed text-only |
|
|
|
|
## Testing Procedure
|
|
|
|
Send a single image with a simple "describe this image" prompt:
|
|
|
|
```python
|
|
import urllib.request, json, base64
|
|
|
|
with open("test.jpg", "rb") as f:
|
|
img_b64 = base64.b64encode(f.read()).decode()
|
|
|
|
payload = {
|
|
"model": "<model_to_test>",
|
|
"messages": [{"role": "user", "content": [
|
|
{"type": "text", "text": "Describe this image in 1 sentence."},
|
|
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}
|
|
]}],
|
|
"max_tokens": 80
|
|
}
|
|
|
|
req = urllib.request.Request(
|
|
"https://ai.noris.de/v1/chat/completions",
|
|
data=json.dumps(payload).encode(),
|
|
headers={"Authorization": "Bearer <key>", "Content-Type": "application/json"}
|
|
)
|
|
resp = urllib.request.urlopen(req, timeout=30)
|
|
print(json.loads(resp.read())["choices"][0]["message"]["content"])
|
|
```
|
|
|
|
If the response describes the image content → multimodal. If it apologizes about not seeing images or returns HTTP 400 → text-only.
|
|
|
|
## Config Gotcha
|
|
|
|
The `auxiliary.vision` section in `~/.hermes/config.yaml` must have:
|
|
- Correct model name WITH prefix (e.g. `vllm/release/gemma-4-31b-it`, not `vllm/gemma-4-31b-it`)
|
|
- Non-empty `base_url` and `api_key`
|
|
|
|
If these are empty or wrong, `vision_analyze` fails with "no keys found that support model" — even if the top-level `vision.*` section is correct. Both sections must be consistent.
|