Document Ollama inference setup

This commit is contained in:
mmckeen-nv
2026-05-20 18:50:24 -07:00
parent 9d0e0e1a16
commit 81f7239bf3
7 changed files with 499 additions and 50 deletions
+118
View File
@@ -0,0 +1,118 @@
# Ollama Inference Runbook
This dashboard can route sandbox inference through OpenShell's `inference.local`
gateway to an OpenAI-compatible Ollama endpoint.
## Recommended WSL2 Path
For large local models, run Ollama inside WSL2 and point the OpenShell provider at
WSL localhost:
```bash
ollama serve
ollama pull nemotron-3-super:120b
ollama show nemotron-3-super:120b
openshell provider create \
--name ollama-local \
--type openai \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
# If the provider already exists:
openshell provider update ollama-local \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
openshell inference set \
--no-verify \
--provider ollama-local \
--model nemotron-3-super:120b \
--timeout 600
```
Use `openshell inference get` to verify the active route.
## Validated Local Models
These values are what Ollama reported on the WSL2 test host.
| Model | Size | Ollama context | Notes |
| --- | ---: | ---: | --- |
| `qwen2.5:7b` | 4.7 GB | 32768 | Fast fallback; enough for short cuFolio card requests after session reset. |
| `nemotron-3-super:120b` | 86 GB | 262144 | Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom. |
The `nemotron-3-super:120b` smoke test loaded entirely into RTX 6000 VRAM:
```text
size_vram: 94227902208
context_length: 262144
```
The end-to-end cuFolio sandbox chat test completed successfully with:
```text
model: inference/nemotron-3-super:120b
pending: false
answer: exact requested token
elapsed: about 91 seconds
```
## OpenClaw Context Behavior
OpenClaw reserves output/recovery tokens before building the prompt. With a 32768
token model and a 20000 token reserve, only about 12768 tokens remain for input
history. That is why long-running `inter-sandbox-chat` sessions can overflow even
when Ollama reports a 32k model context.
The sandbox inference live-apply path now publishes model metadata for known
Ollama models:
- `qwen2.5:7b`: `contextWindow=32768`, `maxTokens=2048`
- `qwen3.5:27b`: `contextWindow=32768`, `maxTokens=2048`
- `nemotron-3-super:120b`: `contextWindow=262144`, `maxTokens=8192`
If a sandbox has accumulated a very large chat transcript, reset or compact the
OpenClaw session before judging model health. In the cuFolio smoke test, an old
`inter-sandbox-chat` transcript caused a pre-inference context overflow; after
OpenClaw reset that session, the same WSL2 Ollama path answered correctly.
## Windows-Host Ollama Notes
Windows-host Ollama can be selected during onboarding and model discovery. From
WSL2, the dashboard probes:
- WSL localhost: `http://127.0.0.1:11434`
- WSL Windows-host gateway candidates from `/etc/resolv.conf` and `/proc/net/route`
- `host.docker.internal`
- Windows localhost through `curl.exe` / PowerShell interop
Useful environment overrides:
```bash
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434
```
During testing, Windows-host `qwen3.5:27b` could expose reasoning in the
OpenAI-compatible response while leaving `message.content` empty. The optional
proxy below converts OpenAI chat requests to Ollama native `/api/chat` with
`think:false` and streams early SSE events to avoid OpenClaw idle timeouts:
```bash
OLLAMA_PROXY_HOST=0.0.0.0 \
OLLAMA_PROXY_PORT=11435 \
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
OLLAMA_PROXY_NUM_CTX=32768 \
node scripts/ollama-thinkless-openai-proxy.mjs
openshell provider update ollama-win \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://172.19.0.1:11435/v1
```
Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is
known to return normal OpenAI `message.content`.