3.9 KiB
Ollama Inference Runbook
This dashboard can route sandbox inference through OpenShell's inference.local
gateway to an OpenAI-compatible Ollama endpoint.
Recommended WSL2 Path
For large local models, run Ollama inside WSL2 and point the OpenShell provider at WSL localhost:
ollama serve
ollama pull nemotron-3-super:120b
ollama show nemotron-3-super:120b
openshell provider create \
--name ollama-local \
--type openai \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
# If the provider already exists:
openshell provider update ollama-local \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
openshell inference set \
--no-verify \
--provider ollama-local \
--model nemotron-3-super:120b \
--timeout 600
Use openshell inference get to verify the active route.
Validated Local Models
These values are what Ollama reported on the WSL2 test host.
| Model | Size | Ollama context | Notes |
|---|---|---|---|
qwen2.5:7b |
4.7 GB | 32768 | Fast fallback; enough for short cuFolio card requests after session reset. |
nemotron-3-super:120b |
86 GB | 262144 | Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom. |
The nemotron-3-super:120b smoke test loaded entirely into RTX 6000 VRAM:
size_vram: 94227902208
context_length: 262144
The end-to-end cuFolio sandbox chat test completed successfully with:
model: inference/nemotron-3-super:120b
pending: false
answer: exact requested token
elapsed: about 91 seconds
OpenClaw Context Behavior
OpenClaw reserves output/recovery tokens before building the prompt. With a 32768
token model and a 20000 token reserve, only about 12768 tokens remain for input
history. That is why long-running inter-sandbox-chat sessions can overflow even
when Ollama reports a 32k model context.
The sandbox inference live-apply path now publishes model metadata for known Ollama models:
qwen2.5:7b:contextWindow=32768,maxTokens=2048qwen3.5:27b:contextWindow=32768,maxTokens=2048nemotron-3-super:120b:contextWindow=262144,maxTokens=8192
If a sandbox has accumulated a very large chat transcript, reset or compact the
OpenClaw session before judging model health. In the cuFolio smoke test, an old
inter-sandbox-chat transcript caused a pre-inference context overflow; after
OpenClaw reset that session, the same WSL2 Ollama path answered correctly.
Windows-Host Ollama Notes
Windows-host Ollama can be selected during onboarding and model discovery. From WSL2, the dashboard probes:
- WSL localhost:
http://127.0.0.1:11434 - WSL Windows-host gateway candidates from
/etc/resolv.confand/proc/net/route host.docker.internal- Windows localhost through
curl.exe/ PowerShell interop
Useful environment overrides:
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434
During testing, Windows-host qwen3.5:27b could expose reasoning in the
OpenAI-compatible response while leaving message.content empty. The optional
proxy below converts OpenAI chat requests to Ollama native /api/chat with
think:false and streams early SSE events to avoid OpenClaw idle timeouts:
OLLAMA_PROXY_HOST=0.0.0.0 \
OLLAMA_PROXY_PORT=11435 \
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
OLLAMA_PROXY_NUM_CTX=32768 \
node scripts/ollama-thinkless-openai-proxy.mjs
openshell provider update ollama-win \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://172.19.0.1:11435/v1
Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is
known to return normal OpenAI message.content.