Files

3.9 KiB

Ollama Inference Runbook

This dashboard can route sandbox inference through OpenShell's inference.local gateway to an OpenAI-compatible Ollama endpoint.

For large local models, run Ollama inside WSL2 and point the OpenShell provider at WSL localhost:

ollama serve
ollama pull nemotron-3-super:120b
ollama show nemotron-3-super:120b

openshell provider create \
  --name ollama-local \
  --type openai \
  --credential OPENAI_API_KEY=ollama \
  --config OPENAI_BASE_URL=http://127.0.0.1:11434/v1

# If the provider already exists:
openshell provider update ollama-local \
  --credential OPENAI_API_KEY=ollama \
  --config OPENAI_BASE_URL=http://127.0.0.1:11434/v1

openshell inference set \
  --no-verify \
  --provider ollama-local \
  --model nemotron-3-super:120b \
  --timeout 600

Use openshell inference get to verify the active route.

Validated Local Models

These values are what Ollama reported on the WSL2 test host.

Model Size Ollama context Notes
qwen2.5:7b 4.7 GB 32768 Fast fallback; enough for short cuFolio card requests after session reset.
nemotron-3-super:120b 86 GB 262144 Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom.

The nemotron-3-super:120b smoke test loaded entirely into RTX 6000 VRAM:

size_vram: 94227902208
context_length: 262144

The end-to-end cuFolio sandbox chat test completed successfully with:

model: inference/nemotron-3-super:120b
pending: false
answer: exact requested token
elapsed: about 91 seconds

OpenClaw Context Behavior

OpenClaw reserves output/recovery tokens before building the prompt. With a 32768 token model and a 20000 token reserve, only about 12768 tokens remain for input history. That is why long-running inter-sandbox-chat sessions can overflow even when Ollama reports a 32k model context.

The sandbox inference live-apply path now publishes model metadata for known Ollama models:

  • qwen2.5:7b: contextWindow=32768, maxTokens=2048
  • qwen3.5:27b: contextWindow=32768, maxTokens=2048
  • nemotron-3-super:120b: contextWindow=262144, maxTokens=8192

If a sandbox has accumulated a very large chat transcript, reset or compact the OpenClaw session before judging model health. In the cuFolio smoke test, an old inter-sandbox-chat transcript caused a pre-inference context overflow; after OpenClaw reset that session, the same WSL2 Ollama path answered correctly.

Windows-Host Ollama Notes

Windows-host Ollama can be selected during onboarding and model discovery. From WSL2, the dashboard probes:

  • WSL localhost: http://127.0.0.1:11434
  • WSL Windows-host gateway candidates from /etc/resolv.conf and /proc/net/route
  • host.docker.internal
  • Windows localhost through curl.exe / PowerShell interop

Useful environment overrides:

OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434

During testing, Windows-host qwen3.5:27b could expose reasoning in the OpenAI-compatible response while leaving message.content empty. The optional proxy below converts OpenAI chat requests to Ollama native /api/chat with think:false and streams early SSE events to avoid OpenClaw idle timeouts:

OLLAMA_PROXY_HOST=0.0.0.0 \
OLLAMA_PROXY_PORT=11435 \
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
OLLAMA_PROXY_NUM_CTX=32768 \
node scripts/ollama-thinkless-openai-proxy.mjs

openshell provider update ollama-win \
  --credential OPENAI_API_KEY=ollama \
  --config OPENAI_BASE_URL=http://172.19.0.1:11435/v1

Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is known to return normal OpenAI message.content.