119 lines
3.9 KiB
Markdown
119 lines
3.9 KiB
Markdown
# Ollama Inference Runbook
|
|
|
|
This dashboard can route sandbox inference through OpenShell's `inference.local`
|
|
gateway to an OpenAI-compatible Ollama endpoint.
|
|
|
|
## Recommended WSL2 Path
|
|
|
|
For large local models, run Ollama inside WSL2 and point the OpenShell provider at
|
|
WSL localhost:
|
|
|
|
```bash
|
|
ollama serve
|
|
ollama pull nemotron-3-super:120b
|
|
ollama show nemotron-3-super:120b
|
|
|
|
openshell provider create \
|
|
--name ollama-local \
|
|
--type openai \
|
|
--credential OPENAI_API_KEY=ollama \
|
|
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
|
|
|
# If the provider already exists:
|
|
openshell provider update ollama-local \
|
|
--credential OPENAI_API_KEY=ollama \
|
|
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
|
|
|
openshell inference set \
|
|
--no-verify \
|
|
--provider ollama-local \
|
|
--model nemotron-3-super:120b \
|
|
--timeout 600
|
|
```
|
|
|
|
Use `openshell inference get` to verify the active route.
|
|
|
|
## Validated Local Models
|
|
|
|
These values are what Ollama reported on the WSL2 test host.
|
|
|
|
| Model | Size | Ollama context | Notes |
|
|
| --- | ---: | ---: | --- |
|
|
| `qwen2.5:7b` | 4.7 GB | 32768 | Fast fallback; enough for short cuFolio card requests after session reset. |
|
|
| `nemotron-3-super:120b` | 86 GB | 262144 | Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom. |
|
|
|
|
The `nemotron-3-super:120b` smoke test loaded entirely into RTX 6000 VRAM:
|
|
|
|
```text
|
|
size_vram: 94227902208
|
|
context_length: 262144
|
|
```
|
|
|
|
The end-to-end cuFolio sandbox chat test completed successfully with:
|
|
|
|
```text
|
|
model: inference/nemotron-3-super:120b
|
|
pending: false
|
|
answer: exact requested token
|
|
elapsed: about 91 seconds
|
|
```
|
|
|
|
## OpenClaw Context Behavior
|
|
|
|
OpenClaw reserves output/recovery tokens before building the prompt. With a 32768
|
|
token model and a 20000 token reserve, only about 12768 tokens remain for input
|
|
history. That is why long-running `inter-sandbox-chat` sessions can overflow even
|
|
when Ollama reports a 32k model context.
|
|
|
|
The sandbox inference live-apply path now publishes model metadata for known
|
|
Ollama models:
|
|
|
|
- `qwen2.5:7b`: `contextWindow=32768`, `maxTokens=2048`
|
|
- `qwen3.5:27b`: `contextWindow=32768`, `maxTokens=2048`
|
|
- `nemotron-3-super:120b`: `contextWindow=262144`, `maxTokens=8192`
|
|
|
|
If a sandbox has accumulated a very large chat transcript, reset or compact the
|
|
OpenClaw session before judging model health. In the cuFolio smoke test, an old
|
|
`inter-sandbox-chat` transcript caused a pre-inference context overflow; after
|
|
OpenClaw reset that session, the same WSL2 Ollama path answered correctly.
|
|
|
|
## Windows-Host Ollama Notes
|
|
|
|
Windows-host Ollama can be selected during onboarding and model discovery. From
|
|
WSL2, the dashboard probes:
|
|
|
|
- WSL localhost: `http://127.0.0.1:11434`
|
|
- WSL Windows-host gateway candidates from `/etc/resolv.conf` and `/proc/net/route`
|
|
- `host.docker.internal`
|
|
- Windows localhost through `curl.exe` / PowerShell interop
|
|
|
|
Useful environment overrides:
|
|
|
|
```bash
|
|
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
|
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
|
|
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
|
|
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
|
|
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
|
```
|
|
|
|
During testing, Windows-host `qwen3.5:27b` could expose reasoning in the
|
|
OpenAI-compatible response while leaving `message.content` empty. The optional
|
|
proxy below converts OpenAI chat requests to Ollama native `/api/chat` with
|
|
`think:false` and streams early SSE events to avoid OpenClaw idle timeouts:
|
|
|
|
```bash
|
|
OLLAMA_PROXY_HOST=0.0.0.0 \
|
|
OLLAMA_PROXY_PORT=11435 \
|
|
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
|
|
OLLAMA_PROXY_NUM_CTX=32768 \
|
|
node scripts/ollama-thinkless-openai-proxy.mjs
|
|
|
|
openshell provider update ollama-win \
|
|
--credential OPENAI_API_KEY=ollama \
|
|
--config OPENAI_BASE_URL=http://172.19.0.1:11435/v1
|
|
```
|
|
|
|
Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is
|
|
known to return normal OpenAI `message.content`.
|