Document Ollama inference setup
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# Ollama Inference Runbook
|
||||
|
||||
This dashboard can route sandbox inference through OpenShell's `inference.local`
|
||||
gateway to an OpenAI-compatible Ollama endpoint.
|
||||
|
||||
## Recommended WSL2 Path
|
||||
|
||||
For large local models, run Ollama inside WSL2 and point the OpenShell provider at
|
||||
WSL localhost:
|
||||
|
||||
```bash
|
||||
ollama serve
|
||||
ollama pull nemotron-3-super:120b
|
||||
ollama show nemotron-3-super:120b
|
||||
|
||||
openshell provider create \
|
||||
--name ollama-local \
|
||||
--type openai \
|
||||
--credential OPENAI_API_KEY=ollama \
|
||||
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
||||
|
||||
# If the provider already exists:
|
||||
openshell provider update ollama-local \
|
||||
--credential OPENAI_API_KEY=ollama \
|
||||
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
||||
|
||||
openshell inference set \
|
||||
--no-verify \
|
||||
--provider ollama-local \
|
||||
--model nemotron-3-super:120b \
|
||||
--timeout 600
|
||||
```
|
||||
|
||||
Use `openshell inference get` to verify the active route.
|
||||
|
||||
## Validated Local Models
|
||||
|
||||
These values are what Ollama reported on the WSL2 test host.
|
||||
|
||||
| Model | Size | Ollama context | Notes |
|
||||
| --- | ---: | ---: | --- |
|
||||
| `qwen2.5:7b` | 4.7 GB | 32768 | Fast fallback; enough for short cuFolio card requests after session reset. |
|
||||
| `nemotron-3-super:120b` | 86 GB | 262144 | Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom. |
|
||||
|
||||
The `nemotron-3-super:120b` smoke test loaded entirely into RTX 6000 VRAM:
|
||||
|
||||
```text
|
||||
size_vram: 94227902208
|
||||
context_length: 262144
|
||||
```
|
||||
|
||||
The end-to-end cuFolio sandbox chat test completed successfully with:
|
||||
|
||||
```text
|
||||
model: inference/nemotron-3-super:120b
|
||||
pending: false
|
||||
answer: exact requested token
|
||||
elapsed: about 91 seconds
|
||||
```
|
||||
|
||||
## OpenClaw Context Behavior
|
||||
|
||||
OpenClaw reserves output/recovery tokens before building the prompt. With a 32768
|
||||
token model and a 20000 token reserve, only about 12768 tokens remain for input
|
||||
history. That is why long-running `inter-sandbox-chat` sessions can overflow even
|
||||
when Ollama reports a 32k model context.
|
||||
|
||||
The sandbox inference live-apply path now publishes model metadata for known
|
||||
Ollama models:
|
||||
|
||||
- `qwen2.5:7b`: `contextWindow=32768`, `maxTokens=2048`
|
||||
- `qwen3.5:27b`: `contextWindow=32768`, `maxTokens=2048`
|
||||
- `nemotron-3-super:120b`: `contextWindow=262144`, `maxTokens=8192`
|
||||
|
||||
If a sandbox has accumulated a very large chat transcript, reset or compact the
|
||||
OpenClaw session before judging model health. In the cuFolio smoke test, an old
|
||||
`inter-sandbox-chat` transcript caused a pre-inference context overflow; after
|
||||
OpenClaw reset that session, the same WSL2 Ollama path answered correctly.
|
||||
|
||||
## Windows-Host Ollama Notes
|
||||
|
||||
Windows-host Ollama can be selected during onboarding and model discovery. From
|
||||
WSL2, the dashboard probes:
|
||||
|
||||
- WSL localhost: `http://127.0.0.1:11434`
|
||||
- WSL Windows-host gateway candidates from `/etc/resolv.conf` and `/proc/net/route`
|
||||
- `host.docker.internal`
|
||||
- Windows localhost through `curl.exe` / PowerShell interop
|
||||
|
||||
Useful environment overrides:
|
||||
|
||||
```bash
|
||||
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
|
||||
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
|
||||
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
|
||||
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||
```
|
||||
|
||||
During testing, Windows-host `qwen3.5:27b` could expose reasoning in the
|
||||
OpenAI-compatible response while leaving `message.content` empty. The optional
|
||||
proxy below converts OpenAI chat requests to Ollama native `/api/chat` with
|
||||
`think:false` and streams early SSE events to avoid OpenClaw idle timeouts:
|
||||
|
||||
```bash
|
||||
OLLAMA_PROXY_HOST=0.0.0.0 \
|
||||
OLLAMA_PROXY_PORT=11435 \
|
||||
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
|
||||
OLLAMA_PROXY_NUM_CTX=32768 \
|
||||
node scripts/ollama-thinkless-openai-proxy.mjs
|
||||
|
||||
openshell provider update ollama-win \
|
||||
--credential OPENAI_API_KEY=ollama \
|
||||
--config OPENAI_BASE_URL=http://172.19.0.1:11435/v1
|
||||
```
|
||||
|
||||
Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is
|
||||
known to return normal OpenAI `message.content`.
|
||||
Reference in New Issue
Block a user