Document Ollama inference setup
This commit is contained in:
@@ -18,6 +18,14 @@ OPENSHELL_CONTROL_CREATE_GPU_MODE=none
|
|||||||
# OPENSHELL_GATEWAY_PORT=8080
|
# OPENSHELL_GATEWAY_PORT=8080
|
||||||
# OPENSHELL_GATEWAY_URL=http://host.docker.internal:8080
|
# OPENSHELL_GATEWAY_URL=http://host.docker.internal:8080
|
||||||
|
|
||||||
|
# Ollama discovery and inference routing.
|
||||||
|
# In WSL2, prefer a WSL-local Ollama server for large models. The model picker also
|
||||||
|
# probes the Windows host gateway and Windows localhost through WSL interop.
|
||||||
|
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||||
|
# OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
|
||||||
|
# OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
|
||||||
|
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
|
||||||
|
|
||||||
# Dashboard port. PORT controls the server listener; NEXT_PUBLIC_DASHBOARD_PORT is exposed to the UI.
|
# Dashboard port. PORT controls the server listener; NEXT_PUBLIC_DASHBOARD_PORT is exposed to the UI.
|
||||||
PORT=3000
|
PORT=3000
|
||||||
NEXT_PUBLIC_DASHBOARD_PORT=3000
|
NEXT_PUBLIC_DASHBOARD_PORT=3000
|
||||||
|
|||||||
@@ -0,0 +1,118 @@
|
|||||||
|
# Ollama Inference Runbook
|
||||||
|
|
||||||
|
This dashboard can route sandbox inference through OpenShell's `inference.local`
|
||||||
|
gateway to an OpenAI-compatible Ollama endpoint.
|
||||||
|
|
||||||
|
## Recommended WSL2 Path
|
||||||
|
|
||||||
|
For large local models, run Ollama inside WSL2 and point the OpenShell provider at
|
||||||
|
WSL localhost:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ollama serve
|
||||||
|
ollama pull nemotron-3-super:120b
|
||||||
|
ollama show nemotron-3-super:120b
|
||||||
|
|
||||||
|
openshell provider create \
|
||||||
|
--name ollama-local \
|
||||||
|
--type openai \
|
||||||
|
--credential OPENAI_API_KEY=ollama \
|
||||||
|
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
||||||
|
|
||||||
|
# If the provider already exists:
|
||||||
|
openshell provider update ollama-local \
|
||||||
|
--credential OPENAI_API_KEY=ollama \
|
||||||
|
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
||||||
|
|
||||||
|
openshell inference set \
|
||||||
|
--no-verify \
|
||||||
|
--provider ollama-local \
|
||||||
|
--model nemotron-3-super:120b \
|
||||||
|
--timeout 600
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `openshell inference get` to verify the active route.
|
||||||
|
|
||||||
|
## Validated Local Models
|
||||||
|
|
||||||
|
These values are what Ollama reported on the WSL2 test host.
|
||||||
|
|
||||||
|
| Model | Size | Ollama context | Notes |
|
||||||
|
| --- | ---: | ---: | --- |
|
||||||
|
| `qwen2.5:7b` | 4.7 GB | 32768 | Fast fallback; enough for short cuFolio card requests after session reset. |
|
||||||
|
| `nemotron-3-super:120b` | 86 GB | 262144 | Fits on a 98 GB RTX 6000 Blackwell at Q4_K_M; much slower, but handles the cuFolio/OpenClaw prompt with far more context headroom. |
|
||||||
|
|
||||||
|
The `nemotron-3-super:120b` smoke test loaded entirely into RTX 6000 VRAM:
|
||||||
|
|
||||||
|
```text
|
||||||
|
size_vram: 94227902208
|
||||||
|
context_length: 262144
|
||||||
|
```
|
||||||
|
|
||||||
|
The end-to-end cuFolio sandbox chat test completed successfully with:
|
||||||
|
|
||||||
|
```text
|
||||||
|
model: inference/nemotron-3-super:120b
|
||||||
|
pending: false
|
||||||
|
answer: exact requested token
|
||||||
|
elapsed: about 91 seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
## OpenClaw Context Behavior
|
||||||
|
|
||||||
|
OpenClaw reserves output/recovery tokens before building the prompt. With a 32768
|
||||||
|
token model and a 20000 token reserve, only about 12768 tokens remain for input
|
||||||
|
history. That is why long-running `inter-sandbox-chat` sessions can overflow even
|
||||||
|
when Ollama reports a 32k model context.
|
||||||
|
|
||||||
|
The sandbox inference live-apply path now publishes model metadata for known
|
||||||
|
Ollama models:
|
||||||
|
|
||||||
|
- `qwen2.5:7b`: `contextWindow=32768`, `maxTokens=2048`
|
||||||
|
- `qwen3.5:27b`: `contextWindow=32768`, `maxTokens=2048`
|
||||||
|
- `nemotron-3-super:120b`: `contextWindow=262144`, `maxTokens=8192`
|
||||||
|
|
||||||
|
If a sandbox has accumulated a very large chat transcript, reset or compact the
|
||||||
|
OpenClaw session before judging model health. In the cuFolio smoke test, an old
|
||||||
|
`inter-sandbox-chat` transcript caused a pre-inference context overflow; after
|
||||||
|
OpenClaw reset that session, the same WSL2 Ollama path answered correctly.
|
||||||
|
|
||||||
|
## Windows-Host Ollama Notes
|
||||||
|
|
||||||
|
Windows-host Ollama can be selected during onboarding and model discovery. From
|
||||||
|
WSL2, the dashboard probes:
|
||||||
|
|
||||||
|
- WSL localhost: `http://127.0.0.1:11434`
|
||||||
|
- WSL Windows-host gateway candidates from `/etc/resolv.conf` and `/proc/net/route`
|
||||||
|
- `host.docker.internal`
|
||||||
|
- Windows localhost through `curl.exe` / PowerShell interop
|
||||||
|
|
||||||
|
Useful environment overrides:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||||
|
OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
|
||||||
|
OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
|
||||||
|
OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
|
||||||
|
OPENSHELL_WINDOWS_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||||
|
```
|
||||||
|
|
||||||
|
During testing, Windows-host `qwen3.5:27b` could expose reasoning in the
|
||||||
|
OpenAI-compatible response while leaving `message.content` empty. The optional
|
||||||
|
proxy below converts OpenAI chat requests to Ollama native `/api/chat` with
|
||||||
|
`think:false` and streams early SSE events to avoid OpenClaw idle timeouts:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OLLAMA_PROXY_HOST=0.0.0.0 \
|
||||||
|
OLLAMA_PROXY_PORT=11435 \
|
||||||
|
OLLAMA_PROXY_UPSTREAM=http://172.18.16.1:11434 \
|
||||||
|
OLLAMA_PROXY_NUM_CTX=32768 \
|
||||||
|
node scripts/ollama-thinkless-openai-proxy.mjs
|
||||||
|
|
||||||
|
openshell provider update ollama-win \
|
||||||
|
--credential OPENAI_API_KEY=ollama \
|
||||||
|
--config OPENAI_BASE_URL=http://172.19.0.1:11435/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
Prefer the WSL2-local path for the cuFolio demo unless the Windows-host model is
|
||||||
|
known to return normal OpenAI `message.content`.
|
||||||
+38
-46
@@ -4,12 +4,12 @@
|
|||||||
|
|
||||||
1. **Navigate to project directory:**
|
1. **Navigate to project directory:**
|
||||||
```bash
|
```bash
|
||||||
cd /path/to/nemoclaw-dashboard
|
cd /path/to/openshell_controller
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Install dependencies:**
|
2. **Install dependencies:**
|
||||||
```bash
|
```bash
|
||||||
npm install
|
./install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Start the development server:**
|
3. **Start the development server:**
|
||||||
@@ -24,40 +24,38 @@ http://localhost:3000
|
|||||||
|
|
||||||
## What You'll See
|
## What You'll See
|
||||||
|
|
||||||
- **Configuration Panel** at the top - set your OpenShell and Ollama URLs
|
- **Sandbox inventory** from the OpenShell gateway
|
||||||
- **Telemetry Cards** - CPU, memory, disk, and GPU usage (updates every 3 seconds)
|
- **Operator terminal** and OpenClaw dashboard proxy links
|
||||||
- **One-Click Actions** - Start Sandbox, Stop Sandbox, Create Instance, Switch Model buttons
|
- **Inference settings** for OpenAI-compatible providers, Ollama, NIM, and vLLM
|
||||||
- **Model Switcher** - Dropdown to select Ollama models
|
- **Ollama model discovery** across WSL2 localhost and optional Windows-host Ollama
|
||||||
- **Configuration Panels** - View settings for OpenShell, NemoClaw, and Ollama
|
- **MCP broker settings** and per-sandbox MCP access controls
|
||||||
|
|
||||||
## Testing the Dashboard
|
## Ollama Smoke Test
|
||||||
|
|
||||||
### Test 1: Verify Dashboard Loads
|
For WSL2 demo hosts, run Ollama in WSL2:
|
||||||
1. Open `http://localhost:3000`
|
|
||||||
2. All sections should render without errors
|
|
||||||
3. Telemetry should show values (randomly generated)
|
|
||||||
|
|
||||||
### Test 2: Configuration Panel
|
```bash
|
||||||
1. Edit the OpenShell URL field
|
ollama serve
|
||||||
2. Edit the Ollama URL field
|
ollama pull qwen2.5:7b
|
||||||
3. Settings are saved to localStorage (persists on refresh)
|
ollama pull nemotron-3-super:120b
|
||||||
|
```
|
||||||
|
|
||||||
### Test 3: One-Click Actions
|
Configure the gateway:
|
||||||
1. Click "Start Sandbox"
|
|
||||||
2. Click "Stop Sandbox"
|
|
||||||
3. Click "Create Instance"
|
|
||||||
4. Click "Switch Model"
|
|
||||||
5. Check that status messages appear
|
|
||||||
|
|
||||||
### Test 4: Model Switcher
|
```bash
|
||||||
1. Select a different model from the dropdown
|
openshell provider update ollama-local \
|
||||||
2. Click "Switch Model"
|
--credential OPENAI_API_KEY=ollama \
|
||||||
3. Verify success message
|
--config OPENAI_BASE_URL=http://127.0.0.1:11434/v1
|
||||||
|
|
||||||
### Test 5: Telemetry Updates
|
openshell inference set \
|
||||||
1. Wait 3 seconds
|
--no-verify \
|
||||||
2. Telemetry values should update automatically
|
--provider ollama-local \
|
||||||
3. No console errors should appear
|
--model nemotron-3-super:120b \
|
||||||
|
--timeout 600
|
||||||
|
```
|
||||||
|
|
||||||
|
`qwen2.5:7b` reports a 32768 token context. `nemotron-3-super:120b` reports a
|
||||||
|
262144 token context and needs about 94 GB of RTX 6000 VRAM at Q4_K_M.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
@@ -67,27 +65,21 @@ http://localhost:3000
|
|||||||
- Restart the server
|
- Restart the server
|
||||||
|
|
||||||
### API endpoints returning errors
|
### API endpoints returning errors
|
||||||
- This is expected - endpoints are mocked for development
|
- Check `openshell sandbox list`
|
||||||
- Check browser console for specific errors
|
- Check that the OpenShell gateway container is running
|
||||||
- No real connection to OpenShell/Ollama is required
|
- Check browser console and server logs for specific errors
|
||||||
|
|
||||||
|
### Ollama models do not appear
|
||||||
|
- Confirm WSL2 Ollama is reachable: `curl http://127.0.0.1:11434/api/tags`
|
||||||
|
- If using Windows-host Ollama, confirm it is reachable from WSL2 through the
|
||||||
|
detected host gateway or Windows interop
|
||||||
|
- Review `OLLAMA_INFERENCE.md` for the supported environment overrides
|
||||||
|
|
||||||
### Dependencies not installing
|
### Dependencies not installing
|
||||||
- Make sure you're running Node.js 18 or later
|
- Make sure you're running Node.js 20 or later
|
||||||
- Clear npm cache: `npm cache clean --force`
|
- Clear npm cache: `npm cache clean --force`
|
||||||
- Try reinstalling: `rm -rf node_modules package-lock.json && npm install`
|
- Try reinstalling: `rm -rf node_modules package-lock.json && npm install`
|
||||||
|
|
||||||
## Next Steps (Production)
|
|
||||||
|
|
||||||
To make this production-ready:
|
|
||||||
|
|
||||||
1. Replace mock API routes with real API calls to OpenShell (port 8080) and Ollama (port 11434)
|
|
||||||
2. Add authentication to API routes
|
|
||||||
3. Implement proper error handling
|
|
||||||
4. Add loading states and skeletons
|
|
||||||
5. Add dark mode toggle
|
|
||||||
6. Add more detailed telemetry charts
|
|
||||||
7. Add deployment configuration (Docker, Vercel, etc.)
|
|
||||||
|
|
||||||
## Support
|
## Support
|
||||||
|
|
||||||
For issues or questions:
|
For issues or questions:
|
||||||
|
|||||||
@@ -167,6 +167,22 @@ Default ports:
|
|||||||
|
|
||||||
The dashboard WebSocket proxy is served on the same listener by default, using `/api/openshell/dashboard/proxy` or `/api/openshell/instances/[instanceId]/dashboard/proxy`. Set `OPENCLAW_DASHBOARD_WS_PROXY_PORT=3001` only if you intentionally want the legacy dedicated sidecar listener.
|
The dashboard WebSocket proxy is served on the same listener by default, using `/api/openshell/dashboard/proxy` or `/api/openshell/instances/[instanceId]/dashboard/proxy`. Set `OPENCLAW_DASHBOARD_WS_PROXY_PORT=3001` only if you intentionally want the legacy dedicated sidecar listener.
|
||||||
|
|
||||||
|
## Ollama Inference
|
||||||
|
|
||||||
|
For WSL2 demo hosts, prefer running Ollama inside WSL2 and routing OpenShell
|
||||||
|
through `ollama-local` at `http://127.0.0.1:11434/v1`. This avoids Windows/WSL
|
||||||
|
host routing surprises and was validated with both `qwen2.5:7b` and
|
||||||
|
`nemotron-3-super:120b`.
|
||||||
|
|
||||||
|
The `nemotron-3-super:120b` Ollama tag reports a `262144` token context window
|
||||||
|
and was smoke-tested end to end through cuFolio sandbox chat on an RTX 6000
|
||||||
|
Blackwell. It fit in VRAM at Q4_K_M, but responses are much slower than the 7B
|
||||||
|
fallback.
|
||||||
|
|
||||||
|
See [OLLAMA_INFERENCE.md](OLLAMA_INFERENCE.md) for setup commands, context-window
|
||||||
|
notes, and the optional Windows-host thinkless proxy used for models that return
|
||||||
|
reasoning without OpenAI `message.content`.
|
||||||
|
|
||||||
## Authentication
|
## Authentication
|
||||||
|
|
||||||
This project currently uses a simple local password and signed HTTP-only cookie.
|
This project currently uses a simple local password and signed HTTP-only cookie.
|
||||||
|
|||||||
@@ -7,16 +7,36 @@ const execFileAsync = promisify(execFile)
|
|||||||
const DOCKER_BIN = process.env.DOCKER_BIN || "docker"
|
const DOCKER_BIN = process.env.DOCKER_BIN || "docker"
|
||||||
const OPENSHELL_CLUSTER_CONTAINER = process.env.OPENSHELL_CLUSTER_CONTAINER || "openshell-cluster-nemoclaw"
|
const OPENSHELL_CLUSTER_CONTAINER = process.env.OPENSHELL_CLUSTER_CONTAINER || "openshell-cluster-nemoclaw"
|
||||||
|
|
||||||
|
function modelContextWindow(modelId: string) {
|
||||||
|
const normalized = modelId.toLowerCase()
|
||||||
|
if (normalized.includes("nemotron-3-super") && normalized.includes("120b")) return 262144
|
||||||
|
if (normalized.includes("qwen2.5:7b")) return 32768
|
||||||
|
if (normalized.includes("qwen3.5:27b")) return 32768
|
||||||
|
return 131072
|
||||||
|
}
|
||||||
|
|
||||||
|
function modelMaxTokens(modelId: string) {
|
||||||
|
const contextWindow = modelContextWindow(modelId)
|
||||||
|
if (contextWindow >= 262144) return 8192
|
||||||
|
if (contextWindow <= 32768) return 2048
|
||||||
|
return 4096
|
||||||
|
}
|
||||||
|
|
||||||
|
function modelSupportsReasoning(modelId: string) {
|
||||||
|
const normalized = modelId.toLowerCase()
|
||||||
|
return normalized.includes("nemotron-3-super")
|
||||||
|
}
|
||||||
|
|
||||||
function modelEntry(modelId: string, modelName: string, compat: Record<string, unknown> | null) {
|
function modelEntry(modelId: string, modelName: string, compat: Record<string, unknown> | null) {
|
||||||
return {
|
return {
|
||||||
...(compat ? { compat } : {}),
|
...(compat ? { compat } : {}),
|
||||||
id: modelId,
|
id: modelId,
|
||||||
name: modelName,
|
name: modelName,
|
||||||
reasoning: false,
|
reasoning: modelSupportsReasoning(modelId),
|
||||||
input: ["text"],
|
input: ["text"],
|
||||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||||
contextWindow: 131072,
|
contextWindow: modelContextWindow(modelId),
|
||||||
maxTokens: 4096,
|
maxTokens: modelMaxTokens(modelId),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+11
-1
@@ -299,7 +299,7 @@ upsert_env() {
|
|||||||
set_env() {
|
set_env() {
|
||||||
local key="$1"
|
local key="$1"
|
||||||
local value="$2"
|
local value="$2"
|
||||||
[[ -n "$value" ]] || return
|
[[ -n "$value" ]] || return 0
|
||||||
touch "$ENV_FILE"
|
touch "$ENV_FILE"
|
||||||
if grep -qE "^${key}=" "$ENV_FILE"; then
|
if grep -qE "^${key}=" "$ENV_FILE"; then
|
||||||
sed -i.bak -E "s|^${key}=.*$|${key}=${value}|" "$ENV_FILE"
|
sed -i.bak -E "s|^${key}=.*$|${key}=${value}|" "$ENV_FILE"
|
||||||
@@ -408,6 +408,12 @@ OPENSHELL_CONTROL_CREATE_GPU_MODE=none
|
|||||||
# OPENSHELL_GATEWAY_HOST=host.docker.internal
|
# OPENSHELL_GATEWAY_HOST=host.docker.internal
|
||||||
# OPENSHELL_GATEWAY_PORT=8080
|
# OPENSHELL_GATEWAY_PORT=8080
|
||||||
# OPENSHELL_GATEWAY_URL=http://host.docker.internal:8080
|
# OPENSHELL_GATEWAY_URL=http://host.docker.internal:8080
|
||||||
|
# Ollama model discovery prefers WSL2 localhost, then the Windows host gateway.
|
||||||
|
# Keep Ollama in WSL2 for large local models unless you intentionally configure Windows-host Ollama.
|
||||||
|
# OPENSHELL_OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||||
|
# OPENSHELL_OLLAMA_HOSTS=127.0.0.1,172.19.0.1
|
||||||
|
# OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS=2500
|
||||||
|
# OPENSHELL_OLLAMA_WINDOWS_INTEROP=1
|
||||||
# For reverse proxies, OpenClaw dashboard websockets use the same-origin proxy path by default.
|
# For reverse proxies, OpenClaw dashboard websockets use the same-origin proxy path by default.
|
||||||
# OPENCLAW_DASHBOARD_BASE_WS_URL=wss://control.example.com
|
# OPENCLAW_DASHBOARD_BASE_WS_URL=wss://control.example.com
|
||||||
# BASE_WS_URL=wss://control.example.com
|
# BASE_WS_URL=wss://control.example.com
|
||||||
@@ -424,9 +430,13 @@ upsert_env "NEXT_PUBLIC_ENABLE_SANDBOX_OPERATIONS" "true"
|
|||||||
upsert_env "OPEN_SHELL_CONTAINER" "$OPEN_SHELL_CONTAINER_DEFAULT"
|
upsert_env "OPEN_SHELL_CONTAINER" "$OPEN_SHELL_CONTAINER_DEFAULT"
|
||||||
upsert_env "OPENSHELL_GATEWAY" "nemoclaw"
|
upsert_env "OPENSHELL_GATEWAY" "nemoclaw"
|
||||||
upsert_env "OPENSHELL_CONTROL_CREATE_GPU_MODE" "none"
|
upsert_env "OPENSHELL_CONTROL_CREATE_GPU_MODE" "none"
|
||||||
|
upsert_env "OPENSHELL_OLLAMA_BASE_URL" "http://127.0.0.1:11434"
|
||||||
|
upsert_env "OPENSHELL_OLLAMA_WINDOWS_INTEROP" "1"
|
||||||
set_env "OPENSHELL_GATEWAY_HOST" "${OPENSHELL_GATEWAY_HOST:-}"
|
set_env "OPENSHELL_GATEWAY_HOST" "${OPENSHELL_GATEWAY_HOST:-}"
|
||||||
set_env "OPENSHELL_GATEWAY_PORT" "${OPENSHELL_GATEWAY_PORT:-}"
|
set_env "OPENSHELL_GATEWAY_PORT" "${OPENSHELL_GATEWAY_PORT:-}"
|
||||||
set_env "OPENSHELL_GATEWAY_URL" "${OPENSHELL_GATEWAY_URL:-}"
|
set_env "OPENSHELL_GATEWAY_URL" "${OPENSHELL_GATEWAY_URL:-}"
|
||||||
|
set_env "OPENSHELL_OLLAMA_HOSTS" "${OPENSHELL_OLLAMA_HOSTS:-}"
|
||||||
|
set_env "OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS" "${OPENSHELL_OLLAMA_PROBE_TIMEOUT_MS:-}"
|
||||||
set_env "OPENSHELL_CONTROL_VENV" "${VIRTUAL_ENV:-$PROJECT_VENV}"
|
set_env "OPENSHELL_CONTROL_VENV" "${VIRTUAL_ENV:-$PROJECT_VENV}"
|
||||||
set_env "OPENSHELL_HOME" "${HOME:-}"
|
set_env "OPENSHELL_HOME" "${HOME:-}"
|
||||||
set_env "OPENSHELL_BIN" "${OPENSHELL_BIN:-}"
|
set_env "OPENSHELL_BIN" "${OPENSHELL_BIN:-}"
|
||||||
|
|||||||
@@ -0,0 +1,285 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
import http from "node:http";
|
||||||
|
|
||||||
|
const listenHost = process.env.OLLAMA_PROXY_HOST || "0.0.0.0";
|
||||||
|
const listenPort = Number.parseInt(process.env.OLLAMA_PROXY_PORT || "11435", 10);
|
||||||
|
const upstream = (process.env.OLLAMA_PROXY_UPSTREAM || "http://172.18.16.1:11434").replace(/\/+$/, "");
|
||||||
|
const numCtx = Number.parseInt(process.env.OLLAMA_PROXY_NUM_CTX || "8192", 10);
|
||||||
|
|
||||||
|
function sendJson(res, status, payload) {
|
||||||
|
const body = JSON.stringify(payload);
|
||||||
|
res.writeHead(status, {
|
||||||
|
"content-type": "application/json",
|
||||||
|
"content-length": Buffer.byteLength(body),
|
||||||
|
});
|
||||||
|
res.end(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function readJson(req) {
|
||||||
|
let body = "";
|
||||||
|
for await (const chunk of req) body += String(chunk);
|
||||||
|
return body.trim() ? JSON.parse(body) : {};
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeMessages(messages) {
|
||||||
|
return Array.isArray(messages)
|
||||||
|
? messages.map((message) => ({
|
||||||
|
role: String(message?.role || "user"),
|
||||||
|
content: Array.isArray(message?.content)
|
||||||
|
? message.content.map((part) => part?.text || "").join("\n")
|
||||||
|
: String(message?.content ?? ""),
|
||||||
|
}))
|
||||||
|
: [];
|
||||||
|
}
|
||||||
|
|
||||||
|
function finishReason(doneReason) {
|
||||||
|
if (doneReason === "stop") return "stop";
|
||||||
|
if (doneReason === "length") return "length";
|
||||||
|
return doneReason || "stop";
|
||||||
|
}
|
||||||
|
|
||||||
|
function sendSse(res, payloads) {
|
||||||
|
res.writeHead(200, {
|
||||||
|
"content-type": "text/event-stream; charset=utf-8",
|
||||||
|
"cache-control": "no-cache",
|
||||||
|
connection: "keep-alive",
|
||||||
|
});
|
||||||
|
for (const payload of payloads) {
|
||||||
|
res.write(`data: ${JSON.stringify(payload)}\n\n`);
|
||||||
|
}
|
||||||
|
res.write("data: [DONE]\n\n");
|
||||||
|
res.end();
|
||||||
|
}
|
||||||
|
|
||||||
|
function writeSse(res, payload) {
|
||||||
|
res.write(`data: ${JSON.stringify(payload)}\n\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function proxyModels(res) {
|
||||||
|
const response = await fetch(`${upstream}/v1/models`);
|
||||||
|
const text = await response.text();
|
||||||
|
res.writeHead(response.status, { "content-type": response.headers.get("content-type") || "application/json" });
|
||||||
|
res.end(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function proxyStreamingChat(res, model, nativeBody) {
|
||||||
|
const id = `chatcmpl-${Date.now().toString(36)}`;
|
||||||
|
const created = Math.floor(Date.now() / 1000);
|
||||||
|
res.writeHead(200, {
|
||||||
|
"content-type": "text/event-stream; charset=utf-8",
|
||||||
|
"cache-control": "no-cache",
|
||||||
|
connection: "keep-alive",
|
||||||
|
});
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }],
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await fetch(`${upstream}/api/chat`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ ...nativeBody, stream: true }),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const text = await response.text();
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [
|
||||||
|
{
|
||||||
|
index: 0,
|
||||||
|
delta: { content: "" },
|
||||||
|
finish_reason: "stop",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
error: {
|
||||||
|
message: text || `Ollama upstream returned ${response.status}`,
|
||||||
|
type: "upstream_error",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
res.write("data: [DONE]\n\n");
|
||||||
|
res.end();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
let buffer = "";
|
||||||
|
let finish = "stop";
|
||||||
|
let sentContent = false;
|
||||||
|
|
||||||
|
for await (const chunk of response.body) {
|
||||||
|
buffer += decoder.decode(chunk, { stream: true });
|
||||||
|
const lines = buffer.split(/\r?\n/);
|
||||||
|
buffer = lines.pop() || "";
|
||||||
|
for (const line of lines) {
|
||||||
|
if (!line.trim()) continue;
|
||||||
|
const payload = JSON.parse(line);
|
||||||
|
const content = String(payload?.message?.content ?? payload?.response ?? "");
|
||||||
|
if (content) {
|
||||||
|
sentContent = true;
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { content }, finish_reason: null }],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (payload?.done) finish = finishReason(payload?.done_reason);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (buffer.trim()) {
|
||||||
|
const payload = JSON.parse(buffer);
|
||||||
|
const content = String(payload?.message?.content ?? payload?.response ?? "");
|
||||||
|
if (content) {
|
||||||
|
sentContent = true;
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { content }, finish_reason: null }],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (payload?.done) finish = finishReason(payload?.done_reason);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!sentContent) {
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { content: "" }, finish_reason: null }],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
writeSse(res, {
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: {}, finish_reason: finish }],
|
||||||
|
});
|
||||||
|
res.write("data: [DONE]\n\n");
|
||||||
|
res.end();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function proxyChat(req, res) {
|
||||||
|
const body = await readJson(req);
|
||||||
|
const model = String(body.model || process.env.OLLAMA_PROXY_MODEL || "qwen3.5:27b");
|
||||||
|
const nativeBody = {
|
||||||
|
model,
|
||||||
|
messages: normalizeMessages(body.messages),
|
||||||
|
stream: false,
|
||||||
|
think: false,
|
||||||
|
options: {
|
||||||
|
temperature: body.temperature ?? 0,
|
||||||
|
num_predict: body.max_tokens ?? body.max_completion_tokens ?? 512,
|
||||||
|
...(Number.isFinite(numCtx) && numCtx > 0 ? { num_ctx: numCtx } : {}),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
if (body.stream === true) {
|
||||||
|
await proxyStreamingChat(res, model, nativeBody);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const response = await fetch(`${upstream}/api/chat`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify(nativeBody),
|
||||||
|
});
|
||||||
|
const payload = await response.json().catch(async () => ({ error: await response.text() }));
|
||||||
|
if (!response.ok) {
|
||||||
|
sendJson(res, response.status, {
|
||||||
|
error: {
|
||||||
|
message: payload?.error || `Ollama upstream returned ${response.status}`,
|
||||||
|
type: "upstream_error",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const content = String(payload?.message?.content ?? payload?.response ?? "");
|
||||||
|
const completion = {
|
||||||
|
id: `chatcmpl-${Date.now().toString(36)}`,
|
||||||
|
object: "chat.completion",
|
||||||
|
created: Math.floor(Date.now() / 1000),
|
||||||
|
model,
|
||||||
|
system_fingerprint: "fp_ollama_native_think_false",
|
||||||
|
choices: [
|
||||||
|
{
|
||||||
|
index: 0,
|
||||||
|
message: { role: "assistant", content },
|
||||||
|
finish_reason: finishReason(payload?.done_reason),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
usage: {
|
||||||
|
prompt_tokens: payload?.prompt_eval_count ?? 0,
|
||||||
|
completion_tokens: payload?.eval_count ?? 0,
|
||||||
|
total_tokens: (payload?.prompt_eval_count ?? 0) + (payload?.eval_count ?? 0),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if (body.stream === true) {
|
||||||
|
const id = completion.id;
|
||||||
|
const created = completion.created;
|
||||||
|
sendSse(res, [
|
||||||
|
{
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: { content }, finish_reason: null }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id,
|
||||||
|
object: "chat.completion.chunk",
|
||||||
|
created,
|
||||||
|
model,
|
||||||
|
choices: [{ index: 0, delta: {}, finish_reason: completion.choices[0].finish_reason }],
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
sendJson(res, 200, completion);
|
||||||
|
}
|
||||||
|
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
const url = new URL(req.url || "/", `http://${req.headers.host || "localhost"}`);
|
||||||
|
Promise.resolve()
|
||||||
|
.then(async () => {
|
||||||
|
if (req.method === "GET" && (url.pathname === "/v1/models" || url.pathname === "/models")) {
|
||||||
|
await proxyModels(res);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (req.method === "POST" && (url.pathname === "/v1/chat/completions" || url.pathname === "/chat/completions")) {
|
||||||
|
await proxyChat(req, res);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (req.method === "GET" && url.pathname === "/health") {
|
||||||
|
sendJson(res, 200, { ok: true, upstream });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
sendJson(res, 404, { error: { message: "not found", type: "not_found" } });
|
||||||
|
})
|
||||||
|
.catch((error) => {
|
||||||
|
sendJson(res, 500, { error: { message: error?.message || String(error), type: "proxy_error" } });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
server.listen(listenPort, listenHost, () => {
|
||||||
|
console.log(`Ollama thinkless OpenAI proxy listening on http://${listenHost}:${listenPort}/v1 -> ${upstream}`);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user