202 lines
7.2 KiB
Markdown
202 lines
7.2 KiB
Markdown
# OpenWebUI ↔ ComfyUI Integration
|
||
|
||
OpenWebUI's native ComfyUI connector is hard-wired for SD/SDXL pipelines (CLIP-L + T5XXL, `EmptyLatentImage`, standard VAE). It **does not work** with FLUX.2 (Qwen3-VL, `EmptyFlux2LatentImage`, TAE/VAE).
|
||
|
||
The cleanest solution is a lightweight OpenAI-compatible adapter that exposes `/v1/images/generations` and internally submits a FLUX.2 workflow to ComfyUI's REST API.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
OpenWebUI ──POST /v1/images/generations──► Adapter ──POST /api/prompt──► ComfyUI
|
||
(CT 135) (OpenAI format) (Docker) (workflow JSON) (CT 204)
|
||
```
|
||
|
||
The adapter runs as a Docker container (or Python service) on a host that has Layer-2/Layer-3 reachability to ComfyUI.
|
||
|
||
---
|
||
|
||
## Adapter Code
|
||
|
||
### `adapter.py`
|
||
|
||
```python
|
||
from fastapi import FastAPI
|
||
from fastapi.responses import JSONResponse
|
||
from pydantic import BaseModel
|
||
from typing import Optional
|
||
import httpx
|
||
import json
|
||
import base64
|
||
import asyncio
|
||
import time
|
||
import random
|
||
import os
|
||
|
||
app = FastAPI(title="ComfyUI OpenAI Adapter", version="0.1.0")
|
||
|
||
COMFYUI_URL = os.environ.get("COMFYUI_URL", "http://127.0.0.1:8188")
|
||
WORKFLOW_PREFIX = os.environ.get("WORKFLOW_PREFIX", "oai_adapter")
|
||
|
||
FLUX2_WORKFLOW = {
|
||
"1": {"inputs": {"unet_name": "flux-2-klein-9b-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
|
||
"2": {"inputs": {"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf", "type": "flux2"}, "class_type": "CLIPLoaderGGUF"},
|
||
"3": {"inputs": {"vae_name": "taef2"}, "class_type": "VAELoader"},
|
||
"4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
||
"5": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
|
||
"6": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
|
||
"7": {"inputs": {"seed": 42, "steps": 4, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0,
|
||
"model": ["1", 0], "positive": ["5", 0], "negative": ["6", 0], "latent_image": ["4", 0]},
|
||
"class_type": "KSampler"},
|
||
"8": {"inputs": {"samples": ["7", 0], "vae": ["3", 0]}, "class_type": "VAEDecode"},
|
||
"9": {"inputs": {"filename_prefix": WORKFLOW_PREFIX, "images": ["8", 0]}, "class_type": "SaveImage"}
|
||
}
|
||
|
||
class ImageRequest(BaseModel):
|
||
prompt: str
|
||
n: Optional[int] = 1
|
||
size: Optional[str] = "1024x1024"
|
||
response_format: Optional[str] = "b64_json"
|
||
model: Optional[str] = "flux2"
|
||
|
||
async def queue_workflow(workflow: dict) -> str:
|
||
async with httpx.AsyncClient() as client:
|
||
r = await client.post(f"{COMFYUI_URL}/api/prompt", json={"prompt": workflow})
|
||
r.raise_for_status()
|
||
return r.json()["prompt_id"]
|
||
|
||
async def wait_for_image(prompt_id: str, timeout: int = 300) -> list:
|
||
async with httpx.AsyncClient() as client:
|
||
start = time.time()
|
||
while time.time() - start < timeout:
|
||
r = await client.get(f"{COMFYUI_URL}/history/{prompt_id}")
|
||
if r.status_code == 200:
|
||
data = r.json()
|
||
if prompt_id in data and "outputs" in data[prompt_id]:
|
||
outputs = []
|
||
for node_id, node_out in data[prompt_id]["outputs"].items():
|
||
if "images" in node_out:
|
||
outputs.extend(node_out["images"])
|
||
if outputs:
|
||
return outputs
|
||
await asyncio.sleep(1)
|
||
raise TimeoutError(f"Timeout waiting for prompt {prompt_id}")
|
||
|
||
async def fetch_image(filename: str, subfolder: str = "", folder_type: str = "output") -> bytes:
|
||
async with httpx.AsyncClient() as client:
|
||
params = {"filename": filename, "subfolder": subfolder, "type": folder_type}
|
||
r = await client.get(f"{COMFYUI_URL}/view", params=params)
|
||
r.raise_for_status()
|
||
return r.content
|
||
|
||
@app.post("/v1/images/generations")
|
||
async def generate_images(req: ImageRequest):
|
||
if req.n > 4:
|
||
return JSONResponse({"error": {"message": "n must be <= 4", "type": "invalid_request_error"}}, status_code=400)
|
||
|
||
try:
|
||
w, h = map(int, req.size.split("x"))
|
||
except ValueError:
|
||
w, h = 1024, 1024
|
||
|
||
# CRITICAL: clamp size to prevent timeout/OOM on consumer GPUs
|
||
MAX_SIZE = 1536
|
||
if w > MAX_SIZE or h > MAX_SIZE:
|
||
scale = MAX_SIZE / max(w, h)
|
||
w = int(w * scale)
|
||
h = int(h * scale)
|
||
|
||
images = []
|
||
for _ in range(req.n):
|
||
wf = json.loads(json.dumps(FLUX2_WORKFLOW))
|
||
wf["5"]["inputs"]["text"] = req.prompt
|
||
wf["4"]["inputs"]["width"] = w
|
||
wf["4"]["inputs"]["height"] = h
|
||
wf["7"]["inputs"]["seed"] = random.randint(1, 2**32)
|
||
|
||
prompt_id = await queue_workflow(wf)
|
||
imgs = await wait_for_image(prompt_id, timeout=300)
|
||
if not imgs:
|
||
raise RuntimeError("No images returned")
|
||
|
||
img_data = await fetch_image(imgs[0]["filename"], imgs[0].get("subfolder", ""), imgs[0].get("type", "output"))
|
||
b64 = base64.b64encode(img_data).decode()
|
||
images.append({"b64_json": b64})
|
||
|
||
return {"created": int(time.time()), "data": images}
|
||
|
||
@app.get("/health")
|
||
async def health():
|
||
return {"status": "ok"}
|
||
```
|
||
|
||
### `Dockerfile`
|
||
|
||
```dockerfile
|
||
FROM python:3.12-slim
|
||
WORKDIR /app
|
||
COPY requirements.txt .
|
||
RUN pip install --no-cache-dir -r requirements.txt
|
||
COPY adapter.py .
|
||
EXPOSE 9000
|
||
CMD ["uvicorn", "adapter:app", "--host", "0.0.0.0", "--port", "9000"]
|
||
```
|
||
|
||
### `requirements.txt`
|
||
|
||
```text
|
||
fastapi>=0.110.0
|
||
uvicorn>=0.29.0
|
||
httpx>=0.27.0
|
||
pydantic>=2.0.0
|
||
```
|
||
|
||
### `docker-compose.yml`
|
||
|
||
```yaml
|
||
services:
|
||
comfyui-openai-adapter:
|
||
build: .
|
||
container_name: comfyui-openai-adapter
|
||
ports:
|
||
- "9000:9000"
|
||
environment:
|
||
- COMFYUI_URL=http://10.0.30.97:8188 # <-- adapt to your ComfyUI IP
|
||
- WORKFLOW_PREFIX=oai_bifrost
|
||
restart: unless-stopped
|
||
```
|
||
|
||
> **Network tip:** If the adapter and ComfyUI are on the same Layer-2 segment, prefer `--network host` (or attach the container to an existing bridge that reaches ComfyUI) to avoid NAT/port-mapping issues.
|
||
|
||
---
|
||
|
||
## OpenWebUI Configuration
|
||
|
||
1. Admin Panel → **Settings** → **Images**
|
||
2. **Image Generation Engine:** `Open AI`
|
||
3. **OpenAI API Base URL:** `http://<adapter-host>:9000/v1`
|
||
4. **OpenAI API Key:** any dummy string (the adapter does not validate keys)
|
||
5. **Model:** `flux2` (or leave default)
|
||
|
||
---
|
||
|
||
## Critical Pitfalls
|
||
|
||
| Issue | Cause | Fix |
|
||
|---|---|---|
|
||
| `TimeoutError` after 300 s | OpenWebUI/DALL-E 3 sends `size: "4096x4096"` | Adapter clamps to `MAX_SIZE = 1536` |
|
||
| OOM / CUDA out of memory | Resolution too high for 48 GB shared VRAM | Keep ≤ 1536 px on consumer APUs |
|
||
| `500 Internal Server Error` | ComfyUI node missing (`CLIPLoaderGGUF`, etc.) | Run `comfy node install comfyui-gguf` and verify with `curl /object_info` |
|
||
| Adapter cannot reach ComfyUI | Docker bridge isolation | Use `--network host` or place both on same bridge |
|
||
| Empty/black images | Wrong VAE (`ae.safetensors` instead of `taef2` for FLUX.2) | Use `taef2` TAE or the FLUX.2-specific VAE |
|
||
|
||
---
|
||
|
||
## Environment Snapshot (Schön Consulting, 2026-06-20)
|
||
|
||
- **ComfyUI:** PVE CT 204, `10.0.30.97:8188`, ROCm gfx1150 (Radeon 890M)
|
||
- **OpenWebUI:** PVE CT 135, `10.0.30.102:8080`
|
||
- **Adapter:** Docker-Host `10.0.30.99:9000` (Portainer node)
|
||
- **Performance:** 1024×1024 ≈ 110 s, 1536×1536 ≈ 180 s (4 steps, euler/simple)
|