hermes-skills/creative/comfyui/references/openwebui-integration.md

OpenWebUI ↔ ComfyUI Integration

OpenWebUI's native ComfyUI connector is hard-wired for SD/SDXL pipelines (CLIP-L + T5XXL, EmptyLatentImage, standard VAE). It does not work with FLUX.2 (Qwen3-VL, EmptyFlux2LatentImage, TAE/VAE).

The cleanest solution is a lightweight OpenAI-compatible adapter that exposes /v1/images/generations and internally submits a FLUX.2 workflow to ComfyUI's REST API.

---

Architecture

OpenWebUI  ──POST /v1/images/generations──►  Adapter  ──POST /api/prompt──►  ComfyUI
(CT 135)         (OpenAI format)           (Docker)       (workflow JSON)    (CT 204)

The adapter runs as a Docker container (or Python service) on a host that has Layer-2/Layer-3 reachability to ComfyUI.

---

Adapter Code

adapter.py

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from typing import Optional
import httpx
import json
import base64
import asyncio
import time
import random
import os

app = FastAPI(title="ComfyUI OpenAI Adapter", version="0.1.0")

COMFYUI_URL = os.environ.get("COMFYUI_URL", "http://127.0.0.1:8188")
WORKFLOW_PREFIX = os.environ.get("WORKFLOW_PREFIX", "oai_adapter")

FLUX2_WORKFLOW = {
    "1": {"inputs": {"unet_name": "flux-2-klein-9b-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
    "2": {"inputs": {"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf", "type": "flux2"}, "class_type": "CLIPLoaderGGUF"},
    "3": {"inputs": {"vae_name": "taef2"}, "class_type": "VAELoader"},
    "4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
    "5": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
    "6": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
    "7": {"inputs": {"seed": 42, "steps": 4, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0,
            "model": ["1", 0], "positive": ["5", 0], "negative": ["6", 0], "latent_image": ["4", 0]},
        "class_type": "KSampler"},
    "8": {"inputs": {"samples": ["7", 0], "vae": ["3", 0]}, "class_type": "VAEDecode"},
    "9": {"inputs": {"filename_prefix": WORKFLOW_PREFIX, "images": ["8", 0]}, "class_type": "SaveImage"}
}

class ImageRequest(BaseModel):
    prompt: str
    n: Optional[int] = 1
    size: Optional[str] = "1024x1024"
    response_format: Optional[str] = "b64_json"
    model: Optional[str] = "flux2"

async def queue_workflow(workflow: dict) -> str:
    async with httpx.AsyncClient() as client:
        r = await client.post(f"{COMFYUI_URL}/api/prompt", json={"prompt": workflow})
        r.raise_for_status()
        return r.json()["prompt_id"]

async def wait_for_image(prompt_id: str, timeout: int = 300) -> list:
    async with httpx.AsyncClient() as client:
        start = time.time()
        while time.time() - start < timeout:
            r = await client.get(f"{COMFYUI_URL}/history/{prompt_id}")
            if r.status_code == 200:
                data = r.json()
                if prompt_id in data and "outputs" in data[prompt_id]:
                    outputs = []
                    for node_id, node_out in data[prompt_id]["outputs"].items():
                        if "images" in node_out:
                            outputs.extend(node_out["images"])
                    if outputs:
                        return outputs
            await asyncio.sleep(1)
        raise TimeoutError(f"Timeout waiting for prompt {prompt_id}")

async def fetch_image(filename: str, subfolder: str = "", folder_type: str = "output") -> bytes:
    async with httpx.AsyncClient() as client:
        params = {"filename": filename, "subfolder": subfolder, "type": folder_type}
        r = await client.get(f"{COMFYUI_URL}/view", params=params)
        r.raise_for_status()
        return r.content

@app.post("/v1/images/generations")
async def generate_images(req: ImageRequest):
    if req.n > 4:
        return JSONResponse({"error": {"message": "n must be <= 4", "type": "invalid_request_error"}}, status_code=400)

    try:
        w, h = map(int, req.size.split("x"))
    except ValueError:
        w, h = 1024, 1024

    # CRITICAL: clamp size to prevent timeout/OOM on consumer GPUs
    MAX_SIZE = 1536
    if w > MAX_SIZE or h > MAX_SIZE:
        scale = MAX_SIZE / max(w, h)
        w = int(w * scale)
        h = int(h * scale)

    images = []
    for _ in range(req.n):
        wf = json.loads(json.dumps(FLUX2_WORKFLOW))
        wf["5"]["inputs"]["text"] = req.prompt
        wf["4"]["inputs"]["width"] = w
        wf["4"]["inputs"]["height"] = h
        wf["7"]["inputs"]["seed"] = random.randint(1, 2**32)

        prompt_id = await queue_workflow(wf)
        imgs = await wait_for_image(prompt_id, timeout=300)
        if not imgs:
            raise RuntimeError("No images returned")

        img_data = await fetch_image(imgs[0]["filename"], imgs[0].get("subfolder", ""), imgs[0].get("type", "output"))
        b64 = base64.b64encode(img_data).decode()
        images.append({"b64_json": b64})

    return {"created": int(time.time()), "data": images}

@app.get("/health")
async def health():
    return {"status": "ok"}

Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY adapter.py .
EXPOSE 9000
CMD ["uvicorn", "adapter:app", "--host", "0.0.0.0", "--port", "9000"]

requirements.txt

fastapi>=0.110.0
uvicorn>=0.29.0
httpx>=0.27.0
pydantic>=2.0.0

docker-compose.yml

services:
  comfyui-openai-adapter:
    build: .
    container_name: comfyui-openai-adapter
    ports:
      - "9000:9000"
    environment:
      - COMFYUI_URL=http://10.0.30.97:8188   # <-- adapt to your ComfyUI IP
      - WORKFLOW_PREFIX=oai_bifrost
    restart: unless-stopped

Network tip: If the adapter and ComfyUI are on the same Layer-2 segment, prefer --network host (or attach the container to an existing bridge that reaches ComfyUI) to avoid NAT/port-mapping issues.

---

OpenWebUI Configuration

1. Admin Panel → Settings → Images
2. Image Generation Engine: Open AI
3. OpenAI API Base URL: http://<adapter-host>:9000/v1
4. OpenAI API Key: any dummy string (the adapter does not validate keys)
5. Model: flux2 (or leave default)

---

Critical Pitfalls

Issue	Cause	Fix
TimeoutError after 300 s	OpenWebUI/DALL-E 3 sends size: "4096x4096"	Adapter clamps to MAX_SIZE = 1536
OOM / CUDA out of memory	Resolution too high for 48 GB shared VRAM	Keep ≤ 1536 px on consumer APUs
500 Internal Server Error	ComfyUI node missing (CLIPLoaderGGUF, etc.)	Run comfy node install comfyui-gguf and verify with curl /object_info
Adapter cannot reach ComfyUI	Docker bridge isolation	Use --network host or place both on same bridge
Empty/black images	Wrong VAE (ae.safetensors instead of taef2 for FLUX.2)	Use taef2 TAE or the FLUX.2-specific VAE

---

Environment Snapshot (Schön Consulting, 2026-06-20)

• ComfyUI: PVE CT 204, 10.0.30.97:8188, ROCm gfx1150 (Radeon 890M)
• OpenWebUI: PVE CT 135, 10.0.30.102:8080
• Adapter: Docker-Host 10.0.30.99:9000 (Portainer node)
• Performance: 1024×1024 ≈ 110 s, 1536×1536 ≈ 180 s (4 steps, euler/simple)