282 lines
12 KiB
Markdown
282 lines
12 KiB
Markdown
# OpenAI-Compatible Multi-Model API Wrapper for ComfyUI
|
||
|
||
FastAPI proxy that exposes `POST /v1/images/generations` (OpenAI format) and
|
||
routes to **multiple** ComfyUI workflows depending on the requested `model`.
|
||
|
||
Supported models: `ideogram4`, `flux2`, `flux1-dev`, `flux1-schnell`.
|
||
|
||
## When to use
|
||
|
||
- You need one endpoint that can switch between Ideogram 4 and FLUX
|
||
families without reconfiguring the frontend.
|
||
- Your chat UI only speaks OpenAI `/v1/images/generations` but you want
|
||
to use ComfyUI's native pipelines (including non-standard nodes like
|
||
`Ideogram4Scheduler`, `ModelSamplingFlux`, `DualCLIPLoader`).
|
||
- You run ComfyUI inside a Proxmox LXC/VM and want a systemd-managed
|
||
adapter service.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Frontend (OpenWebUI, LibreChat, etc.)
|
||
POST /v1/images/generations {model, prompt, size, ...}
|
||
│
|
||
▼
|
||
FastAPI Adapter (this wrapper)
|
||
─ model switch ──► _build_ideogram4_workflow()
|
||
_build_flux2_workflow()
|
||
_build_flux1_workflow()
|
||
│
|
||
▼
|
||
ComfyUI REST API POST /prompt
|
||
│
|
||
▼
|
||
Poll /history/{prompt_id} → fetch /view
|
||
```
|
||
|
||
## Adapter Code (Multi-Model)
|
||
|
||
The full source is provided as a **template** in this skill:
|
||
`templates/comfyui-openai-adapter.py`
|
||
|
||
Key design choices:
|
||
|
||
- **Model routing** — `req.model` selects the workflow builder.
|
||
- **Size validation / fallback** — Ideogram4 only accepts specific
|
||
resolutions. FLUX is more permissive. Invalid sizes silently fall back to the model's default.
|
||
- **Loader node per model family:**
|
||
- Ideogram4 → `UNETLoader` (safetensors)
|
||
- FLUX (GGUF) → `UnetLoaderGGUF` (from ComfyUI-GGUF custom node)
|
||
- Ideogram4 → single `CLIPLoader` with `type: "ideogram4"`
|
||
- FLUX → `DualCLIPLoader` with `type: "flux"` (clip_l + t5xxl)
|
||
- **Quality → steps mapping:** `low` (8), `medium` (20), `high` (30), `ultra` (50).
|
||
FLUX1-schnell hard-capped to 8 steps.
|
||
- **Ideogram4 minimum steps:** do not go below ~12 steps — the model
|
||
produces near-blank gray blocks at ≤8 steps. Clamp or warn when
|
||
the user explicitly requests fewer.
|
||
- **Ideogram4 asymmetric CFG wiring (CRITICAL):** The adapter's `_build_ideogram4_workflow()`
|
||
MUST use `CFGOverride` + `DualModelGuider` + a separate unconditional UNet
|
||
(`ideogram4_unconditional_fp8_scaled`) + `ConditioningZeroOut`. Using `BasicGuider`
|
||
does NOT throw an error but silently degrades image quality. The `DualModelGuider`
|
||
input for the unconditional model is `model_negative` (NOT `model_1`).
|
||
See `references/ideogram4.md` for the full blueprint.
|
||
- **Gray-block detection:** If Ideogram4 output has PIL variance < 15 across all
|
||
channels, the generation produced a blank/gray image (often at `quality=low` + `style=photorealistic`).
|
||
Flag these as failures and retry with higher steps or different style.
|
||
- **ROCm environment** — if ComfyUI runs on AMD ROCm inside an LXC,
|
||
the *systemd service* for ComfyUI must set
|
||
`Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0` (or whatever matches
|
||
the GPU). The adapter itself does not need ROCm env vars.
|
||
- **Timeout for long generations** — set the adapter's HTTP poll loop
|
||
to at least 1800 s (30 min) for high-resolution workloads.
|
||
Edit `for i in range(720)` (30 min at 5 s intervals) and set
|
||
`httpx.AsyncClient(timeout=2400)` on the initial `/prompt` POST.
|
||
- **Synchronous (blocking) by design** — the adapter does NOT implement
|
||
async jobs. The HTTP response blocks until ComfyUI finishes. For
|
||
multi-hour jobs this means the client must keep the connection open.
|
||
If a true async job system is needed, build `POST → job_id` +
|
||
`GET /v1/jobs/{id}` separately.
|
||
|
||
## Running it
|
||
|
||
### Option A: Python venv (quick test)
|
||
|
||
```bash
|
||
python3 -m venv /opt/ideogram4-api/venv
|
||
source /opt/ideogram4-api/venv/bin/activate
|
||
pip install fastapi uvicorn httpx pydantic
|
||
export COMFYUI_HOST=localhost
|
||
export COMFYUI_PORT=8188
|
||
python3 comfyui-openai-adapter.py
|
||
```
|
||
|
||
### Option B: Docker on dedicated host (persistent)
|
||
|
||
```dockerfile
|
||
FROM python:3.12-slim
|
||
WORKDIR /app
|
||
RUN pip install --no-cache-dir fastapi uvicorn httpx pydantic
|
||
COPY comfyui-openai-adapter.py ./main.py
|
||
EXPOSE 8000
|
||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||
```
|
||
|
||
> The adapter and ComfyUI should be on the same LAN or host. Latency matters because the adapter polls ComfyUI every 5 s for up to 60 min.
|
||
|
||
### Option C: systemd service (LXC/VM — recommended for Proxmox)
|
||
|
||
**Prerequisite:** Kill any previously manual/nohup-started ComfyUI or adapter processes *before* enabling systemd units. Old processes hold ports 8188 and 8000 and cause `address already in use`.
|
||
|
||
```bash
|
||
# Inside LXC container (e.g. CT 204)
|
||
ss -tlnp | grep -E '8188|8000'
|
||
fuser -k 8188/tcp 2>/dev/null
|
||
fuser -k 8000/tcp 2>/dev/null
|
||
```
|
||
|
||
**`comfyui.service`** (note the ROCm env var for gfx1150):
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=ComfyUI
|
||
After=network.target
|
||
|
||
[Service]
|
||
Type=simple
|
||
User=root
|
||
WorkingDirectory=/opt/ComfyUI
|
||
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
|
||
Restart=always
|
||
RestartSec=10
|
||
Environment=HIP_VISIBLE_DEVICES=0
|
||
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
|
||
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
**`ideogram4-api.service`** (rename to `comfyui-api.service` if generic):
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=ComfyUI OpenAI API Wrapper
|
||
After=network.target comfyui.service
|
||
Requires=comfyui.service
|
||
|
||
[Service]
|
||
Type=simple
|
||
User=root
|
||
WorkingDirectory=/opt/ideogram4-api
|
||
ExecStart=/opt/ideogram4-api/venv/bin/python /opt/ideogram4-api/main.py
|
||
Restart=always
|
||
RestartSec=5
|
||
Environment=PATH=/opt/ideogram4-api/venv/bin:/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||
Environment=COMFYUI_HOST=localhost
|
||
Environment=COMFYUI_PORT=8188
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
> **Filename note:** The adapter source may be saved as `main.py`, `app.py`, or `adapter.py`. Check `ls /opt/ideogram4-api/` and match `ExecStart=` exactly.
|
||
|
||
Enable & start:
|
||
```bash
|
||
systemctl daemon-reload
|
||
systemctl enable --now comfyui.service
|
||
sleep 15
|
||
systemctl enable --now ideogram4-api.service
|
||
```
|
||
|
||
## Extended OpenAI-compatible parameters
|
||
|
||
Beyond the standard `model`, `prompt`, `n`, `size`, `response_format`,
|
||
the adapter accepts these ComfyUI extensions:
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `seed` | int | random | Fixed seed. Random if omitted. |
|
||
| `steps` | int | from `quality` | Denoising steps (1–100). Overrides `quality` if set. |
|
||
| `cfg` | float | model-specific | Guidance scale. Ideogram4=1.0, FLUX=3.5, others=7.5 |
|
||
| `quality` | string | `medium` | Maps to steps: `low`(8), `medium`(20), `high`(30), `ultra`(50). |
|
||
| `negative_prompt` | string | `""` | Negative prompt text. |
|
||
| `style` | string | `""` | Style modifier string. |
|
||
| `control_strength` | float | — | Reserved for future ControlNet integration. |
|
||
|
||
Example request body:
|
||
```json
|
||
{
|
||
"model": "flux2",
|
||
"prompt": "a cyberpunk alley at night, neon signs",
|
||
"n": 1,
|
||
"size": "1024x1024",
|
||
"quality": "high",
|
||
"seed": 42,
|
||
"cfg": 3.5,
|
||
"negative_prompt": "blurry, low quality"
|
||
}
|
||
```
|
||
|
||
## Frontend configuration (OpenWebUI example)
|
||
|
||
1. Admin → **Settings** → **Images**
|
||
2. **Image Generation Engine:** `Open AI`
|
||
3. **OpenAI API Base URL:** `http://<adapter-host>:8000/v1`
|
||
4. **OpenAI API Key:** any dummy string (adapter does not validate)
|
||
5. **Model:** `ideogram4` or `flux2` or `flux1-dev`
|
||
|
||
## Pitfalls
|
||
|
||
| Issue | Cause | Fix |
|
||
|---|---|---|
|
||
| `TimeoutError` after 300 s | High resolution or GPU under load | Clamp `size`, increase timeout, reduce `steps` |
|
||
| `500` from ComfyUI + `value_not_in_list` | `size` not in model's allowed resolutions | Check allowed sizes; use defaults |
|
||
| `[Errno 98] address already in use` | Old manual `nohup` process still running | `fuser -k 8000/tcp` / `fuser -k 8188/tcp`, then restart systemd |
|
||
| `can't open file '/opt/ideogram4-api/app.py'` | `ExecStart` mismatches actual filename | Check `ls /opt/ideogram4-api/` and align `ExecStart` |
|
||
| `UNETLoader` node error for `.gguf` model | Using `UNETLoader` for a GGUF file | Switch to `UnetLoaderGGUF` |
|
||
| `RuntimeError` during text-encode (ROCm) | Missing `HSA_OVERRIDE_GFX_VERSION` | Add to `comfyui.service` `Environment=` line |
|
||
| Ideogram4 black/corrupted output | Corrupt `.safetensors` or wrong VAE | Verify file integrity; use `flux2-vae.safetensors` |
|
||
| FLUX output dimensions wrong | Used `EmptyLatentImage` instead of `EmptyFlux2LatentImage` | Ideogram4 uses `EmptyFlux2LatentImage`; FLUX.1 uses `EmptyLatentImage` |
|
||
| `DualModelGuider.execute() got unexpected keyword argument 'model_1'` | Wrong workflow key name | Use `model_negative`, NOT `model_1` |
|
||
| systemd `comfyui.service` keeps failing with `exit-code` | Port 8188 held by old manual ComfyUI | `pkill -f "main.py.*8188"`, then `systemctl restart` |
|
||
|
||
## Model-specific node quick-reference
|
||
|
||
### Ideogram 4 local (with asymmetric CFG — correct wiring)
|
||
|
||
```json
|
||
{
|
||
"1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
||
"2": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"},
|
||
"class_type": "UNETLoader"},
|
||
"3": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4"},
|
||
"class_type": "CLIPLoader"},
|
||
"4": {"inputs": {"text": "...", "clip": ["3", 0]}, "class_type": "CLIPTextEncode"},
|
||
"5": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
||
"6": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
||
"7": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
||
"8": {"inputs": {"steps": 20, "width": 1024, "height": 1024, "mu": 0.5, "std": 1.75},
|
||
"class_type": "Ideogram4Scheduler"},
|
||
"15": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"},
|
||
"class_type": "UNETLoader"},
|
||
"16": {"inputs": {"conditioning": ["4", 0]}, "class_type": "ConditioningZeroOut"},
|
||
"17": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["2", 0]},
|
||
"class_type": "CFGOverride"},
|
||
"9": {"inputs": {"cfg": 7.0, "model": ["17", 0], "positive": ["4", 0],
|
||
"model_negative": ["15", 0], "negative": ["16", 0]},
|
||
"class_type": "DualModelGuider"},
|
||
"10": {"inputs": {"noise": ["6", 0], "guider": ["9", 0], "sampler": ["7", 0],
|
||
"sigmas": ["8", 0], "latent_image": ["5", 0]}, "class_type": "SamplerCustomAdvanced"},
|
||
"11": {"inputs": {"samples": ["10", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
|
||
"12": {"inputs": {"filename_prefix": "ideogram4_api", "images": ["11", 0]}, "class_type": "SaveImage"}
|
||
}
|
||
```
|
||
Allowed sizes: `1024x1024`, `1024x1536`, `1536x1024`, `832x1216`, `1216x832`, `896x1152`, `1152x896`.
|
||
|
||
### FLUX.2 / FLUX.1
|
||
```json
|
||
{
|
||
"1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
||
"2": {"inputs": {"unet_name": "flux1-dev-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
|
||
"3": {"inputs": {"model": ["2", 0], "max_shift": 1.15, "base_shift": 0.5,
|
||
"width": 1024, "height": 1024}, "class_type": "ModelSamplingFlux"},
|
||
"4": {"inputs": {"clip_name1": "clip_l.safetensors", "clip_name2": "t5xxl_fp8_e4m3fn.safetensors",
|
||
"type": "flux", "device": "default"}, "class_type": "DualCLIPLoader"},
|
||
"5": {"inputs": {"text": "...", "clip": ["4", 0]}, "class_type": "CLIPTextEncode"},
|
||
"6": {"inputs": {"conditioning": ["5", 0], "guidance": 3.5}, "class_type": "FluxGuidance"},
|
||
"7": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyLatentImage"},
|
||
"8": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
||
"9": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
||
"10": {"inputs": {"model": ["3", 0], "steps": 20, "denoise": 1.0, "scheduler": "simple",
|
||
"sampler": ["9", 0]}, "class_type": "BasicScheduler"},
|
||
"11": {"inputs": {"model": ["3", 0], "conditioning": ["6", 0]}, "class_type": "BasicGuider"},
|
||
"12": {"inputs": {"noise": ["8", 0], "guider": ["11", 0], "sampler": ["9", 0],
|
||
"sigmas": ["10", 0], "latent_image": ["7", 0]}, "class_type": "SamplerCustomAdvanced"},
|
||
"13": {"inputs": {"samples": ["12", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
|
||
"14": {"inputs": {"filename_prefix": "flux2_api", "images": ["13", 0]}, "class_type": "SaveImage"}
|
||
}
|
||
```
|
||
Allowed sizes: all common resolutions up to 1536×1536. FLUX schnell defaults to 4–8 steps; dev to 20–50.
|