dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/creative/comfyui/references/openai-api-wrapper.md
T

12 KiB
Raw Permalink Blame History

OpenAI-Compatible Multi-Model API Wrapper for ComfyUI

FastAPI proxy that exposes POST /v1/images/generations (OpenAI format) and routes to multiple ComfyUI workflows depending on the requested model.

Supported models: ideogram4, flux2, flux1-dev, flux1-schnell.

When to use

  • You need one endpoint that can switch between Ideogram 4 and FLUX families without reconfiguring the frontend.
  • Your chat UI only speaks OpenAI /v1/images/generations but you want to use ComfyUI's native pipelines (including non-standard nodes like Ideogram4Scheduler, ModelSamplingFlux, DualCLIPLoader).
  • You run ComfyUI inside a Proxmox LXC/VM and want a systemd-managed adapter service.

Architecture

Frontend (OpenWebUI, LibreChat, etc.)
   POST /v1/images/generations  {model, prompt, size, ...}
         │
         ▼
   FastAPI Adapter (this wrapper)
   ─ model switch ──► _build_ideogram4_workflow()
                     _build_flux2_workflow()
                     _build_flux1_workflow()
         │
         ▼
   ComfyUI REST API  POST /prompt
         │
         ▼
   Poll /history/{prompt_id} → fetch /view

Adapter Code (Multi-Model)

The full source is provided as a template in this skill: templates/comfyui-openai-adapter.py

Key design choices:

  • Model routing — req.model selects the workflow builder.
  • Size validation / fallback — Ideogram4 only accepts specific resolutions. FLUX is more permissive. Invalid sizes silently fall back to the model's default.
  • Loader node per model family:
    • Ideogram4 → UNETLoader (safetensors)
    • FLUX (GGUF) → UnetLoaderGGUF (from ComfyUI-GGUF custom node)
    • Ideogram4 → single CLIPLoader with type: "ideogram4"
    • FLUX → DualCLIPLoader with type: "flux" (clip_l + t5xxl)
  • Quality → steps mapping: low (8), medium (20), high (30), ultra (50). FLUX1-schnell hard-capped to 8 steps.
  • Ideogram4 minimum steps: do not go below ~12 steps — the model produces near-blank gray blocks at ≤8 steps. Clamp or warn when the user explicitly requests fewer.
  • Ideogram4 asymmetric CFG wiring (CRITICAL): The adapter's _build_ideogram4_workflow() MUST use CFGOverride + DualModelGuider + a separate unconditional UNet (ideogram4_unconditional_fp8_scaled) + ConditioningZeroOut. Using BasicGuider does NOT throw an error but silently degrades image quality. The DualModelGuider input for the unconditional model is model_negative (NOT model_1). See references/ideogram4.md for the full blueprint.
  • Gray-block detection: If Ideogram4 output has PIL variance < 15 across all channels, the generation produced a blank/gray image (often at quality=low + style=photorealistic). Flag these as failures and retry with higher steps or different style.
  • ROCm environment — if ComfyUI runs on AMD ROCm inside an LXC, the systemd service for ComfyUI must set Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 (or whatever matches the GPU). The adapter itself does not need ROCm env vars.
  • Timeout for long generations — set the adapter's HTTP poll loop to at least 1800 s (30 min) for high-resolution workloads. Edit for i in range(720) (30 min at 5 s intervals) and set httpx.AsyncClient(timeout=2400) on the initial /prompt POST.
  • Synchronous (blocking) by design — the adapter does NOT implement async jobs. The HTTP response blocks until ComfyUI finishes. For multi-hour jobs this means the client must keep the connection open. If a true async job system is needed, build POST → job_id + GET /v1/jobs/{id} separately.

Running it

Option A: Python venv (quick test)

python3 -m venv /opt/ideogram4-api/venv
source /opt/ideogram4-api/venv/bin/activate
pip install fastapi uvicorn httpx pydantic
export COMFYUI_HOST=localhost
export COMFYUI_PORT=8188
python3 comfyui-openai-adapter.py

Option B: Docker on dedicated host (persistent)

FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn httpx pydantic
COPY comfyui-openai-adapter.py ./main.py
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The adapter and ComfyUI should be on the same LAN or host. Latency matters because the adapter polls ComfyUI every 5 s for up to 60 min.

Option C: systemd service (LXC/VM — recommended for Proxmox)

Prerequisite: Kill any previously manual/nohup-started ComfyUI or adapter processes before enabling systemd units. Old processes hold ports 8188 and 8000 and cause address already in use.

# Inside LXC container (e.g. CT 204)
ss -tlnp | grep -E '8188|8000'
fuser -k 8188/tcp 2>/dev/null
fuser -k 8000/tcp 2>/dev/null

comfyui.service (note the ROCm env var for gfx1150):

[Unit]
Description=ComfyUI
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ComfyUI
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
Restart=always
RestartSec=10
Environment=HIP_VISIBLE_DEVICES=0
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib

[Install]
WantedBy=multi-user.target

ideogram4-api.service (rename to comfyui-api.service if generic):

[Unit]
Description=ComfyUI OpenAI API Wrapper
After=network.target comfyui.service
Requires=comfyui.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ideogram4-api
ExecStart=/opt/ideogram4-api/venv/bin/python /opt/ideogram4-api/main.py
Restart=always
RestartSec=5
Environment=PATH=/opt/ideogram4-api/venv/bin:/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
Environment=COMFYUI_HOST=localhost
Environment=COMFYUI_PORT=8188

[Install]
WantedBy=multi-user.target

Filename note: The adapter source may be saved as main.py, app.py, or adapter.py. Check ls /opt/ideogram4-api/ and match ExecStart= exactly.

Enable & start:

systemctl daemon-reload
systemctl enable --now comfyui.service
sleep 15
systemctl enable --now ideogram4-api.service

Extended OpenAI-compatible parameters

Beyond the standard model, prompt, n, size, response_format, the adapter accepts these ComfyUI extensions:

Parameter Type Default Description
seed int random Fixed seed. Random if omitted.
steps int from quality Denoising steps (1–100). Overrides quality if set.
cfg float model-specific Guidance scale. Ideogram4=1.0, FLUX=3.5, others=7.5
quality string medium Maps to steps: low(8), medium(20), high(30), ultra(50).
negative_prompt string "" Negative prompt text.
style string "" Style modifier string.
control_strength float — Reserved for future ControlNet integration.

Example request body:

{
  "model": "flux2",
  "prompt": "a cyberpunk alley at night, neon signs",
  "n": 1,
  "size": "1024x1024",
  "quality": "high",
  "seed": 42,
  "cfg": 3.5,
  "negative_prompt": "blurry, low quality"
}

Frontend configuration (OpenWebUI example)

  1. Admin → Settings → Images
  2. Image Generation Engine: Open AI
  3. OpenAI API Base URL: http://<adapter-host>:8000/v1
  4. OpenAI API Key: any dummy string (adapter does not validate)
  5. Model: ideogram4 or flux2 or flux1-dev

Pitfalls

Issue Cause Fix
TimeoutError after 300 s High resolution or GPU under load Clamp size, increase timeout, reduce steps
500 from ComfyUI + value_not_in_list size not in model's allowed resolutions Check allowed sizes; use defaults
[Errno 98] address already in use Old manual nohup process still running fuser -k 8000/tcp / fuser -k 8188/tcp, then restart systemd
can't open file '/opt/ideogram4-api/app.py' ExecStart mismatches actual filename Check ls /opt/ideogram4-api/ and align ExecStart
UNETLoader node error for .gguf model Using UNETLoader for a GGUF file Switch to UnetLoaderGGUF
RuntimeError during text-encode (ROCm) Missing HSA_OVERRIDE_GFX_VERSION Add to comfyui.service Environment= line
Ideogram4 black/corrupted output Corrupt .safetensors or wrong VAE Verify file integrity; use flux2-vae.safetensors
FLUX output dimensions wrong Used EmptyLatentImage instead of EmptyFlux2LatentImage Ideogram4 uses EmptyFlux2LatentImage; FLUX.1 uses EmptyLatentImage
DualModelGuider.execute() got unexpected keyword argument 'model_1' Wrong workflow key name Use model_negative, NOT model_1
systemd comfyui.service keeps failing with exit-code Port 8188 held by old manual ComfyUI pkill -f "main.py.*8188", then systemctl restart

Model-specific node quick-reference

Ideogram 4 local (with asymmetric CFG — correct wiring)

{
  "1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
  "2": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"},
        "class_type": "UNETLoader"},
  "3": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4"},
        "class_type": "CLIPLoader"},
  "4": {"inputs": {"text": "...", "clip": ["3", 0]}, "class_type": "CLIPTextEncode"},
  "5": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
  "6": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
  "7": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
  "8": {"inputs": {"steps": 20, "width": 1024, "height": 1024, "mu": 0.5, "std": 1.75},
        "class_type": "Ideogram4Scheduler"},
  "15": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"},
         "class_type": "UNETLoader"},
  "16": {"inputs": {"conditioning": ["4", 0]}, "class_type": "ConditioningZeroOut"},
  "17": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["2", 0]},
         "class_type": "CFGOverride"},
  "9": {"inputs": {"cfg": 7.0, "model": ["17", 0], "positive": ["4", 0],
         "model_negative": ["15", 0], "negative": ["16", 0]},
        "class_type": "DualModelGuider"},
  "10": {"inputs": {"noise": ["6", 0], "guider": ["9", 0], "sampler": ["7", 0],
          "sigmas": ["8", 0], "latent_image": ["5", 0]}, "class_type": "SamplerCustomAdvanced"},
  "11": {"inputs": {"samples": ["10", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
  "12": {"inputs": {"filename_prefix": "ideogram4_api", "images": ["11", 0]}, "class_type": "SaveImage"}
}

Allowed sizes: 1024x1024, 1024x1536, 1536x1024, 832x1216, 1216x832, 896x1152, 1152x896.

FLUX.2 / FLUX.1

{
  "1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
  "2": {"inputs": {"unet_name": "flux1-dev-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
  "3": {"inputs": {"model": ["2", 0], "max_shift": 1.15, "base_shift": 0.5,
          "width": 1024, "height": 1024}, "class_type": "ModelSamplingFlux"},
  "4": {"inputs": {"clip_name1": "clip_l.safetensors", "clip_name2": "t5xxl_fp8_e4m3fn.safetensors",
          "type": "flux", "device": "default"}, "class_type": "DualCLIPLoader"},
  "5": {"inputs": {"text": "...", "clip": ["4", 0]}, "class_type": "CLIPTextEncode"},
  "6": {"inputs": {"conditioning": ["5", 0], "guidance": 3.5}, "class_type": "FluxGuidance"},
  "7": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyLatentImage"},
  "8": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
  "9": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
  "10": {"inputs": {"model": ["3", 0], "steps": 20, "denoise": 1.0, "scheduler": "simple",
           "sampler": ["9", 0]}, "class_type": "BasicScheduler"},
  "11": {"inputs": {"model": ["3", 0], "conditioning": ["6", 0]}, "class_type": "BasicGuider"},
  "12": {"inputs": {"noise": ["8", 0], "guider": ["11", 0], "sampler": ["9", 0],
           "sigmas": ["10", 0], "latent_image": ["7", 0]}, "class_type": "SamplerCustomAdvanced"},
  "13": {"inputs": {"samples": ["12", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
  "14": {"inputs": {"filename_prefix": "flux2_api", "images": ["13", 0]}, "class_type": "SaveImage"}
}

Allowed sizes: all common resolutions up to 1536×1536. FLUX schnell defaults to 4–8 steps; dev to 20–50.

Reference in New Issue View Git Blame Copy Permalink