Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,281 @@
+# OpenAI-Compatible Multi-Model API Wrapper for ComfyUI
+
+FastAPI proxy that exposes `POST /v1/images/generations` (OpenAI format) and
+routes to **multiple** ComfyUI workflows depending on the requested `model`.
+
+Supported models: `ideogram4`, `flux2`, `flux1-dev`, `flux1-schnell`.
+
+## When to use
+
+- You need one endpoint that can switch between Ideogram 4 and FLUX
+  families without reconfiguring the frontend.
+- Your chat UI only speaks OpenAI `/v1/images/generations` but you want
+  to use ComfyUI's native pipelines (including non-standard nodes like
+  `Ideogram4Scheduler`, `ModelSamplingFlux`, `DualCLIPLoader`).
+- You run ComfyUI inside a Proxmox LXC/VM and want a systemd-managed
+  adapter service.
+
+## Architecture
+
+```
+Frontend (OpenWebUI, LibreChat, etc.)
+   POST /v1/images/generations  {model, prompt, size, ...}
+         │
+         ▼
+   FastAPI Adapter (this wrapper)
+   ─ model switch ──► _build_ideogram4_workflow()
+                     _build_flux2_workflow()
+                     _build_flux1_workflow()
+         │
+         ▼
+   ComfyUI REST API  POST /prompt
+         │
+         ▼
+   Poll /history/{prompt_id} → fetch /view
+```
+
+## Adapter Code (Multi-Model)
+
+The full source is provided as a **template** in this skill:
+`templates/comfyui-openai-adapter.py`
+
+Key design choices:
+
+- **Model routing** — `req.model` selects the workflow builder.
+- **Size validation / fallback** — Ideogram4 only accepts specific
+  resolutions. FLUX is more permissive. Invalid sizes silently fall back to the model's default.
+- **Loader node per model family:**
+  - Ideogram4 → `UNETLoader` (safetensors)
+  - FLUX (GGUF) → `UnetLoaderGGUF` (from ComfyUI-GGUF custom node)
+  - Ideogram4 → single `CLIPLoader` with `type: "ideogram4"`
+  - FLUX → `DualCLIPLoader` with `type: "flux"` (clip_l + t5xxl)
+- **Quality → steps mapping:** `low` (8), `medium` (20), `high` (30), `ultra` (50).
+  FLUX1-schnell hard-capped to 8 steps.
+- **Ideogram4 minimum steps:** do not go below ~12 steps — the model
+  produces near-blank gray blocks at ≤8 steps. Clamp or warn when
+  the user explicitly requests fewer.
+- **Ideogram4 asymmetric CFG wiring (CRITICAL):** The adapter's `_build_ideogram4_workflow()`
+  MUST use `CFGOverride` + `DualModelGuider` + a separate unconditional UNet
+  (`ideogram4_unconditional_fp8_scaled`) + `ConditioningZeroOut`. Using `BasicGuider`
+  does NOT throw an error but silently degrades image quality. The `DualModelGuider`
+  input for the unconditional model is `model_negative` (NOT `model_1`).
+  See `references/ideogram4.md` for the full blueprint.
+- **Gray-block detection:** If Ideogram4 output has PIL variance < 15 across all
+  channels, the generation produced a blank/gray image (often at `quality=low` + `style=photorealistic`).
+  Flag these as failures and retry with higher steps or different style.
+- **ROCm environment** — if ComfyUI runs on AMD ROCm inside an LXC,
+  the *systemd service* for ComfyUI must set
+  `Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0` (or whatever matches
+  the GPU). The adapter itself does not need ROCm env vars.
+- **Timeout for long generations** — set the adapter's HTTP poll loop
+  to at least 1800 s (30 min) for high-resolution workloads.
+  Edit `for i in range(720)` (30 min at 5 s intervals) and set
+  `httpx.AsyncClient(timeout=2400)` on the initial `/prompt` POST.
+- **Synchronous (blocking) by design** — the adapter does NOT implement
+  async jobs. The HTTP response blocks until ComfyUI finishes. For
+  multi-hour jobs this means the client must keep the connection open.
+  If a true async job system is needed, build `POST → job_id` +
+  `GET /v1/jobs/{id}` separately.
+
+## Running it
+
+### Option A: Python venv (quick test)
+
+```bash
+python3 -m venv /opt/ideogram4-api/venv
+source /opt/ideogram4-api/venv/bin/activate
+pip install fastapi uvicorn httpx pydantic
+export COMFYUI_HOST=localhost
+export COMFYUI_PORT=8188
+python3 comfyui-openai-adapter.py
+```
+
+### Option B: Docker on dedicated host (persistent)
+
+```dockerfile
+FROM python:3.12-slim
+WORKDIR /app
+RUN pip install --no-cache-dir fastapi uvicorn httpx pydantic
+COPY comfyui-openai-adapter.py ./main.py
+EXPOSE 8000
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+```
+
+> The adapter and ComfyUI should be on the same LAN or host. Latency matters because the adapter polls ComfyUI every 5 s for up to 60 min.
+
+### Option C: systemd service (LXC/VM — recommended for Proxmox)
+
+**Prerequisite:** Kill any previously manual/nohup-started ComfyUI or adapter processes *before* enabling systemd units. Old processes hold ports 8188 and 8000 and cause `address already in use`.
+
+```bash
+# Inside LXC container (e.g. CT 204)
+ss -tlnp | grep -E '8188|8000'
+fuser -k 8188/tcp 2>/dev/null
+fuser -k 8000/tcp 2>/dev/null
+```
+
+**`comfyui.service`** (note the ROCm env var for gfx1150):
+
+```ini
+[Unit]
+Description=ComfyUI
+After=network.target
+
+[Service]
+Type=simple
+User=root
+WorkingDirectory=/opt/ComfyUI
+ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
+Restart=always
+RestartSec=10
+Environment=HIP_VISIBLE_DEVICES=0
+Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
+Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
+Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib
+
+[Install]
+WantedBy=multi-user.target
+```
+
+**`ideogram4-api.service`** (rename to `comfyui-api.service` if generic):
+
+```ini
+[Unit]
+Description=ComfyUI OpenAI API Wrapper
+After=network.target comfyui.service
+Requires=comfyui.service
+
+[Service]
+Type=simple
+User=root
+WorkingDirectory=/opt/ideogram4-api
+ExecStart=/opt/ideogram4-api/venv/bin/python /opt/ideogram4-api/main.py
+Restart=always
+RestartSec=5
+Environment=PATH=/opt/ideogram4-api/venv/bin:/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
+Environment=COMFYUI_HOST=localhost
+Environment=COMFYUI_PORT=8188
+
+[Install]
+WantedBy=multi-user.target
+```
+
+> **Filename note:** The adapter source may be saved as `main.py`, `app.py`, or `adapter.py`. Check `ls /opt/ideogram4-api/` and match `ExecStart=` exactly.
+
+Enable & start:
+```bash
+systemctl daemon-reload
+systemctl enable --now comfyui.service
+sleep 15
+systemctl enable --now ideogram4-api.service
+```
+
+## Extended OpenAI-compatible parameters
+
+Beyond the standard `model`, `prompt`, `n`, `size`, `response_format`,
+the adapter accepts these ComfyUI extensions:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `seed` | int | random | Fixed seed. Random if omitted. |
+| `steps` | int | from `quality` | Denoising steps (1–100). Overrides `quality` if set. |
+| `cfg` | float | model-specific | Guidance scale. Ideogram4=1.0, FLUX=3.5, others=7.5 |
+| `quality` | string | `medium` | Maps to steps: `low`(8), `medium`(20), `high`(30), `ultra`(50). |
+| `negative_prompt` | string | `""` | Negative prompt text. |
+| `style` | string | `""` | Style modifier string. |
+| `control_strength` | float | — | Reserved for future ControlNet integration. |
+
+Example request body:
+```json
+{
+  "model": "flux2",
+  "prompt": "a cyberpunk alley at night, neon signs",
+  "n": 1,
+  "size": "1024x1024",
+  "quality": "high",
+  "seed": 42,
+  "cfg": 3.5,
+  "negative_prompt": "blurry, low quality"
+}
+```
+
+## Frontend configuration (OpenWebUI example)
+
+1. Admin → **Settings** → **Images**
+2. **Image Generation Engine:** `Open AI`
+3. **OpenAI API Base URL:** `http://<adapter-host>:8000/v1`
+4. **OpenAI API Key:** any dummy string (adapter does not validate)
+5. **Model:** `ideogram4` or `flux2` or `flux1-dev`
+
+## Pitfalls
+
+| Issue | Cause | Fix |
+|---|---|---|
+| `TimeoutError` after 300 s | High resolution or GPU under load | Clamp `size`, increase timeout, reduce `steps` |
+| `500` from ComfyUI + `value_not_in_list` | `size` not in model's allowed resolutions | Check allowed sizes; use defaults |
+| `[Errno 98] address already in use` | Old manual `nohup` process still running | `fuser -k 8000/tcp` / `fuser -k 8188/tcp`, then restart systemd |
+| `can't open file '/opt/ideogram4-api/app.py'` | `ExecStart` mismatches actual filename | Check `ls /opt/ideogram4-api/` and align `ExecStart` |
+| `UNETLoader` node error for `.gguf` model | Using `UNETLoader` for a GGUF file | Switch to `UnetLoaderGGUF` |
+| `RuntimeError` during text-encode (ROCm) | Missing `HSA_OVERRIDE_GFX_VERSION` | Add to `comfyui.service` `Environment=` line |
+| Ideogram4 black/corrupted output | Corrupt `.safetensors` or wrong VAE | Verify file integrity; use `flux2-vae.safetensors` |
+| FLUX output dimensions wrong | Used `EmptyLatentImage` instead of `EmptyFlux2LatentImage` | Ideogram4 uses `EmptyFlux2LatentImage`; FLUX.1 uses `EmptyLatentImage` |
+| `DualModelGuider.execute() got unexpected keyword argument 'model_1'` | Wrong workflow key name | Use `model_negative`, NOT `model_1` |
+| systemd `comfyui.service` keeps failing with `exit-code` | Port 8188 held by old manual ComfyUI | `pkill -f "main.py.*8188"`, then `systemctl restart` |
+
+## Model-specific node quick-reference
+
+### Ideogram 4 local (with asymmetric CFG — correct wiring)
+
+```json
+{
+  "1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
+  "2": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"},
+        "class_type": "UNETLoader"},
+  "3": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4"},
+        "class_type": "CLIPLoader"},
+  "4": {"inputs": {"text": "...", "clip": ["3", 0]}, "class_type": "CLIPTextEncode"},
+  "5": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
+  "6": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
+  "7": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
+  "8": {"inputs": {"steps": 20, "width": 1024, "height": 1024, "mu": 0.5, "std": 1.75},
+        "class_type": "Ideogram4Scheduler"},
+  "15": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"},
+         "class_type": "UNETLoader"},
+  "16": {"inputs": {"conditioning": ["4", 0]}, "class_type": "ConditioningZeroOut"},
+  "17": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["2", 0]},
+         "class_type": "CFGOverride"},
+  "9": {"inputs": {"cfg": 7.0, "model": ["17", 0], "positive": ["4", 0],
+         "model_negative": ["15", 0], "negative": ["16", 0]},
+        "class_type": "DualModelGuider"},
+  "10": {"inputs": {"noise": ["6", 0], "guider": ["9", 0], "sampler": ["7", 0],
+          "sigmas": ["8", 0], "latent_image": ["5", 0]}, "class_type": "SamplerCustomAdvanced"},
+  "11": {"inputs": {"samples": ["10", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
+  "12": {"inputs": {"filename_prefix": "ideogram4_api", "images": ["11", 0]}, "class_type": "SaveImage"}
+}
+```
+Allowed sizes: `1024x1024`, `1024x1536`, `1536x1024`, `832x1216`, `1216x832`, `896x1152`, `1152x896`.
+
+### FLUX.2 / FLUX.1
+```json
+{
+  "1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
+  "2": {"inputs": {"unet_name": "flux1-dev-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
+  "3": {"inputs": {"model": ["2", 0], "max_shift": 1.15, "base_shift": 0.5,
+          "width": 1024, "height": 1024}, "class_type": "ModelSamplingFlux"},
+  "4": {"inputs": {"clip_name1": "clip_l.safetensors", "clip_name2": "t5xxl_fp8_e4m3fn.safetensors",
+          "type": "flux", "device": "default"}, "class_type": "DualCLIPLoader"},
+  "5": {"inputs": {"text": "...", "clip": ["4", 0]}, "class_type": "CLIPTextEncode"},
+  "6": {"inputs": {"conditioning": ["5", 0], "guidance": 3.5}, "class_type": "FluxGuidance"},
+  "7": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyLatentImage"},
+  "8": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
+  "9": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
+  "10": {"inputs": {"model": ["3", 0], "steps": 20, "denoise": 1.0, "scheduler": "simple",
+           "sampler": ["9", 0]}, "class_type": "BasicScheduler"},
+  "11": {"inputs": {"model": ["3", 0], "conditioning": ["6", 0]}, "class_type": "BasicGuider"},
+  "12": {"inputs": {"noise": ["8", 0], "guider": ["11", 0], "sampler": ["9", 0],
+           "sigmas": ["10", 0], "latent_image": ["7", 0]}, "class_type": "SamplerCustomAdvanced"},
+  "13": {"inputs": {"samples": ["12", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
+  "14": {"inputs": {"filename_prefix": "flux2_api", "images": ["13", 0]}, "class_type": "SaveImage"}
+}
+```
+Allowed sizes: all common resolutions up to 1536×1536. FLUX schnell defaults to 4–8 steps; dev to 20–50.