dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

39 KiB

Raw Permalink Blame History

name, description, version, author, license, platforms, compatibility, prerequisites, setup, metadata

name

description

version

author

license

platforms

compatibility

prerequisites

setup

metadata

comfyui

Generate images, video, and audio with ComfyUI — install, launch, manage nodes/models, run workflows with parameter injection. Uses the official comfy-cli for lifecycle and direct REST/WebSocket API for execution.

5.0.0

kshitijk4poor

alt-glitch

MIT

macos

linux

windows

Requires ComfyUI (local, Comfy Desktop, or Comfy Cloud) and comfy-cli (auto-installed via pipx/uvx by the setup script).

commands

python3

help
Run scripts/hardware_check.py FIRST to decide local vs Comfy Cloud; then scripts/comfyui_setup.sh auto-installs locally (or use Cloud API key for platform.comfy.org).

hermes

ComfyUI

Generate images, video, audio, and 3D content through ComfyUI using the official comfy-cli for setup/lifecycle and direct REST/WebSocket API for workflow execution.

What's in this skill

Reference docs (references/):

official-cli.md — every comfy ... command, with flags
rest-api.md — REST + WebSocket endpoints (local + cloud), payload schemas
workflow-format.md — API-format JSON, common node types, param mapping
flux2-setup.md — FLUX.2 local setup: Qwen3VL, EmptyFlux2LatentImage, ROCm gfx1150
ideogram4-local-setup.md — Ideogram 4 local vs. cloud: model downloads, verification recipe, Ideogram4Scheduler workflow
openai-api-wrapper.md — multi-model OpenAI-compatible FastAPI adapter (Ideogram4 + FLUX family). Extended parameters (seed, steps, cfg, quality, negative_prompt, style). systemd service setup with ROCm env vars. Timeout tuning for 30 min generations. Ideogram4 safety/threshold notes (min steps, gray-block detection). See also templates/comfyui-openai-adapter.py for the full adapter source.
openwebui-integration.md — OpenWebUI ↔ ComfyUI adapter for FLUX.2 (OpenAI-compatible /v1/images/generations bridge)
ideogram4.md — Ideogram 4: cloud API (IdeogramV4) vs local (Ideogram4Scheduler) paths, model checklist, corruption fix, full workflow blueprint
ideogram4.md — Ideogram 4: cloud API (IdeogramV4) vs local (Ideogram4Scheduler) paths, model checklist, corruption fix

Scripts (scripts/):

Script	Purpose
`_common.py`	Shared HTTP, cloud routing, node catalogs (don't run directly)
`adapter_flux2.py`	FastAPI OpenAI adapter for FLUX.2: `/v1/images/generations` → ComfyUI `/api/prompt`
`hardware_check.py`	Probe GPU/VRAM/disk → recommend local vs Comfy Cloud
`comfyui_setup.sh`	Hardware check + comfy-cli + ComfyUI install + launch + verify
`extract_schema.py`	Read a workflow → list controllable params + model deps
`check_deps.py`	Check workflow against running server → list missing nodes/models
`auto_fix_deps.py`	Run check_deps then `comfy node install` / `comfy model download`
`run_workflow.py`	Inject params, submit, monitor, download outputs (HTTP or WS)
`run_batch.py`	Submit a workflow N times with sweeps, parallel up to your tier
`comfy_e2e_test.py`	Systematic end-to-end test: iterate model × quality × style, generate images, write JSON+Markdown report. Configurable via CLI args.
`ws_monitor.py`	Real-time WebSocket viewer for executing jobs (live progress)
`health_check.py`	Verification checklist runner — comfy-cli + server + models + smoke test
`fetch_logs.py`	Pull traceback / status messages for a given prompt_id

Example workflows (workflows/): SD 1.5, SDXL, Flux Dev, SDXL img2img, SDXL inpaint, ESRGAN upscale, AnimateDiff video, Wan T2V. See workflows/README.md.

When to Use

User asks to generate images with Stable Diffusion, SDXL, Flux, SD3, etc.
User wants to run a specific ComfyUI workflow file
User wants to chain generative steps (txt2img → upscale → face restore)
User needs ControlNet, inpainting, img2img, or other advanced pipelines
User asks to manage ComfyUI queue, check models, or install custom nodes
User wants video/audio/3D generation via AnimateDiff, Hunyuan, Wan, AudioCraft, etc.

Architecture: Two Layers

┌─────────────────────────────────────────────────────┐
│ Layer 1: comfy-cli (official lifecycle tool)        │
│   Setup, server lifecycle, custom nodes, models     │
│   → comfy install / launch / stop / node / model    │
└─────────────────────────┬───────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────┐
│ Layer 2: REST/WebSocket API + skill scripts         │
│   Workflow execution, param injection, monitoring   │
│   POST /api/prompt, GET /api/view, WS /ws           │
│   → run_workflow.py, run_batch.py, ws_monitor.py    │
└─────────────────────────────────────────────────────┘

Why two layers? The official CLI is excellent for installation and server management but has minimal workflow execution support. The REST/WS API fills that gap — the scripts handle param injection, execution monitoring, and output download that the CLI doesn't do.

Quick Start

Detect environment

# What's available?
command -v comfy >/dev/null 2>&1 && echo "comfy-cli: installed"
curl -s http://127.0.0.1:8188/system_stats 2>/dev/null && echo "server: running"

# Can this machine run ComfyUI locally? (GPU/VRAM/disk check)
python3 scripts/hardware_check.py

If nothing is installed, see Setup & Onboarding below — but always run the hardware check first.

One-line health check

python3 scripts/health_check.py
# → JSON: comfy_cli on PATH? server reachable? at least one checkpoint? smoke-test passes?

Core Workflow

Step 1: Get a workflow JSON in API format

Workflows must be in API format (each node has class_type). They come from:

ComfyUI web UI → Workflow → Export (API) (newer UI) or the legacy "Save (API Format)" button (older UI)
This skill's workflows/ directory (ready-to-run examples)
Community downloads (civitai, Reddit, Discord) — usually editor format, must be loaded into ComfyUI then re-exported

Editor format (top-level nodes and links arrays) is **not directly The skill scripts detect this and tell you to re-export.

Integration with Chat UIs (OpenWebUI, etc.)

ComfyUI does not expose an OpenAI-compatible /v1/images/generations endpoint. Chat UIs that expect image generation (OpenWebUI, LibreChat, etc.) need an adapter when using ComfyUI as backend. Key considerations:

Native ComfyUI connectors in chat UIs assume the classic SD/SDXL/FLUX.1 pipeline (DualCLIPLoaderGGUF, EmptyLatentImage, standard VAE). These will NOT work with FLUX.2-klein-9B, which needs Qwen3-VL, EmptyFlux2LatentImage, and a FLUX.2-compatible VAE (taef2). See references/flux2-setup.md for the full node mapping.
Recommended approach: build a thin OpenAI-compatible FastAPI proxy that translates POST /v1/images/generations into a ComfyUI POST /api/prompt call with your specific workflow. No public Docker image covers the full FLUX.2 node stack; a custom service (~80 LoC Python/FastAPI) is needed.
Alternative: use OpenWebUI Functions to call ComfyUI directly, but this bypasses the polished image-generation UI overlay.

For a concrete walkthrough including verified network topology, see references/openwebui-integration.md.

Step 2: See what's controllable

python3 scripts/extract_schema.py workflow_api.json --summary-only
# → {"parameter_count": 12, "has_negative_prompt": true, "has_seed": true, ...}

python3 scripts/extract_schema.py workflow_api.json
# → full schema with parameters, model deps, embedding refs

Step 3: Run with parameters

# Local (defaults to http://127.0.0.1:8188)
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "a beautiful sunset over mountains", "seed": -1, "steps": 30}' \
  --output-dir ./outputs

# Cloud (export API key once; uses correct /api routing automatically)
export COMFY_CLOUD_API_KEY="comfyui-..."
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "..."}' \
  --host https://cloud.comfy.org \
  --output-dir ./outputs

# Real-time progress via WebSocket (requires `pip install websocket-client`)
python3 scripts/run_workflow.py \
  --workflow flux_dev.json \
  --args '{"prompt": "..."}' \
  --ws

# img2img / inpaint: pass --input-image to upload + reference automatically
python3 scripts/run_workflow.py \
  --workflow sdxl_img2img.json \
  --input-image image=./photo.png \
  --args '{"prompt": "make it watercolor", "denoise": 0.6}'

# Batch / sweep: 8 random seeds, parallel up to cloud tier limit
python3 scripts/run_batch.py \
  --workflow sdxl.json \
  --args '{"prompt": "abstract"}' \
  --count 8 --randomize-seed --parallel 3 \
  --output-dir ./outputs/batch

-1 for seed (or omitting it with --randomize-seed) generates a fresh random seed per run.

Step 4: Present results

The scripts emit JSON to stdout describing every output file:

{
  "status": "success",
  "prompt_id": "abc-123",
  "outputs": [
    {"file": "./outputs/sdxl_00001_.png", "node_id": "9",
     "type": "image", "filename": "sdxl_00001_.png"}
  ]
}

Decision Tree

User says	Tool	Command
Lifecycle (use comfy-cli)
"install ComfyUI"	comfy-cli	`bash scripts/comfyui_setup.sh`
"start ComfyUI"	comfy-cli	`comfy launch --background`
"stop ComfyUI"	comfy-cli	`comfy stop`
"install X node"	comfy-cli	`comfy node install <name>`
"download X model"	comfy-cli	`comfy model download --url <url> --relative-path models/checkpoints`
"list installed models"	comfy-cli	`comfy model list`
"list installed nodes"	comfy-cli	`comfy node show installed`
Execution (use scripts)
"is everything ready?"	script	`health_check.py` (optionally with `--workflow X --smoke-test`)
"what can I change in this workflow?"	script	`extract_schema.py W.json`
"check if W's deps are met"	script	`check_deps.py W.json`
"fix missing deps"	script	`auto_fix_deps.py W.json`
"generate an image"	script	`run_workflow.py --workflow W --args '{...}'`
"use this image" (img2img)	script	`run_workflow.py --input-image image=./x.png ...`
"8 variations with random seeds"	script	`run_batch.py --count 8 --randomize-seed ...`
"show me live progress"	script	`ws_monitor.py --prompt-id <id>`
"fetch the error from job X"	script	`fetch_logs.py <prompt_id>`
Direct REST
"what's in the queue?"	REST	`curl http://HOST:8188/queue` (local) or `--host https://cloud.comfy.org`
"cancel that"	REST	`curl -X POST http://HOST:8188/interrupt`
"free GPU memory"	REST	`curl -X POST http://HOST:8188/free`

Setup & Onboarding

When a user asks to set up ComfyUI, the FIRST thing to do is ask whether they want Comfy Cloud (hosted, zero install, API key) or Local (install ComfyUI on their machine). Don't start running install commands or hardware checks until they've answered.

Official docs: https://docs.comfy.org/installation CLI docs: https://docs.comfy.org/comfy-cli/getting-started Cloud docs: https://docs.comfy.org/get_started/cloud Cloud API: https://docs.comfy.org/development/cloud/overview

Step 0: Ask Local vs Cloud (ALWAYS FIRST)

Suggested script:

"Do you want to run ComfyUI locally on your machine, or use Comfy Cloud?

Comfy Cloud — hosted on RTX 6000 Pro GPUs, all common models pre-installed, zero setup. Requires an API key (paid subscription required to actually run workflows; free tier is read-only). Best if you don't have a capable GPU.

Local — free, but your machine MUST meet the hardware requirements:

NVIDIA GPU with ≥6 GB VRAM (≥8 GB for SDXL, ≥12 GB for Flux/video), OR

AMD GPU with ROCm support (Linux), OR

Apple Silicon Mac (M1+) with ≥16 GB unified memory (≥32 GB recommended).

Intel Macs and machines with no GPU will NOT work — use Cloud instead.

Which would you like?"

Routing:

Cloud → skip to Path A.
Local → run hardware check first, then pick a path from Paths B–E based on the verdict.
Unsure → run the hardware check and let the verdict decide.

Step 1: Verify Hardware (ONLY if user chose local)

python3 scripts/hardware_check.py --json
# Optional: also probe `torch` for actual CUDA/MPS:
python3 scripts/hardware_check.py --json --check-pytorch

Verdict	Meaning	Action
`ok`	≥8 GB VRAM (discrete) OR ≥32 GB unified (Apple Silicon)	Local install — use `comfy_cli_flag` from report
`marginal`	SD1.5 works; SDXL tight; Flux/video unlikely	Local OK for light workflows, else Path A (Cloud)
`cloud`	No usable GPU, <6 GB VRAM, <16 GB Apple unified, Intel Mac, Rosetta Python	Switch to Cloud unless user explicitly forces local

The script also surfaces wsl: true (WSL2 with NVIDIA passthrough) and rosetta: true (x86_64 Python on Apple Silicon — must reinstall as ARM64).

If verdict is cloud but the user wants local, do not proceed silently. Show the notes array verbatim and ask whether they want to (a) switch to Cloud or (b) force a local install (will OOM or be unusably slow on modern models).

Choosing an Installation Path

Use the hardware check first. The table below is the fallback for when the user has already told you their hardware:

Situation	Recommended Path
`verdict: cloud` from hardware check	Path A: Comfy Cloud
No GPU / want to try without commitment	Path A: Comfy Cloud
Windows + NVIDIA + non-technical	Path B: ComfyUI Desktop
Windows + NVIDIA + technical	Path C: Portable or Path D: comfy-cli
Linux + any GPU	Path D: comfy-cli (easiest)
macOS + Apple Silicon	Path B: Desktop or Path D: comfy-cli
Headless / server / CI / agents	Path D: comfy-cli

For the fully automated path (hardware check → install → launch → verify):

bash scripts/comfyui_setup.sh
# Or with overrides:
bash scripts/comfyui_setup.sh --m-series --port=8190 --workspace=/data/comfy

It runs hardware_check.py internally, refuses to install locally when the verdict is cloud (unless --force-cloud-override), picks the right comfy-cli flag, and prefers pipx/uvx over global pip to avoid polluting system Python.

Path A: Comfy Cloud (No Local Install)

For users without a capable GPU or who want zero setup. Hosted on RTX 6000 Pro.

Docs: https://docs.comfy.org/get_started/cloud

Sign up at https://comfy.org/cloud
Generate an API key at https://platform.comfy.org/login

Set the key:

export COMFY_CLOUD_API_KEY="comfyui-xxxxxxxxxxxx"

Run workflows:

python3 scripts/run_workflow.py \
  --workflow workflows/flux_dev_txt2img.json \
  --args '{"prompt": "..."}' \
  --host https://cloud.comfy.org \
  --output-dir ./outputs

Pricing: https://www.comfy.org/cloud/pricing Concurrent jobs: Free/Standard 1, Creator 3, Pro 5. Free tier cannot run workflows via API — only browse models. Paid subscription required for /api/prompt, /api/upload/*, /api/view, etc.

Path B: ComfyUI Desktop (Windows / macOS)

One-click installer for non-technical users. Currently Beta.

Docs: https://docs.comfy.org/installation/desktop

Windows (NVIDIA): https://download.comfy.org/windows/nsis/x64
macOS (Apple Silicon): https://comfy.org

Linux is not supported for Desktop — use Path D.

Path C: ComfyUI Portable (Windows Only)

Docs: https://docs.comfy.org/installation/comfyui_portable_windows

Download from https://github.com/comfyanonymous/ComfyUI/releases, extract, run run_nvidia_gpu.bat. Update via update/update_comfyui_stable.bat.

Path D: comfy-cli (All Platforms — Recommended for Agents)

The official CLI is the best path for headless/automated setups.

Docs: https://docs.comfy.org/comfy-cli/getting-started

Install comfy-cli

# Recommended:
pipx install comfy-cli
# Or use uvx without installing:
uvx --from comfy-cli comfy --help
# Or (if pipx/uvx unavailable):
pip install --user comfy-cli

Disable analytics non-interactively:

comfy --skip-prompt tracking disable

Install ComfyUI

comfy --skip-prompt install --nvidia              # NVIDIA (CUDA)
comfy --skip-prompt install --amd                 # AMD (ROCm, Linux)
comfy --skip-prompt install --m-series            # Apple Silicon (MPS)
comfy --skip-prompt install --cpu                 # CPU only (slow)
comfy --skip-prompt install --nvidia --fast-deps  # uv-based dep resolution

Default location: ~/comfy/ComfyUI (Linux), ~/Documents/comfy/ComfyUI (macOS/Win). Override with comfy --workspace /custom/path install.

Launch / verify

comfy launch --background                       # background daemon on :8188
comfy launch -- --listen 0.0.0.0 --port 8190    # LAN-accessible custom port
curl -s http://127.0.0.1:8188/system_stats      # health check

Path E: Manual Install (Advanced / Unsupported Hardware)

For Ascend NPU, Cambricon MLU, Intel Arc, or other unsupported hardware.

Docs: https://docs.comfy.org/installation/manual_install

git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt
python main.py

Post-Install: Download Models

Classic Checkpoints (safetensors)

# SDXL (general purpose, ~6.5 GB)
comfy model download \
  --url "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors" \
  --relative-path models/checkpoints

# SD 1.5 (lighter, ~4 GB, good for 6 GB cards)
comfy model download \
  --url "https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors" \
  --relative-path models/checkpoints

# Flux Dev fp8 (smaller variant, ~12 GB)
comfy model download \
  --url "https://huggingface.co/Comfy-Org/flux1-dev/resolve/main/flux1-dev-fp8.safetensors" \
  --relative-path models/checkpoints

# CivitAI (set token first):
comfy model download \
  --url "https://civitai.com/api/download/models/128713" \
  --relative-path models/checkpoints \
  --set-civitai-api-token "YOUR_TOKEN"

FLUX Family — GGUF quantized (VRAM-friendly)

Black Forest Labs models are gated on HuggingFace (login + license acceptance required). Use community GGUF quantizations for direct download.

Model	Q-Level	Size	Source (no-auth)
FLUX.1-dev	Q4_K_S	~6.4 GB	`city96/FLUX.1-dev-gguf`
FLUX.2-klein-9B	Q4_K_S	~5.4 GB	`unsloth/FLUX.2-klein-9B-GGUF`
FLUX.2-klein-4B	Q4_K_S	~? GB	`unsloth/FLUX.2-klein-4B-GGUF`

Example:

curl -L -o models/unet/flux-2-klein-9b-Q4_K_S.gguf \
  "https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF/resolve/main/flux-2-klein-9b-Q4_K_S.gguf"

Required text encoders for FLUX

FLUX.1 (dev / schnell): shares CLIP-L + T5XXL:

# CLIP-L
curl -L -o models/clip/clip_l.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors"

# T5-XXL fp8 (Q4 quant, ~4.9 GB)
curl -L -o models/clip/t5xxl_fp8_e4m3fn.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp8_e4m3fn.safetensors"

FLUX.2-klein-9B: uses an 8B Qwen3-VL text embedder (NOT CLIP-L + T5XXL). These models are incompatible with ComfyUI's DualCLIPLoaderGGUF and its FLUX.1 nodes. Attempting to run FLUX.2 with the FLUX.1 CLIP/VAE pipeline produces:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x4096 and 12288x4096)

FLUX.2 VAE is NOT shared with FLUX.1. FLUX.2 emits 128-channel latents (latent_channels = 128), while the FLUX.1 VAE (ae.safetensors) expects 16 channels. Using the wrong VAE produces:

RuntimeError: Given groups=1, weight of size [512, 16, 3, 3], expected input[1, 128, 32, 32] to have 16 channels, but got 128 channels instead

Required FLUX.2 components:

Component	File	Size	Source
UNet (GGUF)	`flux-2-klein-9b-Q4_K_S.gguf`	~5.4 GB	`unsloth/FLUX.2-klein-9B-GGUF`
Text Encoder (GGUF)	`Qwen3VL-8B-Instruct-Q4_K_M.gguf`	~4.7 GB	`Qwen/Qwen3-VL-8B-Instruct-GGUF`
VAE option A	`flux2_vae.safetensors`	~160 MB	`black-forest-labs/FLUX.2-klein-9B` (gated)
VAE option B	`taef2` (TAE decoder)	~2.6 MB	`madebyollin/taesd` (public, lower quality)

Node workflow for FLUX.2 (ComfyUI API format):

{
  "1": {"inputs": {"unet_name": "flux-2-klein-9b-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
  "2": {"inputs": {"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf", "type": "flux2"}, "class_type": "CLIPLoaderGGUF"},
  "3": {"inputs": {"vae_name": "taef2"}, "class_type": "VAELoader"},
  "4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
  "5": {"inputs": {"text": "prompt here", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
  "6": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
  "7": {"inputs": {"seed": 42, "steps": 4, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": ["1", 0], "positive": ["5", 0], "negative": ["6", 0], "latent_image": ["4", 0]}, "class_type": "KSampler"},
  "8": {"inputs": {"samples": ["7", 0], "vae": ["3", 0]}, "class_type": "VAEDecode"},
  "9": {"inputs": {"filename_prefix": "flux2_out", "images": ["8", 0]}, "class_type": "SaveImage"}
}

Key differences from FLUX.1:

Use CLIPLoaderGGUF (single) with type: "flux2", NOT DualCLIPLoaderGGUF
Use EmptyFlux2LatentImage (128 channels), NOT EmptyLatentImage
Use taef2 or dedicated FLUX.2 VAE, NOT ae.safetensors
Default: 4 steps, cfg=1.0, euler/simple scheduler

See references/flux2-setup.md for full reproduction recipe, ROCm gfx1150 workaround, and performance notes.

VAE (shared across FLUX.1 and FLUX.2):

# Primary source (may require auth for BFL repos)
curl -L -o models/vae/ae.safetensors \
  "https://huggingface.co/black-forest-labs/FLUX.1-schnell/resolve/main/ae.safetensors"

# Mirror without auth (ModelScope)
curl -L -o models/vae/ae.safetensors \
  "https://www.modelscope.cn/models/AI-ModelScope/FLUX.1-dev/resolve/master/ae.safetensors"

List installed: comfy model list.

Post-Install: Install Custom Nodes

comfy node install comfyui-impact-pack             # popular utility pack
comfy node install comfyui-animatediff-evolved     # video generation
comfy node install comfyui-controlnet-aux          # ControlNet preprocessors
comfy node install comfyui-essentials              # common helpers
comfy node update all
comfy node install-deps --workflow=workflow.json   # install everything a workflow needs

Post-Install: Verify

python3 scripts/health_check.py
# → comfy_cli on PATH? server reachable? checkpoints? smoke test?

python3 scripts/check_deps.py my_workflow.json
# → are this workflow's nodes/models/embeddings installed?

python3 scripts/run_workflow.py \
  --workflow workflows/sd15_txt2img.json \
  --args '{"prompt": "test", "steps": 4}' \
  --output-dir ./test-outputs

Image Upload (img2img / Inpainting)

The simplest way is to use --input-image with run_workflow.py:

python3 scripts/run_workflow.py \
  --workflow workflows/sdxl_img2img.json \
  --input-image image=./photo.png \
  --args '{"prompt": "make it cyberpunk", "denoise": 0.6}'

The flag uploads photo.png, then injects its server-side filename into whatever schema parameter is named image. For inpainting, pass both:

python3 scripts/run_workflow.py \
  --workflow workflows/sdxl_inpaint.json \
  --input-image image=./photo.png \
  --input-image mask_image=./mask.png \
  --args '{"prompt": "fill with flowers"}'

Manual upload via REST:

curl -X POST "http://127.0.0.1:8188/upload/image" \
  -F "image=@photo.png" -F "type=input" -F "overwrite=true"
# Returns: {"name": "photo.png", "subfolder": "", "type": "input"}

# Cloud equivalent:
curl -X POST "https://cloud.comfy.org/api/upload/image" \
  -H "X-API-Key: $COMFY_CLOUD_API_KEY" \
  -F "image=@photo.png" -F "type=input" -F "overwrite=true"

Cloud Specifics

Base URL: https://cloud.comfy.org
Auth: X-API-Key header (or ?token=KEY for WebSocket)
API key: set $COMFY_CLOUD_API_KEY once and the scripts pick it up automatically
Output download: /api/view returns a 302 to a signed URL; the scripts follow it and strip X-API-Key before fetching from the storage backend (don't leak the API key to S3/CloudFront).
Endpoint differences from local ComfyUI:
- /api/object_info, /api/queue, /api/userdata — 403 on free tier; paid only.
- /history is renamed to /history_v2 on cloud (the scripts route automatically).
- /models/<folder> is renamed to /experiment/models/<folder> on cloud (the scripts route automatically).
- clientId in WebSocket is currently ignored — all connections for a user receive the same broadcast. Filter by prompt_id client-side.
- subfolder is accepted on uploads but ignored — cloud has a flat namespace.
Concurrent jobs: Free/Standard: 1, Creator: 3, Pro: 5. Extras queue automatically. Use run_batch.py --parallel N to saturate your tier.

Queue & System Management

# Local
curl -s http://127.0.0.1:8188/queue | python3 -m json.tool
curl -X POST http://127.0.0.1:8188/queue -d '{"clear": true}'    # cancel pending
curl -X POST http://127.0.0.1:8188/interrupt                      # cancel running
curl -X POST http://127.0.0.1:8188/free \
  -H "Content-Type: application/json" \
  -d '{"unload_models": true, "free_memory": true}'

# Cloud — same paths under /api/, plus:
python3 scripts/fetch_logs.py --tail-queue --host https://cloud.comfy.org

Pitfalls

API format required — every script and the /api/prompt endpoint expect API-format workflow JSON. The scripts detect editor format (top-level nodes and links arrays) and tell you to re-export via "Workflow → Export (API)" (newer UI) or "Save (API Format)" (older UI).
Server must be running — all execution requires a live server. comfy launch --background starts one. Verify with curl http://127.0.0.1:8188/system_stats.
Model names are exact — case-sensitive, includes file extension. check_deps.py does fuzzy matching (with/without extension and folder prefix), but the workflow itself must use the canonical name. Use comfy model list to discover what's installed.
Missing custom nodes — "class_type not found" means a required node isn't installed. check_deps.py reports which package to install; auto_fix_deps.py runs the install for you.
Working directory — comfy-cli auto-detects the ComfyUI workspace. If commands fail with "no workspace found", use comfy --workspace /path/to/ComfyUI <command> or comfy set-default /path/to/ComfyUI.
Cloud free-tier API limits — /api/prompt, /api/view, /api/upload/*, /api/object_info all return 403 on free accounts. health_check.py and check_deps.py handle this gracefully and surface a clear message.
Timeout for video/audio workflows — auto-detected when an output node is VHS_VideoCombine, SaveVideo, etc.; the default jumps from 300 s to 900 s. Override explicitly with --timeout 1800.
Path traversal in output filenames — server-supplied filenames are passed through safe_path_join to refuse anything escaping --output-dir. Keep this protection on — workflows with custom save nodes can produce arbitrary paths.
Workflow JSON is arbitrary code — custom nodes run Python, so submitting an unknown workflow has the same trust profile as eval. Inspect workflows from untrusted sources before running.
comfyui.service unit already exists — Before creating a new systemd service for ComfyUI, check /etc/systemd/system/comfyui.service; it may already be installed but disabled or stopped. Inspect it first with cat /etc/systemd/system/comfyui.service rather than overwriting blindly.
Auto-randomized seed — pass seed: -1 in --args (or use --randomize-seed and omit the seed) to get a fresh seed per run. The actual seed is logged to stderr.
Ideogram 4 has TWO incompatible runtime paths — do not confuse them:
- IdeogramV1/V2/V3/V4 — comfy_api_nodes cloud API nodes. Require a comfy.org login / API key. These call the Ideogram SaaS API and do NOT use local models. Error: Unauthorized: Please login first to use this node.
- Ideogram4Scheduler + local UNet/CLIP/VAE — purely local pipeline. Uses CLIPLoader with type: "ideogram4", EmptyFlux2LatentImage, Ideogram4Scheduler feeding SamplerCustomAdvanced, plus CFGOverride + DualModelGuider for asymmetric classifier-free guidance. Do NOT use plain BasicGuider or standard KSampler — they lack the dual-model wiring. See references/ideogram4.md for node requirements.
DualModelGuider uses model_negative, NOT model_1 — in the API-format workflow, the DualModelGuider node's unconditional model input is named model_negative (not model_1). Using the wrong key name produces: TypeError: DualModelGuider.execute() got an unexpected keyword argument 'model_1'.
LLM gateways ≠ image backends — tools like Bifrost route /v1/chat/completions, not /v1/images/generations. They have no ComfyUI provider and cannot translate OpenAI image requests to ComfyUI workflow JSON. Connecting ComfyUI to chat frontends (OpenWebUI, etc.) requires either the frontends' native ComfyUI connector (limited to SD/SDXL, breaks on FLUX.2) or a custom OpenAI-compatible adapter that submits workflow JSON to POST /api/prompt. See references/openwebui-integration.md.
Shell-escaping hell when driving LXC over SSH — If your ComfyUI lives inside a Proxmox LXC (e.g. CT 204 on 10.0.30.97), avoid multi-layer nested quotes (ssh → pct exec → python3 -c). They break on !, ( and ". Instead write a script file on the PVE host, pct push it into the container, then pct exec it. See references/proxmox-lxc.md.
Ideogram 4: API vs. Local — IdeogramV1-V4 nodes require a comfy.org API key (Unauthorized without auth). For local execution, use Ideogram4Scheduler + download ~28 GB of models (ideogram4_fp8_scaled, ideogram4_unconditional_fp8_scaled, flux2-vae, qwen3vl_8b_fp8_scaled). See references/ideogram4-local-setup.md.
Safetensors "incomplete metadata" even with correct file size — a .safetensors file can report the expected byte count in ls -lh but still be corrupt if the header's data_offsets don't match the actual payload boundaries. Always verify with a header-content-size check (see references/ideogram4-local-setup.md for the verification recipe).
Proxmox pct exec nested quoting hell — when pushing scripts into an LXC container via pct exec or sshpass, avoid inline Python with single quotes. Instead write a script to a temp file via pct push, then execute it:
```
pct push CTID /local/script.py /tmp/script.py
pct exec CTID -- python3 /tmp/script.py
```
This avoids the "\''\"$ escape maze that breaks multiline Python strings.
Editor-format blueprints are NOT API format — ComfyUI's blueprints/*.json files (e.g. Text to Image (Ideogram v4).json) use the editor format ("nodes" / "links" arrays). They must be loaded into ComfyUI and re-exported as API format before execution via POST /api/prompt. The REST endpoint rejects editor format silently or with cryptic errors.
Switching from manual/nohup to systemd blocks ports — If ComfyUI or the API adapter was previously started via nohup python3 main.py & or python3 adapter.py &, the processes survive backgrounding and continue holding ports 8188 / 8000. A subsequent systemctl start will fail with address already in use. Always terminate the manual processes first (kill -9 <pid> or fuser -k <port>/tcp) before enabling systemd units.
systemd ExecStart filename must match actual file on disk — When templating a systemd service for the API adapter, the source code may be saved as adapter.py, main.py, or app.py. Check ls /opt/ideogram4-api/ before writing ExecStart=/opt/ideogram4-api/venv/bin/python /opt/ideogram4-api/xxx.py; a mismatch produces can't open file.
Model families need DIFFERENT loader nodes — Ideogram4 uses UNETLoader for .safetensors. FLUX GGUF models (*.gguf) require UnetLoaderGGUF from the ComfyUI-GGUF custom node. Using UNETLoader for a GGUF file produces value_not_in_list because the safetensors loader doesn't know .gguf. Using UnetLoaderGGUF for a safetensors file also fails. Never globally replace one loader name for another — patch per-workflow instead.
ComfyUI on AMD ROCm needs HSA_OVERRIDE_GFX_VERSION — When running under systemd (not an interactive shell), HSA_OVERRIDE_GFX_VERSION is NOT inherited from any bashrc or manual export. ComfyUI will crash during text-encode with RuntimeError (hipBLASLt architecture mismatch). The fix is adding Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 (or your GPU's gfx string) to the [Service] section of comfyui.service.
LLM gateways ≠ image backends — tools like Bifrost route /v1/chat/completions, not /v1/images/generations. They have no ComfyUI provider and cannot translate OpenAI image requests to ComfyUI workflow JSON. Connecting ComfyUI to chat frontends (OpenWebUI, etc.) requires either the frontends' native ComfyUI connector (limited to SD/SDXL, breaks on FLUX.2) or a custom OpenAI-compatible adapter that submits workflow JSON to POST /api/prompt. See references/openwebui-integration.md and references/openai-api-wrapper.md.
API timeout for long generations — set the adapter's HTTP poll loop and the FastAPI timeout to at least 1800 s (30 min) for high-resolution or multi-model workloads. FLUX dev at 1024×1024 with 50 steps can take 10–15 min on ROCm; larger resolutions or multiple n images may approach 30 min. The adapter polls ComfyUI every 5 s; the total loop iterations must cover the worst case. In the Python source, increase for i in range(360) → for i in range(720) (30 min at 5 s intervals) and set httpx timeout > 2400 s on the initial /prompt POST.
Model loader type must match file format — never do a global sed replacing UNETLoader with UnetLoaderGGUF (or vice versa). Ideogram4 uses .safetensors loaded by UNETLoader; FLUX GGUF models use UnetLoaderGGUF. A global replacement breaks the other pipeline. Patch per-workflow, not per-file.
Ideogram4 ships WITHOUT an active safety classifier — unlike commercial APIs (Midjourney, DALL-E, Ideogram Cloud) that block prompts with an explicit NSFW score, the local Ideogram4Scheduler pipeline has no classifier. Generations are rejected silently or produce blank/gray output only because the model's internal attention collapses at very low step counts (≤8). This looks like censorship but is actually numerical underflow. Fix: enforce a minimum of ~12 steps for Ideogram4. There is no safety log to grep for; safety is absent, not logged.
Ideogram4 BasicGuider in OpenAI adapter produces inferior quality — The template adapter at templates/comfyui-openai-adapter.py originally used BasicGuider for Ideogram4 (node "9" in _build_ideogram4_workflow). This is wrong: Ideogram4 requires asymmetric CFG wiring via CFGOverride + DualModelGuider + a separate unconditional UNet (ideogram4_unconditional_fp8_scaled) + ConditioningZeroOut. Using BasicGuider silently degrades quality but does NOT throw an error. The correct wiring is: UNETLoader(ideogram4) → CFGOverride → DualModelGuider(model=cfg_overridden, model_negative=unconditional_unet, positive=prompt_cond, negative=zeroed_cond) → SamplerCustomAdvanced. See references/ideogram4.md for the full API-format blueprint.
Gray-block detection via image variance — When Ideogram4 produces blank or uniform gray output (especially at quality=low with style=photorealistic), the PNG file is still valid but the image has near-zero content variance. A quick automated check using PIL: ImageStat.Stat(img).var — if all channels have variance < 15, the generation failed. Healthy images show variance > 40 (typically 50–150 for detailed scenes). This is a reliable proxy for "did the model actually render content" and can gate retries or alerts in automated pipelines. See scripts/comfy_e2e_test.py for implementation.
Systematic E2E testing: constant prompt across all parameter combinations — When validating a new adapter, model, or parameter mapping, use a single fixed prompt (e.g. "a majestic mountain landscape at sunset") across all combinations of model × quality × style. This isolates parameter effects from prompt variance. Do NOT vary the prompt per model — it confounds the results. Store outputs in a timestamped directory, compute variance per image, and flag outliers (< 15 variance) as failures. The watcher script scripts/comfy_e2e_watcher.sh reports completion status and reproduces the prompt for traceability.
OpenAI model list must stay consistent with ALLOWED_MODELS — If a validation set (ALLOWED_MODELS = {"ideogram4", "flux2"}) rejects flux1-dev and flux1-schnell at generation time, the GET /v1/models endpoint must reflect the same restriction. A hardcoded list_models() returning all four models causes frontend confusion. Either make it dynamic (for m in sorted(ALLOWED_MODELS)) or keep it manually in sync.

Verification Checklist

Use python3 scripts/health_check.py to run the whole list at once. Manual:

hardware_check.py verdict is ok OR the user explicitly chose Comfy Cloud
comfy --version works (or uvx --from comfy-cli comfy --help)
curl http://HOST:PORT/system_stats returns JSON
comfy model list shows at least one checkpoint (local) OR /api/experiment/models/checkpoints returns models (cloud)
Workflow JSON is in API format
check_deps.py reports is_ready: true (or only node_check_skipped on cloud free tier)
Test run with a small workflow completes; outputs land in --output-dir

39 KiB Raw Permalink Blame History Unescape Escape