Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,131 @@
|
||||
# FLUX Model Download Reference for ComfyUI on ROCm
|
||||
|
||||
Quick reference for downloading FLUX models for local ComfyUI inference on AMD ROCm (no HuggingFace login).
|
||||
|
||||
## Official FLUX Models from Black Forest Labs
|
||||
|
||||
| Model | Params | Open? | Size (FP16) |
|
||||
|-------|--------|-------|-------------|
|
||||
| FLUX.1 [dev] | 12B | ✅ | ~23 GB |
|
||||
| FLUX.1 [schnell] | 12B | ✅ | ~23 GB |
|
||||
| FLUX.1 [pro] | ? | ❌ (API only) | — |
|
||||
| **FLUX.2 [dev]** | ? | ✅ | 16.9 GB |
|
||||
| **FLUX.2 [klein-9B]** | 9B | ✅ | 16.9 GB |
|
||||
| **FLUX.2 [klein-4B]** | 4B | ✅ | ~3.7 GB (Q4) |
|
||||
|
||||
The `klein` variants are faster, smaller variants of FLUX.2.
|
||||
|
||||
## GGUF Quantizations (No Auth Required)
|
||||
|
||||
When HuggingFace requires login (gated models), use community GGUF repos:
|
||||
|
||||
**Unsloth GGUF repo (unsloth/FLUX.2-klein-9B-GGUF):**
|
||||
- `flux-2-klein-9b-Q4_K_S.gguf` — **5.43 GB** ← best balance for most GPUs
|
||||
- `flux-2-klein-9b-Q4_K_M.gguf` — 5.50 GB
|
||||
- `flux-2-klein-9b-Q4_0.gguf` — 5.23 GB
|
||||
- `flux-2-klein-9b-Q5_K_S.gguf` — 6.46 GB
|
||||
- `flux-2-klein-9b-Q8_0.gguf` — 9.29 GB
|
||||
|
||||
**Leejet GGUF repo (leejet/FLUX.2-klein-9B-GGUF):**
|
||||
- `flux-2-klein-9b-Q4_0.gguf` — 5.23 GB
|
||||
- `flux-2-klein-9b-Q8_0.gguf` — 9.29 GB (fewer options than unsloth)
|
||||
|
||||
**FLUX.1-dev GGUF repo (city96/FLUX.1-dev-gguf or similar):**
|
||||
- `flux1-dev-Q4_K_S.gguf` — **6.4 GB**
|
||||
|
||||
## Authentication Barriers
|
||||
|
||||
| Source | Requires Auth? | Notes |
|
||||
|--------|----------------|-------|
|
||||
| `black-forest-labs/*` (HuggingFace) | ✅ Yes (gated) | Must login + accept license |
|
||||
| `unsloth/*-GGUF` | ❌ No | Community quant, direct download |
|
||||
| `leejet/*-GGUF` | ❌ No | Community quant, direct download |
|
||||
| `civitai.com` | ❌ No (most) | Use `api/download/models/<id>?type=Model&format=SafeTensor` |
|
||||
| `modelscope.cn` | ❌ No | China mirror, works for FLUX files |
|
||||
|
||||
**If a download returns a tiny (~140 byte) file:** The request was redirected to a login page or error page. The HTML response is ~140 bytes. Always verify the downloaded file size matches expectations.
|
||||
|
||||
## Required Auxiliary Models
|
||||
|
||||
**Shared VAE (all FLUX models):**
|
||||
| File | Size | Purpose | Where to get (no auth) |
|
||||
|------|------|---------|------------------------|
|
||||
| `ae.safetensors` (VAE) | 319 MB | Image decoder | **modelscope.cn** (HuggingFace gated) |
|
||||
|
||||
**Text encoders are NOT shared:**
|
||||
- **FLUX.1 (dev / schnell):** CLIP-L + T5XXL → `clip_l.safetensors` + `t5xxl_fp8_e4m3fn.safetensors`
|
||||
- **FLUX.2-klein-9B:** Uses **8B Qwen3 text embedder** (NOT CLIP-L / T5XXL). Incompatible with ComfyUI's `DualCLIPLoaderGGUF`. Attempting to run FLUX.2 with FLUX.1 encoders produces `RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x4096 and 12288x4096)`. Use diffusers `Flux2KleinPipeline` or find ComfyUI Qwen3 nodes.
|
||||
|
||||
| File | Size | Purpose | Where to get | Required for |
|
||||
|------|------|---------|-------------|-------------|
|
||||
| `clip_l.safetensors` | 235 MB | CLIP text encoder | civitai or modelscope | FLUX.1 only |
|
||||
| `t5xxl_fp8_e4m3fn.safetensors` | 4.9 GB | T5 XXL text encoder | civitai or modelscope | FLUX.1 only |
|
||||
### Working VAE download URLs (no-auth)
|
||||
|
||||
```bash
|
||||
# Modelscope (verified working, no auth)
|
||||
curl -L -o ae.safetensors \
|
||||
"https://www.modelscope.cn/models/AI-ModelScope/FLUX.1-dev/resolve/master/ae.safetensors"
|
||||
|
||||
# Direct HuggingFace (REQUIRES login for BFL repos)
|
||||
# curl -L -o ae.safetensors \
|
||||
# "https://huggingface.co/black-forest-labs/FLUX.1-schnell/resolve/main/ae.safetensors"
|
||||
# → Will fail with 401, producing a ~140 byte HTML error page
|
||||
```
|
||||
|
||||
## Background Download Pattern
|
||||
|
||||
For multi-GB model downloads from inside a Proxmox LXC, use `nohup` to survive SSH/session timeouts:
|
||||
|
||||
```bash
|
||||
pct exec 204 -- bash -c '
|
||||
cd /opt/ComfyUI/models/unet
|
||||
nohup bash -c "curl -L -o flux-2-klein-9b-Q4_K_S.gguf \\"
|
||||
https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF/resolve/main/flux-2-klein-9b-Q4_K_S.gguf \\"
|
||||
> /tmp/flux2_download.log 2>&1" &
|
||||
'
|
||||
```
|
||||
|
||||
Verify ongoing download:
|
||||
```bash
|
||||
pct exec 204 -- bash -c 'ls -lh /opt/ComfyUI/models/unet/flux-2-klein-9b-Q4_K_S.gguf'
|
||||
pct exec 204 -- bash -c 'tail -3 /tmp/flux2_download.log'
|
||||
```
|
||||
|
||||
## ComfyUI Model Paths
|
||||
|
||||
```
|
||||
/opt/ComfyUI/models/
|
||||
├── unet/ # GGUF diffusion models (FLUX, SDXL, etc.)
|
||||
│ └── flux-2-klein-9b-Q4_K_S.gguf
|
||||
├── clip/ # Text encoders
|
||||
│ ├── clip_l.safetensors
|
||||
│ └── t5xxl_fp8_e4m3fn.safetensors
|
||||
├── vae/ # Image decoders
|
||||
│ └── ae.safetensors
|
||||
└── checkpoints/ # Full safetensors models
|
||||
```
|
||||
|
||||
## ComfyUI + ROCm + GGUF Verification
|
||||
|
||||
After downloading, verify ComfyUI can load the model:
|
||||
|
||||
```bash
|
||||
# Start ComfyUI
|
||||
pct exec 204 -- bash -c 'cd /opt/ComfyUI && python3 main.py --listen 0.0.0.0 --port 8188'
|
||||
|
||||
# Check API responds
|
||||
curl http://10.0.20.91:8188/system_stats
|
||||
|
||||
# The first run will compile shaders; monitor /tmp/comfyui.log
|
||||
pct exec 204 -- bash -c 'tail -20 /tmp/comfyui.log'
|
||||
```
|
||||
|
||||
## Errors and Fixes
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `hipErrorNoBinaryForGpu` | PyTorch ROCm wheel mismatch with host ROCm | Downgrade/upgrade PyTorch wheel: `pip install torch --index-url https://download.pytorch.org/whl/rocm6.2` |
|
||||
| Downloaded file is ~140 bytes | Gated model, no auth token | Switch to unsloth/leejet GGUF repos or modelscope |
|
||||
| Downloaded file is 6+ GB for VAE | Wrong CivitAI URL (model page, not direct download) | Use correct `api/download/models/<id>?type=Model&format=SafeTensor` URL |
|
||||
| `curl` times out during download | Paramiko SSH timeout | Use `nohup` + background; poll file size separately |
|
||||
@@ -0,0 +1,69 @@
|
||||
# FLUX.2 in ComfyUI — Compatibility Guide
|
||||
|
||||
> **Status: Native ComfyUI support confirmed as of June 2026.**
|
||||
> See `references/flux2-setup.md` for the complete reproduction recipe, model downloads, and workflow JSON.
|
||||
|
||||
## The Problem (Historical)
|
||||
|
||||
FLUX.2-klein (9B) uses an **8B Qwen3-VL text embedder**, replacing the CLIP-L + T5XXL combo used by FLUX.1. Early ComfyUI versions (pre-mid 2026) lacked native nodes for FLUX.2, forcing users to:
|
||||
- Use the diffusers `Flux2KleinPipeline` instead, or
|
||||
- Wait for community custom nodes
|
||||
|
||||
This is no longer necessary. ComfyUI now includes:
|
||||
- `CLIPType.FLUX2` enum (line 1299 in `comfy/sd.py`)
|
||||
- `Flux2` model class (line 1057 in `comfy/model_base.py`)
|
||||
- `EmptyFlux2LatentImage` node (128-channel latents)
|
||||
- `Flux2Scheduler` node (mu-snr shifted timesteps)
|
||||
- `Flux2TEModel` text encoder wrapper with Qwen3VL support
|
||||
- `KleinTokenizer` / `KleinTokenizer8B` tokenizers
|
||||
|
||||
## Error Signature → Fix Mapping
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `mat1 and mat2 shapes cannot be multiplied (512x4096 and 12288x4096)` | Using CLIP-L+T5XXL with FLUX.2 UNet | Load `Qwen3VL-8B-Instruct-Q4_K_M.gguf` via `CLIPLoaderGGUF` with `"type": "flux2"` |
|
||||
| `expected input[1, 128, 32, 32] to have 16 channels, but got 128 channels instead` | Using FLUX.1 VAE (`ae.safetensors`) | Use `taef2` or `flux2_vae.safetensors` |
|
||||
| `HIP error: invalid device function` | ROCm gfx1150 without override | `export HSA_OVERRIDE_GFX_VERSION=11.0.0` |
|
||||
|
||||
## Required Models
|
||||
|
||||
| Component | File | ~Size | Source |
|
||||
|-----------|------|-------|--------|
|
||||
| UNet (GGUF) | `flux-2-klein-9b-Q4_K_S.gguf` | 5.4 GB | `unsloth/FLUX.2-klein-9B-GGUF` |
|
||||
| Text Encoder | `Qwen3VL-8B-Instruct-Q4_K_M.gguf` | 4.7 GB | `Qwen/Qwen3-VL-8B-Instruct-GGUF` |
|
||||
| VAE (fast) | `taef2` encoder+decoder `.pth` | 5 MB | `madebyollin/taesd` (GitHub) |
|
||||
| VAE (full) | `flux2_vae.safetensors` | 160 MB | `black-forest-labs/FLUX.2-klein-9B` (gated) |
|
||||
|
||||
**FLUX.1 and FLUX.2 are NOT component-compatible.** Do not mix:
|
||||
- ❌ FLUX.2 UNet + FLUX.1 CLIP/T5
|
||||
- ❌ FLUX.2 latents + FLUX.1 VAE (`ae.safetensors`)
|
||||
- ❌ `EmptyLatentImage` with FLUX.2 (needs `EmptyFlux2LatentImage`)
|
||||
|
||||
## Workflow Differences (FLUX.1 vs FLUX.2)
|
||||
|
||||
```json
|
||||
// FLUX.1 — two CLIP loaders + shared VAE
|
||||
{"2": {"inputs": {"clip_name1": "t5xxl_fp8_e4m3fn.safetensors", "clip_name2": "clip_l.safetensors", "type": "flux"}, "class_type": "DualCLIPLoaderGGUF"}}
|
||||
{"4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyLatentImage"}}
|
||||
{"3": {"inputs": {"vae_name": "ae.safetensors"}, "class_type": "VAELoader"}}
|
||||
|
||||
// FLUX.2 — single Qwen3 CLIP + TAEF2 VAE + 128-channel latent
|
||||
{"2": {"inputs": {"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf", "type": "flux2"}, "class_type": "CLIPLoaderGGUF"}}
|
||||
{"4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"}}
|
||||
{"3": {"inputs": {"vae_name": "taef2"}, "class_type": "VAELoader"}}
|
||||
```
|
||||
|
||||
## Verified E2E Results (gfx1150, ROCm 6.2)
|
||||
|
||||
| Pipeline | Res | Steps | Time | Status |
|
||||
|----------|-----|-------|------|--------|
|
||||
| FLUX.1-dev Q4_K_S + CLIP/T5 + ae.safetensors | 512×512 | 4 | ~46s | ✅ |
|
||||
| FLUX.2-klein Q4_K_S + Qwen3VL Q4_K_M + taef2 | 1024×1024 | 4 | ~110s | ✅ |
|
||||
|
||||
## References
|
||||
|
||||
- Full setup recipe: `references/flux2-setup.md`
|
||||
- Unsloth GGUF repo: https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF
|
||||
- Qwen3VL GGUF repo: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF
|
||||
- BFL blog: https://bfl.ai/blog/flux2-klein-towards-interactive-visual-intelligence
|
||||
- TAEF2 source: https://github.com/madebyollin/taesd
|
||||
@@ -0,0 +1,196 @@
|
||||
# FLUX.2 Setup in ComfyUI — Full Reproduction Recipe
|
||||
|
||||
## Hardware Context (This Session)
|
||||
|
||||
- Proxmox CT 204, ROCm 6.2, PyTorch 2.6.0 (ROCm build)
|
||||
- AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M (gfx1150)
|
||||
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` required for gfx1150
|
||||
- 18GB "VRAM" reported; ~8.5 GB disk free after downloads
|
||||
|
||||
## Timeline
|
||||
|
||||
| Step | Result |
|
||||
|------|--------|
|
||||
| FLUX.1-dev E2E | ✅ Works in 46s (512×512, 4 steps) |
|
||||
| FLUX.2 with FLUX.1 CLIP/T5 | ❌ `mat1 and mat2 shapes cannot be multiplied (512x4096 and 12288x4096)` |
|
||||
| FLUX.2 with Qwen3VL + ae.safetensors | ❌ `expected input[1, 128, 32, 32] to have 16 channels, but got 128 channels instead` |
|
||||
| FLUX.2 with Qwen3VL + taef2 | ✅ Success ~110s (1024×1024, 4 steps) |
|
||||
|
||||
## Required Models
|
||||
|
||||
### 1. UNet (GGUF)
|
||||
|
||||
```bash
|
||||
# ~5.4 GB
|
||||
wget https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF/resolve/main/flux-2-klein-9b-Q4_K_S.gguf
|
||||
# → models/unet/flux-2-klein-9b-Q4_K_S.gguf
|
||||
```
|
||||
|
||||
### 2. Text Encoder (Qwen3-VL-8B, GGUF)
|
||||
|
||||
**CRITICAL:** FLUX.2 uses Qwen3-VL-8B, NOT CLIP-L + T5XXL.
|
||||
|
||||
```bash
|
||||
# ~4.7 GB — Q4_K_M quantization
|
||||
wget https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF/resolve/main/Qwen3VL-8B-Instruct-Q4_K_M.gguf
|
||||
# → models/clip/Qwen3VL-8B-Instruct-Q4_K_M.gguf
|
||||
```
|
||||
|
||||
Alternative: `Qwen3-VL-8B-Instruct-Q8_0.gguf` (~8.3 GB) for higher quality.
|
||||
|
||||
The GGUF detection logic in ComfyUI (`comfy/sd.py:detect_te_model`) identifies this model by:
|
||||
- `model.layers.0.self_attn.q_proj.weight` shape = `[4096, 4096]`
|
||||
- `num_hidden_layers = 36`
|
||||
→ maps to `TEModel.QWEN3VL_8B`
|
||||
|
||||
### 3. VAE — TWO OPTIONS
|
||||
|
||||
**Option A: TAEF2 (Tiny AutoEncoder, fast, lower quality)**
|
||||
|
||||
```bash
|
||||
# ~5 MB total, public, no auth needed
|
||||
cd models/vae_approx
|
||||
curl -LO https://github.com/madebyollin/taesd/raw/main/taef2_encoder.pth
|
||||
curl -LO https://github.com/madebyollin/taesd/raw/main/taef2_decoder.pth
|
||||
# Reference as vae_name: "taef2" in VAELoader node
|
||||
```
|
||||
|
||||
**Option B: Full FLUX.2 VAE (`flux2_vae.safetensors`)**
|
||||
|
||||
```bash
|
||||
# ~160 MB, gated repo (BFL license acceptance required)
|
||||
# From: black-forest-labs/FLUX.2-klein-9B vae/diffusion_pytorch_model.safetensors
|
||||
# Save as: models/vae/flux2_vae.safetensors
|
||||
```
|
||||
|
||||
The FLUX.1 VAE (`ae.safetensors`, 16 channels) is **incompatible** with FLUX.2 (128 channels).
|
||||
|
||||
## ComfyUI Workflow (API JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {
|
||||
"inputs": {"unet_name": "flux-2-klein-9b-Q4_K_S.gguf"},
|
||||
"class_type": "UnetLoaderGGUF"
|
||||
},
|
||||
"2": {
|
||||
"inputs": {
|
||||
"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf",
|
||||
"type": "flux2"
|
||||
},
|
||||
"class_type": "CLIPLoaderGGUF"
|
||||
},
|
||||
"3": {
|
||||
"inputs": {"vae_name": "taef2"},
|
||||
"class_type": "VAELoader"
|
||||
},
|
||||
"4": {
|
||||
"inputs": {"width": 1024, "height": 1024, "batch_size": 1},
|
||||
"class_type": "EmptyFlux2LatentImage"
|
||||
},
|
||||
"5": {
|
||||
"inputs": {
|
||||
"text": "A beautiful mountain lake at sunset, photorealistic, 4k",
|
||||
"clip": ["2", 0]
|
||||
},
|
||||
"class_type": "CLIPTextEncode"
|
||||
},
|
||||
"6": {
|
||||
"inputs": {"text": "", "clip": ["2", 0]},
|
||||
"class_type": "CLIPTextEncode"
|
||||
},
|
||||
"7": {
|
||||
"inputs": {
|
||||
"seed": 42,
|
||||
"steps": 4,
|
||||
"cfg": 1.0,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "simple",
|
||||
"denoise": 1.0,
|
||||
"model": ["1", 0],
|
||||
"positive": ["5", 0],
|
||||
"negative": ["6", 0],
|
||||
"latent_image": ["4", 0]
|
||||
},
|
||||
"class_type": "KSampler"
|
||||
},
|
||||
"8": {
|
||||
"inputs": {"samples": ["7", 0], "vae": ["3", 0]},
|
||||
"class_type": "VAEDecode"
|
||||
},
|
||||
"9": {
|
||||
"inputs": {"filename_prefix": "flux2_out", "images": ["8", 0]},
|
||||
"class_type": "SaveImage"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key node differences from FLUX.1
|
||||
|
||||
| Node | FLUX.1 | FLUX.2 |
|
||||
|------|--------|--------|
|
||||
| CLIP loader | `DualCLIPLoaderGGUF` (clip_l + t5xxl) | `CLIPLoaderGGUF` (single Qwen3VL) |
|
||||
| CLIP type | `"flux"` | `"flux2"` |
|
||||
| Empty latent | `EmptyLatentImage` (16 channels) | `EmptyFlux2LatentImage` (128 channels) |
|
||||
| VAE | `ae.safetensors` | `taef2` or `flux2_vae.safetensors` |
|
||||
|
||||
## ROCm gfx1150 Workaround
|
||||
|
||||
gfx1150 (RDNA3.5, Radeon 890M) is not fully supported by ROCm 6.2. Set override before launch:
|
||||
|
||||
```bash
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.0.0
|
||||
python main.py --listen 0.0.0.0 --port 8188
|
||||
```
|
||||
|
||||
Without this, CLIP encoding fails with:
|
||||
```
|
||||
RuntimeError: HIP error: invalid device function
|
||||
```
|
||||
|
||||
## Performance (gfx1150, ROCm 6.2)
|
||||
|
||||
| Pipeline | Resolution | Steps | Time |
|
||||
|----------|-----------|-------|------|
|
||||
| FLUX.1-dev Q4_K_S + CLIP/T5 + ae.safetensors | 512×512 | 4 | ~46s |
|
||||
| FLUX.2-klein Q4_K_S + Qwen3VL Q4_K_M + taef2 | 1024×1024 | 4 | ~110s |
|
||||
|
||||
FLUX.2 is slower despite fewer parameters because Qwen3VL-8B text encoding is heavier than CLIP+T5, and the TAEF2 decode takes extra time.
|
||||
|
||||
## Disk Space Budget
|
||||
|
||||
| Component | Size |
|
||||
|-----------|------|
|
||||
| FLUX.2 UNet Q4_K_S | 5.4 GB |
|
||||
| Qwen3VL-8B Q4_K_M | 4.7 GB |
|
||||
| taef2 encoder+decoder | 5 MB |
|
||||
| **Total** | **~10.1 GB** |
|
||||
|
||||
Full FLUX.2 VAE adds ~160 MB. Q8_0 text encoder adds ~3.6 GB more.
|
||||
|
||||
## Common Errors
|
||||
|
||||
### `mat1 and mat2 shapes cannot be multiplied (512x4096 and 12288x4096)`
|
||||
**Cause:** Using CLIP-L + T5XXL (FLUX.1 encoders) with FLUX.2 UNet.
|
||||
**Fix:** Switch to `Qwen3VL-8B-Instruct-Q4_K_M.gguf` + `CLIPLoaderGGUF` with `"type": "flux2"`.
|
||||
|
||||
### `expected input[1, 128, 32, 32] to have 16 channels, but got 128 channels instead`
|
||||
**Cause:** Using FLUX.1 VAE (`ae.safetensors`) with FLUX.2 latents.
|
||||
**Fix:** Use `taef2` or download dedicated `flux2_vae.safetensors`.
|
||||
|
||||
### `HIP error: invalid device function`
|
||||
**Cause:** ROCm doesn't natively support gfx1150.
|
||||
**Fix:** `export HSA_OVERRIDE_GFX_VERSION=11.0.0` before starting ComfyUI.
|
||||
|
||||
### `RuntimeError: The expanded size of the tensor (512) must match ...`
|
||||
**Cause:** Using `EmptyLatentImage` (16-channel) instead of `EmptyFlux2LatentImage` (128-channel).
|
||||
**Fix:** Change node class to `EmptyFlux2LatentImage`.
|
||||
|
||||
## References
|
||||
|
||||
- ComfyUI `comfy/model_base.py` lines 1057-1066: `class Flux2(Flux)`
|
||||
- ComfyUI `comfy/latent_formats.py` line 192: `class Flux2(LatentFormat)` with `latent_channels = 128`
|
||||
- ComfyUI `comfy/sd.py` lines 1631-1634: FLUX2 CLIPType routes to `klein_te` with Qwen3VL
|
||||
- TAEF2 source: https://github.com/madebyollin/taesd
|
||||
- Unsloth GGUF: https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF
|
||||
- Qwen GGUF: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF
|
||||
@@ -0,0 +1,124 @@
|
||||
# Ideogram 4 Local Execution in ComfyUI
|
||||
|
||||
Multiple paths exist to generate with Ideogram 4. The simplest is the cloud API node (`IdeogramV4`), but it requires a `comfy.org` API key. For local execution without cloud dependency, use the native `Ideogram4Scheduler` + model files.
|
||||
|
||||
## Path A: Cloud API Node (Simplest, but requires key)
|
||||
|
||||
**Node:** `IdeogramV4`
|
||||
**Location:** `comfy_api_nodes/nodes_ideogram.py`
|
||||
**Requires:** `COMFY_CLOUD_API_KEY` env var or login
|
||||
|
||||
**Error without auth:**
|
||||
```
|
||||
Unauthorized: Please login first to use this node.
|
||||
```
|
||||
|
||||
Works with structured JSON prompt (scene + style + background blocks). The easiest if you have the key.
|
||||
|
||||
---
|
||||
|
||||
## Path B: Local Native Pipeline (No API key, but 28 GB disk needed)
|
||||
|
||||
**Recommended when:** No comfy.org auth available or prefer full local execution.
|
||||
|
||||
### Required Model Files
|
||||
|
||||
| File | Size | Folder | Source |
|
||||
|---|---|---|---|
|
||||
| `ideogram4_fp8_scaled.safetensors` | ~8.6 GB | `models/diffusion_models/` | HuggingFace: `Comfy-Org/Ideogram-4` |
|
||||
| `ideogram4_unconditional_fp8_scaled.safetensors` | ~8.6 GB | `models/diffusion_models/` | HuggingFace: `Comfy-Org/Ideogram-4` |
|
||||
| `flux2-vae.safetensors` | ~320 MB | `models/vae/` | HuggingFace: `Comfy-Org/flux2-dev` |
|
||||
| `qwen3vl_8b_fp8_scaled.safetensors` | ~9.9 GB | `models/text_encoders/` | HuggingFace: `Comfy-Org/Qwen3-VL` |
|
||||
| `gemma4_e4b_it_fp8_scaled.safetensors` | ~5.4 GB (optional alt) | `models/text_encoders/` | HuggingFace: `Comfy-Org/gemma-4` |
|
||||
|
||||
**Total: ~28 GB**
|
||||
|
||||
### Model Verification Recipe
|
||||
|
||||
A file can have the correct size on disk but still fail to load with `SafetensorError: incomplete metadata`. This happens when a download is interrupted and resumed incorrectly, or when the file was copied incompletely.
|
||||
|
||||
Use this Python script to verify any `.safetensors` file before trusting it:
|
||||
|
||||
```python
|
||||
import struct, json, os
|
||||
|
||||
def verify_safetensors(path):
|
||||
"""Check header metadata against actual file size."""
|
||||
actual = os.path.getsize(path)
|
||||
f = open(path, 'rb')
|
||||
header_len = struct.unpack("<Q", f.read(8))[0]
|
||||
meta = json.loads(f.read(header_len))
|
||||
total = 8 + header_len
|
||||
for k, v in meta.items():
|
||||
if isinstance(v, dict) and "data_offsets" in v:
|
||||
total += v["data_offsets"][1] - v["data_offsets"][0]
|
||||
return {
|
||||
"actual_bytes": actual,
|
||||
"expected_bytes": total,
|
||||
"ok": total == actual,
|
||||
"header_keys": len(meta),
|
||||
"missing": total - actual if total > actual else 0,
|
||||
}
|
||||
|
||||
# Example
|
||||
result = verify_safetensors("models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors")
|
||||
print(result)
|
||||
# If ok=False, the file is corrupt and MUST be re-downloaded.
|
||||
```
|
||||
|
||||
**Command line shortcut (in Container):**
|
||||
```bash
|
||||
python3 -c "
|
||||
import struct, json, os; p='/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors'
|
||||
f=open(p,'rb'); h=struct.unpack('<Q',f.read(8))[0]; m=json.loads(f.read(h))
|
||||
t=8+h; [t:=t+v['data_offsets'][1]-v['data_offsets'][0] for k,v in m.items()
|
||||
if isinstance(v,dict) and 'data_offsets' in v]
|
||||
print('ok=',t==os.path.getsize(p))
|
||||
"
|
||||
```
|
||||
|
||||
### Text Encoder Compatibility Note
|
||||
|
||||
Ideogram 4 supports **two** text encoders:
|
||||
- `qwen3vl_8b_fp8_scaled.safetensors` (Qwen3-VL, ~9.9 GB) — primary, higher quality
|
||||
- `gemma4_e4b_it_fp8_scaled.safetensors` (Gemma 4, ~5.4 GB) — lighter alternative
|
||||
|
||||
If one fails (e.g., corrupt download), try the other.
|
||||
|
||||
### ComfyUI BluePrint Workflow (API Format)
|
||||
|
||||
The canonical workflow is shipped as a BluePrint JSON in ComfyUI:
|
||||
```
|
||||
blueprints/Text to Image (Ideogram v4).json
|
||||
```
|
||||
|
||||
This uses the `Ideogram4Scheduler` + `DualModelGuider` with the two UNet files (conditional + unconditional) for asymmetric classifier-free guidance.
|
||||
|
||||
**Key nodes in the BluePrint:**
|
||||
- `Ideogram4Scheduler` → outputs SIGMAS (replaces BasicScheduler)
|
||||
- `DualModelGuider` → combines positive + unconditional models
|
||||
- `CFGOverride` → cfg schedule override (start/end percent)
|
||||
- `EmptyFlux2LatentImage` → 128-channel latent
|
||||
- `SamplerCustomAdvanced` → with euler + Ideogram4Scheduler sigmas
|
||||
|
||||
### Common Errors — Local Path
|
||||
|
||||
**1. `SafetensorError: incomplete metadata, file not fully covered`**
|
||||
- **Cause:** File is structurally corrupt (wrong tensor offsets in header).
|
||||
- **Fix:** Run verification recipe above. If `ok=False`, delete and re-download from HuggingFace.
|
||||
- **Important:** `ls -lh` showing the right size is NOT sufficient. The header values must match the data offsets.
|
||||
|
||||
**2. `Unauthorized: Please login first to use this node`**
|
||||
- **Cause:** Using `IdeogramV1`/`V2`/`V3`/`V4` API node without comfy.org auth.
|
||||
- **Fix:** Switch to local pipeline (`Ideogram4Scheduler` + model files), or add `COMFY_CLOUD_API_KEY`.
|
||||
|
||||
**3. `expected input[1, 128, 32, 32] to have 16 channels`**
|
||||
- **Cause:** Using FLUX.1 VAE (`ae.safetensors`) instead of `flux2-vae.safetensors`.
|
||||
- **Fix:** VAE must be `flux2-vae.safetensors` (or `taef2`).
|
||||
|
||||
## References
|
||||
|
||||
- ComfyUI BluePrints: `blueprints/Text to Image (Ideogram v4).json`
|
||||
- Model source: https://huggingface.co/Comfy-Org/Ideogram-4
|
||||
- Text encoder: https://huggingface.co/Comfy-Org/Qwen3-VL
|
||||
- Alternative encoder: https://huggingface.co/Comfy-Org/gemma-4
|
||||
@@ -0,0 +1,180 @@
|
||||
# Ideogram 4 in ComfyUI
|
||||
|
||||
Two distinct execution paths exist. Pick the right one based on auth and hardware.
|
||||
|
||||
## Path A: Cloud API (`IdeogramV4` node)
|
||||
|
||||
- **Node:** `IdeogramV4` (built-in, from `comfy_api_nodes.nodes_ideogram`)
|
||||
- **Inputs:** `prompt` (STRING), `resolution` (COMBO), `rendering_speed` (COMBO), `seed` (INT)
|
||||
- **Hidden inputs:** `auth_token_comfy_org`, `api_key_comfy_org`
|
||||
- **Requires:** comfy.org account + API key (set via auth_token or api_key_comfy_org)
|
||||
|
||||
### Common error
|
||||
```
|
||||
Unauthorized: Please login first to use this node.
|
||||
```
|
||||
→ Set the `api_key_comfy_org` hidden input or login in the ComfyUI web UI.
|
||||
|
||||
## Path B: Local inference (`Ideogram4Scheduler` node)
|
||||
|
||||
- **Node:** `Ideogram4Scheduler` (built-in, `comfy_extras.nodes_ideogram4`)
|
||||
- **Outputs:** `SIGMAS` — feed into `SamplerCustomAdvanced`
|
||||
- **Workflow structure:** `Ideogram4Scheduler` → `SamplerCustomAdvanced` → `VAEDecode` → `SaveImage`
|
||||
|
||||
### Why you can't just "use the blueprint directly"
|
||||
|
||||
The official blueprint at `blueprints/Text to Image (Ideogram v4).json` is in **editor format** (`"nodes"` / `"links"` arrays with a top-level `definitions.subgraphs` array). **The ComfyUI REST API (`POST /api/prompt`) does NOT accept editor format.** Submitting it produces HTTP 400 or 500 with no useful message. You must either:
|
||||
|
||||
1. Load the blueprint in the ComfyUI web UI and re-export it as **API format** (Workflow → Export API), OR
|
||||
2. Manually construct an API-format workflow using the verified node wiring below.
|
||||
|
||||
### Critical: Ideogram 4 needs asymmetric CFG wiring
|
||||
|
||||
Unlike standard pipelines that use `BasicGuider`, Ideogram 4 requires `CFGOverride` + `DualModelGuider` for asymmetric classifier-free guidance:
|
||||
|
||||
- `CFGOverride` sits on the **main model** to override the CFG value in a specific timestep range
|
||||
- `DualModelGuider` takes the CFG-overridden main model, the positive conditioning, a **separate unconditional model** (via `model_negative`), and the zeroed-out conditioning (via `ConditioningZeroOut`)
|
||||
- The guider output (`GUIDER`) feeds into `SamplerCustomAdvanced`
|
||||
|
||||
**Do NOT use:** plain `BasicGuider` (only supports single model + pos/neg), standard `KSampler` (doesn't use `Ideogram4Scheduler` sigmas), or `model_1` as the DualModelGuider input name (it's `model_negative`).
|
||||
|
||||
### Node input names for API-format workflows (verified against `/object_info`)
|
||||
|
||||
| Node | Required inputs | Optional inputs | Notes |
|
||||
|------|-----------------|-----------------|-------|
|
||||
| `Ideogram4Scheduler` | `steps`, `width`, `height`, `mu`, `std` | — | Outputs `SIGMAS` for `SamplerCustomAdvanced` |
|
||||
| `CFGOverride` | `model`, `cfg`, `start_percent`, `end_percent` | — | Place on main UNet between `UNETLoader` and `DualModelGuider` |
|
||||
| `DualModelGuider` | `model`, `positive`, `cfg` | `model_negative` (MODEL), `negative` (CONDITIONING) | `model_negative` is the unconditional UNet (NOT `model_1`) |
|
||||
| `CLIPLoader` | `clip_name` | `type` (use `"ideogram4"`), `device` | Must use `type: "ideogram4"` for Qwen3VL text encoder |
|
||||
| `EmptyFlux2LatentImage` | `width`, `height`, `batch_size` | — | 128-channel latent (NOT standard `EmptyLatentImage`) |
|
||||
| `SamplerCustomAdvanced` | `noise`, `guider`, `sampler`, `sigmas`, `latent_image` | — | `guider` comes from `DualModelGuider`, `sigmas` from `Ideogram4Scheduler` |
|
||||
|
||||
### Required models
|
||||
|
||||
| Component | Filename | Path | Size | Source |
|
||||
|-----------|----------|------|------|--------|
|
||||
| UNet (main) | `ideogram4_fp8_scaled.safetensors` | `models/diffusion_models/` | ~8.6 GB | `Comfy-Org/Ideogram-4` |
|
||||
| UNet (unconditional) | `ideogram4_unconditional_fp8_scaled.safetensors` | `models/diffusion_models/` | ~8.6 GB | `Comfy-Org/Ideogram-4` |
|
||||
| VAE | `flux2-vae.safetensors` | `models/vae/` | ~320 MB | `Comfy-Org/flux2-dev` |
|
||||
| Text encoder (A) | `qwen3vl_8b_fp8_scaled.safetensors` | `models/text_encoders/` | ~9.8 GB | `Comfy-Org/Qwen3-VL` |
|
||||
| Text encoder (B) | `gemma4_e4b_it_fp8_scaled.safetensors` | `models/text_encoders/` | ~? GB | `Comfy-Org/gemma-4` |
|
||||
|
||||
### Known corruption: `qwen3vl_8b_fp8_scaled.safetensors`
|
||||
|
||||
A truncated download of this ~9.8 GB file produces:
|
||||
```
|
||||
Error while deserializing header: incomplete metadata, file not fully covered
|
||||
```
|
||||
at ComfyUI's `CLIPLoader` (node `load_clip` → `comfy.utils.load_torch_file`).
|
||||
|
||||
**Quick integrity check** (run inside the ComfyUI container or on the host):
|
||||
```bash
|
||||
python3 -c "
|
||||
import struct, json, os
|
||||
p = 'models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors'
|
||||
f = open(p, 'rb'); hl = struct.unpack('<Q', f.read(8))[0]
|
||||
m = json.loads(f.read(hl)); t = 8 + hl
|
||||
for k,v in m.items():
|
||||
if isinstance(v, dict) and 'data_offsets' in v:
|
||||
t += v['data_offsets'][1] - v['data_offsets'][0]
|
||||
a = os.path.getsize(p); print(f'expected={t} actual={a} ok={t==a}')
|
||||
"
|
||||
```
|
||||
If `ok=False`, the download is incomplete. Re-download via:
|
||||
```bash
|
||||
curl -L -o models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors \
|
||||
"https://huggingface.co/Comfy-Org/Qwen3-VL/resolve/main/text_encoders/qwen3vl_8b_fp8_scaled.safetensors"
|
||||
```
|
||||
|
||||
**Alternative:** `safetensors.safe_open` also catches corruption early:
|
||||
```python
|
||||
from safetensors import safe_open
|
||||
f = safe_open("qwen3vl_8b_fp8_scaled.safetensors", framework="pt", device="cpu")
|
||||
print(len(f.keys())) # Should be ~1254; crashes on truncated files
|
||||
```
|
||||
|
||||
### Pitfall: wrong input name for DualModelGuider
|
||||
|
||||
The Blueprint user's `Subgraphs`-based `DualModelGuider` uses the internal **editor-format** link name `model_1`, but the **API-format** input name is **`model_negative`**:
|
||||
|
||||
```
|
||||
# WRONG (editor convention leaked into API format):
|
||||
"inputs": {"model": ["X", 0], "model_1": ["Y", 0], ...}
|
||||
|
||||
# RIGHT (API format):
|
||||
"inputs": {"model": ["X", 0], "model_negative": ["Y", 0], ...}
|
||||
```
|
||||
|
||||
Error if wrong: `TypeError: DualModelGuider.execute() got an unexpected keyword argument 'model_1'`
|
||||
|
||||
### Full local workflow blueprint (extracted from ComfyUI `/history`)
|
||||
|
||||
The official blueprint at `blueprints/Text to Image (Ideogram v4).json` is in **editor format** (`"nodes"` / `"links"` arrays). It must be converted to **API format** (`"class_type"` per node) before execution. Below is a verified API-format workflow:
|
||||
|
||||
```json
|
||||
{
|
||||
"37": {
|
||||
"inputs": {"aspect_ratio": "1:1 (Square)", "megapixels": 1.0, "multiple": 8},
|
||||
"class_type": "ResolutionSelector"
|
||||
},
|
||||
"98:9": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
||||
"98:11": {"inputs": {"width": ["98:31", 1], "height": ["98:32", 1], "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
||||
"98:12": {"inputs": {"noise": ["98:18", 0], "guider": ["98:155", 0], "sampler": ["98:16", 0], "sigmas": ["98:17", 0], "latent_image": ["98:11", 0]}, "class_type": "SamplerCustomAdvanced"},
|
||||
"98:13": {"inputs": {"samples": ["98:12", 0], "vae": ["98:9", 0]}, "class_type": "VAEDecode"},
|
||||
"98:14": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4", "device": "default"}, "class_type": "CLIPLoader"},
|
||||
"98:16": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
||||
"98:17": {"inputs": {"steps": ["98:151", 1], "width": ["98:31", 1], "height": ["98:32", 1], "mu": ["98:144", 0], "std": ["98:146", 0]}, "class_type": "Ideogram4Scheduler"},
|
||||
"98:18": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
||||
"98:23": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
|
||||
"98:24": {"inputs": {"text": "PROMPT_HERE", "clip": ["98:14", 0]}, "class_type": "CLIPTextEncode"},
|
||||
"98:10": {"inputs": {"conditioning": ["98:24", 0]}, "class_type": "ConditioningZeroOut"},
|
||||
"98:31": {"inputs": {"expression": "max(((a + 15) // 16) * 16, 256)", "values.a": ["98:27", 0]}, "class_type": "ComfyMathExpression"},
|
||||
"98:32": {"inputs": {"expression": "max(((a + 15) // 16) * 16, 256)", "values.a": ["98:28", 0]}, "class_type": "ComfyMathExpression"},
|
||||
"98:27": {"inputs": {"value": ["37", 0]}, "class_type": "PrimitiveInt"},
|
||||
"98:28": {"inputs": {"value": ["37", 1]}, "class_type": "PrimitiveInt"},
|
||||
"98:144": {"inputs": {"value": ["98:145", 0]}, "class_type": "ComfyNumberConvert"},
|
||||
"98:145": {"inputs": {"json_string": ["98:148", 0], "key": "mu"}, "class_type": "JsonExtractString"},
|
||||
"98:146": {"inputs": {"value": ["98:150", 0]}, "class_type": "ComfyNumberConvert"},
|
||||
"98:147": {"inputs": {"json_string": "{\\\"Quality\\\":{\\\"num_steps\\\":48,\\\"mu\\\":0.0,\\\"std\\\":1.5,\\\"preset\\\":\\\"Quality\\\"}}", "key": "Quality"}, "class_type": "JsonExtractString"},
|
||||
"98:148": {"inputs": {"string": ["98:147", 0], "find": "'", "replace": "\\""}, "class_type": "StringReplace"},
|
||||
"98:149": {"inputs": {"json_string": ["98:148", 0], "key": "num_steps"}, "class_type": "JsonExtractString"},
|
||||
"98:150": {"inputs": {"json_string": ["98:148", 0], "key": "std"}, "class_type": "JsonExtractString"},
|
||||
"98:151": {"inputs": {"value": ["98:149", 0]}, "class_type": "ComfyNumberConvert"},
|
||||
"98:154": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
|
||||
"98:155": {"inputs": {"cfg": 7.0, "model": ["98:157", 0], "positive": ["98:24", 0], "model_negative": ["98:154", 0], "negative": ["98:10", 0]}, "class_type": "DualModelGuider"},
|
||||
"98:156": {"inputs": {"choice": "Quality", "index": 1, "option1": "Quality", "option2": "Default", "option3": "Turbo", "option4": ""}, "class_type": "CustomCombo"},
|
||||
"98:157": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["98:23", 0]}, "class_type": "CFGOverride"},
|
||||
"158": {"inputs": {"filename_prefix": "Ideogram_4.0", "images": ["98:13", 0]}, "class_type": "SaveImage"}
|
||||
}
|
||||
```
|
||||
|
||||
Replace `"PROMPT_HERE"` (node `98:24`, field `text`) with your prompt. Ideogram 4 accepts either plain text or structured JSON prompts.
|
||||
|
||||
### Accessing a ComfyUI LXC container via Proxmox VE
|
||||
|
||||
If ComfyUI runs inside a Proxmox LXC container without direct SSH:
|
||||
|
||||
```bash
|
||||
# Proxmox node shell → run inside container
|
||||
pct exec CTID -- bash -c "COMMAND"
|
||||
|
||||
# Push a file into the container (avoid nested quote hell)
|
||||
pct push CTID /local/script.py /tmp/script.py
|
||||
pct exec CTID -- python3 /tmp/script.py
|
||||
|
||||
# Common CTID in this environment: 204 (ComfyUI+FLUX.2/ROCm)
|
||||
```
|
||||
|
||||
### Key differences from FLUX.1
|
||||
|
||||
- Text encoder: `qwen3vl_8b_fp8_scaled` or `gemma4_e4b_it_fp8_scaled` — NOT CLIP-L + T5XXL
|
||||
- Latent: `EmptyFlux2LatentImage` (128 channels)
|
||||
- VAE: `flux2-vae.safetensors` — NOT `ae.safetensors`
|
||||
- CFG: Uses asymmetric CFG (unconditional conditioning is zeroed out via `ConditioningZeroOut`, not a separate negative prompt string)
|
||||
|
||||
## Quick verification checklist
|
||||
|
||||
1. `curl $HOST:8188/models/diffusion_models | grep ideogram4` — models present?
|
||||
2. Check file integrity of `qwen3vl_8b_fp8_scaled.safetensors` with the Python check above.
|
||||
3. For cloud: ensure `api_key_comfy_org` is set or the user is logged into comfy.org via the web UI.
|
||||
4. For local: verify the blueprint was exported as **API format**, not editor format (which has top-level `"nodes"` / `"links"` arrays).
|
||||
@@ -0,0 +1,255 @@
|
||||
# comfy-cli Command Reference
|
||||
|
||||
Official CLI from [Comfy-Org/comfy-cli](https://github.com/Comfy-Org/comfy-cli).
|
||||
Docs: https://docs.comfy.org/comfy-cli/getting-started
|
||||
|
||||
## Installation
|
||||
|
||||
Order of preference:
|
||||
|
||||
```bash
|
||||
pipx install comfy-cli # recommended (isolated env)
|
||||
uvx --from comfy-cli comfy --help # zero-install via uv
|
||||
pip install --user comfy-cli # fallback
|
||||
```
|
||||
|
||||
The skill's `comfyui_setup.sh` picks the best available method.
|
||||
|
||||
First run may prompt for analytics. Disable non-interactively:
|
||||
```bash
|
||||
comfy --skip-prompt tracking disable
|
||||
```
|
||||
|
||||
## Global Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--workspace <path>` | Target a specific ComfyUI workspace |
|
||||
| `--recent` | Use most recently used workspace |
|
||||
| `--here` | Use current directory as workspace |
|
||||
| `--skip-prompt` | No interactive prompts (use defaults) |
|
||||
| `-v` / `--version` | Print version |
|
||||
|
||||
Workspace resolution priority:
|
||||
1. `--workspace` (explicit path)
|
||||
2. `--recent` (from config)
|
||||
3. `--here` (cwd)
|
||||
4. `comfy set-default` path
|
||||
5. Most recently used
|
||||
6. `~/comfy/ComfyUI` (Linux) or `~/Documents/comfy/ComfyUI` (macOS/Win)
|
||||
|
||||
## Lifecycle Commands
|
||||
|
||||
### `comfy install`
|
||||
|
||||
Download and install ComfyUI + ComfyUI-Manager.
|
||||
|
||||
```bash
|
||||
comfy install # interactive GPU selection
|
||||
comfy install --nvidia
|
||||
comfy install --amd # ROCm (Linux)
|
||||
comfy install --m-series # Apple Silicon (MPS)
|
||||
comfy install --cpu # CPU only (slow)
|
||||
comfy install --fast-deps # use uv for deps
|
||||
comfy install --skip-manager # skip ComfyUI-Manager
|
||||
```
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--nvidia` / `--amd` / `--m-series` / `--cpu` | GPU type |
|
||||
| `--cuda-version` | 11.8, 12.1, 12.4, 12.6, 12.8, 12.9, 13.0 |
|
||||
| `--rocm-version` | 6.1, 6.2, 6.3, 7.0, 7.1 |
|
||||
| `--fast-deps` | uv-based dependency resolution |
|
||||
| `--skip-manager` | Don't install ComfyUI-Manager |
|
||||
| `--skip-torch-or-directml` | Skip PyTorch install |
|
||||
| `--version <ver>` | `0.2.0`, `latest`, `nightly` |
|
||||
| `--commit <hash>` | Install specific commit |
|
||||
| `--pr "#1234"` | Install from a PR |
|
||||
| `--restore` | Restore deps for existing install |
|
||||
|
||||
### `comfy launch`
|
||||
|
||||
```bash
|
||||
comfy launch # foreground :8188
|
||||
comfy launch --background # background daemon
|
||||
comfy launch -- --listen 0.0.0.0 # LAN-accessible
|
||||
comfy launch -- --port 8190 # custom port
|
||||
comfy launch -- --cpu # force CPU mode
|
||||
comfy launch -- --lowvram # 6 GB cards
|
||||
comfy launch --background -- --listen 0.0.0.0 --port 8190
|
||||
```
|
||||
|
||||
Common extra args after `--`: `--listen`, `--port`, `--cpu`, `--lowvram`,
|
||||
`--novram`, `--fp16-vae`, `--force-fp32`, `--disable-cuda-malloc`.
|
||||
|
||||
### `comfy stop`
|
||||
|
||||
```bash
|
||||
comfy stop
|
||||
```
|
||||
|
||||
### `comfy run`
|
||||
|
||||
Submit a raw workflow JSON to a running server. **Limited** — no parameter
|
||||
injection, no structured output download. For agents, use
|
||||
`scripts/run_workflow.py` instead.
|
||||
|
||||
```bash
|
||||
comfy run --workflow workflow_api.json
|
||||
comfy run --workflow workflow_api.json --host 10.0.0.5 --port 8188
|
||||
comfy run --workflow workflow_api.json --timeout 300 --wait
|
||||
```
|
||||
|
||||
### `comfy which`
|
||||
|
||||
```bash
|
||||
comfy which # show targeted workspace
|
||||
comfy --recent which
|
||||
```
|
||||
|
||||
### `comfy set-default`
|
||||
|
||||
```bash
|
||||
comfy set-default /path/to/ComfyUI
|
||||
comfy set-default /path/to/ComfyUI --launch-extras="--listen 0.0.0.0"
|
||||
```
|
||||
|
||||
### `comfy update`
|
||||
|
||||
```bash
|
||||
comfy update # update ComfyUI core
|
||||
comfy node update all # update all custom nodes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `comfy node` — Custom Node Management
|
||||
|
||||
All node operations use ComfyUI-Manager (`cm-cli`) under the hood.
|
||||
|
||||
```bash
|
||||
comfy node show installed # list installed
|
||||
comfy node show enabled # list enabled
|
||||
comfy node show all # all available in registry
|
||||
comfy node simple-show installed # compact list
|
||||
|
||||
comfy node install comfyui-impact-pack
|
||||
comfy node install <name> --uv-compile # ComfyUI-Manager v4.1+ unified resolver
|
||||
comfy node uninstall <name>
|
||||
comfy node update <name> | all
|
||||
comfy node enable <name>
|
||||
comfy node disable <name>
|
||||
comfy node fix <name> # fix broken deps
|
||||
|
||||
comfy node install-deps --workflow=workflow.json
|
||||
comfy node deps-in-workflow --workflow=w.json --output=deps.json
|
||||
|
||||
comfy node save-snapshot
|
||||
comfy node restore-snapshot <file>
|
||||
|
||||
comfy node bisect start # binary-search a culprit node
|
||||
comfy node bisect good
|
||||
comfy node bisect bad
|
||||
comfy node bisect reset
|
||||
```
|
||||
|
||||
### Dependency Resolution Options
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--fast-deps` | comfy-cli built-in uv resolver |
|
||||
| `--uv-compile` | ComfyUI-Manager v4.1+ unified resolver (recommended) |
|
||||
| `--no-deps` | Skip dep installation |
|
||||
|
||||
Make `uv-compile` default: `comfy manager uv-compile-default true`
|
||||
|
||||
---
|
||||
|
||||
## `comfy model` — Model Management
|
||||
|
||||
```bash
|
||||
comfy model list
|
||||
comfy model list --relative-path models/checkpoints
|
||||
|
||||
comfy model download --url <URL>
|
||||
comfy model download --url <URL> --relative-path models/loras
|
||||
comfy model download --url <URL> --filename custom_name.safetensors
|
||||
|
||||
comfy model remove # interactive
|
||||
comfy model remove --relative-path models/checkpoints --model-names "model.safetensors"
|
||||
```
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--url` | Download URL (CivitAI, HuggingFace, direct) |
|
||||
| `--relative-path` | Subdirectory under workspace (e.g. `models/checkpoints`) |
|
||||
| `--filename` | Custom save filename |
|
||||
| `--set-civitai-api-token` | Persist CivitAI token |
|
||||
| `--set-hf-api-token` | Persist HuggingFace token |
|
||||
| `--downloader` | `httpx` (default) or `aria2` |
|
||||
|
||||
Standard model directories:
|
||||
```
|
||||
ComfyUI/models/
|
||||
├── checkpoints/ # Full model files
|
||||
├── loras/ # LoRA adapters
|
||||
├── vae/ # VAE models
|
||||
├── controlnet/ # ControlNet models
|
||||
├── clip/ # CLIP / T5 text encoders
|
||||
├── clip_vision/ # CLIP vision encoders
|
||||
├── upscale_models/ # ESRGAN / SwinIR / etc.
|
||||
├── embeddings/ # Textual inversion embeddings
|
||||
├── unet/ # Standalone UNet weights
|
||||
├── diffusion_models/ # Flux / SD3 / Wan diffusion models
|
||||
├── animatediff_models/ # AnimateDiff motion modules
|
||||
├── ipadapter/ # IPAdapter weights
|
||||
└── style_models/ # Style adapters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `comfy manager` — ComfyUI-Manager Settings
|
||||
|
||||
```bash
|
||||
comfy manager disable # disable Manager completely
|
||||
comfy manager enable-gui # enable new GUI
|
||||
comfy manager disable-gui # API-only
|
||||
comfy manager enable-legacy-gui # legacy GUI
|
||||
comfy manager uv-compile-default true # make --uv-compile the default
|
||||
comfy manager clear # clear startup action
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `comfy pr-cache` — Frontend PR Cache
|
||||
|
||||
```bash
|
||||
comfy pr-cache list
|
||||
comfy pr-cache clean
|
||||
comfy pr-cache clean 456
|
||||
```
|
||||
|
||||
Cache expires after 7 days; max 10 builds.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
| OS | Path |
|
||||
|----|------|
|
||||
| Linux | `~/.config/comfy-cli/config.ini` |
|
||||
| macOS | `~/Library/Application Support/comfy-cli/config.ini` |
|
||||
| Windows | `~/AppData/Local/comfy-cli/config.ini` |
|
||||
|
||||
Stores: default workspace, recent workspace, background server PID, API
|
||||
tokens, manager GUI mode, launch extras.
|
||||
|
||||
## Discovery
|
||||
|
||||
Custom-node registry:
|
||||
- https://registry.comfy.org/
|
||||
|
||||
Model browsers:
|
||||
- https://huggingface.co/models
|
||||
- https://civitai.com (NSFW; requires API token for many)
|
||||
- https://comfyworkflows.com (community workflows)
|
||||
@@ -0,0 +1,107 @@
|
||||
# OpenAI-Compatible HTTP Adapter for ComfyUI
|
||||
|
||||
Small FastAPI bridge that exposes `POST /v1/images/generations` (OpenAI format) and maps it to ComfyUI's `POST /api/prompt`.
|
||||
|
||||
## When to use
|
||||
|
||||
- OpenWebUI, Bifrost, or any other tool that speaks OpenAI `/v1/images/generations` needs to generate images with ComfyUI.
|
||||
- ComfyUI itself has no native OpenAI-compatible endpoint.
|
||||
- The adapter is especially useful for **FLUX.2 workflows** that OpenWebUI's built-in ComfyUI connector cannot handle (different nodes: Qwen3VL, `EmptyFlux2LatentImage`, TAEF2).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
OpenWebUI / Bifrost / Client
|
||||
POST /v1/images/generations
|
||||
|
|
||||
v
|
||||
[FastAPI Adapter] (Docker or systemd)
|
||||
|
|
||||
v
|
||||
POST http://<comfyui>:8188/api/prompt
|
||||
GET http://<comfyui>:8188/history/<prompt_id>
|
||||
GET http://<comfyui>:8188/view
|
||||
```
|
||||
|
||||
## Example: Minimal FLUX.2 Adapter (Docker)
|
||||
|
||||
### `Dockerfile`
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY adapter.py .
|
||||
EXPOSE 9000
|
||||
CMD ["uvicorn", "adapter:app", "--host", "0.0.0.0", "--port", "9000"]
|
||||
```
|
||||
|
||||
### `requirements.txt`
|
||||
|
||||
```
|
||||
fastapi>=0.110.0
|
||||
uvicorn>=0.29.0
|
||||
httpx>=0.27.0
|
||||
pydantic>=2.0.0
|
||||
```
|
||||
|
||||
### `adapter.py` — FLUX.2 Variant
|
||||
|
||||
The node names must match what is installed on the target ComfyUI instance.
|
||||
Query the running server first with `GET /object_info` to confirm.
|
||||
|
||||
Key requirements for FLUX.2:
|
||||
- `CLIPLoaderGGUF` with `"type": "flux2"` (loads Qwen3VL)
|
||||
- `EmptyFlux2LatentImage` (128-channel latents)
|
||||
- `VAELoader` with `"vae_name": "taef2"` (or dedicated `flux2_vae.safetensors`)
|
||||
- `UnetLoaderGGUF` with the FLUX.2 UNet GGUF
|
||||
|
||||
See `scripts/adapter_flux2.py` for a ready-to-use file.
|
||||
|
||||
### Build & run
|
||||
|
||||
```bash
|
||||
docker build -t comfyui-openai-adapter:latest .
|
||||
docker run -d --name comfyui-openai-adapter --network host \
|
||||
-e COMFYUI_URL=http://10.0.30.97:8188 \
|
||||
-e WORKFLOW_PREFIX=oai_adapter \
|
||||
--restart unless-stopped \
|
||||
comfyui-openai-adapter:latest
|
||||
```
|
||||
|
||||
## Verifying node availability
|
||||
|
||||
Before deploying the adapter, confirm the target ComfyUI instance has the required nodes:
|
||||
|
||||
```bash
|
||||
curl -s http://<comfyui>:8188/object_info | \
|
||||
python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||
[print(f'{n}: {n in d}') for n in \
|
||||
['CLIPLoaderGGUF','UnetLoaderGGUF','EmptyFlux2LatentImage','VAELoader','KSampler']]"
|
||||
```
|
||||
|
||||
## OpenWebUI configuration
|
||||
|
||||
Admin Panel → Settings → Images → Image Generation
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Engine | `Open AI` |
|
||||
| API Base URL | `http://<adapter-ip>:9000/v1` |
|
||||
| API Key | dummy (adapter does not enforce auth) |
|
||||
| Model | `flux2` |
|
||||
|
||||
## Bifrost integration
|
||||
|
||||
Bifrost is an LLM gateway with no native ComfyUI provider. The adapter fills that gap:
|
||||
- Register the adapter as an OpenAI-compatible provider in Bifrost, or
|
||||
- Point OpenWebUI directly at the adapter (recommended; keeps image path separate from chat routing).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **Node names are exact.** Use `GET /object_info` to verify every `class_type` exists; missing custom nodes produce cryptic "class_type not found" errors from ComfyUI.
|
||||
2. **Model names are exact and case-sensitive.** `flux-2-klein-9b-Q4_K_S.gguf` != `flux-2-klein-9b-q4_k_s.gguf`.
|
||||
3. **TAEF2 vs full VAE.** `taef2` resolves tiny encoder/decoder `pth` files; the full `flux2_vae.safetensors` requires manual download from a gated repo.
|
||||
4. **Seed injection.** Always rewrite the seed per image when `n > 1`; otherwise every image in the batch is identical.
|
||||
5. **Timeout tuning.** FLUX.2 on ROCm gfx1150 takes ~110s for 1024x1024 @ 4 steps. Set `wait_for_image` timeout >= 300s.
|
||||
@@ -0,0 +1,281 @@
|
||||
# OpenAI-Compatible Multi-Model API Wrapper for ComfyUI
|
||||
|
||||
FastAPI proxy that exposes `POST /v1/images/generations` (OpenAI format) and
|
||||
routes to **multiple** ComfyUI workflows depending on the requested `model`.
|
||||
|
||||
Supported models: `ideogram4`, `flux2`, `flux1-dev`, `flux1-schnell`.
|
||||
|
||||
## When to use
|
||||
|
||||
- You need one endpoint that can switch between Ideogram 4 and FLUX
|
||||
families without reconfiguring the frontend.
|
||||
- Your chat UI only speaks OpenAI `/v1/images/generations` but you want
|
||||
to use ComfyUI's native pipelines (including non-standard nodes like
|
||||
`Ideogram4Scheduler`, `ModelSamplingFlux`, `DualCLIPLoader`).
|
||||
- You run ComfyUI inside a Proxmox LXC/VM and want a systemd-managed
|
||||
adapter service.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Frontend (OpenWebUI, LibreChat, etc.)
|
||||
POST /v1/images/generations {model, prompt, size, ...}
|
||||
│
|
||||
▼
|
||||
FastAPI Adapter (this wrapper)
|
||||
─ model switch ──► _build_ideogram4_workflow()
|
||||
_build_flux2_workflow()
|
||||
_build_flux1_workflow()
|
||||
│
|
||||
▼
|
||||
ComfyUI REST API POST /prompt
|
||||
│
|
||||
▼
|
||||
Poll /history/{prompt_id} → fetch /view
|
||||
```
|
||||
|
||||
## Adapter Code (Multi-Model)
|
||||
|
||||
The full source is provided as a **template** in this skill:
|
||||
`templates/comfyui-openai-adapter.py`
|
||||
|
||||
Key design choices:
|
||||
|
||||
- **Model routing** — `req.model` selects the workflow builder.
|
||||
- **Size validation / fallback** — Ideogram4 only accepts specific
|
||||
resolutions. FLUX is more permissive. Invalid sizes silently fall back to the model's default.
|
||||
- **Loader node per model family:**
|
||||
- Ideogram4 → `UNETLoader` (safetensors)
|
||||
- FLUX (GGUF) → `UnetLoaderGGUF` (from ComfyUI-GGUF custom node)
|
||||
- Ideogram4 → single `CLIPLoader` with `type: "ideogram4"`
|
||||
- FLUX → `DualCLIPLoader` with `type: "flux"` (clip_l + t5xxl)
|
||||
- **Quality → steps mapping:** `low` (8), `medium` (20), `high` (30), `ultra` (50).
|
||||
FLUX1-schnell hard-capped to 8 steps.
|
||||
- **Ideogram4 minimum steps:** do not go below ~12 steps — the model
|
||||
produces near-blank gray blocks at ≤8 steps. Clamp or warn when
|
||||
the user explicitly requests fewer.
|
||||
- **Ideogram4 asymmetric CFG wiring (CRITICAL):** The adapter's `_build_ideogram4_workflow()`
|
||||
MUST use `CFGOverride` + `DualModelGuider` + a separate unconditional UNet
|
||||
(`ideogram4_unconditional_fp8_scaled`) + `ConditioningZeroOut`. Using `BasicGuider`
|
||||
does NOT throw an error but silently degrades image quality. The `DualModelGuider`
|
||||
input for the unconditional model is `model_negative` (NOT `model_1`).
|
||||
See `references/ideogram4.md` for the full blueprint.
|
||||
- **Gray-block detection:** If Ideogram4 output has PIL variance < 15 across all
|
||||
channels, the generation produced a blank/gray image (often at `quality=low` + `style=photorealistic`).
|
||||
Flag these as failures and retry with higher steps or different style.
|
||||
- **ROCm environment** — if ComfyUI runs on AMD ROCm inside an LXC,
|
||||
the *systemd service* for ComfyUI must set
|
||||
`Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0` (or whatever matches
|
||||
the GPU). The adapter itself does not need ROCm env vars.
|
||||
- **Timeout for long generations** — set the adapter's HTTP poll loop
|
||||
to at least 1800 s (30 min) for high-resolution workloads.
|
||||
Edit `for i in range(720)` (30 min at 5 s intervals) and set
|
||||
`httpx.AsyncClient(timeout=2400)` on the initial `/prompt` POST.
|
||||
- **Synchronous (blocking) by design** — the adapter does NOT implement
|
||||
async jobs. The HTTP response blocks until ComfyUI finishes. For
|
||||
multi-hour jobs this means the client must keep the connection open.
|
||||
If a true async job system is needed, build `POST → job_id` +
|
||||
`GET /v1/jobs/{id}` separately.
|
||||
|
||||
## Running it
|
||||
|
||||
### Option A: Python venv (quick test)
|
||||
|
||||
```bash
|
||||
python3 -m venv /opt/ideogram4-api/venv
|
||||
source /opt/ideogram4-api/venv/bin/activate
|
||||
pip install fastapi uvicorn httpx pydantic
|
||||
export COMFYUI_HOST=localhost
|
||||
export COMFYUI_PORT=8188
|
||||
python3 comfyui-openai-adapter.py
|
||||
```
|
||||
|
||||
### Option B: Docker on dedicated host (persistent)
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app
|
||||
RUN pip install --no-cache-dir fastapi uvicorn httpx pydantic
|
||||
COPY comfyui-openai-adapter.py ./main.py
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
> The adapter and ComfyUI should be on the same LAN or host. Latency matters because the adapter polls ComfyUI every 5 s for up to 60 min.
|
||||
|
||||
### Option C: systemd service (LXC/VM — recommended for Proxmox)
|
||||
|
||||
**Prerequisite:** Kill any previously manual/nohup-started ComfyUI or adapter processes *before* enabling systemd units. Old processes hold ports 8188 and 8000 and cause `address already in use`.
|
||||
|
||||
```bash
|
||||
# Inside LXC container (e.g. CT 204)
|
||||
ss -tlnp | grep -E '8188|8000'
|
||||
fuser -k 8188/tcp 2>/dev/null
|
||||
fuser -k 8000/tcp 2>/dev/null
|
||||
```
|
||||
|
||||
**`comfyui.service`** (note the ROCm env var for gfx1150):
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=ComfyUI
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/ComfyUI
|
||||
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
Environment=HIP_VISIBLE_DEVICES=0
|
||||
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
|
||||
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**`ideogram4-api.service`** (rename to `comfyui-api.service` if generic):
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=ComfyUI OpenAI API Wrapper
|
||||
After=network.target comfyui.service
|
||||
Requires=comfyui.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/ideogram4-api
|
||||
ExecStart=/opt/ideogram4-api/venv/bin/python /opt/ideogram4-api/main.py
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=PATH=/opt/ideogram4-api/venv/bin:/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Environment=COMFYUI_HOST=localhost
|
||||
Environment=COMFYUI_PORT=8188
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
> **Filename note:** The adapter source may be saved as `main.py`, `app.py`, or `adapter.py`. Check `ls /opt/ideogram4-api/` and match `ExecStart=` exactly.
|
||||
|
||||
Enable & start:
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now comfyui.service
|
||||
sleep 15
|
||||
systemctl enable --now ideogram4-api.service
|
||||
```
|
||||
|
||||
## Extended OpenAI-compatible parameters
|
||||
|
||||
Beyond the standard `model`, `prompt`, `n`, `size`, `response_format`,
|
||||
the adapter accepts these ComfyUI extensions:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `seed` | int | random | Fixed seed. Random if omitted. |
|
||||
| `steps` | int | from `quality` | Denoising steps (1–100). Overrides `quality` if set. |
|
||||
| `cfg` | float | model-specific | Guidance scale. Ideogram4=1.0, FLUX=3.5, others=7.5 |
|
||||
| `quality` | string | `medium` | Maps to steps: `low`(8), `medium`(20), `high`(30), `ultra`(50). |
|
||||
| `negative_prompt` | string | `""` | Negative prompt text. |
|
||||
| `style` | string | `""` | Style modifier string. |
|
||||
| `control_strength` | float | — | Reserved for future ControlNet integration. |
|
||||
|
||||
Example request body:
|
||||
```json
|
||||
{
|
||||
"model": "flux2",
|
||||
"prompt": "a cyberpunk alley at night, neon signs",
|
||||
"n": 1,
|
||||
"size": "1024x1024",
|
||||
"quality": "high",
|
||||
"seed": 42,
|
||||
"cfg": 3.5,
|
||||
"negative_prompt": "blurry, low quality"
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend configuration (OpenWebUI example)
|
||||
|
||||
1. Admin → **Settings** → **Images**
|
||||
2. **Image Generation Engine:** `Open AI`
|
||||
3. **OpenAI API Base URL:** `http://<adapter-host>:8000/v1`
|
||||
4. **OpenAI API Key:** any dummy string (adapter does not validate)
|
||||
5. **Model:** `ideogram4` or `flux2` or `flux1-dev`
|
||||
|
||||
## Pitfalls
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `TimeoutError` after 300 s | High resolution or GPU under load | Clamp `size`, increase timeout, reduce `steps` |
|
||||
| `500` from ComfyUI + `value_not_in_list` | `size` not in model's allowed resolutions | Check allowed sizes; use defaults |
|
||||
| `[Errno 98] address already in use` | Old manual `nohup` process still running | `fuser -k 8000/tcp` / `fuser -k 8188/tcp`, then restart systemd |
|
||||
| `can't open file '/opt/ideogram4-api/app.py'` | `ExecStart` mismatches actual filename | Check `ls /opt/ideogram4-api/` and align `ExecStart` |
|
||||
| `UNETLoader` node error for `.gguf` model | Using `UNETLoader` for a GGUF file | Switch to `UnetLoaderGGUF` |
|
||||
| `RuntimeError` during text-encode (ROCm) | Missing `HSA_OVERRIDE_GFX_VERSION` | Add to `comfyui.service` `Environment=` line |
|
||||
| Ideogram4 black/corrupted output | Corrupt `.safetensors` or wrong VAE | Verify file integrity; use `flux2-vae.safetensors` |
|
||||
| FLUX output dimensions wrong | Used `EmptyLatentImage` instead of `EmptyFlux2LatentImage` | Ideogram4 uses `EmptyFlux2LatentImage`; FLUX.1 uses `EmptyLatentImage` |
|
||||
| `DualModelGuider.execute() got unexpected keyword argument 'model_1'` | Wrong workflow key name | Use `model_negative`, NOT `model_1` |
|
||||
| systemd `comfyui.service` keeps failing with `exit-code` | Port 8188 held by old manual ComfyUI | `pkill -f "main.py.*8188"`, then `systemctl restart` |
|
||||
|
||||
## Model-specific node quick-reference
|
||||
|
||||
### Ideogram 4 local (with asymmetric CFG — correct wiring)
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
||||
"2": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"},
|
||||
"class_type": "UNETLoader"},
|
||||
"3": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4"},
|
||||
"class_type": "CLIPLoader"},
|
||||
"4": {"inputs": {"text": "...", "clip": ["3", 0]}, "class_type": "CLIPTextEncode"},
|
||||
"5": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
||||
"6": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
||||
"7": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
||||
"8": {"inputs": {"steps": 20, "width": 1024, "height": 1024, "mu": 0.5, "std": 1.75},
|
||||
"class_type": "Ideogram4Scheduler"},
|
||||
"15": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"},
|
||||
"class_type": "UNETLoader"},
|
||||
"16": {"inputs": {"conditioning": ["4", 0]}, "class_type": "ConditioningZeroOut"},
|
||||
"17": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["2", 0]},
|
||||
"class_type": "CFGOverride"},
|
||||
"9": {"inputs": {"cfg": 7.0, "model": ["17", 0], "positive": ["4", 0],
|
||||
"model_negative": ["15", 0], "negative": ["16", 0]},
|
||||
"class_type": "DualModelGuider"},
|
||||
"10": {"inputs": {"noise": ["6", 0], "guider": ["9", 0], "sampler": ["7", 0],
|
||||
"sigmas": ["8", 0], "latent_image": ["5", 0]}, "class_type": "SamplerCustomAdvanced"},
|
||||
"11": {"inputs": {"samples": ["10", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
|
||||
"12": {"inputs": {"filename_prefix": "ideogram4_api", "images": ["11", 0]}, "class_type": "SaveImage"}
|
||||
}
|
||||
```
|
||||
Allowed sizes: `1024x1024`, `1024x1536`, `1536x1024`, `832x1216`, `1216x832`, `896x1152`, `1152x896`.
|
||||
|
||||
### FLUX.2 / FLUX.1
|
||||
```json
|
||||
{
|
||||
"1": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
||||
"2": {"inputs": {"unet_name": "flux1-dev-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
|
||||
"3": {"inputs": {"model": ["2", 0], "max_shift": 1.15, "base_shift": 0.5,
|
||||
"width": 1024, "height": 1024}, "class_type": "ModelSamplingFlux"},
|
||||
"4": {"inputs": {"clip_name1": "clip_l.safetensors", "clip_name2": "t5xxl_fp8_e4m3fn.safetensors",
|
||||
"type": "flux", "device": "default"}, "class_type": "DualCLIPLoader"},
|
||||
"5": {"inputs": {"text": "...", "clip": ["4", 0]}, "class_type": "CLIPTextEncode"},
|
||||
"6": {"inputs": {"conditioning": ["5", 0], "guidance": 3.5}, "class_type": "FluxGuidance"},
|
||||
"7": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyLatentImage"},
|
||||
"8": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
||||
"9": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
||||
"10": {"inputs": {"model": ["3", 0], "steps": 20, "denoise": 1.0, "scheduler": "simple",
|
||||
"sampler": ["9", 0]}, "class_type": "BasicScheduler"},
|
||||
"11": {"inputs": {"model": ["3", 0], "conditioning": ["6", 0]}, "class_type": "BasicGuider"},
|
||||
"12": {"inputs": {"noise": ["8", 0], "guider": ["11", 0], "sampler": ["9", 0],
|
||||
"sigmas": ["10", 0], "latent_image": ["7", 0]}, "class_type": "SamplerCustomAdvanced"},
|
||||
"13": {"inputs": {"samples": ["12", 0], "vae": ["1", 0]}, "class_type": "VAEDecode"},
|
||||
"14": {"inputs": {"filename_prefix": "flux2_api", "images": ["13", 0]}, "class_type": "SaveImage"}
|
||||
}
|
||||
```
|
||||
Allowed sizes: all common resolutions up to 1536×1536. FLUX schnell defaults to 4–8 steps; dev to 20–50.
|
||||
@@ -0,0 +1,201 @@
|
||||
# OpenWebUI ↔ ComfyUI Integration
|
||||
|
||||
OpenWebUI's native ComfyUI connector is hard-wired for SD/SDXL pipelines (CLIP-L + T5XXL, `EmptyLatentImage`, standard VAE). It **does not work** with FLUX.2 (Qwen3-VL, `EmptyFlux2LatentImage`, TAE/VAE).
|
||||
|
||||
The cleanest solution is a lightweight OpenAI-compatible adapter that exposes `/v1/images/generations` and internally submits a FLUX.2 workflow to ComfyUI's REST API.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
OpenWebUI ──POST /v1/images/generations──► Adapter ──POST /api/prompt──► ComfyUI
|
||||
(CT 135) (OpenAI format) (Docker) (workflow JSON) (CT 204)
|
||||
```
|
||||
|
||||
The adapter runs as a Docker container (or Python service) on a host that has Layer-2/Layer-3 reachability to ComfyUI.
|
||||
|
||||
---
|
||||
|
||||
## Adapter Code
|
||||
|
||||
### `adapter.py`
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
import httpx
|
||||
import json
|
||||
import base64
|
||||
import asyncio
|
||||
import time
|
||||
import random
|
||||
import os
|
||||
|
||||
app = FastAPI(title="ComfyUI OpenAI Adapter", version="0.1.0")
|
||||
|
||||
COMFYUI_URL = os.environ.get("COMFYUI_URL", "http://127.0.0.1:8188")
|
||||
WORKFLOW_PREFIX = os.environ.get("WORKFLOW_PREFIX", "oai_adapter")
|
||||
|
||||
FLUX2_WORKFLOW = {
|
||||
"1": {"inputs": {"unet_name": "flux-2-klein-9b-Q4_K_S.gguf"}, "class_type": "UnetLoaderGGUF"},
|
||||
"2": {"inputs": {"clip_name": "Qwen3VL-8B-Instruct-Q4_K_M.gguf", "type": "flux2"}, "class_type": "CLIPLoaderGGUF"},
|
||||
"3": {"inputs": {"vae_name": "taef2"}, "class_type": "VAELoader"},
|
||||
"4": {"inputs": {"width": 1024, "height": 1024, "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
||||
"5": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
|
||||
"6": {"inputs": {"text": "", "clip": ["2", 0]}, "class_type": "CLIPTextEncode"},
|
||||
"7": {"inputs": {"seed": 42, "steps": 4, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0,
|
||||
"model": ["1", 0], "positive": ["5", 0], "negative": ["6", 0], "latent_image": ["4", 0]},
|
||||
"class_type": "KSampler"},
|
||||
"8": {"inputs": {"samples": ["7", 0], "vae": ["3", 0]}, "class_type": "VAEDecode"},
|
||||
"9": {"inputs": {"filename_prefix": WORKFLOW_PREFIX, "images": ["8", 0]}, "class_type": "SaveImage"}
|
||||
}
|
||||
|
||||
class ImageRequest(BaseModel):
|
||||
prompt: str
|
||||
n: Optional[int] = 1
|
||||
size: Optional[str] = "1024x1024"
|
||||
response_format: Optional[str] = "b64_json"
|
||||
model: Optional[str] = "flux2"
|
||||
|
||||
async def queue_workflow(workflow: dict) -> str:
|
||||
async with httpx.AsyncClient() as client:
|
||||
r = await client.post(f"{COMFYUI_URL}/api/prompt", json={"prompt": workflow})
|
||||
r.raise_for_status()
|
||||
return r.json()["prompt_id"]
|
||||
|
||||
async def wait_for_image(prompt_id: str, timeout: int = 300) -> list:
|
||||
async with httpx.AsyncClient() as client:
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
r = await client.get(f"{COMFYUI_URL}/history/{prompt_id}")
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
if prompt_id in data and "outputs" in data[prompt_id]:
|
||||
outputs = []
|
||||
for node_id, node_out in data[prompt_id]["outputs"].items():
|
||||
if "images" in node_out:
|
||||
outputs.extend(node_out["images"])
|
||||
if outputs:
|
||||
return outputs
|
||||
await asyncio.sleep(1)
|
||||
raise TimeoutError(f"Timeout waiting for prompt {prompt_id}")
|
||||
|
||||
async def fetch_image(filename: str, subfolder: str = "", folder_type: str = "output") -> bytes:
|
||||
async with httpx.AsyncClient() as client:
|
||||
params = {"filename": filename, "subfolder": subfolder, "type": folder_type}
|
||||
r = await client.get(f"{COMFYUI_URL}/view", params=params)
|
||||
r.raise_for_status()
|
||||
return r.content
|
||||
|
||||
@app.post("/v1/images/generations")
|
||||
async def generate_images(req: ImageRequest):
|
||||
if req.n > 4:
|
||||
return JSONResponse({"error": {"message": "n must be <= 4", "type": "invalid_request_error"}}, status_code=400)
|
||||
|
||||
try:
|
||||
w, h = map(int, req.size.split("x"))
|
||||
except ValueError:
|
||||
w, h = 1024, 1024
|
||||
|
||||
# CRITICAL: clamp size to prevent timeout/OOM on consumer GPUs
|
||||
MAX_SIZE = 1536
|
||||
if w > MAX_SIZE or h > MAX_SIZE:
|
||||
scale = MAX_SIZE / max(w, h)
|
||||
w = int(w * scale)
|
||||
h = int(h * scale)
|
||||
|
||||
images = []
|
||||
for _ in range(req.n):
|
||||
wf = json.loads(json.dumps(FLUX2_WORKFLOW))
|
||||
wf["5"]["inputs"]["text"] = req.prompt
|
||||
wf["4"]["inputs"]["width"] = w
|
||||
wf["4"]["inputs"]["height"] = h
|
||||
wf["7"]["inputs"]["seed"] = random.randint(1, 2**32)
|
||||
|
||||
prompt_id = await queue_workflow(wf)
|
||||
imgs = await wait_for_image(prompt_id, timeout=300)
|
||||
if not imgs:
|
||||
raise RuntimeError("No images returned")
|
||||
|
||||
img_data = await fetch_image(imgs[0]["filename"], imgs[0].get("subfolder", ""), imgs[0].get("type", "output"))
|
||||
b64 = base64.b64encode(img_data).decode()
|
||||
images.append({"b64_json": b64})
|
||||
|
||||
return {"created": int(time.time()), "data": images}
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
```
|
||||
|
||||
### `Dockerfile`
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY adapter.py .
|
||||
EXPOSE 9000
|
||||
CMD ["uvicorn", "adapter:app", "--host", "0.0.0.0", "--port", "9000"]
|
||||
```
|
||||
|
||||
### `requirements.txt`
|
||||
|
||||
```text
|
||||
fastapi>=0.110.0
|
||||
uvicorn>=0.29.0
|
||||
httpx>=0.27.0
|
||||
pydantic>=2.0.0
|
||||
```
|
||||
|
||||
### `docker-compose.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
comfyui-openai-adapter:
|
||||
build: .
|
||||
container_name: comfyui-openai-adapter
|
||||
ports:
|
||||
- "9000:9000"
|
||||
environment:
|
||||
- COMFYUI_URL=http://10.0.30.97:8188 # <-- adapt to your ComfyUI IP
|
||||
- WORKFLOW_PREFIX=oai_bifrost
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
> **Network tip:** If the adapter and ComfyUI are on the same Layer-2 segment, prefer `--network host` (or attach the container to an existing bridge that reaches ComfyUI) to avoid NAT/port-mapping issues.
|
||||
|
||||
---
|
||||
|
||||
## OpenWebUI Configuration
|
||||
|
||||
1. Admin Panel → **Settings** → **Images**
|
||||
2. **Image Generation Engine:** `Open AI`
|
||||
3. **OpenAI API Base URL:** `http://<adapter-host>:9000/v1`
|
||||
4. **OpenAI API Key:** any dummy string (the adapter does not validate keys)
|
||||
5. **Model:** `flux2` (or leave default)
|
||||
|
||||
---
|
||||
|
||||
## Critical Pitfalls
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `TimeoutError` after 300 s | OpenWebUI/DALL-E 3 sends `size: "4096x4096"` | Adapter clamps to `MAX_SIZE = 1536` |
|
||||
| OOM / CUDA out of memory | Resolution too high for 48 GB shared VRAM | Keep ≤ 1536 px on consumer APUs |
|
||||
| `500 Internal Server Error` | ComfyUI node missing (`CLIPLoaderGGUF`, etc.) | Run `comfy node install comfyui-gguf` and verify with `curl /object_info` |
|
||||
| Adapter cannot reach ComfyUI | Docker bridge isolation | Use `--network host` or place both on same bridge |
|
||||
| Empty/black images | Wrong VAE (`ae.safetensors` instead of `taef2` for FLUX.2) | Use `taef2` TAE or the FLUX.2-specific VAE |
|
||||
|
||||
---
|
||||
|
||||
## Environment Snapshot (Schön Consulting, 2026-06-20)
|
||||
|
||||
- **ComfyUI:** PVE CT 204, `10.0.30.97:8188`, ROCm gfx1150 (Radeon 890M)
|
||||
- **OpenWebUI:** PVE CT 135, `10.0.30.102:8080`
|
||||
- **Adapter:** Docker-Host `10.0.30.99:9000` (Portainer node)
|
||||
- **Performance:** 1024×1024 ≈ 110 s, 1536×1536 ≈ 180 s (4 steps, euler/simple)
|
||||
@@ -0,0 +1,194 @@
|
||||
# Working with ComfyUI inside a Proxmox LXC over SSH
|
||||
|
||||
When ComfyUI is hosted inside a Proxmox CT (e.g. CT 204 on 10.0.30.97) and you
|
||||
need to interact with it from a machine that only has SSH access to the PVE host
|
||||
(10.0.20.91), the most reliable pattern is:
|
||||
|
||||
## ❌ Don't: multi-layer nested quotes
|
||||
|
||||
```bash
|
||||
# FAILS — 4-level escaping breaks on !, (, ", etc.
|
||||
ssh root@pve "pct exec 204 -- python3 -c \"print('hello')\""
|
||||
ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'
|
||||
```
|
||||
|
||||
Result: `bash: -c: line 34: syntax error near unexpected token ('`
|
||||
|
||||
## ✅ Do: write-then-execute
|
||||
|
||||
```bash
|
||||
# 1. SSH to PVE host and create a script file there
|
||||
ssh root@10.0.20.91 << 'HEREDOC'
|
||||
cat > /tmp/my_script.py << 'PYEOF'
|
||||
import struct, json, os
|
||||
path = "/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors"
|
||||
f = open(path, "rb")
|
||||
hlen = struct.unpack("<Q", f.read(8))[0]
|
||||
meta = json.loads(f.read(hlen))
|
||||
total = 8 + hlen
|
||||
for k, v in meta.items():
|
||||
if isinstance(v, dict) and "data_offsets" in v:
|
||||
total += v["data_offsets"][1] - v["data_offsets"][0]
|
||||
actual = os.path.getsize(path)
|
||||
print(f"expected={total} actual={actual} ok={total==actual}")
|
||||
PYEOF
|
||||
HEREDOC
|
||||
|
||||
# 2. Push script into LXC container
|
||||
ssh root@10.0.20.91 "pct push 204 /tmp/my_script.py /tmp/my_script.py"
|
||||
|
||||
# 3. Execute inside container
|
||||
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/my_script.py"
|
||||
|
||||
# 4. Clean up
|
||||
ssh root@10.0.20.91 "rm /tmp/my_script.py"
|
||||
```
|
||||
|
||||
## Template: compact one-liner for checks
|
||||
|
||||
```bash
|
||||
ssh root@10.0.20.91 'bash -s' << 'SSHSCRIPT'
|
||||
cat > /tmp/probe.py << 'PYEOF'
|
||||
from safetensors import safe_open
|
||||
f = safe_open("/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors",
|
||||
framework="pt", device="cpu")
|
||||
print("OK, tensors:", len(list(f.keys())))
|
||||
PYEOF
|
||||
pct push 204 /tmp/probe.py /tmp/probe.py
|
||||
pct exec 204 -- python3 /tmp/probe.py
|
||||
rm /tmp/probe.py
|
||||
SSHSCRIPT
|
||||
```
|
||||
|
||||
## Copying large files into LXC (e.g. model downloads)
|
||||
|
||||
```bash
|
||||
# Download to PVE host tmp
|
||||
ssh root@10.0.20.91 "cd /tmp && curl -L -o model.safetensors <URL>"
|
||||
|
||||
# Push into container
|
||||
ssh root@10.0.20.91 "pct push 204 /tmp/model.safetensors /opt/ComfyUI/models/text_encoders/model.safetensors"
|
||||
|
||||
# Verify
|
||||
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/probe.py"
|
||||
|
||||
# Clean up host
|
||||
ssh root@10.0.20.91 "rm /tmp/model.safetensors"
|
||||
```
|
||||
|
||||
## Persistent services inside LXC
|
||||
|
||||
ComfyUI and adapters should run as systemd services so they survive container restarts and reboots without manual intervention. The pattern:
|
||||
|
||||
1. Check if a service already exists before creating: `ls /etc/systemd/system/comfyui.service`
|
||||
2. Stop any manual `nohup` / background processes (they hold ports 8188/8000)
|
||||
3. Create a `.service` file on the PVE host, `pct push` into LXC `/etc/systemd/system/`
|
||||
4. `systemctl daemon-reload && systemctl enable --now <service>`
|
||||
|
||||
### Stopping manual processes before systemd takeover
|
||||
|
||||
If ComfyUI was previously started with `nohup python3 main.py --listen 0.0.0.0 --port 8188 &`, the process survives and keeps port 8188 occupied. `systemctl start comfyui.service` will then fail with:
|
||||
```
|
||||
[ERROR] Port 8188 is already in use on address 0.0.0.0
|
||||
```
|
||||
|
||||
Fix:
|
||||
```bash
|
||||
pct exec 204 -- bash -c "pkill -9 -f 'main.py.*8188'; fuser -k 8000/tcp 2>/dev/null || true"
|
||||
```
|
||||
|
||||
### Example: ComfyUI (may already exist on CT 204)
|
||||
|
||||
```bash
|
||||
# Check existing service first
|
||||
pct exec 204 -- cat /etc/systemd/system/comfyui.service 2>/dev/null || echo "NOT_FOUND"
|
||||
```
|
||||
|
||||
Already installed on Schön Consulting CT 204:
|
||||
```ini
|
||||
# /etc/systemd/system/comfyui.service
|
||||
[Unit]
|
||||
Description=ComfyUI
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/ComfyUI
|
||||
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
Environment=HIP_VISIBLE_DEVICES=0
|
||||
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Example: Ideogram4 OpenAI API Wrapper
|
||||
|
||||
Template in `templates/ideogram4-api.service`:
|
||||
|
||||
```bash
|
||||
# Write on PVE host, then push
|
||||
pct push 204 /tmp/ideogram4-api.service /etc/systemd/system/ideogram4-api.service
|
||||
pct exec 204 -- systemctl daemon-reload
|
||||
pct exec 204 -- systemctl enable --now ideogram4-api.service
|
||||
pct exec 204 -- systemctl status ideogram4-api.service
|
||||
```
|
||||
|
||||
Key points:
|
||||
- `Requires=comfyui.service` so the wrapper only starts when ComfyUI is up
|
||||
- Use full paths to venv Python binary (`/opt/ideogram4-api/venv/bin/python`) so systemd doesn't need PATH activation
|
||||
- `ExecStart` filename must match actual file on disk (`main.py`, `app.py`, or `adapter.py` — check `ls /opt/ideogram4-api/`)
|
||||
- `Restart=always` keeps the wrapper alive across ComfyUI restarts
|
||||
|
||||
## Key takeaways
|
||||
- Use `'%help'` heredoc delimiter on outer SSH to prevent local variable substitution
|
||||
- Use `'PYEOF'` heredoc for inner Python to prevent expansion of Python braces
|
||||
- `pct push` is faster and more reliable than `cat | pct exec tee`
|
||||
- For systemd service files, `pct push` is also the cleanest way to install them
|
||||
- Always verify model integrity after copy: struct-unpack safetensors header + compare data offsets
|
||||
|
||||
## Concrete environment defaults
|
||||
|
||||
For the Schön Consulting Proxmox VE setup:
|
||||
- **PVE host:** `10.0.20.91`, user `root`, password `28acaneltO!#`
|
||||
- **ComfyUI+ROCm LXC:** CTID `204`, IP `10.0.30.97`
|
||||
- **ComfyUI root:** `/opt/ComfyUI`
|
||||
- **API adapter root:** `/opt/ideogram4-api`
|
||||
- **Common model paths:**
|
||||
- `/opt/ComfyUI/models/diffusion_models/`
|
||||
- `/opt/ComfyUI/models/unet/` (FLUX GGUF models)
|
||||
- `/opt/ComfyUI/models/text_encoders/` (also used as `/opt/ComfyUI/models/clip/`)
|
||||
- `/opt/ComfyUI/models/vae/`
|
||||
- `/opt/ComfyUI/models/checkpoints/`
|
||||
|
||||
### Fast probe via sshpass (single-step)
|
||||
When already authenticated, use `sshpass` to skip interactive password entry:
|
||||
```bash
|
||||
export SSHPASS="28acaneltO!#"
|
||||
|
||||
# Check if ComfyUI server is responsive
|
||||
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
|
||||
"pct exec 204 -- bash -c 'curl -s http://localhost:8188/system_stats | head -c 200'"
|
||||
|
||||
# List models in a folder
|
||||
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
|
||||
"pct exec 204 -- ls -la /opt/ComfyUI/models/diffusion_models/"
|
||||
|
||||
# Check a safetensors file without parsing
|
||||
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
|
||||
"pct exec 204 -- python3 -c 'from safetensors import safe_open; f=safe_open(\"/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors\", framework=\"pt\", device=\"cpu\"); print(\"OK, tensors:\", len(list(f.keys())))'"
|
||||
```
|
||||
|
||||
### Pitfall: `pct exec` with inline Python over `sshpass` still breaks
|
||||
Even with sshpass, inline `-c` Python strings fail due to double-shell escaping. The pattern below **WILL NOT WORK**:
|
||||
```bash
|
||||
# FAILS — nested quotes collapse
|
||||
sshpass -e ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'"
|
||||
# → bash: eval: line N: unexpected EOF while looking for matching `"'
|
||||
```
|
||||
|
||||
The **only reliable approach** is `pct push` + `pct exec` (see write-then-execute above).
|
||||
@@ -0,0 +1,312 @@
|
||||
# ComfyUI REST + WebSocket API Reference
|
||||
|
||||
ComfyUI exposes a REST + WebSocket interface for workflow execution and
|
||||
management. **The same surface is used locally and on Comfy Cloud, with
|
||||
auth/path differences.**
|
||||
|
||||
## Connection
|
||||
|
||||
| | Local ComfyUI | Comfy Cloud |
|
||||
|---|---|---|
|
||||
| Base URL | `http://127.0.0.1:8188` | `https://cloud.comfy.org` |
|
||||
| API path prefix | none (`/prompt`, `/view`, …) | `/api/...` (`/api/prompt`, `/api/view`, …) |
|
||||
| Auth | none (or bearer token if configured) | `X-API-Key` header |
|
||||
| WebSocket | `ws://host:port/ws?clientId={uuid}` | `wss://cloud.comfy.org/ws?clientId={uuid}&token={API_KEY}` |
|
||||
| `/api/view` response | direct bytes | 302 redirect → signed URL (use `curl -L`) |
|
||||
|
||||
The skill scripts route URLs automatically via `_common.resolve_url()`.
|
||||
|
||||
## Endpoint differences on Comfy Cloud
|
||||
|
||||
The cloud surface diverges from local ComfyUI in several ways. The skill
|
||||
scripts handle these transparently; document them here so anyone calling
|
||||
`curl` directly knows.
|
||||
|
||||
| Local path | Cloud path | Notes |
|
||||
|------------|-----------|-------|
|
||||
| `/system_stats` | `/api/system_stats` | Cloud version is **public** (no auth required) |
|
||||
| `/object_info` | `/api/object_info` | **Paid tier only** — free returns 403 |
|
||||
| `/queue` | `/api/queue` | Paid tier only |
|
||||
| `/userdata` | `/api/userdata` | Paid tier only |
|
||||
| `/prompt` (POST) | `/api/prompt` (POST) | Paid tier only |
|
||||
| `/upload/image` | `/api/upload/image` | Paid tier only; `subfolder` accepted but ignored |
|
||||
| `/upload/mask` | `/api/upload/mask` | Same as above |
|
||||
| `/view` | `/api/view` | Paid tier only; **returns 302** to signed URL |
|
||||
| `/history` | `/api/history_v2` | **Renamed**; old path returns 404 |
|
||||
| `/history/{id}` | `/api/history_v2/{id}` or `/api/jobs/{id}` | Both work; `/jobs` returns full job |
|
||||
| `/models` | `/api/experiment/models` | **Renamed** |
|
||||
| `/models/{folder}` | `/api/experiment/models/{folder}` | **Renamed**; response shape differs (see below) |
|
||||
|
||||
### Cloud model-list response shape
|
||||
|
||||
- **Local:** `["a.safetensors", "b.safetensors", …]` — flat list of strings.
|
||||
- **Cloud:** `[{"name": "a.safetensors", "pathIndex": 0}, …]` — list of objects.
|
||||
- **Cloud 404 with `code: "folder_not_found"`** — folder is empty or unknown,
|
||||
not an "endpoint missing" error. Distinguish by reading the body.
|
||||
|
||||
The skill helper `_common.parse_model_list()` normalizes both.
|
||||
|
||||
## Workflow Execution
|
||||
|
||||
### Submit Workflow
|
||||
|
||||
```bash
|
||||
# Local
|
||||
curl -X POST "http://127.0.0.1:8188/prompt" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt": '"$(cat workflow_api.json)"', "client_id": "'"$(uuidgen)"'"}'
|
||||
|
||||
# Cloud
|
||||
curl -X POST "https://cloud.comfy.org/api/prompt" \
|
||||
-H "X-API-Key: $COMFY_CLOUD_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt": '"$(cat workflow_api.json)"'}'
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"prompt_id": "abc-123-def", "number": 1, "node_errors": {}}
|
||||
```
|
||||
|
||||
If `node_errors` is non-empty, the workflow has validation errors (missing
|
||||
nodes, bad inputs).
|
||||
|
||||
### Check Job Status (Cloud)
|
||||
|
||||
```bash
|
||||
curl -X GET "https://cloud.comfy.org/api/job/{prompt_id}/status" \
|
||||
-H "X-API-Key: $COMFY_CLOUD_API_KEY"
|
||||
```
|
||||
|
||||
| Status | Description |
|
||||
| ------------- | ---------------------------------- |
|
||||
| `pending` | Job is queued and waiting to start |
|
||||
| `in_progress` | Job is currently executing |
|
||||
| `completed` | Job finished successfully |
|
||||
| `failed` | Job encountered an error |
|
||||
| `cancelled` | Job was cancelled by user |
|
||||
|
||||
### Job detail with outputs (Cloud)
|
||||
|
||||
```bash
|
||||
curl -X GET "https://cloud.comfy.org/api/jobs/{prompt_id}" \
|
||||
-H "X-API-Key: $COMFY_CLOUD_API_KEY"
|
||||
```
|
||||
|
||||
Response includes `outputs` keyed by node ID. Cloud uses `video` (singular)
|
||||
in the output structure; local uses `videos` (plural). The skill scripts
|
||||
accept both.
|
||||
|
||||
### Get History (Local)
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8188/history" # all
|
||||
curl -s "http://127.0.0.1:8188/history/{id}" # one prompt_id
|
||||
```
|
||||
|
||||
Local entry shape:
|
||||
```json
|
||||
{
|
||||
"<prompt_id>": {
|
||||
"prompt": [...],
|
||||
"outputs": {"<node_id>": {"images": [...]}},
|
||||
"status": {
|
||||
"status_str": "success" | "error",
|
||||
"completed": true | false,
|
||||
"messages": [["execution_start", {...}], ["execution_error", {...}], …]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Important:** when reading status, check `status_str == "error"` BEFORE
|
||||
checking `completed`, because both can be true for failed runs.
|
||||
|
||||
### Download Output
|
||||
|
||||
```bash
|
||||
# Local (direct bytes)
|
||||
curl -s "http://127.0.0.1:8188/view?filename=ComfyUI_00001_.png&subfolder=&type=output" \
|
||||
-o output.png
|
||||
|
||||
# Cloud (302 → signed URL; -L follows; STRIP X-API-Key for the second hop)
|
||||
curl -L "https://cloud.comfy.org/api/view?filename=...&type=output" \
|
||||
-H "X-API-Key: $COMFY_CLOUD_API_KEY" \
|
||||
-o output.png
|
||||
```
|
||||
|
||||
The skill's `run_workflow.py` strips `X-API-Key` automatically on the
|
||||
cross-host redirect, so the signed URL never sees your auth.
|
||||
|
||||
## WebSocket Monitoring
|
||||
|
||||
Connect for real-time execution events.
|
||||
|
||||
```bash
|
||||
# Local
|
||||
wscat -c "ws://127.0.0.1:8188/ws?clientId=MY-UUID"
|
||||
|
||||
# Cloud
|
||||
wscat -c "wss://cloud.comfy.org/ws?clientId=MY-UUID&token=$COMFY_CLOUD_API_KEY"
|
||||
```
|
||||
|
||||
**Note:** on Cloud the `clientId` is currently ignored — all messages for a
|
||||
user are broadcast to every connection. Filter messages client-side by
|
||||
`data.prompt_id`.
|
||||
|
||||
### JSON Message Types
|
||||
|
||||
| Type | When | Key Fields |
|
||||
|------|------|------------|
|
||||
| `status` | Queue change | `status.exec_info.queue_remaining` |
|
||||
| `notification` | User-friendly status string | `value` |
|
||||
| `execution_start` | Workflow begins | `prompt_id` |
|
||||
| `executing` | Node running (or end-of-run if `node` is null on local) | `node`, `prompt_id` |
|
||||
| `progress` | Sampling steps | `node`, `value`, `max` |
|
||||
| `progress_state` | Extended progress with per-node metadata | `nodes` (dict) |
|
||||
| `executed` | Node output ready | `node`, `output` (with `images`/`video`/etc.) |
|
||||
| `execution_cached` | Nodes skipped because of cache | `nodes` (list of IDs) |
|
||||
| `execution_success` | All done | `prompt_id` |
|
||||
| `execution_error` | Failure | `exception_type`, `exception_message`, `traceback`, `node_id` |
|
||||
| `execution_interrupted` | Cancelled | `prompt_id` |
|
||||
|
||||
### Binary Frames (Preview Images)
|
||||
|
||||
| Type code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0x00000001` | `PREVIEW_IMAGE` — `[type:4][image_type:4][data]` (image_type 1=JPEG, 2=PNG) |
|
||||
| `0x00000003` | `TEXT` — `[type:4][nid_len:4][nid][text]` (UTF-8) |
|
||||
| `0x00000004` | `PREVIEW_IMAGE_WITH_METADATA` — `[type:4][meta_len:4][json][image_data]` |
|
||||
|
||||
`scripts/ws_monitor.py --previews <dir>` saves preview frames to disk.
|
||||
|
||||
## File Upload
|
||||
|
||||
```bash
|
||||
# Image
|
||||
curl -X POST "http://127.0.0.1:8188/upload/image" \
|
||||
-F "image=@photo.png" -F "type=input" -F "overwrite=true"
|
||||
# Returns: {"name": "photo.png", "subfolder": "", "type": "input"}
|
||||
|
||||
# Mask (linked to a previously uploaded image)
|
||||
curl -X POST "http://127.0.0.1:8188/upload/mask" \
|
||||
-F "image=@mask.png" -F "type=input" \
|
||||
-F 'original_ref={"filename":"photo.png","subfolder":"","type":"input"}'
|
||||
```
|
||||
|
||||
Cloud equivalent: prepend `https://cloud.comfy.org/api` and add `-H "X-API-Key: $COMFY_CLOUD_API_KEY"`.
|
||||
|
||||
## Node & Model Discovery
|
||||
|
||||
```bash
|
||||
# All node types and their input specs
|
||||
curl -s "http://127.0.0.1:8188/object_info" | python3 -m json.tool
|
||||
|
||||
# Specific node
|
||||
curl -s "http://127.0.0.1:8188/object_info/KSampler"
|
||||
|
||||
# Models per folder (local)
|
||||
curl -s "http://127.0.0.1:8188/models/checkpoints"
|
||||
curl -s "http://127.0.0.1:8188/models/loras"
|
||||
|
||||
# Models per folder (cloud — note the experimental prefix)
|
||||
curl -s "https://cloud.comfy.org/api/experiment/models/checkpoints" \
|
||||
-H "X-API-Key: $COMFY_CLOUD_API_KEY"
|
||||
```
|
||||
|
||||
## Queue Management
|
||||
|
||||
```bash
|
||||
# View queue
|
||||
curl -s "http://127.0.0.1:8188/queue"
|
||||
|
||||
# Clear all pending
|
||||
curl -X POST "http://127.0.0.1:8188/queue" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"clear": true}'
|
||||
|
||||
# Delete specific items
|
||||
curl -X POST "http://127.0.0.1:8188/queue" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"delete": ["prompt_id_1", "prompt_id_2"]}'
|
||||
|
||||
# Cancel currently-running job
|
||||
curl -X POST "http://127.0.0.1:8188/interrupt"
|
||||
```
|
||||
|
||||
## System Management
|
||||
|
||||
```bash
|
||||
# Stats (VRAM, RAM, GPU, ComfyUI version)
|
||||
curl -s "http://127.0.0.1:8188/system_stats"
|
||||
|
||||
# Free GPU memory
|
||||
curl -X POST "http://127.0.0.1:8188/free" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"unload_models": true, "free_memory": true}'
|
||||
```
|
||||
|
||||
## ComfyUI-Manager Endpoints (Optional)
|
||||
|
||||
These require ComfyUI-Manager installed. Useful for installing nodes/models
|
||||
via the API instead of `comfy-cli`.
|
||||
|
||||
```bash
|
||||
# Install a custom node from a git URL
|
||||
curl -X POST "http://127.0.0.1:8188/manager/queue/install" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"git_url": "https://github.com/user/comfyui-node.git"}'
|
||||
|
||||
# Check install queue status
|
||||
curl -s "http://127.0.0.1:8188/manager/queue/status"
|
||||
|
||||
# Install model
|
||||
curl -X POST "http://127.0.0.1:8188/manager/queue/install_model" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://...", "path": "models/checkpoints", "filename": "model.safetensors"}'
|
||||
```
|
||||
|
||||
## POST /prompt Payload Format
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": {
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {
|
||||
"seed": 42,
|
||||
"steps": 20,
|
||||
"cfg": 7.5,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "normal",
|
||||
"denoise": 1.0,
|
||||
"model": ["4", 0],
|
||||
"positive": ["6", 0],
|
||||
"negative": ["7", 0],
|
||||
"latent_image": ["5", 0]
|
||||
}
|
||||
}
|
||||
},
|
||||
"client_id": "unique-uuid-for-ws-filtering",
|
||||
"extra_data": {
|
||||
"api_key_comfy_org": "optional-PARTNER-NODE-key (NOT the cloud auth key)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `prompt`: workflow graph in API format
|
||||
- `client_id`: UUID — local server uses it to filter WebSocket events; cloud
|
||||
ignores it.
|
||||
- `extra_data.api_key_comfy_org`: ONLY required when the workflow uses
|
||||
partner nodes (Flux Pro, Ideogram, etc.). Don't conflate with `X-API-Key`.
|
||||
|
||||
## Error Categories (cloud `execution_error` `exception_type`)
|
||||
|
||||
| Type | Meaning |
|
||||
|------|---------|
|
||||
| `ValidationError` | Bad workflow / inputs (often nicer to surface from `node_errors`) |
|
||||
| `ModelDownloadError` | Required model not available |
|
||||
| `ImageDownloadError` | Failed to fetch input image from URL |
|
||||
| `OOMError` | Out of GPU memory |
|
||||
| `InsufficientFundsError` | Account balance too low (partner nodes) |
|
||||
| `InactiveSubscriptionError` | Subscription not active |
|
||||
@@ -0,0 +1,226 @@
|
||||
# ComfyUI Workflow JSON Format
|
||||
|
||||
## Two Formats — Only API Format Is Executable
|
||||
|
||||
**API format** is required for `/api/prompt` and every script in this skill.
|
||||
The web UI also produces an "editor format" used for visual editing, which
|
||||
**cannot** be submitted directly.
|
||||
|
||||
### API Format
|
||||
|
||||
Top-level keys are string node IDs. Each node has `class_type` and `inputs`:
|
||||
|
||||
```json
|
||||
{
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {
|
||||
"seed": 156680208700286,
|
||||
"steps": 20,
|
||||
"cfg": 8,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "normal",
|
||||
"denoise": 1.0,
|
||||
"model": ["4", 0],
|
||||
"positive": ["6", 0],
|
||||
"negative": ["7", 0],
|
||||
"latent_image": ["5", 0]
|
||||
},
|
||||
"_meta": {"title": "KSampler"}
|
||||
},
|
||||
"4": {
|
||||
"class_type": "CheckpointLoaderSimple",
|
||||
"inputs": {"ckpt_name": "v1-5-pruned-emaonly.safetensors"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:** every top-level value has `class_type`. The skill's
|
||||
`_common.is_api_format()` does this check.
|
||||
|
||||
### Editor Format (not directly executable)
|
||||
|
||||
Has `nodes[]` and `links[]` arrays — the visual graph. To convert: open in
|
||||
ComfyUI's web UI and use **Workflow → Export (API)** (newer UI) or the
|
||||
"Save (API Format)" button (older UI).
|
||||
|
||||
**Detection:** top-level has `"nodes"` and `"links"` keys.
|
||||
|
||||
## Inputs: Literals vs Links
|
||||
|
||||
```json
|
||||
"inputs": {
|
||||
"text": "a cat", // literal — modifiable
|
||||
"seed": 42, // literal — modifiable
|
||||
"clip": ["4", 1] // link — wiring; do NOT overwrite
|
||||
}
|
||||
```
|
||||
|
||||
Links are length-2 arrays of `[upstream_node_id, output_slot]`. The skill's
|
||||
parameter injector refuses to overwrite a link with a literal (logs a
|
||||
warning and skips).
|
||||
|
||||
## Common Node Types and Their Controllable Parameters
|
||||
|
||||
The full catalog lives in `scripts/_common.py` (`PARAM_PATTERNS` and
|
||||
`MODEL_LOADERS`). Highlights:
|
||||
|
||||
### Text Prompts
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `CLIPTextEncode` | `text` |
|
||||
| `CLIPTextEncodeSDXL` | `text_g`, `text_l`, `width`, `height` |
|
||||
| `CLIPTextEncodeFlux` | `clip_l`, `t5xxl`, `guidance` |
|
||||
|
||||
To distinguish positive from negative the skill traces `KSampler.negative`
|
||||
back through Reroute / Primitive nodes to the source CLIPTextEncode. Falls
|
||||
back to `_meta.title` heuristics ("negative", "neg", "anti").
|
||||
|
||||
### Sampling
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `KSampler` | `seed`, `steps`, `cfg`, `sampler_name`, `scheduler`, `denoise` |
|
||||
| `KSamplerAdvanced` | `noise_seed`, `steps`, `cfg`, `start_at_step`, `end_at_step` |
|
||||
| `SamplerCustom` | `noise_seed`, `cfg`, `sampler`, `sigmas` |
|
||||
| `SamplerCustomAdvanced` | `noise_seed` (via RandomNoise input) |
|
||||
| `RandomNoise` | `noise_seed` |
|
||||
| `BasicScheduler` | `steps`, `scheduler`, `denoise` |
|
||||
| `KSamplerSelect` | `sampler_name` |
|
||||
| `BasicGuider` / `CFGGuider` | `cfg` |
|
||||
| `ModelSamplingFlux` | `max_shift`, `base_shift`, `width`, `height` |
|
||||
| `SDTurboScheduler` | `steps`, `denoise` |
|
||||
|
||||
### Latent / Dimensions
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `EmptyLatentImage` | `width`, `height`, `batch_size` |
|
||||
| `EmptySD3LatentImage` | `width`, `height`, `batch_size` |
|
||||
| `EmptyHunyuanLatentVideo` | `width`, `height`, `length`, `batch_size` |
|
||||
| `EmptyMochiLatentVideo` | `width`, `height`, `length`, `batch_size` |
|
||||
| `EmptyLTXVLatentVideo` | `width`, `height`, `length`, `batch_size` |
|
||||
|
||||
### Model Loading
|
||||
|
||||
| Node Class | Key Fields | Folder |
|
||||
|------------|------------|--------|
|
||||
| `CheckpointLoaderSimple` | `ckpt_name` | `checkpoints` |
|
||||
| `LoraLoader` | `lora_name`, `strength_model`, `strength_clip` | `loras` |
|
||||
| `LoraLoaderModelOnly` | `lora_name`, `strength_model` | `loras` |
|
||||
| `VAELoader` | `vae_name` | `vae` |
|
||||
| `ControlNetLoader` | `control_net_name` | `controlnet` |
|
||||
| `CLIPLoader` | `clip_name` | `clip` |
|
||||
| `DualCLIPLoader` | `clip_name1`, `clip_name2` | `clip` |
|
||||
| `TripleCLIPLoader` | `clip_name1/2/3` | `clip` |
|
||||
| `UNETLoader` | `unet_name` | `unet` |
|
||||
| `DiffusionModelLoader` | `model_name` | `diffusion_models` |
|
||||
| `UpscaleModelLoader` | `model_name` | `upscale_models` |
|
||||
| `IPAdapterModelLoader` | `ipadapter_file` | `ipadapter` |
|
||||
| `ADE_AnimateDiffLoaderWithContext` | `model_name`, `motion_scale` | `animatediff_models` |
|
||||
|
||||
### Image Input/Output
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `LoadImage` | `image` (server-side filename, after upload) |
|
||||
| `LoadImageMask` | `image`, `channel` (`red` / `green` / `blue` / `alpha`) |
|
||||
| `VAEEncode` / `VAEDecode` | (no controllable fields) |
|
||||
| `VAEEncodeForInpaint` | `grow_mask_by` |
|
||||
| `SaveImage` | `filename_prefix` |
|
||||
| `VHS_VideoCombine` | `frame_rate`, `format`, `filename_prefix`, `loop_count`, `pingpong` |
|
||||
|
||||
### ControlNet
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `ControlNetApply` | `strength` |
|
||||
| `ControlNetApplyAdvanced` | `strength`, `start_percent`, `end_percent` |
|
||||
|
||||
### IPAdapter (community pack `comfyui_ipadapter_plus`)
|
||||
|
||||
| Node Class | Key Fields |
|
||||
|------------|------------|
|
||||
| `IPAdapterAdvanced` | `weight`, `start_at`, `end_at` |
|
||||
| `IPAdapter` | `weight` |
|
||||
|
||||
### Embeddings (referenced inside prompt strings)
|
||||
|
||||
ComfyUI scans prompt text for `embedding:NAME` syntax. The skill's
|
||||
`_common.iter_embedding_refs()` extracts these as model dependencies.
|
||||
|
||||
```text
|
||||
"a beautiful cat, embedding:goodvibes:1.2, embedding:art-style"
|
||||
```
|
||||
|
||||
`extract_schema.py` and `check_deps.py` surface these in
|
||||
`embedding_dependencies` / `missing_embeddings`.
|
||||
|
||||
## Parameter Injection Pattern
|
||||
|
||||
```python
|
||||
import json, copy
|
||||
|
||||
with open("workflow_api.json") as f:
|
||||
workflow = json.load(f)
|
||||
|
||||
wf = copy.deepcopy(workflow)
|
||||
wf["6"]["inputs"]["text"] = "a beautiful sunset"
|
||||
wf["7"]["inputs"]["text"] = "ugly, blurry"
|
||||
wf["3"]["inputs"]["seed"] = 42
|
||||
wf["3"]["inputs"]["steps"] = 30
|
||||
wf["5"]["inputs"]["width"] = 1024
|
||||
wf["5"]["inputs"]["height"] = 1024
|
||||
```
|
||||
|
||||
`scripts/extract_schema.py` automates discovering which node IDs/fields
|
||||
correspond to which user-facing parameters. It returns a `parameters` dict
|
||||
that `run_workflow.py` reads to inject values from `--args`.
|
||||
|
||||
## Identifying Controllable Parameters (Heuristics)
|
||||
|
||||
For unknown workflows:
|
||||
|
||||
1. **Prompt text** — any `CLIPTextEncode.text`. Use connection tracing back
|
||||
from `KSampler.positive` / `.negative` to disambiguate (don't trust
|
||||
meta-title alone).
|
||||
2. **Seed** — `KSampler.seed` / `KSamplerAdvanced.noise_seed` / `RandomNoise.noise_seed`.
|
||||
3. **Dimensions** — `Empty*LatentImage.width/height` (must be multiples of 8).
|
||||
4. **Steps / CFG** — `KSampler.steps`, `KSampler.cfg`. Steps 20–50 typical.
|
||||
CFG 5–15 typical (Flux uses guidance, not CFG).
|
||||
5. **Model / checkpoint** — `CheckpointLoaderSimple.ckpt_name`. Filename must
|
||||
match an installed file *exactly*.
|
||||
6. **LoRA** — `LoraLoader.lora_name`, `.strength_model`.
|
||||
7. **Images for img2img / inpaint** — `LoadImage.image`. Server-side filename
|
||||
after upload.
|
||||
8. **Denoise** — `KSampler.denoise`. 0.0–1.0; 1.0 = ignore input image,
|
||||
0.0 = pass through. Sweet spot for img2img: 0.4–0.7.
|
||||
|
||||
## Output Nodes
|
||||
|
||||
Output is produced by these node types. The skill's `OUTPUT_NODES` set
|
||||
extends to common community packs.
|
||||
|
||||
| Node | Output Key | Content |
|
||||
|------|-----------|---------|
|
||||
| `SaveImage` | `images` | List of `{filename, subfolder, type}` |
|
||||
| `PreviewImage` | `images` | Temporary preview (not saved) |
|
||||
| `VHS_VideoCombine` | `gifs` (older) or `videos`/`video` (newer cloud) | Video file refs |
|
||||
| `SaveAudio` | `audio` | Audio file refs |
|
||||
| `SaveAnimatedWEBP` / `SaveAnimatedPNG` | `images` | Animated images |
|
||||
| `Save3D` | `3d` | 3D asset refs |
|
||||
|
||||
After execution, fetch outputs from `/history/{prompt_id}` (local) or
|
||||
`/api/jobs/{prompt_id}` (cloud) → `outputs` → `{node_id}` → `{key}`.
|
||||
|
||||
## Wrapper Variants
|
||||
|
||||
Some saved JSON files wrap the workflow under a `"prompt"` key (matching
|
||||
the `/api/prompt` payload shape). The skill's `_common.unwrap_workflow()`
|
||||
handles this — pass any of:
|
||||
|
||||
- raw API format: `{"3": {...}, "4": {...}}`
|
||||
- wrapped: `{"prompt": {"3": {...}}, "client_id": "..."}`
|
||||
|
||||
It rejects editor format with a clear error and a re-export instruction.
|
||||
Reference in New Issue
Block a user