180 lines
11 KiB
Markdown
180 lines
11 KiB
Markdown
# Ideogram 4 in ComfyUI
|
|
|
|
Two distinct execution paths exist. Pick the right one based on auth and hardware.
|
|
|
|
## Path A: Cloud API (`IdeogramV4` node)
|
|
|
|
- **Node:** `IdeogramV4` (built-in, from `comfy_api_nodes.nodes_ideogram`)
|
|
- **Inputs:** `prompt` (STRING), `resolution` (COMBO), `rendering_speed` (COMBO), `seed` (INT)
|
|
- **Hidden inputs:** `auth_token_comfy_org`, `api_key_comfy_org`
|
|
- **Requires:** comfy.org account + API key (set via auth_token or api_key_comfy_org)
|
|
|
|
### Common error
|
|
```
|
|
Unauthorized: Please login first to use this node.
|
|
```
|
|
→ Set the `api_key_comfy_org` hidden input or login in the ComfyUI web UI.
|
|
|
|
## Path B: Local inference (`Ideogram4Scheduler` node)
|
|
|
|
- **Node:** `Ideogram4Scheduler` (built-in, `comfy_extras.nodes_ideogram4`)
|
|
- **Outputs:** `SIGMAS` — feed into `SamplerCustomAdvanced`
|
|
- **Workflow structure:** `Ideogram4Scheduler` → `SamplerCustomAdvanced` → `VAEDecode` → `SaveImage`
|
|
|
|
### Why you can't just "use the blueprint directly"
|
|
|
|
The official blueprint at `blueprints/Text to Image (Ideogram v4).json` is in **editor format** (`"nodes"` / `"links"` arrays with a top-level `definitions.subgraphs` array). **The ComfyUI REST API (`POST /api/prompt`) does NOT accept editor format.** Submitting it produces HTTP 400 or 500 with no useful message. You must either:
|
|
|
|
1. Load the blueprint in the ComfyUI web UI and re-export it as **API format** (Workflow → Export API), OR
|
|
2. Manually construct an API-format workflow using the verified node wiring below.
|
|
|
|
### Critical: Ideogram 4 needs asymmetric CFG wiring
|
|
|
|
Unlike standard pipelines that use `BasicGuider`, Ideogram 4 requires `CFGOverride` + `DualModelGuider` for asymmetric classifier-free guidance:
|
|
|
|
- `CFGOverride` sits on the **main model** to override the CFG value in a specific timestep range
|
|
- `DualModelGuider` takes the CFG-overridden main model, the positive conditioning, a **separate unconditional model** (via `model_negative`), and the zeroed-out conditioning (via `ConditioningZeroOut`)
|
|
- The guider output (`GUIDER`) feeds into `SamplerCustomAdvanced`
|
|
|
|
**Do NOT use:** plain `BasicGuider` (only supports single model + pos/neg), standard `KSampler` (doesn't use `Ideogram4Scheduler` sigmas), or `model_1` as the DualModelGuider input name (it's `model_negative`).
|
|
|
|
### Node input names for API-format workflows (verified against `/object_info`)
|
|
|
|
| Node | Required inputs | Optional inputs | Notes |
|
|
|------|-----------------|-----------------|-------|
|
|
| `Ideogram4Scheduler` | `steps`, `width`, `height`, `mu`, `std` | — | Outputs `SIGMAS` for `SamplerCustomAdvanced` |
|
|
| `CFGOverride` | `model`, `cfg`, `start_percent`, `end_percent` | — | Place on main UNet between `UNETLoader` and `DualModelGuider` |
|
|
| `DualModelGuider` | `model`, `positive`, `cfg` | `model_negative` (MODEL), `negative` (CONDITIONING) | `model_negative` is the unconditional UNet (NOT `model_1`) |
|
|
| `CLIPLoader` | `clip_name` | `type` (use `"ideogram4"`), `device` | Must use `type: "ideogram4"` for Qwen3VL text encoder |
|
|
| `EmptyFlux2LatentImage` | `width`, `height`, `batch_size` | — | 128-channel latent (NOT standard `EmptyLatentImage`) |
|
|
| `SamplerCustomAdvanced` | `noise`, `guider`, `sampler`, `sigmas`, `latent_image` | — | `guider` comes from `DualModelGuider`, `sigmas` from `Ideogram4Scheduler` |
|
|
|
|
### Required models
|
|
|
|
| Component | Filename | Path | Size | Source |
|
|
|-----------|----------|------|------|--------|
|
|
| UNet (main) | `ideogram4_fp8_scaled.safetensors` | `models/diffusion_models/` | ~8.6 GB | `Comfy-Org/Ideogram-4` |
|
|
| UNet (unconditional) | `ideogram4_unconditional_fp8_scaled.safetensors` | `models/diffusion_models/` | ~8.6 GB | `Comfy-Org/Ideogram-4` |
|
|
| VAE | `flux2-vae.safetensors` | `models/vae/` | ~320 MB | `Comfy-Org/flux2-dev` |
|
|
| Text encoder (A) | `qwen3vl_8b_fp8_scaled.safetensors` | `models/text_encoders/` | ~9.8 GB | `Comfy-Org/Qwen3-VL` |
|
|
| Text encoder (B) | `gemma4_e4b_it_fp8_scaled.safetensors` | `models/text_encoders/` | ~? GB | `Comfy-Org/gemma-4` |
|
|
|
|
### Known corruption: `qwen3vl_8b_fp8_scaled.safetensors`
|
|
|
|
A truncated download of this ~9.8 GB file produces:
|
|
```
|
|
Error while deserializing header: incomplete metadata, file not fully covered
|
|
```
|
|
at ComfyUI's `CLIPLoader` (node `load_clip` → `comfy.utils.load_torch_file`).
|
|
|
|
**Quick integrity check** (run inside the ComfyUI container or on the host):
|
|
```bash
|
|
python3 -c "
|
|
import struct, json, os
|
|
p = 'models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors'
|
|
f = open(p, 'rb'); hl = struct.unpack('<Q', f.read(8))[0]
|
|
m = json.loads(f.read(hl)); t = 8 + hl
|
|
for k,v in m.items():
|
|
if isinstance(v, dict) and 'data_offsets' in v:
|
|
t += v['data_offsets'][1] - v['data_offsets'][0]
|
|
a = os.path.getsize(p); print(f'expected={t} actual={a} ok={t==a}')
|
|
"
|
|
```
|
|
If `ok=False`, the download is incomplete. Re-download via:
|
|
```bash
|
|
curl -L -o models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors \
|
|
"https://huggingface.co/Comfy-Org/Qwen3-VL/resolve/main/text_encoders/qwen3vl_8b_fp8_scaled.safetensors"
|
|
```
|
|
|
|
**Alternative:** `safetensors.safe_open` also catches corruption early:
|
|
```python
|
|
from safetensors import safe_open
|
|
f = safe_open("qwen3vl_8b_fp8_scaled.safetensors", framework="pt", device="cpu")
|
|
print(len(f.keys())) # Should be ~1254; crashes on truncated files
|
|
```
|
|
|
|
### Pitfall: wrong input name for DualModelGuider
|
|
|
|
The Blueprint user's `Subgraphs`-based `DualModelGuider` uses the internal **editor-format** link name `model_1`, but the **API-format** input name is **`model_negative`**:
|
|
|
|
```
|
|
# WRONG (editor convention leaked into API format):
|
|
"inputs": {"model": ["X", 0], "model_1": ["Y", 0], ...}
|
|
|
|
# RIGHT (API format):
|
|
"inputs": {"model": ["X", 0], "model_negative": ["Y", 0], ...}
|
|
```
|
|
|
|
Error if wrong: `TypeError: DualModelGuider.execute() got an unexpected keyword argument 'model_1'`
|
|
|
|
### Full local workflow blueprint (extracted from ComfyUI `/history`)
|
|
|
|
The official blueprint at `blueprints/Text to Image (Ideogram v4).json` is in **editor format** (`"nodes"` / `"links"` arrays). It must be converted to **API format** (`"class_type"` per node) before execution. Below is a verified API-format workflow:
|
|
|
|
```json
|
|
{
|
|
"37": {
|
|
"inputs": {"aspect_ratio": "1:1 (Square)", "megapixels": 1.0, "multiple": 8},
|
|
"class_type": "ResolutionSelector"
|
|
},
|
|
"98:9": {"inputs": {"vae_name": "flux2-vae.safetensors"}, "class_type": "VAELoader"},
|
|
"98:11": {"inputs": {"width": ["98:31", 1], "height": ["98:32", 1], "batch_size": 1}, "class_type": "EmptyFlux2LatentImage"},
|
|
"98:12": {"inputs": {"noise": ["98:18", 0], "guider": ["98:155", 0], "sampler": ["98:16", 0], "sigmas": ["98:17", 0], "latent_image": ["98:11", 0]}, "class_type": "SamplerCustomAdvanced"},
|
|
"98:13": {"inputs": {"samples": ["98:12", 0], "vae": ["98:9", 0]}, "class_type": "VAEDecode"},
|
|
"98:14": {"inputs": {"clip_name": "qwen3vl_8b_fp8_scaled.safetensors", "type": "ideogram4", "device": "default"}, "class_type": "CLIPLoader"},
|
|
"98:16": {"inputs": {"sampler_name": "euler"}, "class_type": "KSamplerSelect"},
|
|
"98:17": {"inputs": {"steps": ["98:151", 1], "width": ["98:31", 1], "height": ["98:32", 1], "mu": ["98:144", 0], "std": ["98:146", 0]}, "class_type": "Ideogram4Scheduler"},
|
|
"98:18": {"inputs": {"noise_seed": 42}, "class_type": "RandomNoise"},
|
|
"98:23": {"inputs": {"unet_name": "ideogram4_fp8_scaled.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
|
|
"98:24": {"inputs": {"text": "PROMPT_HERE", "clip": ["98:14", 0]}, "class_type": "CLIPTextEncode"},
|
|
"98:10": {"inputs": {"conditioning": ["98:24", 0]}, "class_type": "ConditioningZeroOut"},
|
|
"98:31": {"inputs": {"expression": "max(((a + 15) // 16) * 16, 256)", "values.a": ["98:27", 0]}, "class_type": "ComfyMathExpression"},
|
|
"98:32": {"inputs": {"expression": "max(((a + 15) // 16) * 16, 256)", "values.a": ["98:28", 0]}, "class_type": "ComfyMathExpression"},
|
|
"98:27": {"inputs": {"value": ["37", 0]}, "class_type": "PrimitiveInt"},
|
|
"98:28": {"inputs": {"value": ["37", 1]}, "class_type": "PrimitiveInt"},
|
|
"98:144": {"inputs": {"value": ["98:145", 0]}, "class_type": "ComfyNumberConvert"},
|
|
"98:145": {"inputs": {"json_string": ["98:148", 0], "key": "mu"}, "class_type": "JsonExtractString"},
|
|
"98:146": {"inputs": {"value": ["98:150", 0]}, "class_type": "ComfyNumberConvert"},
|
|
"98:147": {"inputs": {"json_string": "{\\\"Quality\\\":{\\\"num_steps\\\":48,\\\"mu\\\":0.0,\\\"std\\\":1.5,\\\"preset\\\":\\\"Quality\\\"}}", "key": "Quality"}, "class_type": "JsonExtractString"},
|
|
"98:148": {"inputs": {"string": ["98:147", 0], "find": "'", "replace": "\\""}, "class_type": "StringReplace"},
|
|
"98:149": {"inputs": {"json_string": ["98:148", 0], "key": "num_steps"}, "class_type": "JsonExtractString"},
|
|
"98:150": {"inputs": {"json_string": ["98:148", 0], "key": "std"}, "class_type": "JsonExtractString"},
|
|
"98:151": {"inputs": {"value": ["98:149", 0]}, "class_type": "ComfyNumberConvert"},
|
|
"98:154": {"inputs": {"unet_name": "ideogram4_unconditional_fp8_scaled.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
|
|
"98:155": {"inputs": {"cfg": 7.0, "model": ["98:157", 0], "positive": ["98:24", 0], "model_negative": ["98:154", 0], "negative": ["98:10", 0]}, "class_type": "DualModelGuider"},
|
|
"98:156": {"inputs": {"choice": "Quality", "index": 1, "option1": "Quality", "option2": "Default", "option3": "Turbo", "option4": ""}, "class_type": "CustomCombo"},
|
|
"98:157": {"inputs": {"cfg": 3.0, "start_percent": 0.7, "end_percent": 1.0, "model": ["98:23", 0]}, "class_type": "CFGOverride"},
|
|
"158": {"inputs": {"filename_prefix": "Ideogram_4.0", "images": ["98:13", 0]}, "class_type": "SaveImage"}
|
|
}
|
|
```
|
|
|
|
Replace `"PROMPT_HERE"` (node `98:24`, field `text`) with your prompt. Ideogram 4 accepts either plain text or structured JSON prompts.
|
|
|
|
### Accessing a ComfyUI LXC container via Proxmox VE
|
|
|
|
If ComfyUI runs inside a Proxmox LXC container without direct SSH:
|
|
|
|
```bash
|
|
# Proxmox node shell → run inside container
|
|
pct exec CTID -- bash -c "COMMAND"
|
|
|
|
# Push a file into the container (avoid nested quote hell)
|
|
pct push CTID /local/script.py /tmp/script.py
|
|
pct exec CTID -- python3 /tmp/script.py
|
|
|
|
# Common CTID in this environment: 204 (ComfyUI+FLUX.2/ROCm)
|
|
```
|
|
|
|
### Key differences from FLUX.1
|
|
|
|
- Text encoder: `qwen3vl_8b_fp8_scaled` or `gemma4_e4b_it_fp8_scaled` — NOT CLIP-L + T5XXL
|
|
- Latent: `EmptyFlux2LatentImage` (128 channels)
|
|
- VAE: `flux2-vae.safetensors` — NOT `ae.safetensors`
|
|
- CFG: Uses asymmetric CFG (unconditional conditioning is zeroed out via `ConditioningZeroOut`, not a separate negative prompt string)
|
|
|
|
## Quick verification checklist
|
|
|
|
1. `curl $HOST:8188/models/diffusion_models | grep ideogram4` — models present?
|
|
2. Check file integrity of `qwen3vl_8b_fp8_scaled.safetensors` with the Python check above.
|
|
3. For cloud: ensure `api_key_comfy_org` is set or the user is logged into comfy.org via the web UI.
|
|
4. For local: verify the blueprint was exported as **API format**, not editor format (which has top-level `"nodes"` / `"links"` arrays). |