# Working with ComfyUI inside a Proxmox LXC over SSH

When ComfyUI is hosted inside a Proxmox CT (e.g. CT 204 on 10.0.30.97) and you
need to interact with it from a machine that only has SSH access to the PVE host
(10.0.20.91), the most reliable pattern is:

## ❌ Don't: multi-layer nested quotes

```bash
# FAILS — 4-level escaping breaks on !, (, ", etc.
ssh root@pve "pct exec 204 -- python3 -c \"print('hello')\""
ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'
```

Result: `bash: -c: line 34: syntax error near unexpected token ('`

## ✅ Do: write-then-execute

```bash
# 1. SSH to PVE host and create a script file there
ssh root@10.0.20.91 << 'HEREDOC'
cat > /tmp/my_script.py << 'PYEOF'
import struct, json, os
path = "/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors"
f = open(path, "rb")
hlen = struct.unpack("<Q", f.read(8))[0]
meta = json.loads(f.read(hlen))
total = 8 + hlen
for k, v in meta.items():
    if isinstance(v, dict) and "data_offsets" in v:
        total += v["data_offsets"][1] - v["data_offsets"][0]
actual = os.path.getsize(path)
print(f"expected={total} actual={actual} ok={total==actual}")
PYEOF
HEREDOC

# 2. Push script into LXC container
ssh root@10.0.20.91 "pct push 204 /tmp/my_script.py /tmp/my_script.py"

# 3. Execute inside container
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/my_script.py"

# 4. Clean up
ssh root@10.0.20.91 "rm /tmp/my_script.py"
```

## Template: compact one-liner for checks

```bash
ssh root@10.0.20.91 'bash -s' << 'SSHSCRIPT'
cat > /tmp/probe.py << 'PYEOF'
from safetensors import safe_open
f = safe_open("/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors",
              framework="pt", device="cpu")
print("OK, tensors:", len(list(f.keys())))
PYEOF
pct push 204 /tmp/probe.py /tmp/probe.py
pct exec 204 -- python3 /tmp/probe.py
rm /tmp/probe.py
SSHSCRIPT
```

## Copying large files into LXC (e.g. model downloads)

```bash
# Download to PVE host tmp
ssh root@10.0.20.91 "cd /tmp && curl -L -o model.safetensors <URL>"

# Push into container
ssh root@10.0.20.91 "pct push 204 /tmp/model.safetensors /opt/ComfyUI/models/text_encoders/model.safetensors"

# Verify
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/probe.py"

# Clean up host
ssh root@10.0.20.91 "rm /tmp/model.safetensors"
```

## Persistent services inside LXC

ComfyUI and adapters should run as systemd services so they survive container restarts and reboots without manual intervention. The pattern:

1. Check if a service already exists before creating: `ls /etc/systemd/system/comfyui.service`
2. Stop any manual `nohup` / background processes (they hold ports 8188/8000)
3. Create a `.service` file on the PVE host, `pct push` into LXC `/etc/systemd/system/`
4. `systemctl daemon-reload && systemctl enable --now <service>`

### Stopping manual processes before systemd takeover

If ComfyUI was previously started with `nohup python3 main.py --listen 0.0.0.0 --port 8188 &`, the process survives and keeps port 8188 occupied. `systemctl start comfyui.service` will then fail with:
```
[ERROR] Port 8188 is already in use on address 0.0.0.0
```

Fix:
```bash
pct exec 204 -- bash -c "pkill -9 -f 'main.py.*8188'; fuser -k 8000/tcp 2>/dev/null || true"
```

### Example: ComfyUI (may already exist on CT 204)

```bash
# Check existing service first
pct exec 204 -- cat /etc/systemd/system/comfyui.service 2>/dev/null || echo "NOT_FOUND"
```

Already installed on Schön Consulting CT 204:
```ini
# /etc/systemd/system/comfyui.service
[Unit]
Description=ComfyUI
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ComfyUI
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
Restart=always
RestartSec=10
Environment=HIP_VISIBLE_DEVICES=0
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib

[Install]
WantedBy=multi-user.target
```

### Example: Ideogram4 OpenAI API Wrapper

Template in `templates/ideogram4-api.service`:

```bash
# Write on PVE host, then push
pct push 204 /tmp/ideogram4-api.service /etc/systemd/system/ideogram4-api.service
pct exec 204 -- systemctl daemon-reload
pct exec 204 -- systemctl enable --now ideogram4-api.service
pct exec 204 -- systemctl status ideogram4-api.service
```

Key points:
- `Requires=comfyui.service` so the wrapper only starts when ComfyUI is up
- Use full paths to venv Python binary (`/opt/ideogram4-api/venv/bin/python`) so systemd doesn't need PATH activation
- `ExecStart` filename must match actual file on disk (`main.py`, `app.py`, or `adapter.py` — check `ls /opt/ideogram4-api/`)
- `Restart=always` keeps the wrapper alive across ComfyUI restarts

## Key takeaways
- Use `'%help'` heredoc delimiter on outer SSH to prevent local variable substitution
- Use `'PYEOF'` heredoc for inner Python to prevent expansion of Python braces
- `pct push` is faster and more reliable than `cat | pct exec tee`
- For systemd service files, `pct push` is also the cleanest way to install them
- Always verify model integrity after copy: struct-unpack safetensors header + compare data offsets

## Concrete environment defaults

For the Schön Consulting Proxmox VE setup:
- **PVE host:** `10.0.20.91`, user `root`, password `28acaneltO!#`
- **ComfyUI+ROCm LXC:** CTID `204`, IP `10.0.30.97`
- **ComfyUI root:** `/opt/ComfyUI`
- **API adapter root:** `/opt/ideogram4-api`
- **Common model paths:**
  - `/opt/ComfyUI/models/diffusion_models/`
  - `/opt/ComfyUI/models/unet/` (FLUX GGUF models)
  - `/opt/ComfyUI/models/text_encoders/` (also used as `/opt/ComfyUI/models/clip/`)
  - `/opt/ComfyUI/models/vae/`
  - `/opt/ComfyUI/models/checkpoints/`

### Fast probe via sshpass (single-step)
When already authenticated, use `sshpass` to skip interactive password entry:
```bash
export SSHPASS="28acaneltO!#"

# Check if ComfyUI server is responsive
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- bash -c 'curl -s http://localhost:8188/system_stats | head -c 200'"

# List models in a folder
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- ls -la /opt/ComfyUI/models/diffusion_models/"

# Check a safetensors file without parsing
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- python3 -c 'from safetensors import safe_open; f=safe_open(\"/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors\", framework=\"pt\", device=\"cpu\"); print(\"OK, tensors:\", len(list(f.keys())))'"
```

### Pitfall: `pct exec` with inline Python over `sshpass` still breaks
Even with sshpass, inline `-c` Python strings fail due to double-shell escaping. The pattern below **WILL NOT WORK**:
```bash
# FAILS — nested quotes collapse
sshpass -e ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'"
# → bash: eval: line N: unexpected EOF while looking for matching `"'
```

The **only reliable approach** is `pct push` + `pct exec` (see write-then-execute above).