hermes-skills/creative/comfyui/references/proxmox-lxc.md

Working with ComfyUI inside a Proxmox LXC over SSH

When ComfyUI is hosted inside a Proxmox CT (e.g. CT 204 on 10.0.30.97) and you need to interact with it from a machine that only has SSH access to the PVE host (10.0.20.91), the most reliable pattern is:

❌ Don't: multi-layer nested quotes

# FAILS — 4-level escaping breaks on !, (, ", etc.
ssh root@pve "pct exec 204 -- python3 -c \"print('hello')\""
ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'

Result: bash: -c: line 34: syntax error near unexpected token ('

✅ Do: write-then-execute

# 1. SSH to PVE host and create a script file there
ssh root@10.0.20.91 << 'HEREDOC'
cat > /tmp/my_script.py << 'PYEOF'
import struct, json, os
path = "/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors"
f = open(path, "rb")
hlen = struct.unpack("<Q", f.read(8))[0]
meta = json.loads(f.read(hlen))
total = 8 + hlen
for k, v in meta.items():
    if isinstance(v, dict) and "data_offsets" in v:
        total += v["data_offsets"][1] - v["data_offsets"][0]
actual = os.path.getsize(path)
print(f"expected={total} actual={actual} ok={total==actual}")
PYEOF
HEREDOC

# 2. Push script into LXC container
ssh root@10.0.20.91 "pct push 204 /tmp/my_script.py /tmp/my_script.py"

# 3. Execute inside container
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/my_script.py"

# 4. Clean up
ssh root@10.0.20.91 "rm /tmp/my_script.py"

Template: compact one-liner for checks

ssh root@10.0.20.91 'bash -s' << 'SSHSCRIPT'
cat > /tmp/probe.py << 'PYEOF'
from safetensors import safe_open
f = safe_open("/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors",
              framework="pt", device="cpu")
print("OK, tensors:", len(list(f.keys())))
PYEOF
pct push 204 /tmp/probe.py /tmp/probe.py
pct exec 204 -- python3 /tmp/probe.py
rm /tmp/probe.py
SSHSCRIPT

Copying large files into LXC (e.g. model downloads)

# Download to PVE host tmp
ssh root@10.0.20.91 "cd /tmp && curl -L -o model.safetensors <URL>"

# Push into container
ssh root@10.0.20.91 "pct push 204 /tmp/model.safetensors /opt/ComfyUI/models/text_encoders/model.safetensors"

# Verify
ssh root@10.0.20.91 "pct exec 204 -- python3 /tmp/probe.py"

# Clean up host
ssh root@10.0.20.91 "rm /tmp/model.safetensors"

Persistent services inside LXC

ComfyUI and adapters should run as systemd services so they survive container restarts and reboots without manual intervention. The pattern:

1. Check if a service already exists before creating: ls /etc/systemd/system/comfyui.service
2. Stop any manual nohup / background processes (they hold ports 8188/8000)
3. Create a .service file on the PVE host, pct push into LXC /etc/systemd/system/
4. systemctl daemon-reload && systemctl enable --now <service>

Stopping manual processes before systemd takeover

If ComfyUI was previously started with nohup python3 main.py --listen 0.0.0.0 --port 8188 &, the process survives and keeps port 8188 occupied. systemctl start comfyui.service will then fail with:

[ERROR] Port 8188 is already in use on address 0.0.0.0

Fix:

pct exec 204 -- bash -c "pkill -9 -f 'main.py.*8188'; fuser -k 8000/tcp 2>/dev/null || true"

Example: ComfyUI (may already exist on CT 204)

# Check existing service first
pct exec 204 -- cat /etc/systemd/system/comfyui.service 2>/dev/null || echo "NOT_FOUND"

Already installed on Schön Consulting CT 204:

# /etc/systemd/system/comfyui.service
[Unit]
Description=ComfyUI
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ComfyUI
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
Restart=always
RestartSec=10
Environment=HIP_VISIBLE_DEVICES=0
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
Environment=LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/torch/lib:/opt/rocm/lib

[Install]
WantedBy=multi-user.target

Example: Ideogram4 OpenAI API Wrapper

Template in templates/ideogram4-api.service:

# Write on PVE host, then push
pct push 204 /tmp/ideogram4-api.service /etc/systemd/system/ideogram4-api.service
pct exec 204 -- systemctl daemon-reload
pct exec 204 -- systemctl enable --now ideogram4-api.service
pct exec 204 -- systemctl status ideogram4-api.service

Key points:

• Requires=comfyui.service so the wrapper only starts when ComfyUI is up
• Use full paths to venv Python binary (/opt/ideogram4-api/venv/bin/python) so systemd doesn't need PATH activation
• ExecStart filename must match actual file on disk (main.py, app.py, or adapter.py — check ls /opt/ideogram4-api/)
• Restart=always keeps the wrapper alive across ComfyUI restarts

Key takeaways

• Use '%help' heredoc delimiter on outer SSH to prevent local variable substitution
• Use 'PYEOF' heredoc for inner Python to prevent expansion of Python braces
• pct push is faster and more reliable than cat | pct exec tee
• For systemd service files, pct push is also the cleanest way to install them
• Always verify model integrity after copy: struct-unpack safetensors header + compare data offsets

Concrete environment defaults

For the Schön Consulting Proxmox VE setup:

• PVE host: 10.0.20.91, user root, password 28acaneltO!#
• ComfyUI+ROCm LXC: CTID 204, IP 10.0.30.97
• ComfyUI root: /opt/ComfyUI
• API adapter root: /opt/ideogram4-api
• Common model paths:
  • /opt/ComfyUI/models/diffusion_models/
  • /opt/ComfyUI/models/unet/ (FLUX GGUF models)
  • /opt/ComfyUI/models/text_encoders/ (also used as /opt/ComfyUI/models/clip/)
  • /opt/ComfyUI/models/vae/
  • /opt/ComfyUI/models/checkpoints/

Fast probe via sshpass (single-step)

When already authenticated, use sshpass to skip interactive password entry:

export SSHPASS="28acaneltO!#"

# Check if ComfyUI server is responsive
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- bash -c 'curl -s http://localhost:8188/system_stats | head -c 200'"

# List models in a folder
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- ls -la /opt/ComfyUI/models/diffusion_models/"

# Check a safetensors file without parsing
sshpass -e ssh -o StrictHostKeyChecking=no root@10.0.20.91 \
  "pct exec 204 -- python3 -c 'from safetensors import safe_open; f=safe_open(\"/opt/ComfyUI/models/text_encoders/qwen3vl_8b_fp8_scaled.safetensors\", framework=\"pt\", device=\"cpu\"); print(\"OK, tensors:\", len(list(f.keys())))'"

Pitfall: pct exec with inline Python over sshpass still breaks

Even with sshpass, inline -c Python strings fail due to double-shell escaping. The pattern below WILL NOT WORK:

# FAILS — nested quotes collapse
sshpass -e ssh root@pve "pct exec 204 -- python3 -c 'import struct; ...'"
# → bash: eval: line N: unexpected EOF while looking for matching `"'

The only reliable approach is pct push + pct exec (see write-then-execute above).