# LXC GPU Autosetup — Background Script Pattern

## Problem

Provisioning GPU-accelerated LXC containers involves long-running operations:

1. `apt update && apt install` inside LXC (first setup)
2. `amdgpu-install --usecase=rocm --no-dkms` (30-90 min, 100+ packages)
3. `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2` (~4 GB download, 10-20 min)
4. `pip install -r requirements.txt` for ComfyUI/Ollama/vLLM

Running any of these in the foreground via `pct exec` is brittle:
- SSH timeout kills the session
- `pct exec` double-shell-quoting breaks complex commands
- No way to resume if interrupted

## Solution: Background Script on Host

Write a bash script on the Proxmox **host**, run it via `nohup`, and poll completion markers.

### Step 1: Write the Script

```bash
# /tmp/ctNNN_autosetup.sh — runs ON the Proxmox host, orchestrates INSIDE the LXC
CTID=${1:-204}
LOG=/tmp/ct${CTID}_autosetup.log
DONE=/tmp/ct${CTID}_autosetup_done

echo "$(date): Starting auto-setup for CT ${CTID}..." > "$LOG"

# --- Wait for dpkg lock inside CT ---
while pct exec "$CTID" -- fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do
    echo "$(date): dpkg still locked..." >> "$LOG"
    sleep 30
done
echo "$(date): dpkg free." >> "$LOG"

# --- Install ROCm (inside CT) ---
pct exec "$CTID" -- bash -c '
    export DEBIAN_FRONTEND=noninteractive
    export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
    dpkg --configure -a 2>&1
    apt-get -f install -y -qq 2>&1
    cd /tmp
    wget -q https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb
    dpkg -i amdgpu-install_6.3.60303-1_all.deb 2>&1 | tail -3
    apt-get update -qq
    amdgpu-install --usecase=rocm --no-dkms -y 2>&1 | tail -10
' >> "$LOG" 2>&1

# --- Verify ROCm ---
echo "$(date): Verifying ROCm..." >> "$LOG"
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocminfo 2>&1 | grep -E "Name:" | head -5' >> "$LOG" 2>&1
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocm-smi 2>&1 | head -5' >> "$LOG" 2>&1

# --- PyTorch ROCm ---
echo "$(date): Installing PyTorch ROCm..." >> "$LOG"
pct exec "$CTID" -- bash -c 'pip3 install --break-system-packages torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2 2>&1 | tail -10' >> "$LOG" 2>&1

# --- ComfyUI / App ---
echo "$(date): Installing app..." >> "$LOG"
pct exec "$CTID" -- bash -c '
    cd /opt
    git clone https://github.com/comfyanonymous/ComfyUI.git 2>&1 | tail -3
    cd ComfyUI
    pip3 install --break-system-packages -r requirements.txt 2>&1 | tail -10
' >> "$LOG" 2>&1

# --- systemd service ---
pct exec "$CTID" -- bash -c 'cat > /etc/systemd/system/comfyui.service << "EOF"
[Unit]
Description=ComfyUI
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ComfyUI
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
Restart=always
RestartSec=10
Environment=HIP_VISIBLE_DEVICES=0
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable comfyui
' >> "$LOG" 2>&1

pct exec "$CTID" -- systemctl start comfyui >> "$LOG" 2>&1
sleep 5
pct exec "$CTID" -- ss -tlnp | grep 8188 >> "$LOG" 2>&1

echo "$(date): Setup complete." >> "$LOG"
echo "DONE" >> "$DONE"
```

### Step 2: Launch on Host

```bash
chmod +x /tmp/ct204_autosetup.sh
nohup bash /tmp/ct204_autosetup.sh > /dev/null 2>&1 &
```

### Step 3: Poll from Client

```bash
# Check completion
cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET

# Check progress
tail -20 /tmp/ct204_autosetup.log
```

## Python Paramiko Polling Pattern

When automating from a remote client (not the Proxmox host):

```python
import paramiko, time

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('10.0.20.91', username='root', password='28acaneltO!#', timeout=15)

for attempt in range(20):  # ~10 minutes
    stdin, stdout, stderr = client.exec_command(
        "cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET"
    )
    status = stdout.read().decode().strip()
    if status == "DONE":
        print("Setup complete!")
        break
    
    stdin2, stdout2, stderr2 = client.exec_command(
        "tail -5 /tmp/ct204_autosetup.log"
    )
    print(f"Progress: {stdout2.read().decode().strip()[-200:]}")
    time.sleep(30)

client.close()
```

## PEP 668 Fix for Ubuntu 24.04+ Containers

Ubuntu 24.04 Noble enforces `externally-managed` Python environment. Inside LXC:

```bash
# WRONG — fails with PEP 668
pip3 install torch

# RIGHT — bypass inside disposable container
pip3 install --break-system-packages torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/rocm6.2
```

Alternative: Create a venv inside the container (overkill for single-purpose inference boxes).

## ROCm PATH Gotcha

ROCm 6.3 does NOT add `/opt/rocm/bin` to PATH by default. Every `pct exec` ROCm call needs:

```bash
export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
rocminfo
rocm-smi
```

Or set it globally:

```bash
pct exec 204 -- bash -c 'echo "export PATH=/opt/rocm/bin:\$PATH" > /etc/profile.d/rocm.sh'
```

## Quoting Rule for pct exec

`pct exec` always adds a shell layer. Complex commands should be written to a file inside the container, then executed:

```bash
# BAD — quoting nightmare, easy breakage
pct exec 204 -- python3 -c "import torch; print('HIP:', torch.cuda.is_available())"

# GOOD — write script, execute
pct exec 204 -- bash -c 'cat > /tmp/verify.py << "EOF"
import torch
print("HIP available:", torch.cuda.is_available())
print("Device:", torch.cuda.get_device_name(0))
EOF
python3 /tmp/verify.py'
```

## Monitoring via Cronjob

Set up a cronjob that polls the `DONE` marker and reports status:

```bash
# ~/.hermes/cron/ct204_monitor.sh
if [ -f /tmp/ct204_autosetup_done ]; then
    echo "CT 204 SETUP COMPLETE"
    tail -10 /tmp/ct204_autosetup.log
    pct exec 204 -- python3 -c 'import torch; print(torch.cuda.is_available())'
else
    echo "CT 204 SETUP RUNNING"
    tail -5 /tmp/ct204_autosetup.log
fi
```