209 lines
6.0 KiB
Markdown
209 lines
6.0 KiB
Markdown
# LXC GPU Autosetup — Background Script Pattern
|
|
|
|
## Problem
|
|
|
|
Provisioning GPU-accelerated LXC containers involves long-running operations:
|
|
|
|
1. `apt update && apt install` inside LXC (first setup)
|
|
2. `amdgpu-install --usecase=rocm --no-dkms` (30-90 min, 100+ packages)
|
|
3. `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2` (~4 GB download, 10-20 min)
|
|
4. `pip install -r requirements.txt` for ComfyUI/Ollama/vLLM
|
|
|
|
Running any of these in the foreground via `pct exec` is brittle:
|
|
- SSH timeout kills the session
|
|
- `pct exec` double-shell-quoting breaks complex commands
|
|
- No way to resume if interrupted
|
|
|
|
## Solution: Background Script on Host
|
|
|
|
Write a bash script on the Proxmox **host**, run it via `nohup`, and poll completion markers.
|
|
|
|
### Step 1: Write the Script
|
|
|
|
```bash
|
|
# /tmp/ctNNN_autosetup.sh — runs ON the Proxmox host, orchestrates INSIDE the LXC
|
|
CTID=${1:-204}
|
|
LOG=/tmp/ct${CTID}_autosetup.log
|
|
DONE=/tmp/ct${CTID}_autosetup_done
|
|
|
|
echo "$(date): Starting auto-setup for CT ${CTID}..." > "$LOG"
|
|
|
|
# --- Wait for dpkg lock inside CT ---
|
|
while pct exec "$CTID" -- fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do
|
|
echo "$(date): dpkg still locked..." >> "$LOG"
|
|
sleep 30
|
|
done
|
|
echo "$(date): dpkg free." >> "$LOG"
|
|
|
|
# --- Install ROCm (inside CT) ---
|
|
pct exec "$CTID" -- bash -c '
|
|
export DEBIAN_FRONTEND=noninteractive
|
|
export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
|
|
dpkg --configure -a 2>&1
|
|
apt-get -f install -y -qq 2>&1
|
|
cd /tmp
|
|
wget -q https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb
|
|
dpkg -i amdgpu-install_6.3.60303-1_all.deb 2>&1 | tail -3
|
|
apt-get update -qq
|
|
amdgpu-install --usecase=rocm --no-dkms -y 2>&1 | tail -10
|
|
' >> "$LOG" 2>&1
|
|
|
|
# --- Verify ROCm ---
|
|
echo "$(date): Verifying ROCm..." >> "$LOG"
|
|
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocminfo 2>&1 | grep -E "Name:" | head -5' >> "$LOG" 2>&1
|
|
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocm-smi 2>&1 | head -5' >> "$LOG" 2>&1
|
|
|
|
# --- PyTorch ROCm ---
|
|
echo "$(date): Installing PyTorch ROCm..." >> "$LOG"
|
|
pct exec "$CTID" -- bash -c 'pip3 install --break-system-packages torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2 2>&1 | tail -10' >> "$LOG" 2>&1
|
|
|
|
# --- ComfyUI / App ---
|
|
echo "$(date): Installing app..." >> "$LOG"
|
|
pct exec "$CTID" -- bash -c '
|
|
cd /opt
|
|
git clone https://github.com/comfyanonymous/ComfyUI.git 2>&1 | tail -3
|
|
cd ComfyUI
|
|
pip3 install --break-system-packages -r requirements.txt 2>&1 | tail -10
|
|
' >> "$LOG" 2>&1
|
|
|
|
# --- systemd service ---
|
|
pct exec "$CTID" -- bash -c 'cat > /etc/systemd/system/comfyui.service << "EOF"
|
|
[Unit]
|
|
Description=ComfyUI
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=root
|
|
WorkingDirectory=/opt/ComfyUI
|
|
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
|
|
Restart=always
|
|
RestartSec=10
|
|
Environment=HIP_VISIBLE_DEVICES=0
|
|
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
EOF
|
|
systemctl daemon-reload
|
|
systemctl enable comfyui
|
|
' >> "$LOG" 2>&1
|
|
|
|
pct exec "$CTID" -- systemctl start comfyui >> "$LOG" 2>&1
|
|
sleep 5
|
|
pct exec "$CTID" -- ss -tlnp | grep 8188 >> "$LOG" 2>&1
|
|
|
|
echo "$(date): Setup complete." >> "$LOG"
|
|
echo "DONE" >> "$DONE"
|
|
```
|
|
|
|
### Step 2: Launch on Host
|
|
|
|
```bash
|
|
chmod +x /tmp/ct204_autosetup.sh
|
|
nohup bash /tmp/ct204_autosetup.sh > /dev/null 2>&1 &
|
|
```
|
|
|
|
### Step 3: Poll from Client
|
|
|
|
```bash
|
|
# Check completion
|
|
cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET
|
|
|
|
# Check progress
|
|
tail -20 /tmp/ct204_autosetup.log
|
|
```
|
|
|
|
## Python Paramiko Polling Pattern
|
|
|
|
When automating from a remote client (not the Proxmox host):
|
|
|
|
```python
|
|
import paramiko, time
|
|
|
|
client = paramiko.SSHClient()
|
|
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
|
client.connect('10.0.20.91', username='root', password='28acaneltO!#', timeout=15)
|
|
|
|
for attempt in range(20): # ~10 minutes
|
|
stdin, stdout, stderr = client.exec_command(
|
|
"cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET"
|
|
)
|
|
status = stdout.read().decode().strip()
|
|
if status == "DONE":
|
|
print("Setup complete!")
|
|
break
|
|
|
|
stdin2, stdout2, stderr2 = client.exec_command(
|
|
"tail -5 /tmp/ct204_autosetup.log"
|
|
)
|
|
print(f"Progress: {stdout2.read().decode().strip()[-200:]}")
|
|
time.sleep(30)
|
|
|
|
client.close()
|
|
```
|
|
|
|
## PEP 668 Fix for Ubuntu 24.04+ Containers
|
|
|
|
Ubuntu 24.04 Noble enforces `externally-managed` Python environment. Inside LXC:
|
|
|
|
```bash
|
|
# WRONG — fails with PEP 668
|
|
pip3 install torch
|
|
|
|
# RIGHT — bypass inside disposable container
|
|
pip3 install --break-system-packages torch torchvision torchaudio \
|
|
--index-url https://download.pytorch.org/whl/rocm6.2
|
|
```
|
|
|
|
Alternative: Create a venv inside the container (overkill for single-purpose inference boxes).
|
|
|
|
## ROCm PATH Gotcha
|
|
|
|
ROCm 6.3 does NOT add `/opt/rocm/bin` to PATH by default. Every `pct exec` ROCm call needs:
|
|
|
|
```bash
|
|
export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
|
|
rocminfo
|
|
rocm-smi
|
|
```
|
|
|
|
Or set it globally:
|
|
|
|
```bash
|
|
pct exec 204 -- bash -c 'echo "export PATH=/opt/rocm/bin:\$PATH" > /etc/profile.d/rocm.sh'
|
|
```
|
|
|
|
## Quoting Rule for pct exec
|
|
|
|
`pct exec` always adds a shell layer. Complex commands should be written to a file inside the container, then executed:
|
|
|
|
```bash
|
|
# BAD — quoting nightmare, easy breakage
|
|
pct exec 204 -- python3 -c "import torch; print('HIP:', torch.cuda.is_available())"
|
|
|
|
# GOOD — write script, execute
|
|
pct exec 204 -- bash -c 'cat > /tmp/verify.py << "EOF"
|
|
import torch
|
|
print("HIP available:", torch.cuda.is_available())
|
|
print("Device:", torch.cuda.get_device_name(0))
|
|
EOF
|
|
python3 /tmp/verify.py'
|
|
```
|
|
|
|
## Monitoring via Cronjob
|
|
|
|
Set up a cronjob that polls the `DONE` marker and reports status:
|
|
|
|
```bash
|
|
# ~/.hermes/cron/ct204_monitor.sh
|
|
if [ -f /tmp/ct204_autosetup_done ]; then
|
|
echo "CT 204 SETUP COMPLETE"
|
|
tail -10 /tmp/ct204_autosetup.log
|
|
pct exec 204 -- python3 -c 'import torch; print(torch.cuda.is_available())'
|
|
else
|
|
echo "CT 204 SETUP RUNNING"
|
|
tail -5 /tmp/ct204_autosetup.log
|
|
fi
|
|
```
|