Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
+208
@@ -0,0 +1,208 @@
|
||||
# LXC GPU Autosetup — Background Script Pattern
|
||||
|
||||
## Problem
|
||||
|
||||
Provisioning GPU-accelerated LXC containers involves long-running operations:
|
||||
|
||||
1. `apt update && apt install` inside LXC (first setup)
|
||||
2. `amdgpu-install --usecase=rocm --no-dkms` (30-90 min, 100+ packages)
|
||||
3. `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2` (~4 GB download, 10-20 min)
|
||||
4. `pip install -r requirements.txt` for ComfyUI/Ollama/vLLM
|
||||
|
||||
Running any of these in the foreground via `pct exec` is brittle:
|
||||
- SSH timeout kills the session
|
||||
- `pct exec` double-shell-quoting breaks complex commands
|
||||
- No way to resume if interrupted
|
||||
|
||||
## Solution: Background Script on Host
|
||||
|
||||
Write a bash script on the Proxmox **host**, run it via `nohup`, and poll completion markers.
|
||||
|
||||
### Step 1: Write the Script
|
||||
|
||||
```bash
|
||||
# /tmp/ctNNN_autosetup.sh — runs ON the Proxmox host, orchestrates INSIDE the LXC
|
||||
CTID=${1:-204}
|
||||
LOG=/tmp/ct${CTID}_autosetup.log
|
||||
DONE=/tmp/ct${CTID}_autosetup_done
|
||||
|
||||
echo "$(date): Starting auto-setup for CT ${CTID}..." > "$LOG"
|
||||
|
||||
# --- Wait for dpkg lock inside CT ---
|
||||
while pct exec "$CTID" -- fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do
|
||||
echo "$(date): dpkg still locked..." >> "$LOG"
|
||||
sleep 30
|
||||
done
|
||||
echo "$(date): dpkg free." >> "$LOG"
|
||||
|
||||
# --- Install ROCm (inside CT) ---
|
||||
pct exec "$CTID" -- bash -c '
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
|
||||
dpkg --configure -a 2>&1
|
||||
apt-get -f install -y -qq 2>&1
|
||||
cd /tmp
|
||||
wget -q https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb
|
||||
dpkg -i amdgpu-install_6.3.60303-1_all.deb 2>&1 | tail -3
|
||||
apt-get update -qq
|
||||
amdgpu-install --usecase=rocm --no-dkms -y 2>&1 | tail -10
|
||||
' >> "$LOG" 2>&1
|
||||
|
||||
# --- Verify ROCm ---
|
||||
echo "$(date): Verifying ROCm..." >> "$LOG"
|
||||
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocminfo 2>&1 | grep -E "Name:" | head -5' >> "$LOG" 2>&1
|
||||
pct exec "$CTID" -- bash -c 'export PATH=/opt/rocm/bin:$PATH; rocm-smi 2>&1 | head -5' >> "$LOG" 2>&1
|
||||
|
||||
# --- PyTorch ROCm ---
|
||||
echo "$(date): Installing PyTorch ROCm..." >> "$LOG"
|
||||
pct exec "$CTID" -- bash -c 'pip3 install --break-system-packages torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2 2>&1 | tail -10' >> "$LOG" 2>&1
|
||||
|
||||
# --- ComfyUI / App ---
|
||||
echo "$(date): Installing app..." >> "$LOG"
|
||||
pct exec "$CTID" -- bash -c '
|
||||
cd /opt
|
||||
git clone https://github.com/comfyanonymous/ComfyUI.git 2>&1 | tail -3
|
||||
cd ComfyUI
|
||||
pip3 install --break-system-packages -r requirements.txt 2>&1 | tail -10
|
||||
' >> "$LOG" 2>&1
|
||||
|
||||
# --- systemd service ---
|
||||
pct exec "$CTID" -- bash -c 'cat > /etc/systemd/system/comfyui.service << "EOF"
|
||||
[Unit]
|
||||
Description=ComfyUI
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/ComfyUI
|
||||
ExecStart=/usr/bin/python3 main.py --listen 0.0.0.0 --port 8188
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
Environment=HIP_VISIBLE_DEVICES=0
|
||||
Environment=PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl enable comfyui
|
||||
' >> "$LOG" 2>&1
|
||||
|
||||
pct exec "$CTID" -- systemctl start comfyui >> "$LOG" 2>&1
|
||||
sleep 5
|
||||
pct exec "$CTID" -- ss -tlnp | grep 8188 >> "$LOG" 2>&1
|
||||
|
||||
echo "$(date): Setup complete." >> "$LOG"
|
||||
echo "DONE" >> "$DONE"
|
||||
```
|
||||
|
||||
### Step 2: Launch on Host
|
||||
|
||||
```bash
|
||||
chmod +x /tmp/ct204_autosetup.sh
|
||||
nohup bash /tmp/ct204_autosetup.sh > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
### Step 3: Poll from Client
|
||||
|
||||
```bash
|
||||
# Check completion
|
||||
cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET
|
||||
|
||||
# Check progress
|
||||
tail -20 /tmp/ct204_autosetup.log
|
||||
```
|
||||
|
||||
## Python Paramiko Polling Pattern
|
||||
|
||||
When automating from a remote client (not the Proxmox host):
|
||||
|
||||
```python
|
||||
import paramiko, time
|
||||
|
||||
client = paramiko.SSHClient()
|
||||
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
client.connect('10.0.20.91', username='root', password='28acaneltO!#', timeout=15)
|
||||
|
||||
for attempt in range(20): # ~10 minutes
|
||||
stdin, stdout, stderr = client.exec_command(
|
||||
"cat /tmp/ct204_autosetup_done 2>/dev/null || echo NOT_YET"
|
||||
)
|
||||
status = stdout.read().decode().strip()
|
||||
if status == "DONE":
|
||||
print("Setup complete!")
|
||||
break
|
||||
|
||||
stdin2, stdout2, stderr2 = client.exec_command(
|
||||
"tail -5 /tmp/ct204_autosetup.log"
|
||||
)
|
||||
print(f"Progress: {stdout2.read().decode().strip()[-200:]}")
|
||||
time.sleep(30)
|
||||
|
||||
client.close()
|
||||
```
|
||||
|
||||
## PEP 668 Fix for Ubuntu 24.04+ Containers
|
||||
|
||||
Ubuntu 24.04 Noble enforces `externally-managed` Python environment. Inside LXC:
|
||||
|
||||
```bash
|
||||
# WRONG — fails with PEP 668
|
||||
pip3 install torch
|
||||
|
||||
# RIGHT — bypass inside disposable container
|
||||
pip3 install --break-system-packages torch torchvision torchaudio \
|
||||
--index-url https://download.pytorch.org/whl/rocm6.2
|
||||
```
|
||||
|
||||
Alternative: Create a venv inside the container (overkill for single-purpose inference boxes).
|
||||
|
||||
## ROCm PATH Gotcha
|
||||
|
||||
ROCm 6.3 does NOT add `/opt/rocm/bin` to PATH by default. Every `pct exec` ROCm call needs:
|
||||
|
||||
```bash
|
||||
export PATH=/opt/rocm/bin:/usr/local/bin:$PATH
|
||||
rocminfo
|
||||
rocm-smi
|
||||
```
|
||||
|
||||
Or set it globally:
|
||||
|
||||
```bash
|
||||
pct exec 204 -- bash -c 'echo "export PATH=/opt/rocm/bin:\$PATH" > /etc/profile.d/rocm.sh'
|
||||
```
|
||||
|
||||
## Quoting Rule for pct exec
|
||||
|
||||
`pct exec` always adds a shell layer. Complex commands should be written to a file inside the container, then executed:
|
||||
|
||||
```bash
|
||||
# BAD — quoting nightmare, easy breakage
|
||||
pct exec 204 -- python3 -c "import torch; print('HIP:', torch.cuda.is_available())"
|
||||
|
||||
# GOOD — write script, execute
|
||||
pct exec 204 -- bash -c 'cat > /tmp/verify.py << "EOF"
|
||||
import torch
|
||||
print("HIP available:", torch.cuda.is_available())
|
||||
print("Device:", torch.cuda.get_device_name(0))
|
||||
EOF
|
||||
python3 /tmp/verify.py'
|
||||
```
|
||||
|
||||
## Monitoring via Cronjob
|
||||
|
||||
Set up a cronjob that polls the `DONE` marker and reports status:
|
||||
|
||||
```bash
|
||||
# ~/.hermes/cron/ct204_monitor.sh
|
||||
if [ -f /tmp/ct204_autosetup_done ]; then
|
||||
echo "CT 204 SETUP COMPLETE"
|
||||
tail -10 /tmp/ct204_autosetup.log
|
||||
pct exec 204 -- python3 -c 'import torch; print(torch.cuda.is_available())'
|
||||
else
|
||||
echo "CT 204 SETUP RUNNING"
|
||||
tail -5 /tmp/ct204_autosetup.log
|
||||
fi
|
||||
```
|
||||
Reference in New Issue
Block a user