hermes-skills/devops/multi-agent-team-architecture/SKILL.md

---
name: multi-agent-team-architecture
description: Designing and implementing a multi-agent team using Hermes profiles, SOUL.md templates, and CEO-level orchestration.
tags:
  - multi-agent
  - profiles
  - architecture
---

# Multi-Agent Team Architecture

## Trigger Conditions
- User wants to set up a virtual team or multiple specialized agents.
- User asks to separate tasks by domain (e.g., SRE vs. Nutrition).
- User wants to scale the AI team with new "hires" (profiles).

## Core Concept: The "Schön Consulting" Model
The team is organized as a **CEO (Orchestrator) + Specialized Agents (Profiles)** structure.
- **CEO (You/Standard Profile):** Strategic oversight, task delegation, cross-domain coordination, and user interaction.
- **Agents (Hermes Profiles):** Fully isolated environments (`SOUL.md`, Memory, Skills, Cron) for specific domains (e.g., `infra-sre`, `nutrition-coach`).

## Step 1: Profile Creation
Every team member is a distinct Hermes profile. This ensures no "context bleed" between domains.

```bash
# Create a blank profile (fresh state)
hermes profile create <agent-name>

# Or clone existing config/SOUL to start quickly
hermes profile create <agent-name> --clone
```

## Step 2: The `SOUL.md` Template
Every agent requires a structured `SOUL.md` to define their "identity." Use this template:

```markdown
# <Agent-Name> (Role Title)

## Role
[Brief professional description of the agent's purpose and expertise.]

## Persona & Tone
- **Style:** (e.g., Direct, Empathetic, Technical, Pragmatic)
- **Key Trait:** (e.g., Security-first, Family-oriented, Budget-conscious)

## Responsibilities
1. [Primary Task 1]
2. [Primary Task 2]
3. [Primary Task 3]

## Tools & Workflow
- [Specific tools, scripts, or skills this agent should use]
- [How it handles data/results]

## Escalation Rules (An CEO)
- **CRITICAL:** [Condition for immediate, high-priority alert]
- **WARNING:** [Condition for next-report inclusion]
- **INFO:** [Condition for routine logging]

## Communication Style
"[Example of how the agent addresses the user/CEO and reports status]"
```

## Step 3: Model & Provider Alignment Across Profiles

When a user says "set model X for all agents", update **three keys per profile** — not just `model.default`. Hermes checks them in a priority order, and leftover stale values cause silent fallbacks:

```bash
hermes config set --profile <agent> model.provider <PROVIDER>
hermes config set --profile <agent> model.default    <MODEL>
hermes config set --profile <agent> model.model      <MODEL>   # explicit mirror
```

**Verification:**
```bash
hermes profile list
```
shows the model string column; if it disagrees with what you expect, the wrong key is carrying it.

### Auxiliary tasks stale-endpoint fix
After a provider migration, agent profiles may carry stale `auxiliary.*` overrides pointing to a dead custom endpoint (e.g. `base_url: http://10.0.30.99:8080/v1`). Symptoms: `Auxiliary title generation failed: HTTP 400: network error occurred while connecting to provider API`.

**Bulk reset all auxiliary slots across all profiles to `auto` (falls back to main model):**
```python
import subprocess

profiles = ['default', 'infra-sre', 'nutrition-coach']  # adapt to your stack
slots = [
    'vision', 'web_extract', 'compression', 'session_search',
    'skills_hub', 'approval', 'mcp', 'title_generation', 'flush_memories',
]
for p in profiles:
    for s in slots:
        for k, v in [('provider', 'auto'), ('model', ''), ('base_url', ''), ('api_key', '')]:
            subprocess.run(
                ['hermes', 'config', 'set', '--profile', p, f'auxiliary.{s}.{k}', v],
                capture_output=True)
```

### Provider Config Inheritance (Critical — Before Gateway Start)

**Agent profiles do NOT automatically inherit `providers.*` blocks from the default profile.**

If your default profile defines a custom provider (e.g., `providers.noris` with `base_url` and `api_key`), every agent profile that uses this provider MUST explicitly copy that block. Otherwise you get:

```
⚠️ Provider authentication failed: Unknown provider 'noris'.
```

**Fix:** Copy the provider definition into the agent's `config.yaml`:

```bash
# 1. Read provider from default
grep -A4 'providers:' ~/.hermes/config.yaml

# 2. Inject into agent config (example: nutrition-coach)
# Edit ~/.hermes/profiles/nutrition-coach/config.yaml:
providers:
  noris:
    type: openai
    base_url: https://ai.noris.de/v1
    api_key: sk-...
```

Also remove stale `api_key` / `base_url` entries from the agent's `model:` block — they should come from `providers.*`.

**Verification after fix:**
```bash
hermes profile show <name>  # should show "(noris)" not "(custom)"
```

---

## Step 4: Telegram Bot Setup & Verification (Critical)
Before an agent can communicate via Telegram, the systemd service must be installed.
1. **Configure .env:** Create `~/.hermes/profiles/<name>/.env` with `TELEGRAM_BOT_TOKEN=...`
2. **Reload:** `systemctl --user daemon-reload`
3. **Install Service:** Run `<name> gateway install`.
   - *⚠️ **Critical:** `gateway start` fails without this.*
4. **Start Service:** `systemctl --user start hermes-gateway-<name>.service`
5. **Verification:**
   - Check process: `systemctl status hermes-gateway-<name>.service` (Active: running)
   - Check logs: `journalctl --user -u hermes-gateway-<name>.service -n 20`
   - Test API: `curl https://api.telegram.org/bot<TOKEN>/getMe` -> expect `{"ok": true}`

### Telegram DM Limitation (Two Bots, Same User)
**A Telegram user has ONE DM chat ID regardless of which bot messages them.** If you register a second bot (`@schoen_essensplaner_bot`) and expect it to create a "separate chat" from the CEO bot, both bots will deliver to the **same DM thread** (the user's personal chat ID).

**True separation requires a group or topics:**
- **New Group:** Create a dedicated group, invite the bot → negative group chat ID
- **Topics:** Enable Topics in an existing group, post into a dedicated topic thread
- The `infra-sre` agent delivers to `telegram:-5265906503` because it targets a **group**, not a DM.

## Step 4: Operational Workflow
### Task Distribution
- **Direct Command:** User asks a specific agent (e.g., via Telegram Topic or `chef, check infra`).
- **Delegation:** CEO uses `delegate_task` to offload complex sub-tasks to isolated contexts.
- **Automation:** CEO sets `cronjob` for routine tasks (e.g., Sunday night infra scan).

### Success Measurement
The CEO monitors specific metrics per agent:
- **Infra:** Uptime, TLS-Expiration, Port-Drift.
- **Family:** Plan-adherence, Feedback-Adaption, Budget-accuracy.

## Step 4: Team Growth (Gap Analysis)
The CEO must proactively identify "hiring" needs:
1. **Pattern Recognition:** Notice repetitive user questions or manual tasks.
2. **Proposal:** "Chef, we need a new agent for [Task] because [Gap]."
3. **Execution:** User approves -> CEO creates Profile + `SOUL.md` + Cron -> Agent goes live.

## Pitfalls
- **Context Bleed:** Do not share `Memory` across profiles unless explicitly intended (use `memory` tool to copy).
- **Over-Engineering:** Start with 1-2 agents; expand based on actual need, not theoretical ones.
- **Telegram Routing:** If using a single bot, rely on `Topics` in groups for separation. If using multiple bots, ensure each has a unique `@BotFather` token.
- **Cannot restart gateway from inside the gateway process.** When running as a Telegram gateway session, `systemctl --user restart hermes-gateway` or `hermes gateway restart` will be blocked with `"Blocked: cannot restart or stop the gateway from inside the gateway process"` because SIGTERM propagates to child processes (including the agent itself). Instead: use the `/restart` slash command in the Telegram chat, or ask the user to run `hermes gateway restart` from a separate SSH session.
- **Cross-profile gateway restarts also blocked from inside any gateway.** `systemctl --user restart hermes-gateway-infra-sre` also fails from inside the main gateway because systemd sends SIGTERM to the entire cgroup. Must be done from an external shell or via `/restart` in each respective bot chat.
- **`hermes update` procedure for multi-gateway setups.** After `hermes update` completes (pulls code + syncs skills + updates Python deps), each gateway service must be restarted individually to load the new code. The update command itself does NOT restart gateways. For a 3-profile setup (default + infra-sre + nutrition-coach), all three need restart. From inside any gateway session, use `/restart`; from SSH, use `systemctl --user restart hermes-gateway hermes-gateway-infra-sre hermes-gateway-nutrition-coach`.
- **Gateway may hang in "deactivating" state after restart attempt.** If `systemctl --user is-active` shows `deactivating` for an extended period, use `systemctl --user reset-failed <service>` followed by `systemctl --user start <service>`. Alternatively, `systemctl --user kill <service>` then `start`. This is a systemd race condition, not a Hermes bug.
- **MCP servers and toolset changes require gateway restart.** Adding `mcp_servers` to `config.yaml` or enabling a new toolset (e.g., `hermes tools enable homeassistant`) does not take effect until the gateway restarts. MCP tools are discovered at startup only — no hot-reload.

## Skill Update History
- **2026-04-21**: Initial creation based on "Schön Consulting Inc." virtual team setup (Infra SRE + Nutrition Coach).