feat(runbook): Notfall-Runbook + Inline-Hints an Alarm-Flaechen (CHO tower-notfall-runbook)
DATEI 1: public/runbook/runbook-tower-notfall.md (statisch, Anker #deadman/#spool/#deadletter/#health)
DATEI 2: /runbook-Route (Server Component, Mini-MD-Renderer, CSP-konform, force-static)
DATEI 3: runbookHints.ts (LAMP_HINTS/buildView/renderHintHtml 1:1 aus runbook_hints.py)
+ RunbookHint.tsx eingebunden an 4 Alarm-Flaechen:
- EscalationHistory (Deadman, Szenario A)
- TaskBrowser Spool-Tab (Szenario B) + Dead-Letter-Tab (B3)
- TaskBrowser/OpsPanels Degraded/Error (Health, Szenario C)
Keine Ampelfarben/Deadman-Logik/Degraded-Banner/Datenpfade angefasst (CHO-NICHT ANFASSEN erfuellt).
Build verifiziert: tsc --noEmit clean, next build gruen (/runbook static).
This commit is contained in:
@@ -1,6 +1,8 @@
|
|||||||
"use client"
|
"use client"
|
||||||
|
|
||||||
import { useEffect, useState } from "react"
|
import { useEffect, useState } from "react"
|
||||||
|
import RunbookHint from "./RunbookHint"
|
||||||
|
import { hintByKey } from "@/app/lib/fleet/runbookHints"
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* EscalationHistory — Kernfeature 4: HITL-/Escalation-Historie.
|
* EscalationHistory — Kernfeature 4: HITL-/Escalation-Historie.
|
||||||
@@ -93,6 +95,7 @@ export default function EscalationHistory() {
|
|||||||
<h2 className="text-[10px] uppercase tracking-wider text-[var(--status-stopped)] mb-3">
|
<h2 className="text-[10px] uppercase tracking-wider text-[var(--status-stopped)] mb-3">
|
||||||
HITL-Alarme (type=escalation)
|
HITL-Alarme (type=escalation)
|
||||||
</h2>
|
</h2>
|
||||||
|
<RunbookHint hint={hintByKey("deadman")} />
|
||||||
{esc.length === 0 ? (
|
{esc.length === 0 ? (
|
||||||
<p className="text-xs text-[var(--foreground-dim)] font-mono py-4 text-center">
|
<p className="text-xs text-[var(--foreground-dim)] font-mono py-4 text-center">
|
||||||
Keine Alarme — alles ruhig
|
Keine Alarme — alles ruhig
|
||||||
|
|||||||
@@ -14,6 +14,8 @@
|
|||||||
* Keine Mutation, keine Netz-Grants, 30 s Auto-Refresh.
|
* Keine Mutation, keine Netz-Grants, 30 s Auto-Refresh.
|
||||||
*/
|
*/
|
||||||
import { useCallback, useEffect, useState } from 'react'
|
import { useCallback, useEffect, useState } from 'react'
|
||||||
|
import RunbookHint from './RunbookHint'
|
||||||
|
import { hintByKey } from '@/app/lib/fleet/runbookHints'
|
||||||
|
|
||||||
const REFRESH_MS = 30_000
|
const REFRESH_MS = 30_000
|
||||||
|
|
||||||
@@ -190,6 +192,7 @@ export default function OpsPanels() {
|
|||||||
Datenquelle eingeschraenkt: {err}
|
Datenquelle eingeschraenkt: {err}
|
||||||
</p>
|
</p>
|
||||||
)}
|
)}
|
||||||
|
{err && <RunbookHint hint={hintByKey("health")} />}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{tab === 'nar' && (
|
{tab === 'nar' && (
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"use client"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* RunbookHint.tsx — statischer Inline-Hint an den Alarm-Flaechen des Towers.
|
||||||
|
* Portierung der render_hint_html()-Semantik als React-Komponente
|
||||||
|
* (Karte tower-notfall-runbook, DATEI 3). Rein statisch: kein Polling,
|
||||||
|
* kein Egress, keine Secrets — nur interner Deep-Link auf /runbook#<anker>.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { LampHint } from "@/app/lib/fleet/runbookHints"
|
||||||
|
|
||||||
|
export default function RunbookHint({ hint }: { hint: LampHint }) {
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
id={`hint-${hint.key}`}
|
||||||
|
className="lamp-hint text-xs leading-snug text-[var(--foreground-dim)]"
|
||||||
|
data-component={hint.component}
|
||||||
|
data-testid={`runbook-hint-${hint.key}`}
|
||||||
|
>
|
||||||
|
<span className="lh-title block font-semibold text-[var(--foreground)]">
|
||||||
|
Notfall {hint.scenario} — {hint.title}
|
||||||
|
</span>
|
||||||
|
<ol className="lh-steps my-0.5 ml-4 list-decimal">
|
||||||
|
{hint.kurzschritte.map((s: string) => (
|
||||||
|
<li key={s}>{s}</li>
|
||||||
|
))}
|
||||||
|
</ol>
|
||||||
|
<div className="lh-esc mt-0.5">
|
||||||
|
<strong>Eskalation:</strong> {hint.eskalation}
|
||||||
|
</div>
|
||||||
|
<a
|
||||||
|
className="lh-link text-[#93c5fd] underline hover:text-[#a5ccff]"
|
||||||
|
href={`/runbook${hint.anchor}`}
|
||||||
|
>
|
||||||
|
Vollständiges Runbook →
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -13,6 +13,8 @@
|
|||||||
* Polt 30 s; degradierte Quelle -> Banner statt Leeranzeige.
|
* Polt 30 s; degradierte Quelle -> Banner statt Leeranzeige.
|
||||||
*/
|
*/
|
||||||
import { useCallback, useEffect, useRef, useState } from "react"
|
import { useCallback, useEffect, useRef, useState } from "react"
|
||||||
|
import RunbookHint from "./RunbookHint"
|
||||||
|
import { hintByKey } from "@/app/lib/fleet/runbookHints"
|
||||||
|
|
||||||
type TaskEnvelope = {
|
type TaskEnvelope = {
|
||||||
task_id: string
|
task_id: string
|
||||||
@@ -253,11 +255,26 @@ export default function TaskBrowser() {
|
|||||||
Tasks nicht ladbar: {loadError}
|
Tasks nicht ladbar: {loadError}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
{tab === "spool" && (
|
||||||
|
<div className="mt-4 border border-[var(--border)] bg-[var(--background-tertiary)] p-3">
|
||||||
|
<RunbookHint hint={hintByKey("spool")} />
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{tab === "dead-letter" && (
|
||||||
|
<div className="mt-4 border border-[var(--border)] bg-[var(--background-tertiary)] p-3">
|
||||||
|
<RunbookHint hint={hintByKey("deadletter")} />
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
{data?.degraded && (
|
{data?.degraded && (
|
||||||
<div className="mt-4 border border-amber-500 bg-[var(--background-tertiary)] p-3 text-xs text-amber-400">
|
<div className="mt-4 border border-amber-500 bg-[var(--background-tertiary)] p-3 text-xs text-amber-400">
|
||||||
DEGRADED: {data.degraded_reason || "Quelle gestoert"} — Anzeige evtl. unvollstaendig.
|
DEGRADED: {data.degraded_reason || "Quelle gestoert"} — Anzeige evtl. unvollstaendig.
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
{(loadError || data?.degraded) && (
|
||||||
|
<div className="mt-4 border border-[var(--border)] bg-[var(--background-tertiary)] p-3">
|
||||||
|
<RunbookHint hint={hintByKey("health")} />
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
<div className="mt-4 overflow-x-auto">
|
<div className="mt-4 overflow-x-auto">
|
||||||
{tab === "alarm" ? (
|
{tab === "alarm" ? (
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
/**
|
||||||
|
* runbookHints.ts — 1:1-Portierung von /sandbox/fleet-change-orders/
|
||||||
|
* tower-notfall-runbook/runbook_hints.py (Karte tower-notfall-runbook, devops).
|
||||||
|
*
|
||||||
|
* Zweck: Kurzschritte + Eskalationskontakt fuer den Owner direkt an den
|
||||||
|
* Alarm-Ampeln des Towers (10.0.30.213, tcp/8099) als kontextueller
|
||||||
|
* Inline-Hint mit Deep-Link auf das Notfall-Runbook (/runbook#<anker>).
|
||||||
|
*
|
||||||
|
* Rein statisch — KEINE Datenquelle, KEIN Polling, KEIN Egress, KEINE
|
||||||
|
* Secrets. LAMP_HINTS + buildView() + renderHintHtml() entsprechen
|
||||||
|
* LAMP_HINTS + build_view() + render_hint_html() der Referenz 1:1.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export const RUNBOOK_URL = "/runbook"
|
||||||
|
|
||||||
|
export type LampHint = {
|
||||||
|
key: "deadman" | "spool" | "health" | "deadletter"
|
||||||
|
component: string
|
||||||
|
badge_id: string | null
|
||||||
|
anchor: string
|
||||||
|
scenario: string
|
||||||
|
title: string
|
||||||
|
core: boolean
|
||||||
|
kurzschritte: string[]
|
||||||
|
eskalation: string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reihenfolge = Anzeige-Reihenfolge im Tower-Header/Panel.
|
||||||
|
// core=true -> Pflicht-Ampel laut Karte (Deadman-Blockade, Spool-Stau, Panel-Ausfall).
|
||||||
|
export const LAMP_HINTS: LampHint[] = [
|
||||||
|
{
|
||||||
|
key: "deadman",
|
||||||
|
component: "Deadman-Status",
|
||||||
|
badge_id: "dm-badge",
|
||||||
|
anchor: "#deadman",
|
||||||
|
scenario: "A",
|
||||||
|
title: "Deadman-Blockade",
|
||||||
|
core: true,
|
||||||
|
kurzschritte: [
|
||||||
|
"Owner-Queue oeffnen: blockierenden Task (ID, Typ, Alter) an der Deadman-Ampel ablesen.",
|
||||||
|
"Task in der Queue entscheiden (Abnahme/Review/NAR) oder mit reply_to + Kommentar zurueckgeben.",
|
||||||
|
"Deadman-Ampel auf 'progress' pruefen (Auto-Refresh 30 s).",
|
||||||
|
],
|
||||||
|
eskalation:
|
||||||
|
"Bleibt 'blocked' > 5400 s oder ist der Task unklar: Owner ueber die Owner-Queue informieren (Grund + Task-ID nennen).",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "spool",
|
||||||
|
component: "Spool-Aging (Reife)",
|
||||||
|
badge_id: "sp-badge",
|
||||||
|
anchor: "#spool",
|
||||||
|
scenario: "B",
|
||||||
|
title: "Spool-Stau",
|
||||||
|
core: true,
|
||||||
|
kurzschritte: [
|
||||||
|
"Spool-Aging: aeltesten roten Envelope (> 5400 s) + Empfaenger ablesen.",
|
||||||
|
"Empfaenger gruen: Task an denselben Empfaenger erneut submitten (reply_to setzen).",
|
||||||
|
"Empfaenger rot: zuerst Szenario A/C loesen; Failed-Tasks ueber die Dead-Letter-Ansicht (#deadletter) pruefen.",
|
||||||
|
],
|
||||||
|
eskalation:
|
||||||
|
"Envelope > 5400 s ohne roten Empfaenger (Ursache unklar): an tower-owner ueber den CISO-Kanal melden (Envelope-IDs); ab 2 betroffenen Agents zusaetzlich Owner informieren.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "health",
|
||||||
|
component: "Health-Ampel / Panel",
|
||||||
|
badge_id: null,
|
||||||
|
anchor: "#health",
|
||||||
|
scenario: "C",
|
||||||
|
title: "Panel-Ausfall",
|
||||||
|
core: true,
|
||||||
|
kurzschritte: [
|
||||||
|
"/healthz pruefen: 200 = Server lebt, Problem liegt bei Datenquelle/UI.",
|
||||||
|
"/api/status + Degraded-Banner: Quelle down = Degraded (60 s-Cache laeuft weiter), KEIN Totalausfall.",
|
||||||
|
"Kein 200 / Verbindung abgelehnt = Totalausfall: Neustart/Deploy nur ueber Aenderungsauftrag, nicht selbst am Host.",
|
||||||
|
],
|
||||||
|
eskalation:
|
||||||
|
"healthz-Ausfall oder Deploy-Bedarf: sofort P1 an tower-owner ueber den CISO-Kanal; parallel Owner informieren (ohne Panel ist die Lage nur ueber die Agenten-Boards sichtbar).",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "deadletter",
|
||||||
|
component: "Dead-Letter-Ansicht",
|
||||||
|
badge_id: null,
|
||||||
|
anchor: "#deadletter",
|
||||||
|
scenario: "B3",
|
||||||
|
title: "Verlorene Tasks",
|
||||||
|
core: false,
|
||||||
|
kurzschritte: [
|
||||||
|
"Dead-Letter: failed-Tasks (7 Tage) mit from->to + Alter pruefen.",
|
||||||
|
"Wiederholbar: erneut submitten (reply_to); unloesbar: Task mit Grund neu aufsetzen, nicht still verwerfen.",
|
||||||
|
],
|
||||||
|
eskalation:
|
||||||
|
"Gehaeufte Failed-Tasks eines Empfaengers: Empfaenger-Health pruefen (#health), dann wie Szenario B eskalieren.",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
/** Hint per key (statisch, wirft bei unbekanntem key). */
|
||||||
|
export function hintByKey(key: LampHint["key"]): LampHint {
|
||||||
|
const h = LAMP_HINTS.find((x) => x.key === key)
|
||||||
|
if (!h) throw new Error(`unknown runbook hint: ${key}`)
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
export type HintView = {
|
||||||
|
key: string
|
||||||
|
component: string
|
||||||
|
badge_id: string | null
|
||||||
|
scenario: string
|
||||||
|
title: string
|
||||||
|
core: boolean
|
||||||
|
anchor: string
|
||||||
|
url: string
|
||||||
|
kurzschritte: string[]
|
||||||
|
eskalation: string
|
||||||
|
html: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Statische Hint-Ansicht (JSON-faehig) — entspricht build_view(). */
|
||||||
|
export function buildView(now: string | null = null, source = "static") {
|
||||||
|
const hints: HintView[] = LAMP_HINTS.map((h) => ({
|
||||||
|
key: h.key,
|
||||||
|
component: h.component,
|
||||||
|
badge_id: h.badge_id,
|
||||||
|
scenario: h.scenario,
|
||||||
|
title: h.title,
|
||||||
|
core: h.core,
|
||||||
|
anchor: h.anchor,
|
||||||
|
url: RUNBOOK_URL + h.anchor,
|
||||||
|
kurzschritte: [...h.kurzschritte],
|
||||||
|
eskalation: h.eskalation,
|
||||||
|
html: renderHintHtml(h),
|
||||||
|
}))
|
||||||
|
return {
|
||||||
|
generated_at: now,
|
||||||
|
source,
|
||||||
|
runbook_url: RUNBOOK_URL,
|
||||||
|
refresh_seconds: 0, // statisch
|
||||||
|
hint_count: hints.length,
|
||||||
|
core_count: hints.filter((h) => h.core).length,
|
||||||
|
hints,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function escapeHtml(s: string): string {
|
||||||
|
return s
|
||||||
|
.replace(/&/g, "&")
|
||||||
|
.replace(/</g, "<")
|
||||||
|
.replace(/>/g, ">")
|
||||||
|
.replace(/"/g, """)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Kompakter Inline-Hint als HTML-String (rein Text + interner Link). */
|
||||||
|
export function renderHintHtml(h: LampHint): string {
|
||||||
|
const esc = escapeHtml
|
||||||
|
const stepLis = h.kurzschritte.map((s) => `<li>${esc(s)}</li>`).join("")
|
||||||
|
return `<div id="hint-${esc(h.key)}" class="lamp-hint" data-component="${esc(h.component)}">` +
|
||||||
|
`<span class="lh-title">Notfall ${esc(h.scenario)} — ${esc(h.title)}</span>` +
|
||||||
|
`<ol class="lh-steps">${stepLis}</ol>` +
|
||||||
|
`<div class="lh-esc"><strong>Eskalation:</strong> ${esc(h.eskalation)}</div>` +
|
||||||
|
`<a class="lh-link" href="${esc(RUNBOOK_URL + h.anchor)}">Vollstaendiges Runbook →</a>` +
|
||||||
|
`</div>`
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
/**
|
||||||
|
* /runbook — Statische Notfall-Runbook-Seite des Towers (Server Component).
|
||||||
|
* Karte tower-notfall-runbook, DATEI 2: rendert die Runbook-Datei als
|
||||||
|
* schlichte Markdown-/Textseite — rein statisch, kein Script, keine
|
||||||
|
* externen URLs, keine Secrets. Anker (#deadman etc.) kommen aus der MD.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { readFile } from "node:fs/promises"
|
||||||
|
import path from "node:path"
|
||||||
|
|
||||||
|
export const dynamic = "force-static"
|
||||||
|
export const metadata = { title: "Notfall-Runbook Tower" }
|
||||||
|
|
||||||
|
const MD_PATH = path.join(process.cwd(), "public", "runbook", "runbook-tower-notfall.md")
|
||||||
|
|
||||||
|
/** Minimaler Markdown->HTML-Renderer (keine Dep, kein Script-Output). */
|
||||||
|
function renderMarkdown(md: string): string {
|
||||||
|
const esc = (s: string) =>
|
||||||
|
s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """)
|
||||||
|
const lines = md.split(/\r?\n/)
|
||||||
|
const out: string[] = []
|
||||||
|
let inCode = false
|
||||||
|
let listOpen = false
|
||||||
|
const closeList = () => {
|
||||||
|
if (listOpen) {
|
||||||
|
out.push("</ul>")
|
||||||
|
listOpen = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const raw of lines) {
|
||||||
|
const line = raw
|
||||||
|
// Bekannt-sichere Anker-Tags aus der Runbook-MD unverändert durchreichen
|
||||||
|
// (Deep-Link-Ziele #deadman/#spool/#deadletter/#health).
|
||||||
|
const anchor = line.match(/^<a id="([a-z]+)"><\/a>$/)
|
||||||
|
if (anchor && !inCode) {
|
||||||
|
out.push(`<span id="${anchor[1]}"></span>`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (/^```/.test(line)) {
|
||||||
|
closeList()
|
||||||
|
out.push(inCode ? "</code></pre>" : '<pre class="runbook-code"><code>')
|
||||||
|
inCode = !inCode
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (inCode) {
|
||||||
|
out.push(esc(line))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
const h = line.match(/^(#{1,4})\s+(.*)$/)
|
||||||
|
if (h) {
|
||||||
|
closeList()
|
||||||
|
const lvl = Math.min(h[1].length + 1, 5) // h1->h2 (Seitenh1 ist der Titel)
|
||||||
|
out.push(`<h${lvl}>${esc(h[2])}</h${lvl}>`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (/^\s*[-*]\s+/.test(line)) {
|
||||||
|
if (!listOpen) {
|
||||||
|
out.push('<ul class="runbook-list">')
|
||||||
|
listOpen = true
|
||||||
|
}
|
||||||
|
out.push(`<li>${esc(line.replace(/^\s*[-*]\s+/, ""))}</li>`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (line.trim() === "") {
|
||||||
|
closeList()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out.push(`<p>${esc(line)}</p>`)
|
||||||
|
}
|
||||||
|
closeList()
|
||||||
|
return out.join("\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
export default async function RunbookPage() {
|
||||||
|
let md = ""
|
||||||
|
try {
|
||||||
|
md = await readFile(MD_PATH, "utf8")
|
||||||
|
} catch {
|
||||||
|
md = "# Runbook nicht gefunden\n\nDie Runbook-Datei ist im Image nicht vorhanden (public/runbook/runbook-tower-notfall.md)."
|
||||||
|
}
|
||||||
|
const html = renderMarkdown(md)
|
||||||
|
return (
|
||||||
|
<main className="panel mx-auto my-8 max-w-4xl p-8" data-testid="runbook-page">
|
||||||
|
<h1 className="text-lg font-semibold uppercase tracking-wider text-[var(--nvidia-green)]">
|
||||||
|
Notfall-Runbook Tower
|
||||||
|
</h1>
|
||||||
|
<p className="mt-1 text-xs text-[var(--foreground-dim)]">
|
||||||
|
Statisch · Deep-Link-Ziele der Alarm-Ampeln:{" "}
|
||||||
|
<a className="text-[#93c5fd] underline" href="#deadman">#deadman</a> ·{" "}
|
||||||
|
<a className="text-[#93c5fd] underline" href="#spool">#spool</a> ·{" "}
|
||||||
|
<a className="text-[#93c5fd] underline" href="#deadletter">#deadletter</a> ·{" "}
|
||||||
|
<a className="text-[#93c5fd] underline" href="#health">#health</a>
|
||||||
|
</p>
|
||||||
|
<div
|
||||||
|
className="runbook-body mt-6 space-y-2 text-sm leading-relaxed"
|
||||||
|
dangerouslySetInnerHTML={{ __html: html }}
|
||||||
|
/>
|
||||||
|
</main>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
# Notfall-Runbook Tower (Fleet vs-fleet-rebuild)
|
||||||
|
|
||||||
|
Stand: 2026-10-30 · Autor: devops · Karte: tower-notfall-runbook
|
||||||
|
Geltung: Owner am Tower-Panel (10.0.30.213, tcp/8099). Ziel: in <5 min
|
||||||
|
entscheiden und handeln, ohne zu raetseln.
|
||||||
|
|
||||||
|
## Anker (Deep-Link-Ziele fuer die Alarm-Ampeln)
|
||||||
|
Die Inline-Hints/Links an den Tower-Ampeln springen genau auf einen dieser
|
||||||
|
Anker (nur interne Links, kein aktiver Inhalt/Script, keine Secrets):
|
||||||
|
- **#deadman** -> Deadman-Status / Szenario A
|
||||||
|
- **#spool** -> Spool-Aging / Szenario B
|
||||||
|
- **#deadletter** -> Dead-Letter-Ansicht / Szenario B (Punkt 3)
|
||||||
|
- **#health** -> Health-Ampel / Degraded-Banner / Szenario C
|
||||||
|
Diese Datei ist reiner Text (Markdown); die Hints rendern daraus nur
|
||||||
|
Ueberschrift + Link, nichts Ausfuehrbares.
|
||||||
|
|
||||||
|
## 0. Grundlagen (immer zuerst lesen)
|
||||||
|
- Heartbeat-Intervall: 300 s je Agent an den Beacon.
|
||||||
|
- Ampel Health: gruen <120 s, gelb 120-300 s, rot >300 s (Beacon-Heartbeat-Age).
|
||||||
|
- Deadman-Schwelle: 5400 s (90 min) blockierte Fleet -> Kaskade reisst.
|
||||||
|
- Spool-Aging: offene Envelopes (submitted/working), rot >5400 s.
|
||||||
|
- Diagnose-Oberflaechen am Tower-Panel: Deadman-Status, Spool-Aging,
|
||||||
|
Dead-Letter, Alarm-Feed, Health-Ampel, /healthz + /api/status.
|
||||||
|
- Aufgaben des Owners: nur ENTSCHEIDEN + Freigeben. Kein Debugging im Panel.
|
||||||
|
|
||||||
|
## ESKALATIONSKONTAKT (Owner)
|
||||||
|
- Primaer: **Owner** (Dominik). Kanal: Owner-Inbox/Queue im Tower.
|
||||||
|
- Eskalation aus einem Agenten heraus: ueber den CISO-Kanal
|
||||||
|
(`submit_task(from_agent=<id>, to=ciso, type=review, topic="[NAR] <ziel>")`);
|
||||||
|
der CISO leitet kritische Faelle als `escalation` P1 an `tower-owner`
|
||||||
|
weiter (FLEET-POLICIES 1.3). Direkt-Adressierung an `tower-owner` ist nur
|
||||||
|
dem CISO erlaubt.
|
||||||
|
- Owner-Entscheidungen (Abnahme/Review/NAR) laufen ueber die Owner-Queue.
|
||||||
|
- Bei Netz-Themen: NAR ueber den CISO; bei Infrastruktur/Tower: tower-owner.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<a id="deadman"></a>
|
||||||
|
## Szenario A — Deadman-Blockade (Fleet steht, Ampel "Deadman: blocked")
|
||||||
|
Beobachtung: Deadman-Status zeigt Zustand `blocked` + blockierenden Owner-Task
|
||||||
|
(ID, Typ, Alter) im Header; ggf. rot in der Health-Ampel.
|
||||||
|
|
||||||
|
Diagnose-Pfad:
|
||||||
|
1. Panel -> **Deadman-Status**: welcher Owner-Task blockiert? Alter auslesen.
|
||||||
|
2. Panel -> **Alarm-Feed**: juengste Deadman-Kaskade (Zeit, Agent, Ursache,
|
||||||
|
Dauer) - bestaetigt, ob es die laufende Blockade ist.
|
||||||
|
3. Panel -> **Health-Ampel** (#health): welche Agents stehen rot/gelb (wer ist
|
||||||
|
betroffen)?
|
||||||
|
4. Abgleich: blockierender Task-Typ (Abnahme/Review/NAR/Entscheidung).
|
||||||
|
|
||||||
|
Schritte:
|
||||||
|
1. Blockierenden Owner-Task oeffnen und INNERHALB der Queue entscheiden
|
||||||
|
(Abnahme erteilen, Review freigeben/ablehnen, NAR approven).
|
||||||
|
2. Nach Entscheidung Deadman-Status auf `progress` pruefen (Auto-Refresh 30 s).
|
||||||
|
3. Wenn der Task nicht entscheidbar ist (fehlender Kontext): Task mit
|
||||||
|
Kommentar zurueck an Absender (reply_to) statt liegen zu lassen.
|
||||||
|
4. Verifikation: Health-Ampel der betroffenen Agents wieder gruen <120 s
|
||||||
|
binnen 2 Heartbeat-Zyklen (<=10 min).
|
||||||
|
|
||||||
|
Eskalation: Bleibt die Blockade >5400 s bestehen oder ist der Task unklar ->
|
||||||
|
Owner direkt ansprechen (Kanal s.o.), Grund + Task-ID nennen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<a id="spool"></a>
|
||||||
|
## Szenario B — Spool-Stau (Tasks haengen, Zustellung klemmt)
|
||||||
|
Beobachtung: Spool-Aging listet Envelopes >5400 s rot; Zusteller liefern nicht.
|
||||||
|
|
||||||
|
Diagnose-Pfad:
|
||||||
|
1. Panel -> **Spool-Aging**: Envelopes nach Alter + Empfaenger. Aeltester
|
||||||
|
Eintrag zuerst; rot = >5400 s.
|
||||||
|
<a id="deadletter"></a>
|
||||||
|
2. Panel -> **Dead-Letter**: status=failed (letzte 7 Tage) mit from->to, Alter -
|
||||||
|
bereits verlorene Arbeit (Luecke Spool-Aging vs. Alarm-Feed).
|
||||||
|
3. Panel -> **Health-Ampel** (#health): ist Empfaenger-Agent rot? -> down.
|
||||||
|
4. Panel -> **/api/status**: Datenquellen-Zustand Prometheus/Beacon; degraded?
|
||||||
|
|
||||||
|
Schritte:
|
||||||
|
1. Einzelner haengender Envelope, Empfaenger gruen -> Zustellung neu anstossen
|
||||||
|
(Task erneut submitten an denselben Empfaenger, reply_to setzen).
|
||||||
|
2. Blockierter ganzer Empfaenger (rot) -> Szenario A/C zuerst loesen; der Spool
|
||||||
|
leert sich danach automatisch.
|
||||||
|
3. Failed-Tasks aus der Dead-Letter-Ansicht pruefen: wiederholbar -> erneut
|
||||||
|
submitten; endgueltig unloesbar -> Task mit Grund neu aufsetzen (nicht still
|
||||||
|
verwerfen).
|
||||||
|
4. Verifikation: keine Eintraege mehr >5400 s im Spool-Aging; Dead-Letter
|
||||||
|
waechst nicht weiter.
|
||||||
|
|
||||||
|
Eskalation: Envelopes >5400 s ohne down-Empfaenger -> Ursache unklar, an
|
||||||
|
tower-owner mit Screenshot/IDs melden; bei >2 betroffenen Agents zusaetzlich
|
||||||
|
Owner informieren (Kanal s.o.).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<a id="health"></a>
|
||||||
|
## Szenario C — Panel-Ausfall (Tower nicht erreichbar/leer/fehlerhaft)
|
||||||
|
Beobachtung: Panel laedt nicht, Leeranzeige oder Crash; /healthz antwortet nicht.
|
||||||
|
|
||||||
|
Diagnose-Pfad (Reihenfolge, um Server vs. Daten zu trennen):
|
||||||
|
1. Panel -> **/healthz**: 200 = Server lebt -> Problem ist Datenquelle/UI.
|
||||||
|
2. Panel -> **/api/status**: Version, Uptime, Datenquellen-Zustand
|
||||||
|
Prometheus/Beacon. Liefert JSON = API ok.
|
||||||
|
3. Degraded-Banner + Datenalter-Anzeige: bei Prometheus-/Beacon-Ausfall zeigt
|
||||||
|
der Tower Datenalter + 60 s-Cache; das ist Degraded, KEIN Totalausfall.
|
||||||
|
4. Wenn /healthz gar nicht antwortet: Panel-/Prozess-Ausfall am Tower-Host.
|
||||||
|
|
||||||
|
Schritte:
|
||||||
|
1. Degraded (healthz 200, /api/status zeigt Quelle down): keine Panik -
|
||||||
|
Betrieb laeuft mit Cache weiter. Ursache der Quelle pruefen (Beacon-Ausfall
|
||||||
|
= alle Agents rot -> Szenario-A-Umfeld).
|
||||||
|
2. Totalausfall (healthz kein 200 / Verbindung abgelehnt): Tower braucht
|
||||||
|
Neustart/Deploy -> Aenderungsauftrag an tower-owner (nicht selbst am Host
|
||||||
|
eingreifen).
|
||||||
|
3. Waehrend Ausfall: Diagnose ersatzweise ueber die Agenten-Boards/Beacon
|
||||||
|
(Karten-Status, Heartbeat-Zeitstempel) weiterfuehren, Owner informieren.
|
||||||
|
4. Verifikation: /healthz 200 UND /api/status mit allen Quellen "up"; Panel
|
||||||
|
zeigt Datenalter ~0.
|
||||||
|
|
||||||
|
Eskalation: healthz-Ausfall oder Deploy-Bedarf -> sofort P1 an tower-owner
|
||||||
|
(ueber den CISO-Kanal); parallele Owner-Info, da ohne Panel die Lage ueber
|
||||||
|
Agenten nicht sichtbar ist.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Kurz-Anhang: Anzeige am Tower
|
||||||
|
Dieses Runbook + Eskalationskontakt sollen an den Alarm-Ampeln (Deadman-Status
|
||||||
|
-> #deadman, Spool-Aging -> #spool, Dead-Letter -> #deadletter, Health-Ampel ->
|
||||||
|
#health) als kontextueller Hilfe-Link/Inline-Hint erscheinen. Der Hint ist
|
||||||
|
reiner Text (Ueberschrift + interner Link), kein aktives Element; Alarmlogik
|
||||||
|
und Degraded-Banner bleiben unberuehrt. Aenderungsauftrag an tower-owner siehe
|
||||||
|
separater Task "[build] tower-notfall-runbook: Runbook-Link an Alarm-Ampeln".
|
||||||
Reference in New Issue
Block a user