feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
File diff suppressed because it is too large
Load Diff
+66
@@ -0,0 +1,66 @@
|
||||
# ArgoCD Self-Management Cutover — 2026-07-14
|
||||
|
||||
## Context
|
||||
|
||||
Migrating ArgoCD's Git source from external Gitea (CT108, LXC, `10.0.30.105:3000`)
|
||||
to K8s-internal Gitea (Helm chart in `gitea` namespace, reachable via
|
||||
Traefik VIP `10.0.30.203`). The chicken-and-egg problem: ArgoCD deploys
|
||||
Gitea, but ArgoCD also pulls from Gitea.
|
||||
|
||||
## Sequence
|
||||
|
||||
### 1. CoreDNS Internal DNS Override
|
||||
|
||||
Added `hosts` plugin to CoreDNS via `HelmChartConfig` so `git.schoen.codes`
|
||||
resolves to `10.0.30.203` (Traefik VIP) inside K8s — no public DNS needed.
|
||||
|
||||
**File:** `clusters/main/operators/coredns-config.yaml`
|
||||
|
||||
**Key pitfall:** Initial attempt used `valuesContent: corefile: |` —
|
||||
silently ignored. The RKE2 CoreDNS chart uses `servers[].plugins[]`
|
||||
structure. Fixed by using the correct chart values structure.
|
||||
|
||||
**Key pitfall:** HelmChartConfig was synched by ArgoCD but ConfigMap
|
||||
didn't update until we directly patched it + waited 30s for CoreDNS
|
||||
`reload` plugin to pick up the change.
|
||||
|
||||
### 2. ArgoCD Repository Secret
|
||||
|
||||
Created `argocd-repo-k8s-gitea` secret in `argocd` namespace with
|
||||
URL, username, password, type=git.
|
||||
|
||||
**Critical pitfall:** Secret existed but ArgoCD showed
|
||||
`authentication required: Unauthorized` because the secret was NOT
|
||||
labeled with `argocd.argoproj.io/secret-type=repository`. Without this
|
||||
label, ArgoCD does not recognize the secret as a repository credential.
|
||||
|
||||
### 3. ArgoCD Application URL Patching
|
||||
|
||||
Patched all 7 Applications (root + 6 children) from
|
||||
`http://10.0.30.105:3000/dominik/iac-homelab.git` to
|
||||
`http://git.schoen.codes/dominik/iac-homelab.git` using
|
||||
`kubectl patch --type json`.
|
||||
|
||||
### 4. Verification
|
||||
|
||||
- Root app: `Synced/Healthy` ✅
|
||||
- Revision: commit hash from new Gitea ✅
|
||||
- All child apps converging ✅
|
||||
|
||||
## Result
|
||||
|
||||
ArgoCD now pulls from K8s Gitea via:
|
||||
```
|
||||
ArgoCD → git.schoen.codes → CoreDNS → 10.0.30.203 (Cilium VIP)
|
||||
→ Traefik DaemonSet → Gitea pod
|
||||
```
|
||||
|
||||
The entire GitOps loop is self-contained in K8s. CT108 kept as fallback.
|
||||
|
||||
## Files Changed
|
||||
|
||||
- `clusters/main/operators/coredns-config.yaml` — CoreDNS HelmChartConfig
|
||||
- `clusters/main/proxy/traefik-lb-service.yaml` — Traefik LoadBalancer VIP
|
||||
- `clusters/main/proxy/namespace.yaml` — proxy namespace
|
||||
- `clusters/main/apps/proxy.yaml` — ArgoCD app for proxy namespace
|
||||
- `docs/plans/2026-07-14-traefik-ct99999-k8s-migration.md` — migration plan
|
||||
@@ -0,0 +1,274 @@
|
||||
# Gitea Backup CronJob — Session Detail (2026-07-14)
|
||||
|
||||
## Context
|
||||
|
||||
After deploying InfluxDB backup CronJob (§4.3i), user requested the
|
||||
same backup pattern for Gitea. Unlike InfluxDB (single PVC, single
|
||||
backup command), Gitea has two data stores:
|
||||
1. **PVC** (`/data`) — repos, LFS, config, attachments, avatars
|
||||
2. **External Galera DB** (10.0.30.70:3306) — users, issues, PRs, labels
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### tar over `gitea dump`
|
||||
- `gitea dump` produces a zip but can miss LFS objects and repos outside
|
||||
the default path
|
||||
- `tar czf /data` captures everything reliably
|
||||
- Trade-off: larger archive (143 MB), but complete
|
||||
|
||||
### kubectl exec + kubectl cp for PVC access
|
||||
- PVC is RWO, mounted by the running `gitea-0` pod
|
||||
- Can't mount it in the backup pod (Multi-Attach error)
|
||||
- Solution: exec into the running pod to tar, then cp the archive out
|
||||
- Requires RBAC: ServiceAccount with pods/exec permission
|
||||
|
||||
### Separate mysqldump initContainer
|
||||
- `mysql:8.0` image has `mysqldump`
|
||||
- Credentials from existing `gitea-db` K8s secret (already used by Gitea)
|
||||
- `--single-transaction` for consistent dump without locking
|
||||
|
||||
### S3 credentials reuse
|
||||
- Same Norris S3 account as Postgres/InfluxDB backups
|
||||
- New ExternalSecret per namespace, same 1Password item keys
|
||||
- New bucket: `homelab-gitea-backup`
|
||||
|
||||
## Manifest Files Created
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `clusters/main/gitea/backup-rbac.yaml` | ServiceAccount + Role (pods/exec, pods/get) + RoleBinding |
|
||||
| `clusters/main/gitea/backup-s3-secret.yaml` | ExternalSecret → 1Password `postgres-s3-backup` creds |
|
||||
| `clusters/main/gitea/backup-cronjob.yaml` | CronJob: 2 initContainers (files + db) + 1 upload container |
|
||||
|
||||
Commit: `1679786`
|
||||
|
||||
## CronJob Spec
|
||||
|
||||
- **Schedule:** `30 3 * * *` (03:30 UTC — 30 min after InfluxDB backup)
|
||||
- **initContainer 1 (gitea-files):** `bitnami/kubectl:1.31` — kubectl exec tar + kubectl cp
|
||||
- **initContainer 2 (gitea-db):** `mysql:8.0` — mysqldump against Galera
|
||||
- **Container (upload):** `amazon/aws-cli:2.35.22` — aws s3 cp to Norris S3
|
||||
- **Shared volume:** emptyDir (passed between all containers)
|
||||
- **ServiceAccount:** `gitea-backup` (with RBAC for kubectl exec/cp)
|
||||
|
||||
## S3 Upload Paths
|
||||
|
||||
```
|
||||
s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-data.tar.gz
|
||||
s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-db.sql.gz
|
||||
```
|
||||
|
||||
## Execution Results
|
||||
|
||||
### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`
|
||||
|
||||
Both buckets (`homelab-influxdb-backup` + `homelab-gitea-backup`) were
|
||||
created successfully via one-off pods. **CRITICAL PITFALL**: `aws s3 mb`
|
||||
with `--region nsc-nbg` fails with `InvalidLocationConstraint: The
|
||||
nsc-nbg location constraint is not valid`. Must use `--region us-east-1`
|
||||
for bucket creation. The `nsc-nbg` region works for subsequent `aws s3 cp`
|
||||
operations but NOT for `s3 mb`.
|
||||
|
||||
```bash
|
||||
# CORRECT — us-east-1 for bucket creation
|
||||
kubectl run aws-mb-$bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
|
||||
--env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
|
||||
--command -- sh -c "aws s3 mb s3://$bucket \
|
||||
--endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
|
||||
```
|
||||
|
||||
### InfluxDB Backup — VERIFIED SUCCESSFUL ✅
|
||||
|
||||
InfluxDB backup CronJob ran successfully:
|
||||
- 77 shards backed up
|
||||
- `backup.tar.gz` = 25.4 MB
|
||||
- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
|
||||
- Total time: ~45 seconds
|
||||
|
||||
### Gitea Backup — BLOCKED by Image Pull Issues
|
||||
|
||||
The Gitea CronJob requires a kubectl image for the initContainer that
|
||||
runs `kubectl exec` + `kubectl cp`. Multiple images failed:
|
||||
|
||||
1. **`bitnami/kubectl:1.31`** — tag does NOT exist on Docker Hub
|
||||
(`failed to resolve reference: docker.io/bitnami/kubectl:1.31: not found`)
|
||||
2. **`bitnami/kubectl:1.31.4`** — same NotFound error
|
||||
3. **`bitnami/kubectl:latest`** — pulling but extremely slow (Docker Hub rate-limit)
|
||||
4. **`alpine/k8s:1.31.4`** — pulling but took 3+ minutes, still Pending
|
||||
|
||||
The `amazon/aws-cli:2.35.22` and `mysql:8.0` images pull fine (already
|
||||
cached from the InfluxDB backup CronJob). Only the kubectl image is
|
||||
problematic.
|
||||
|
||||
### kubectl cp Fails on Large Files
|
||||
|
||||
Attempting `kubectl cp` of the 143 MB gitea-data.tar.gz from the
|
||||
mgmt-runner into an upload pod failed with:
|
||||
```
|
||||
write tcp 10.0.30.124:35976->10.0.30.51:6443: write: connection reset by peer
|
||||
websocket: close 1006 (abnormal closure): unexpected EOF
|
||||
```
|
||||
|
||||
`kubectl cp` uses the K8s API websocket to stream file data. Large files
|
||||
(>~50 MB) are unreliable — the websocket connection resets before
|
||||
transfer completes. Small files (<1 MB) work fine.
|
||||
|
||||
### ArgoCD Reports Success But Doesn't Create New Resources
|
||||
|
||||
When new manifest files (backup-rbac.yaml, backup-s3-secret.yaml,
|
||||
backup-cronjob.yaml) were added to the existing `gitea-config` Application's
|
||||
directory (`clusters/main/gitea/`), ArgoCD showed:
|
||||
- Sync status: `Succeeded`
|
||||
- Operation phase: `Succeeded`
|
||||
- Message: `successfully synced (all tasks run)`
|
||||
- BUT: `kubectl get cronjob -n gitea` returned `No resources found`
|
||||
|
||||
This is a known ArgoCD behavior with `ServerSideApply=true` — new files
|
||||
in an existing directory aren't always detected on the first sync cycle.
|
||||
The hard-refresh annotation was applied but still showed `OutOfSync`.
|
||||
|
||||
**Fix that worked:** Transfer the manifest files to VM200 via base64
|
||||
encoding through SSH, then `kubectl apply -f` manually. The CronJob,
|
||||
RBAC, and ExternalSecret were all created successfully this way.
|
||||
|
||||
```bash
|
||||
# Transfer files from Hermes host to VM200 via base64
|
||||
B64=$(base64 -w0 /path/to/manifest.yaml)
|
||||
ssh root@10.0.30.124 "echo '$B64' | base64 -d > /root/iac-homelab/path/to/manifest.yaml"
|
||||
ssh root@10.0.30.124 "KUBECONFIG=/root/.kube/config kubectl apply -f /root/iac-homelab/path/to/manifest.yaml"
|
||||
```
|
||||
|
||||
### VM200 Repo Remote Not Updated After Gitea Migration
|
||||
|
||||
The mgmt-runner's `/root/iac-homelab` repo still pointed at the old
|
||||
Gitea on CT108 (`http://...@10.0.30.105:3000/...`). After ArgoCD was
|
||||
switched to K8s Gitea, the VM200 repo became stale. New commits pushed
|
||||
to K8s Gitea were invisible to VM200.
|
||||
|
||||
**Fix**: Update the remote URL on VM200 to point at K8s Gitea, or
|
||||
transfer files via base64 encoding through SSH.
|
||||
|
||||
### Manual Backup from mgmt-runner — Successful Workaround
|
||||
|
||||
When the CronJob pod approach is blocked by image issues, run the backup
|
||||
directly from the mgmt-runner (VM200, 10.0.30.124) which has kubectl +
|
||||
mysqldump already installed:
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
|
||||
KUBECONFIG=/root/.kube/config
|
||||
set -e
|
||||
|
||||
GITEA_POD=$(kubectl -n gitea get pod -l app.kubernetes.io/name=gitea \
|
||||
-o jsonpath="{.items[0].metadata.name}")
|
||||
AKI=$(kubectl -n gitea get secret gitea-s3-backup \
|
||||
-o jsonpath="{.data.aws_access_key_id}" | base64 -d)
|
||||
SAK=$(kubectl -n gitea get secret gitea-s3-backup \
|
||||
-o jsonpath="{.data.aws_secret_access_key}" | base64 -d)
|
||||
DB_PASS=$(kubectl -n gitea get secret gitea-db \
|
||||
-o jsonpath="{.data.password}" | base64 -d)
|
||||
DATE=$(date +%Y-%m-%d)
|
||||
|
||||
# 1. tar /data from gitea pod
|
||||
kubectl -n gitea exec $GITEA_POD -- tar czf /tmp/gitea-data.tar.gz -C / data
|
||||
kubectl -n gitea cp $GITEA_POD:/tmp/gitea-data.tar.gz /tmp/gitea-data.tar.gz
|
||||
kubectl -n gitea exec $GITEA_POD -- rm -f /tmp/gitea-data.tar.gz
|
||||
|
||||
# 2. mysqldump against Galera
|
||||
mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
|
||||
--password="$DB_PASS" --single-transaction \
|
||||
--routines --triggers gitea | gzip > /tmp/gitea-db.sql.gz
|
||||
|
||||
# 3. Upload via one-off pod with emptyDir (avoid hostPath — pods run on
|
||||
# different nodes!)
|
||||
# See pitfall below about hostPath
|
||||
'
|
||||
```
|
||||
|
||||
**Pitfall: hostPath volumes are node-local.** A pod scheduled on
|
||||
worker-01 cannot read files from the mgmt-runner's `/tmp` via hostPath.
|
||||
The pod lands on whichever node the scheduler chooses. Use `kubectl cp`
|
||||
to transfer files into a pod with emptyDir, or use `nodeName:` to pin
|
||||
the pod to a specific node (fragile — not recommended for production).
|
||||
|
||||
**Pitfall: `kubectl cp` to upload pod fails for large files.** The
|
||||
143 MB gitea-data.tar.gz caused websocket connection resets. Workaround:
|
||||
split the upload into smaller chunks, or use an initContainer that
|
||||
generates the data directly (avoiding the cp step entirely).
|
||||
|
||||
## Recommended Fix for CronJob
|
||||
|
||||
The CronJob approach needs a kubectl image that can be reliably pulled.
|
||||
Options:
|
||||
|
||||
1. **Pre-pull the image on all worker nodes** — `crictl pull alpine/k8s:1.31.4`
|
||||
on each worker, then the CronJob pulls instantly from local cache.
|
||||
2. **Use a different kubectl image from a non-Docker-Hub registry** — e.g.
|
||||
`registry.k8s.io/kubectl-sidecar` or an internally hosted image.
|
||||
3. **Restructure the CronJob to avoid kubectl entirely** — mount the PVC
|
||||
read-only (requires PVC to be RWX, or stop the pod first).
|
||||
4. **Run backup as a script on the mgmt-runner** via systemd timer or
|
||||
crontab — pragmatic but not GitOps-managed.
|
||||
|
||||
Until the image issue is resolved, the manual mgmt-runner approach
|
||||
(above) produces correct backups.
|
||||
|
||||
## Remaining Steps
|
||||
|
||||
1. ~~Create S3 bucket `homelab-gitea-backup`~~ ✅ Done
|
||||
2. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
|
||||
3. ~~InfluxDB backup test~~ ✅ Verified (25.4 MB uploaded)
|
||||
4. **Fix Gitea CronJob kubectl image** — pre-pull or use alternative
|
||||
5. **Gitea backup test run** — verify both files in S3
|
||||
6. Monitor first scheduled runs (03:00 + 03:30 UTC)
|
||||
|
||||
## Pitfalls Summary
|
||||
|
||||
### `bitnami/kubectl:1.31` tag does NOT exist
|
||||
The bitnami/kubectl image uses full semver tags (`1.31.4`, not `1.31`).
|
||||
However, even with correct tags, Docker Hub rate-limiting causes
|
||||
extremely slow pulls (3+ minutes to Pending). Pre-pull on all workers
|
||||
or use an alternative registry.
|
||||
|
||||
### S3 bucket creation: `--region us-east-1` not `nsc-nbg`
|
||||
`aws s3 mb` with `--region nsc-nbg` fails with `InvalidLocationConstraint`.
|
||||
Use `--region us-east-1` for bucket creation. The `nsc-nbg` region works
|
||||
for subsequent `aws s3 cp` operations.
|
||||
|
||||
### `kubectl cp` unreliable for large files (>50 MB)
|
||||
WebSocket connection resets before transfer completes. Small files
|
||||
(<1 MB) work fine. For large file transfers, consider splitting or
|
||||
avoiding `kubectl cp` entirely.
|
||||
|
||||
### VM200 repo remote may be stale after Gitea migration
|
||||
After switching ArgoCD from CT108 to K8s Gitea, the mgmt-runner's local
|
||||
repo remote still points at the old CT108 URL. New commits are invisible.
|
||||
Fix: `git remote set-url origin <new-url>` on VM200, or transfer files
|
||||
via base64 through SSH.
|
||||
|
||||
### hostPath volumes are node-local
|
||||
A pod using `hostPath: /tmp` reads from the NODE it's scheduled on, not
|
||||
from the machine that created the files. If files are on the mgmt-runner
|
||||
but the pod runs on a worker, the files are invisible. Use emptyDir +
|
||||
`kubectl cp`, or pin the pod with `nodeName:` (fragile).
|
||||
|
||||
### RBAC required for kubectl exec/cp
|
||||
Without explicit Role granting `pods/exec` + `pods/get`, the backup
|
||||
pod's kubectl commands fail with `forbidden`. Must create ServiceAccount,
|
||||
Role, and RoleBinding before the CronJob can function.
|
||||
|
||||
### kubectl cp namespace syntax
|
||||
`kubectl cp` requires `namespace/pod:path` format. Missing the
|
||||
namespace prefix causes failures when the backup pod and target pod
|
||||
are in different namespaces (though typically same namespace here).
|
||||
|
||||
## Pattern Reuse
|
||||
|
||||
This pattern (PVC tar via kubectl exec + external DB dump + S3 upload)
|
||||
applies to any K8s service with both PVC data and an external database:
|
||||
- Gitea (PVC + Galera)
|
||||
- Future services with similar dual-data-store architecture
|
||||
|
||||
For services with only PVC data (no external DB), skip the mysqldump
|
||||
initContainer. For services with only an external DB (no PVC), skip
|
||||
the kubectl exec initContainer.
|
||||
+216
@@ -0,0 +1,216 @@
|
||||
# Gitea Helm Chart v12 Deployment — Session 2026-07-14
|
||||
|
||||
## Context
|
||||
Deploying Gitea on K8s via ArgoCD using the official `gitea-charts/gitea`
|
||||
chart v12.6.0 (app v1.26.1). DB = external MariaDB Galera via MaxScale
|
||||
VIP 10.0.30.70:3306. Secrets via 1Password ESO.
|
||||
|
||||
## Issue 1: DB Password Not Reaching Init Container
|
||||
|
||||
### Symptom
|
||||
```
|
||||
[F] Failed to initialize ORM engine: Error 1045 (28000):
|
||||
Access denied for user 'gitea'@'10.0.30.62' (using password: NO)
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
The `configure-gitea` init container runs `gitea migrate` which reads
|
||||
`app.ini`. The chart generates `app.ini` from `gitea.config` values —
|
||||
but `PASSWD` was not set there. Env vars `GITEA__database__PASSWD`
|
||||
override app.ini at runtime, but only if they reach the init container.
|
||||
|
||||
### Failed Attempts
|
||||
1. `gitea.env` — not a valid chart value path; silently ignored
|
||||
2. `gitea.database.existingSecret` — not a valid chart value; ignored
|
||||
3. `gitea.config.database.PASSWD: __placeholder__` — literal string
|
||||
reaches DB, not a secret reference
|
||||
|
||||
### Solution
|
||||
`gitea.additionalConfigFromEnvs` — templated into ALL containers
|
||||
(init-directories, init-app-ini, configure-gitea, main):
|
||||
```yaml
|
||||
gitea:
|
||||
additionalConfigFromEnvs:
|
||||
- name: GITEA__database__PASSWD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: gitea-db
|
||||
key: password
|
||||
```
|
||||
|
||||
### Verification
|
||||
After fix, init-app-ini log showed `database + PASSWD` in config,
|
||||
and configure-gitea logged `PING DATABASE mysql` followed by
|
||||
successful schema migration.
|
||||
|
||||
## Issue 2: Valkey Auto-Deployed
|
||||
|
||||
### Symptom
|
||||
3 `gitea-valkey-cluster-*` pods appeared despite `redis-cluster.enabled: false`.
|
||||
|
||||
### Root Cause
|
||||
Chart v12 renamed `redis-cluster` → `valkey-cluster`. The old key
|
||||
no longer controls the cache subsystem.
|
||||
|
||||
### Fix
|
||||
```yaml
|
||||
valkey-cluster:
|
||||
enabled: false
|
||||
```
|
||||
ArgoCD pruned the Valkey StatefulSet after the updated values synced.
|
||||
|
||||
## Issue 3: RWO PVC Blocks Rolling Update
|
||||
|
||||
### Symptom
|
||||
New pod stuck in `Init:0/3` with:
|
||||
```
|
||||
Multi-Attach error for volume "pvc-..."
|
||||
Volume is already used by pod(s) gitea-OLD_POD
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
`ReadWriteOnce` PVC + single replica + ArgoCD selfHeal creates new
|
||||
pod before old one terminates. Both compete for the same volume.
|
||||
|
||||
### Fix
|
||||
```bash
|
||||
kubectl scale deploy -n gitea gitea --replicas=0
|
||||
sleep 15
|
||||
kubectl scale deploy -n gitea gitea --replicas=1
|
||||
```
|
||||
|
||||
## Issue 4: Galera Schema Migration Slowness
|
||||
|
||||
### Symptom
|
||||
`configure-gitea` init container stayed at `Init:2/3` for 5+ minutes.
|
||||
|
||||
### Diagnosis
|
||||
- 104 tables created by Gitea 1.26.1
|
||||
- Galera replicates each CREATE TABLE + index synchronously to 3 nodes
|
||||
- `SHOW PROCESSLIST` on Galera showed `creating table` / `Committing alter table`
|
||||
- Total time: ~8 minutes from first table to pod Running
|
||||
|
||||
### Key Insight
|
||||
This is normal for Galera — do NOT CrashLoop the pod or delete it.
|
||||
Monitor progress via `SELECT COUNT(*) FROM information_schema.tables
|
||||
WHERE table_schema='gitea'`.
|
||||
|
||||
## ArgoCD App Architecture Used
|
||||
|
||||
Two separate ArgoCD Applications:
|
||||
1. `gitea-config` (sync-wave 1): Git source → `clusters/main/gitea/`
|
||||
(namespace.yaml + external-secrets.yaml)
|
||||
2. `gitea` (sync-wave 2): Helm source → `https://dl.gitea.com/charts/`
|
||||
chart `gitea` v12.6.0
|
||||
|
||||
The root App-of-Apps (`clusters/main/apps/`) auto-discovers both.
|
||||
|
||||
## Final Working Values (Key Sections)
|
||||
|
||||
```yaml
|
||||
gitea:
|
||||
config:
|
||||
database:
|
||||
DB_TYPE: mysql
|
||||
HOST: 10.0.30.70:3306
|
||||
NAME: gitea
|
||||
USER: gitea
|
||||
additionalConfigFromEnvs:
|
||||
- name: GITEA__database__PASSWD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: gitea-db
|
||||
key: password
|
||||
existingSecret:
|
||||
name: gitea-security
|
||||
keys:
|
||||
internal_token: internal_token
|
||||
jwt_secret: jwt_secret
|
||||
secret_key: secret_key
|
||||
lfs_jwt_secret: lfs_jwt_secret
|
||||
|
||||
mysql:
|
||||
enabled: false
|
||||
postgresql:
|
||||
enabled: false
|
||||
postgresql-ha:
|
||||
enabled: false
|
||||
redis-cluster:
|
||||
enabled: false
|
||||
valkey-cluster:
|
||||
enabled: false
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: ceph-hdd-replica
|
||||
size: 10Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
```
|
||||
|
||||
## Repo Migration via Gitea Migration API (Task 7)
|
||||
|
||||
### Prerequisites
|
||||
- API tokens on BOTH Gitea instances (source CT108 + dest K8s)
|
||||
- Orgs and users created on destination before migrating
|
||||
- `migrations.ALLOWED_HOST_LIST` configured (see above)
|
||||
|
||||
### Token Generation
|
||||
|
||||
```bash
|
||||
# CT108 (source, LXC — Gitea runs as user 'gitea'):
|
||||
ssh root@10.0.30.105 "su - gitea -s /bin/bash -c \
|
||||
'gitea admin user generate-access-token -u dominik -t migrator --raw \
|
||||
--config /etc/gitea/app.ini'"
|
||||
|
||||
# K8s Gitea (destination):
|
||||
kubectl exec -n gitea deploy/gitea -- \
|
||||
gitea admin user generate-access-token -u dominik -t migrator --raw \
|
||||
--config /data/gitea/conf/app.ini
|
||||
```
|
||||
|
||||
### Migration Script Pattern
|
||||
|
||||
```python
|
||||
for repo in repos:
|
||||
migrate_data = {
|
||||
"clone_addr": f"{CT108_URL}/{owner}/{name}.git",
|
||||
"repo_owner": owner, "repo_name": name,
|
||||
"mirror": False, "wiki": True, "issues": True,
|
||||
"labels": True, "milestones": True,
|
||||
"pull_requests": True, "releases": True,
|
||||
"service": "gitea", "auth_token": CT108_TOKEN,
|
||||
}
|
||||
resp = k8s_api("POST", "/api/v1/repos/migrate", migrate_data)
|
||||
```
|
||||
|
||||
API calls go through `kubectl exec` on the K8s Gitea pod (no
|
||||
LoadBalancer/Ingress needed during migration):
|
||||
```bash
|
||||
kubectl exec -n gitea deploy/gitea -- curl -s -X POST \
|
||||
-H "Authorization: token TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{...}' http://localhost:3000/api/v1/repos/migrate
|
||||
```
|
||||
|
||||
### Results (23 repos)
|
||||
- 18/23 migrated on first attempt ✅
|
||||
- 1/23 skipped (already existed from test) ⏭️
|
||||
- 4/23 failed with PR import errors → retried with `pull_requests: false` → all ✅
|
||||
- Final: 23/23 repos migrated with issues + wikis
|
||||
|
||||
### Failed Repos (PR Import Bug)
|
||||
- `dominik/hermes-skills` — `initRepository: getRepositoryByID`
|
||||
- `pm-infra/homelab-board` — `error while listing pull requests`
|
||||
- `d.schoen/packer-proxmox` — `initRepository: getRepositoryByID`
|
||||
- `codecamp-n/qr-reader-webapp` — `error while listing pull requests`
|
||||
|
||||
Fix: `DELETE /api/v1/repos/{owner}/{name}` (cleanup), then retry
|
||||
migration with `pull_requests: false`.
|
||||
|
||||
## Commits
|
||||
- `48a26e6` — Initial scaffold (ArgoCD apps + namespace + ES + helm values)
|
||||
- `68eda13` — Fix: GITEA__database__PASSWD env + disable valkey-cluster
|
||||
- `effb49b` — Fix: use additionalConfigFromEnvs (correct chart mechanism)
|
||||
- `40da393` — Fix: allow migration from CT108 host (wrong setting name)
|
||||
- `68adfab` — Fix: correct migrations setting names (ALLOWED_HOST_LIST + ALLOW_LOCALNETWORKS)
|
||||
@@ -0,0 +1,83 @@
|
||||
# Gitea K8s Migration Plan (2026-07-13)
|
||||
|
||||
Session where the user decided to migrate Gitea from LXC CT108 to RKE2 K8s.
|
||||
|
||||
## Current State (CT108)
|
||||
|
||||
- Host: proxmox6, LXC container
|
||||
- Gitea binary (not Docker), SQLite (4.7 MB)
|
||||
- 23 repos (160 MB), 5 orgs/users (dominik, clawdia, codecamp-n, pm-infra, d.schoen)
|
||||
- Config: `/etc/gitea/app.ini`, data: `/var/lib/gitea/data/`
|
||||
- RAM: ~142 MB RSS, Disk: 1.3 GB / 7.8 GB
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### 1. Galera over CNPG (user-directed)
|
||||
|
||||
User instinctively preferred Galera, then asked for evaluation. After
|
||||
comparison, Galera won decisively:
|
||||
|
||||
- HA: Galera has 3 nodes + MaxScale. CNPG plan was 1 instance = no HA.
|
||||
- Resources: Galera = 0 new. CNPG = 512Mi + 10Gi.
|
||||
- MySQL is Gitea's native DB — best-tested combination.
|
||||
- SQLite→MySQL migration via Gitea Migration API (no schema conversion).
|
||||
|
||||
### 2. Gitea Migration API over manual SQLite dump
|
||||
|
||||
The Gitea Migration API (`POST /api/v1/repos/migrate`) eliminates the
|
||||
hardest task entirely. One call per repo brings:
|
||||
- Git data (clone)
|
||||
- Issues + labels + milestones
|
||||
- Pull requests
|
||||
- Releases
|
||||
- Wiki
|
||||
|
||||
No `pgloader`, no `sed` on SQLite dumps, no schema conversion.
|
||||
|
||||
### 3. External URL: git.schoen.codes
|
||||
|
||||
User specified `git.schoen.codes` as the new external URL (not
|
||||
`git.familie-schoen.com`).
|
||||
|
||||
## Plan Structure (10 Tasks)
|
||||
|
||||
1. Create 1Password items (gitea-db, gitea-security)
|
||||
2. Create database on Galera via MaxScale VIP
|
||||
3. Scaffold iac-homelab directory structure + ArgoCD app manifest
|
||||
4. Create ExternalSecrets (DB creds + security tokens)
|
||||
5. Create Gitea Helm release values (Galera DB, PVC, Ingress)
|
||||
6. Verify K8s Gitea reachable + DB connected
|
||||
7. Migrate repos + metadata via Gitea Migration API
|
||||
8. DNS cutover (git.schoen.codes → K8s LoadBalancer IP)
|
||||
9. Update ArgoCD source URL (chicken-egg cutover)
|
||||
10. Shutdown CT108 after 7-day stability window
|
||||
|
||||
## Resource Footprint
|
||||
|
||||
| Component | CPU req | RAM req | Storage |
|
||||
|-----------|---------|---------|---------|
|
||||
| Gitea pod | 100m | 256Mi | 10Gi (HDD) |
|
||||
| Total | 100m | 256Mi | 10Gi |
|
||||
|
||||
Previous (CT108): 1 core, 1 GB RAM, 8 GB disk → K8s footprint smaller.
|
||||
|
||||
## iac-homelab Repo Structure
|
||||
|
||||
```
|
||||
clusters/main/
|
||||
├── apps/gitea.yaml # ArgoCD Application (sync-wave 2)
|
||||
├── gitea/
|
||||
│ ├── namespace.yaml
|
||||
│ ├── external-secrets.yaml # gitea-db + gitea-security (sync-wave 0)
|
||||
│ ├── values.yaml # HelmChart CR (sync-wave 2)
|
||||
│ └── README.md
|
||||
└── docs/plans/2026-07-13-gitea-k8s-migration.md
|
||||
```
|
||||
|
||||
## Established Patterns Applied
|
||||
|
||||
- ArgoCD App manifest → `clusters/main/apps/{name}.yaml` with sync-waves
|
||||
- ExternalSecrets → `external-secrets.io/v1`, ClusterSecretStore `onepassword-store`
|
||||
- StorageClasses: `ceph-flash` (SSD) for DB, `ceph-hdd-replica` (HDD) for repo data
|
||||
- Traefik ingress with TLS
|
||||
- `CreateNamespace=true` in ArgoCD syncOptions
|
||||
@@ -0,0 +1,155 @@
|
||||
# Hermes Host Git Remote Cutover to K8s Gitea — 2026-07-14
|
||||
|
||||
## Context
|
||||
|
||||
After ArgoCD self-management cutover (Section 4.3e), the Hermes host's
|
||||
own `iac-homelab` git remote still pointed to CT108
|
||||
(`10.0.30.105:3000`). User instructed: "nutze du intern in hermes ab
|
||||
jetzt auch das k8s gitea" — switch Hermes host's git remote to K8s Gitea.
|
||||
|
||||
## Key Learnings
|
||||
|
||||
### CoreDNS Is K8s-Internal Only
|
||||
|
||||
The CoreDNS `hosts` override (Section 4.3d) makes `git.schoen.codes`
|
||||
resolve to `10.0.30.203` **only inside K8s pods**. The Hermes host,
|
||||
being outside K8s, cannot use CoreDNS. Public DNS was explicitly
|
||||
rejected by the user ("Nutze für Internet Zwecke nicht den externen DNS
|
||||
Resolver, sondern einen internen").
|
||||
|
||||
Solution: `/etc/hosts` entry on the Hermes host:
|
||||
```bash
|
||||
echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts
|
||||
```
|
||||
|
||||
### Gitea API Tokens Do NOT Migrate
|
||||
|
||||
When repos are migrated from CT108 Gitea to K8s Gitea via the Gitea
|
||||
Migration API, **API tokens are NOT copied**. Tokens are stored hashed
|
||||
in the database and are instance-specific. The old token
|
||||
(`07efa...`) from CT108 returns `401 Unauthorized` on K8s Gitea.
|
||||
|
||||
New token must be generated on K8s Gitea:
|
||||
```bash
|
||||
# Via mgmt-runner (VM200) — Hermes host has no kubectl:
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
||||
"KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
|
||||
gitea admin user generate-access-token \
|
||||
-u dominik -t hermes-gitops --scopes all --raw"
|
||||
```
|
||||
|
||||
Note: The `--config /data/gitea/conf/app.ini` flag is optional —
|
||||
`gitea admin user generate-access-token` auto-discovers the config when
|
||||
running inside the pod. The command outputs the raw token to stdout.
|
||||
|
||||
### JWT ≠ Gitea API Token (Critical Distinction)
|
||||
|
||||
The 1Password ExternalSecrets for Gitea (`gitea-security`) contain
|
||||
`internal_token`, `jwt_secret`, `secret_key`, `lfs_jwt_secret`. These
|
||||
are **internal security tokens** (JWTs), NOT API access tokens.
|
||||
|
||||
**How to distinguish:**
|
||||
- **Gitea API token**: hex string, e.g. `07efa534ea1a147fce1bf5813c669cb0ebc7b522`
|
||||
- **JWT / internal security token**: three base64 parts separated by dots,
|
||||
e.g. `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3ODM5NjM1NDd9.Lyr6fqc...`
|
||||
|
||||
A JWT passed as `Authorization: token <JWT>` or `Authorization: Bearer
|
||||
<JWT>` returns `401 Unauthorized` on both old and new Gitea instances.
|
||||
The `nbf` (Not Before) claim in the JWT payload identifies it as a
|
||||
security token, not an access token.
|
||||
|
||||
**If a user provides a token that looks like a JWT (dots separating
|
||||
base64 segments), it is NOT a Gitea API token.** Direct them to
|
||||
Settings → Applications → Generate New Token in the Gitea web UI, or
|
||||
generate one via `gitea admin user generate-access-token` (see above).
|
||||
|
||||
### K8s Gitea Reachability from Outside K8s
|
||||
|
||||
| Path | Works? | Notes |
|
||||
|------|--------|-------|
|
||||
| `http://10.0.30.203:3000` | ❌ Timeout | Port 3000 not exposed on VIP |
|
||||
| `http://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | Traefik routes by Host header |
|
||||
| `https://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | TLS via Traefik |
|
||||
| `10.0.30.202:2222` (SSH) | Separate LB | Gitea SSH LoadBalancer service |
|
||||
| `git.schoen.codes` (via /etc/hosts) | ✅ | Resolves to 10.0.30.203 |
|
||||
|
||||
### Hermes Host Has No kubectl
|
||||
|
||||
The Hermes host (`/home/debian`) does not have `kubectl`, `helm`, or
|
||||
`kubeconfig`. All K8s operations go through SSH to mgmt-runner (VM200)
|
||||
at `10.0.30.124`. See SKILL.md Section 1.2.
|
||||
|
||||
### ArgoCD Repo Secret Also Needs Token Update
|
||||
|
||||
After generating a new Gitea API token, the ArgoCD repository secret
|
||||
must also be patched with the new token — otherwise ArgoCD's repo
|
||||
connection breaks even though the Hermes host git remote works:
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
||||
"KUBECONFIG=/root/.kube/config kubectl patch secret \
|
||||
argocd-repo-k8s-gitea -n argocd --type merge \
|
||||
-p '{\"data\":{\"password\":\"$(echo -n TOKEN | base64)\"}}'"
|
||||
```
|
||||
|
||||
### Gitea Helm Chart Admin Password Extraction
|
||||
|
||||
The Gitea Helm Chart v12 auto-generates an admin password on first
|
||||
install (unless explicitly set in values). The password is stored as
|
||||
the `GITEA_ADMIN_PASSWORD` environment variable in the `configure-gitea`
|
||||
init container — NOT in any K8s Secret or Helm values file.
|
||||
|
||||
```bash
|
||||
# Extract admin password from the init container env vars:
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
||||
"KUBECONFIG=/root/.kube/config kubectl get deploy gitea -n gitea \
|
||||
-o jsonpath='{.spec.template.spec.initContainers[?(@.name==\"configure-gitea\")].env}'" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
for e in json.load(sys.stdin):
|
||||
if e.get('name')=='GITEA_ADMIN_PASSWORD':
|
||||
print(e['value'])
|
||||
"
|
||||
```
|
||||
|
||||
Additional env vars in the same container:
|
||||
- `GITEA_ADMIN_USERNAME`: the admin username (e.g. `dominik`)
|
||||
- `GITEA_ADMIN_PASSWORD_MODE`: `keepUpdated` (chart syncs password on
|
||||
every sync) or `initialOnlyRequireReset`
|
||||
|
||||
**Pitfall:** The password is NOT in the `gitea-security` K8s Secret
|
||||
(that contains JWT-based internal tokens only — see "JWT ≠ Gitea API
|
||||
Token" above). It is also NOT in the Helm values `gitea.admin` section
|
||||
(unless explicitly set — by default the chart generates a random one).
|
||||
|
||||
### Storing Credentials in 1Password
|
||||
|
||||
After extracting the admin password and generating an API token, store
|
||||
both in 1Password vault "Hermes":
|
||||
|
||||
- **API Token**: Category `API Credential`, title `K8s Gitea Token
|
||||
(hermes-gitops)`. Requires two-step create+edit (see 1password-cli
|
||||
skill — `op item create` does not populate the `credential` field
|
||||
for API Credential items).
|
||||
- **Admin Login**: Category `Login`, title `K8s Gitea Admin (dominik)`.
|
||||
Works in one step with `op item create --category="Login"`.
|
||||
|
||||
## Steps Performed (Completed)
|
||||
|
||||
1. Checked current remote: `git remote -v` → CT108 (`10.0.30.105:3000`)
|
||||
2. Verified K8s Gitea reachable via Traefik VIP: `curl -H "Host: git.schoen.codes" http://10.0.30.203/` → 200
|
||||
3. Added `/etc/hosts` entry: `10.0.30.203 git.schoen.codes`
|
||||
4. Changed git remote URL to `http://dominik:TOKEN@git.schoen.codes/dominik/iac-homelab.git`
|
||||
5. `git fetch origin` → `Authentication failed` (old CT108 token invalid on K8s Gitea)
|
||||
6. User provided a JWT (internal security token from 1Password) — identified as NOT an API token (see pitfall above)
|
||||
7. Generated new API token via mgmt-runner SSH: `gitea admin user generate-access-token -u dominik -t hermes-gitops --scopes all --raw` → `4a3fd...`
|
||||
8. Updated git remote with new token: `git remote set-url origin "http://dominik:4a3fd...@git.schoen.codes/..."`
|
||||
9. `git fetch origin` → ✅ success (forced update from `d9135ca` → `68adfab`)
|
||||
10. `git push --dry-run origin main` → ✅ "Everything up-to-date"
|
||||
11. Patched ArgoCD repo secret `argocd-repo-k8s-gitea` with new token
|
||||
|
||||
## Result
|
||||
|
||||
Hermes host git operations now use K8s Gitea exclusively. ArgoCD also
|
||||
pulls from K8s Gitea. The entire GitOps loop is self-contained in K8s.
|
||||
CT108 can be decommissioned (Task 10 — pending user approval).
|
||||
@@ -0,0 +1,151 @@
|
||||
# Hermes Memory System Architecture (2026-07-14)
|
||||
|
||||
## Overview
|
||||
|
||||
Hermes has two persistent memory stores injected into the system prompt:
|
||||
|
||||
- **MEMORY.md** — agent's personal notes (environment facts, infrastructure
|
||||
topology, tool quirks). Default limit: 2,200 chars.
|
||||
- **USER.md** — what the agent knows about the user (preferences, style,
|
||||
communication rules). Default limit: 1,375 chars.
|
||||
|
||||
Both files live at `~/.hermes/memories/` and use `§` (section sign) as
|
||||
entry delimiter: `\n§\n` between entries.
|
||||
|
||||
## Config
|
||||
|
||||
```yaml
|
||||
# ~/.hermes/config.yaml
|
||||
memory:
|
||||
memory_enabled: true
|
||||
user_profile_enabled: true
|
||||
memory_char_limit: 2200
|
||||
user_char_limit: 1375
|
||||
nudge_interval: 10
|
||||
```
|
||||
|
||||
Adjust limits:
|
||||
```bash
|
||||
hermes config set memory.user_char_limit 3000
|
||||
hermes config set memory.memory_char_limit 4000
|
||||
# Takes effect on next /reset or gateway restart
|
||||
```
|
||||
|
||||
## System Prompt Injection Pipeline
|
||||
|
||||
### 1. Loading (once per session)
|
||||
|
||||
`agent_init.py` creates a `MemoryStore` instance and calls
|
||||
`load_from_disk()`. This reads both `.md` files, splits on `§`, deduplicates
|
||||
entries, and creates a **frozen snapshot**.
|
||||
|
||||
### 2. Frozen Snapshot (prefix-cache invariant)
|
||||
|
||||
```python
|
||||
self._system_prompt_snapshot = {
|
||||
"memory": self._render_block("memory", sanitized_memory),
|
||||
"user": self._render_block("user", sanitized_user),
|
||||
}
|
||||
```
|
||||
|
||||
The snapshot is **never mutated mid-session** — even when `memory()` tool
|
||||
calls write new entries to disk. This keeps the system prompt byte-stable,
|
||||
preserving the LLM prefix cache across all turns.
|
||||
|
||||
### 3. Injection Point
|
||||
|
||||
`system_prompt.py` injects both blocks into the **volatile tier** of the
|
||||
system prompt (alongside date, model, provider):
|
||||
|
||||
```python
|
||||
if agent._memory_enabled:
|
||||
mem_block = agent._memory_store.format_for_system_prompt("memory")
|
||||
volatile_parts.append(mem_block)
|
||||
if agent._user_profile_enabled:
|
||||
user_block = agent._memory_store.format_for_system_prompt("user")
|
||||
volatile_parts.append(user_block)
|
||||
```
|
||||
|
||||
### 4. Rendered Format
|
||||
|
||||
```
|
||||
════════════════════════════════════════════
|
||||
MEMORY (your personal notes) [100% — 2,217/2,200 chars]
|
||||
════════════════════════════════════════════
|
||||
entry1
|
||||
§
|
||||
entry2
|
||||
§
|
||||
...
|
||||
|
||||
════════════════════════════════════════════
|
||||
USER PROFILE (who the user is) [86% — 1,184/1,375 chars]
|
||||
════════════════════════════════════════════
|
||||
entry1
|
||||
§
|
||||
entry2
|
||||
...
|
||||
```
|
||||
|
||||
### 5. Threat Scanning
|
||||
|
||||
Each entry is scanned for prompt-injection patterns at load time
|
||||
(`_sanitize_entries_for_snapshot`). Matches are replaced with
|
||||
`[BLOCKED: ...]` in the snapshot, but the original text remains in the
|
||||
live file so the user can inspect and delete it.
|
||||
|
||||
## System Prompt Tiers
|
||||
|
||||
```
|
||||
STABLE (byte-stable, cached for entire session)
|
||||
1. SOUL.md — identity ("You are Hermes Agent...")
|
||||
2. Tool schemas, platform hints, guardrails
|
||||
|
||||
CONTEXT (cwd-dependent, changes between sessions)
|
||||
3. .hermes.md / HERMES.md (walks to git root)
|
||||
4. AGENTS.md (cwd only)
|
||||
5. CLAUDE.md (cwd only)
|
||||
6. .cursorrules (cwd only)
|
||||
(first match wins — only ONE loaded, ~20K char cap)
|
||||
|
||||
VOLATILE (per-session, not cached)
|
||||
7. MEMORY.md snapshot (memory_char_limit)
|
||||
8. USER.md snapshot (user_char_limit)
|
||||
9. External memory provider (optional, e.g. Hindsight)
|
||||
10. Date, model, provider, session ID
|
||||
```
|
||||
|
||||
## Alternative Context Injection Points
|
||||
|
||||
Beyond MEMORY.md/USER.md, additional context can be injected via:
|
||||
|
||||
- **SOUL.md** (`~/.hermes/SOUL.md`) — global identity/personality, always
|
||||
loaded, no char limit
|
||||
- **AGENTS.md** (in repo root) — project conventions, ~20K char cap,
|
||||
loaded when cwd is the repo
|
||||
- **CLAUDE.md / .cursorrules** — same as AGENTS.md, first-match-wins
|
||||
- **.hermes.md** — walks up to git root, highest priority among context files
|
||||
|
||||
## Live State vs Snapshot
|
||||
|
||||
| Aspect | Snapshot (System Prompt) | Live State (Tool Calls) |
|
||||
|--------|--------------------------|--------------------------|
|
||||
| When read | Once at session start | Every `memory()` call |
|
||||
| When written | Never (read-only) | Every `memory()` call → disk |
|
||||
| Prefix cache | Byte-stable for session | N/A (not in prompt) |
|
||||
| Changes visible | Next `/reset` or restart | Immediately in tool response |
|
||||
|
||||
## Key Insight
|
||||
|
||||
Memory writes via the `memory()` tool are **not visible in the current
|
||||
session's system prompt** — they persist to disk but the snapshot was
|
||||
already frozen. This is by design: mutating the system prompt mid-session
|
||||
would invalidate the prefix cache and multiply API costs.
|
||||
|
||||
## Source Files
|
||||
|
||||
- `tools/memory_tool.py` — `MemoryStore` class, file I/O, rendering
|
||||
- `agent/agent_init.py` lines 1199-1218 — initialization
|
||||
- `agent/system_prompt.py` lines 426-435 — injection into volatile tier
|
||||
- `agent/prompt_builder.py` lines 1921-1968 — context files (AGENTS.md etc.)
|
||||
- `agent/coding_context.py` — project detection, context file discovery
|
||||
@@ -0,0 +1,276 @@
|
||||
# InfluxDB K8s Migration — Session Detail (2026-07-14)
|
||||
|
||||
## Context
|
||||
|
||||
Migrating InfluxDB v2 from CT109 (hot, 10.0.30.109) + CT134 (archive, 10.0.30.110)
|
||||
to a single K8s StatefulSet. Design was developed through the Compound Engineering
|
||||
brainstorming process (brainstorming → writing-plans → execution).
|
||||
|
||||
Design doc: `docs/plans/2026-07-14-influxdb-k8s-migration-design.md`
|
||||
Implementation plan: `docs/plans/2026-07-14-influxdb-k8s-migration-plan.md`
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Fresh Deploy + Replication** (Option B) over Backup/Restore (Option A)
|
||||
- **Single instance** with both buckets + local downsampling task (not two instances)
|
||||
- **Dual-Write Phase** cutover: CT109 stays as fallback for 1-2 weeks
|
||||
- **Daily backup** to Norris S3 via CronJob (same pattern as CNPG)
|
||||
|
||||
## 1Password Item Creation
|
||||
|
||||
### Vault: "Kubernetes ESO" (ID: 334ykdtj5kar3jlpcrztjvx2fu)
|
||||
|
||||
Created two items using the K8s ESO service account token (extracted from
|
||||
`onepassword-token` secret in `external-secrets` namespace):
|
||||
|
||||
1. **influxdb-k8s-admin** (Login category, ID: pc7olfgnbfc34ljjcnafrbf4si)
|
||||
- username: dominik
|
||||
- password: generated via `openssl rand -base64 24`
|
||||
- url: http://10.0.30.204:8086
|
||||
|
||||
2. **influxdb-k8s-token** (API Credential category, ID: ec7u4yb2zumae7ewq3yxciynhy)
|
||||
- TWO-STEP required: `op item create` then `op item edit <ID> credential="<token>"`
|
||||
|
||||
### ExternalSecret Key Mapping
|
||||
|
||||
```yaml
|
||||
data:
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_USERNAME
|
||||
remoteRef:
|
||||
key: influxdb-k8s-admin/username # Login category → "username" field
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_PASSWORD
|
||||
remoteRef:
|
||||
key: influxdb-k8s-admin/password # Login category → "password" field
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
|
||||
remoteRef:
|
||||
key: influxdb-k8s-token/credential # API Credential → "credential" field
|
||||
```
|
||||
|
||||
## Deployment Details
|
||||
|
||||
### Resource Allocation
|
||||
- VIP: 10.0.30.204 (Cilium L2 LB IPAM, pool .200-.250)
|
||||
- PVC data: 30Gi on ceph-hdd-replica
|
||||
- PVC config: 1Gi on ceph-hdd-replica
|
||||
- Resources: 512Mi-2Gi RAM, 250m-2000m CPU
|
||||
|
||||
### Init Mode Behavior
|
||||
- `DOCKER_INFLUXDB_INIT_MODE=setup` creates Org `homelab` + Bucket `ha_hot_90d` (90d retention)
|
||||
- Admin token from 1Password is set as the initial all-access token
|
||||
- PostSync Job creates: `ha_archive_5y_5m` bucket (5y/1825d retention) + downsampling task
|
||||
|
||||
### K8s Org ID differs from CT109
|
||||
- CT109 org `homelab`: ID `d2aeab314f4408ee`
|
||||
- K8s org `homelab`: ID `430395547892375d`
|
||||
- Replication `--remote-org-id` must use the DESTINATION (K8s) org ID
|
||||
|
||||
## Replication Setup
|
||||
|
||||
```bash
|
||||
# On CT109: create remote pointing to K8s
|
||||
influx remote create \
|
||||
--name k8s-influxdb \
|
||||
--remote-url http://10.0.30.204:8086 \
|
||||
--remote-api-token <K8S_TOKEN> \
|
||||
--remote-org-id 430395547892375d \
|
||||
--org homelab
|
||||
|
||||
# Create replication for hot bucket
|
||||
# CRITICAL: flags are --local-bucket-id and --remote-bucket-id (NOT --local-bucket)
|
||||
influx replication create \
|
||||
--name ha-hot-to-k8s \
|
||||
--remote-id <REMOTE_ID> \
|
||||
--local-bucket-id 3ca435503b6d5f59 \
|
||||
--remote-bucket-id 7a2ec96a41099ee8 \
|
||||
--org homelab
|
||||
```
|
||||
|
||||
Verification: `influx replication list` shows `Latest Status Code: 204` (success).
|
||||
|
||||
## Backfill via Flux `to()`
|
||||
|
||||
Chunked backfill (3 parallel ranges) from CT109 to K8s:
|
||||
|
||||
```flux
|
||||
from(bucket: "ha_hot_90d")
|
||||
|> range(start: 2026-04-15T00:00:00Z, stop: 2026-05-15T00:00:00Z)
|
||||
|> to(bucket: "ha_hot_90d", host: "http://10.0.30.204:8086",
|
||||
token: "<K8S_TOKEN>", org: "homelab")
|
||||
```
|
||||
|
||||
Result: 4,253,134 rows transferred. Disk footprint only 51 MB on K8s (vs 4.1 GB
|
||||
on CT109) — InfluxDB v2 columnar compression is extremely effective on fresh shards.
|
||||
|
||||
## Downsampling Task — Pitfall and Fix
|
||||
|
||||
### Error 1: Missing `option task` header
|
||||
```
|
||||
400 Bad Request: invalid options: no task options defined
|
||||
```
|
||||
Fix: Add `option task = { name: "archive_5m_to_hdd_90d", every: 1h }` at the top.
|
||||
|
||||
### Error 2: String fields crash `mean()`
|
||||
HA writes `icon_str`, `state_class_str`, `attribution_str` alongside numeric `value`.
|
||||
Fix: `|> filter(fn: (r) => r._field == "value")` before aggregation.
|
||||
|
||||
### Final working Flux
|
||||
```flux
|
||||
option task = { name: "archive_5m_to_hdd_90d", every: 1h }
|
||||
|
||||
from(bucket: "ha_hot_90d")
|
||||
|> range(start: -91d, stop: -90d)
|
||||
|> filter(fn: (r) => r._field == "value")
|
||||
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|
||||
|> to(bucket: "ha_archive_5y_5m", org: "homelab")
|
||||
```
|
||||
|
||||
## ArgoCD ConfigMap Update Failure
|
||||
|
||||
After fixing the Flux script in the ConfigMap and pushing to Git, ArgoCD showed
|
||||
`OutOfSync/Healthy` but the live ConfigMap retained the old script content.
|
||||
The PostSync Hook Job ran with the OLD ConfigMap and failed again.
|
||||
|
||||
### Workaround
|
||||
1. `kubectl delete cm influxdb-init -n influxdb`
|
||||
2. Hard refresh ArgoCD app
|
||||
3. Created the task manually via `kubectl exec` (faster for one-time fixes)
|
||||
|
||||
## HA Cutover — Home Assistant OS Access
|
||||
|
||||
### Discovery
|
||||
HA was NOT on 10.0.30.100 (Docker host). Found by checking CT109 inbound connections:
|
||||
```bash
|
||||
ssh root@10.0.30.109 'ss -tnp | grep ":8086"'
|
||||
# Found: 10.0.30.10 connecting to CT109:8086
|
||||
```
|
||||
|
||||
### HA OS SSH Access
|
||||
- **Port 22222** (NOT 22 — port 22 is refused)
|
||||
- `ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10`
|
||||
- HA OS is a minimal Alpine-based system (no `hostname`, no `python3`)
|
||||
|
||||
### HA Configuration Path
|
||||
- Config at `/mnt/data/supervisor/homeassistant/configuration.yaml`
|
||||
- NOT at `/config/` (which is empty)
|
||||
- Supervisor metadata at `/mnt/data/supervisor/homeassistant.json` (contains
|
||||
`access_token` and `refresh_token` for HA Supervisor API)
|
||||
|
||||
### InfluxDB Config in HA
|
||||
```yaml
|
||||
influxdb:
|
||||
api_version: 2
|
||||
ssl: false
|
||||
host: 10.0.30.204 # Changed from 10.0.30.109
|
||||
port: 8086
|
||||
token: <K8S_TOKEN> # Changed from CT109 token
|
||||
organization: homelab
|
||||
bucket: ha_hot_90d
|
||||
tags:
|
||||
source: HA
|
||||
tags_attributes:
|
||||
- friendly_name
|
||||
default_measurement: units
|
||||
```
|
||||
|
||||
### Applying Changes
|
||||
Used `sed -i` (HA OS has no python3):
|
||||
```bash
|
||||
ssh -p 22222 root@10.0.30.10 \
|
||||
"sed -i 's/host: 10.0.30.109/host: 10.0.30.204/' /mnt/data/supervisor/homeassistant/configuration.yaml && \
|
||||
sed -i 's/token: 92ec.../token: <K8S_TOKEN>/' /mnt/data/supervisor/homeassistant/configuration.yaml"
|
||||
```
|
||||
|
||||
Restart HA:
|
||||
```bash
|
||||
ssh -p 22222 root@10.0.30.10 'docker restart homeassistant'
|
||||
```
|
||||
|
||||
### Verification
|
||||
After 90s, queried K8s InfluxDB for recent data — confirmed HA writing directly
|
||||
to K8s with `source: HA` tag and timestamps within the last 3 minutes.
|
||||
|
||||
## Backup CronJob — Multi-Container Pattern
|
||||
|
||||
### Image Limitations Discovered
|
||||
- `influxdb:2.7` — has `tar` but NO `python3`, `aws`, `rclone`, working `apt-get`
|
||||
- `amazon/aws-cli:2` — tag DOES NOT EXIST, must use specific version (e.g. `2.35.22`)
|
||||
- `amazon/aws-cli:2.35.22` — has `aws` but NO `tar`
|
||||
|
||||
### Working Pattern: init-container + upload-container
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: backup
|
||||
image: influxdb:2.7
|
||||
command: [/bin/sh, -c, |
|
||||
influx backup /shared/backup --host http://influxdb:8086 \
|
||||
--token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
|
||||
cd /shared && tar czf backup.tar.gz backup/]
|
||||
envFrom:
|
||||
- secretRef: { name: influxdb-admin }
|
||||
volumeMounts:
|
||||
- { name: shared, mountPath: /shared }
|
||||
containers:
|
||||
- name: upload
|
||||
image: amazon/aws-cli:2.35.22
|
||||
command: [/bin/sh, -c, |
|
||||
aws s3 cp /shared/backup.tar.gz s3://homelab-influxdb-backup/$(date +%Y-%m-%d)/backup.tar.gz \
|
||||
--endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
|
||||
env: # MUST be uppercase AWS_* — envFrom with lowercase fails
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_access_key_id } }
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_secret_access_key } }
|
||||
volumeMounts:
|
||||
- { name: shared, mountPath: /shared }
|
||||
volumes:
|
||||
- { name: shared, emptyDir: {} }
|
||||
```
|
||||
|
||||
### S3 Credentials
|
||||
Reused existing `postgres-s3-backup` 1Password item (same Norris S3 account).
|
||||
ExternalSecret references `postgres-s3-backup/access_key_id` and
|
||||
`postgres-s3-backup/secret_access_key`.
|
||||
|
||||
### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`
|
||||
Bucket `homelab-influxdb-backup` must be created before first backup run.
|
||||
**CRITICAL**: `aws s3 mb` with `--region nsc-nbg` fails with
|
||||
`InvalidLocationConstraint: The nsc-nbg location constraint is not valid`.
|
||||
Must use `--region us-east-1` for bucket creation. `nsc-nbg` works for
|
||||
subsequent `aws s3 cp` operations but NOT for `s3 mb`.
|
||||
Use a one-off pod:
|
||||
```bash
|
||||
kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
|
||||
--env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
|
||||
--command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
|
||||
--endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
|
||||
```
|
||||
|
||||
### Backup Verified Successful ✅
|
||||
InfluxDB backup CronJob ran successfully on 2026-07-14:
|
||||
- 77 shards backed up via `influx backup`
|
||||
- `backup.tar.gz` = 25.4 MB (compressed)
|
||||
- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
|
||||
- Total time: ~45 seconds
|
||||
- Both S3 buckets created: `homelab-influxdb-backup` + `homelab-gitea-backup`
|
||||
|
||||
## Commits
|
||||
|
||||
- `c2840f4` — Design + Implementation Plan docs
|
||||
- `85be11f` — InfluxDB StatefulSet + all 6 manifests
|
||||
- `03808c1` — Fix: `option task` header in Flux script
|
||||
- `a4ef1fe` — Backup CronJob + S3 ExternalSecret
|
||||
- `f4208d3` — Fix: rclone/aws/boto3 fallback (attempted)
|
||||
- `2728f2e` — Fix: init-container + aws-cli container split
|
||||
- `39879ed` — Fix: correct aws-cli image tag 2.35.22
|
||||
- `ebd1f5c` — Fix: move tar to backup container
|
||||
- `be26461` — Fix: uppercase AWS env vars
|
||||
|
||||
## Remaining Steps
|
||||
|
||||
1. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
|
||||
2. ~~Trigger backup test run~~ ✅ Verified (25.4 MB uploaded)
|
||||
3. ~~Verify backup uploaded to Norris S3~~ ✅ Confirmed
|
||||
4. After 1-2 weeks stable: decommission CT109 + CT134
|
||||
- Stop replication on CT109
|
||||
- Stop InfluxDB services on CT109 + CT134
|
||||
- Stilllege CTs in Proxmox
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
# Traefik LXC → K8s Migration (2026-07-14)
|
||||
|
||||
## CT99999 Inventory (Source)
|
||||
|
||||
- **Location:** proxmox7, VLAN 60, IP `10.0.60.10/24`
|
||||
- **Traefik version:** 3.4.1
|
||||
- **Static config:** `/etc/traefik/traefik.yaml`
|
||||
- **Dynamic config:** `/etc/traefik/conf.d/` (file provider, watch=true)
|
||||
- **TLS:** Let's Encrypt via TLS challenge, `acme.json` at `/etc/traefik/ssl/acme.json`
|
||||
- **Entry Points:** web (:80 → redirect to :443), websecure (:443), traefik (:8080 dashboard)
|
||||
|
||||
## Services on CT99999
|
||||
|
||||
| Router | Host | Backend | Port | Notes |
|
||||
|--------|------|---------|------|-------|
|
||||
| grafana | grafana.familie-schoen.com | 10.0.30.141 | 3000 | CT141 monitoring |
|
||||
| librechat | chat.familie-schoen.com, chat.schoen.eu | 10.0.30.102 | 8080 | LibreChat VM |
|
||||
| seafile_old | seafile.familie-schoen.com | 10.0.30.100 | 8081 | Seafile old |
|
||||
| imkerei | imkerei.familie-schoen.com | 10.0.30.100 | 80 | Catchall forwarder |
|
||||
| immich | immich.familie-schoen.com | 10.0.30.99 | 2283 | Immich |
|
||||
| paperless | dokumente.familie-schoen.com | 10.0.30.101 | 8000 | CT104 paperless |
|
||||
| homeassistant | homeassistant.familie-schoen.com | 10.0.30.10 | 8123 | VM106 HA |
|
||||
| seafile | cloud.familie-schoen.com | 10.0.30.99 | 80 | Seafile |
|
||||
| seadoc | cloud.familie-schoen.com + /sdoc-server | 10.0.30.99 | 8888 | SeaDoc, stripPrefix middleware |
|
||||
| authelia | auth.familie-schoen.com | 192.168.100.11 | 9091 | CT112 authelia |
|
||||
| authelia-admin | auth-admin.familie-schoen.com | 192.168.100.11 | 9092 | CT112 authelia admin |
|
||||
| gitea | git.familie-schoen.com | 10.0.30.105 | 3000 | CT108 (being replaced by K8s) |
|
||||
| catchall-tcp | HostSNI(`*`) passthrough | 10.0.30.100 | 80 | TCP catchall |
|
||||
|
||||
## K8s Traefik (Target)
|
||||
|
||||
- **Type:** DaemonSet `rke2-traefik` in `kube-system`
|
||||
- **Ports:** hostPort 80 + 443 on all 6 nodes (10.0.30.51-53, .61-63)
|
||||
- **IngressClass:** `traefik`
|
||||
- **Existing Ingresses:** argocd-server (argocd ns), gitea (gitea ns)
|
||||
- **Needs:** Let's Encrypt certResolver (not configured by default)
|
||||
|
||||
## HA Failover VIP via Cilium L2 (Implemented 2026-07-14)
|
||||
|
||||
### Architecture
|
||||
|
||||
Instead of pointing the firewall NAT at a single K8s node (SPOF), a
|
||||
`LoadBalancer` Service requests a VIP from the existing Cilium LB IPAM
|
||||
pool. Cilium announces the VIP via ARP on one node; auto-failover to
|
||||
another node on failure (~1-2s).
|
||||
|
||||
```
|
||||
Firewall NAT → 10.0.30.203 (Cilium VIP)
|
||||
↓ L2 Announcement (ARP)
|
||||
[aktiver K8s Node — Traefik DaemonSet]
|
||||
Automatisches Failover bei Node-Ausfall (~1-2s)
|
||||
```
|
||||
|
||||
### Components Deployed
|
||||
|
||||
| Component | File | Purpose |
|
||||
|-----------|------|---------|
|
||||
| `Service` (LoadBalancer) | `clusters/main/proxy/traefik-lb-service.yaml` | VIP `.203`, selects Traefik pods |
|
||||
| `Namespace` | `clusters/main/proxy/namespace.yaml` | Shared proxy namespace |
|
||||
| ArgoCD App | `clusters/main/apps/proxy.yaml` | GitOps for proxy namespace |
|
||||
|
||||
### traefik-lb-service.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: traefik-lb
|
||||
namespace: kube-system
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
loadBalancerIP: 10.0.30.203
|
||||
selector:
|
||||
app.kubernetes.io/instance: rke2-traefik-kube-system
|
||||
app.kubernetes.io/name: rke2-traefik
|
||||
ports:
|
||||
- name: web
|
||||
port: 80
|
||||
targetPort: web
|
||||
- name: websecure
|
||||
port: 443
|
||||
targetPort: websecure
|
||||
```
|
||||
|
||||
### Why Cilium L2 (not MetalLB/keepalived)
|
||||
|
||||
- Cilium L2 LB IPAM already running (3 services use it: .200, .201, .202)
|
||||
- No additional software needed
|
||||
- Automatic failover — no manual VIP management
|
||||
- Pool: `10.0.30.200–250` (CiliumLoadBalancerIPPool `default-pool`)
|
||||
- L2 policy: `default-l2-policy` announces LB IPs via ARP on `^eth[0-9]+`
|
||||
|
||||
### Verification
|
||||
|
||||
After deploying, tested from mgmt-runner (.124):
|
||||
- `10.0.30.203:80/443` → 404 (no default route — correct)
|
||||
- `git.schoen.codes` via VIP → Gitea 1.26.1 ✅
|
||||
- `argocd` via VIP → ArgoCD UI ✅
|
||||
|
||||
### Firewall Cutover
|
||||
|
||||
Change NAT target: `10.0.60.10` → `10.0.30.203` (one rule, instant cutover).
|
||||
Rollback: revert NAT to `10.0.60.10`. CT99999 stays running throughout.
|
||||
|
||||
## Pattern for External Backends (not in K8s)
|
||||
|
||||
Use `ExternalName` Service + `IngressRoute` CRD:
|
||||
|
||||
```yaml
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: proxy
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`grafana.familie-schoen.com`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: grafana-external
|
||||
port: 3000
|
||||
tls:
|
||||
certResolver: letsencrypt
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana-external
|
||||
namespace: proxy
|
||||
spec:
|
||||
type: ExternalName
|
||||
externalName: 10.0.30.141
|
||||
```
|
||||
|
||||
## SeaDoc StripPath Middleware
|
||||
|
||||
```yaml
|
||||
middlewares:
|
||||
seadoc-strippath:
|
||||
stripPrefix:
|
||||
prefixes:
|
||||
- /sdoc-server
|
||||
```
|
||||
|
||||
## Accessing CT99999
|
||||
|
||||
```bash
|
||||
# CT99999 is on proxmox7 (10.0.20.70)
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.70 "pct exec 99999 -- ..."
|
||||
```
|
||||
|
||||
## Remaining Steps
|
||||
|
||||
1. Add Let's Encrypt certResolver via HelmChartConfig for `rke2-traefik`
|
||||
2. Create IngressRoute + ExternalName Service for each of the 11 external backends
|
||||
3. Handle catchall-TCP (make specific, not HostSNI(`*`) — see pitfall in SKILL.md)
|
||||
4. Cutover firewall NAT: `10.0.60.10` → `10.0.30.203`
|
||||
5. Verify each service, then decommission CT99999 after 1 week
|
||||
|
||||
## Reference
|
||||
|
||||
Full migration plan: `docs/plans/2026-07-14-traefik-ct99999-k8s-migration.md` in iac-homelab repo.
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# Workload Assessment: Authelia Decommission + InfluxDB Evaluation — 2026-07-14
|
||||
|
||||
## Context
|
||||
|
||||
After Gitea migration to K8s, user asked "was wäre die nächste Workload
|
||||
für eine Migration auf den K8s?" — triggering a full infrastructure
|
||||
inventory and workload migration candidate assessment.
|
||||
|
||||
## Authelia (CT112) — Decommissioned Instead of Migrated
|
||||
|
||||
### Assessment Findings
|
||||
|
||||
Authelia appeared in the initial assessment as the #1 migration
|
||||
candidate (dependency for Traefik migration). Deep investigation revealed
|
||||
it was **completely unused**:
|
||||
|
||||
| Check | Finding |
|
||||
|-------|---------|
|
||||
| **Traefik middleware** | NO router used Authelia as `forwardAuth` — only a route TO Authelia existed |
|
||||
| **Registered users** | 2 (admin + dominik), neither actively logging in |
|
||||
| **DB freshness** | SQLite 305 KB, last modified August 2025 (11 months stale) |
|
||||
| **Domain** | Configured for `familie-schoen.com` (old domain, active is `schoen.codes`) |
|
||||
| **2FA** | TOTP configured but `default_policy: one_factor` (password only) |
|
||||
|
||||
### Decision: Stop + Delete (Not Migrate)
|
||||
|
||||
```bash
|
||||
# Stop CT
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
|
||||
"ssh root@proxmox6 'pct stop 112'"
|
||||
# Remove Traefik routes (authelia + authelia-admin routers + services)
|
||||
# Delete CT with storage
|
||||
pct destroy 112 --purge # fails if RBD already gone → see ct-deletion-stale-config ref
|
||||
rm -f /etc/pve/lxc/112.conf # cleanup stale config
|
||||
```
|
||||
|
||||
### Traefik Config Editing on CT99999
|
||||
|
||||
Removed Authelia routers + services from `/etc/traefik/conf.d/explicit-http.yml`
|
||||
on CT99999 (10.0.60.10). Traefik runs as systemd service (not Docker).
|
||||
|
||||
**Backtick escaping:** YAML rules contain `Host(\`domain\`)` with
|
||||
backtick-delimited expressions. SSH heredoc breaks these (shell
|
||||
interprets backticks as command substitution). Solution: base64-encode
|
||||
locally, decode on remote:
|
||||
|
||||
```bash
|
||||
cat << 'EOF' | base64 -w0
|
||||
http:
|
||||
routers:
|
||||
myapp:
|
||||
rule: "Host(`myapp.example.com`)"
|
||||
...
|
||||
EOF
|
||||
# SSH to remote:
|
||||
ssh root@10.0.60.10 "echo '<BASE64>' | base64 -d > /etc/traefik/conf.d/myfile.yml"
|
||||
```
|
||||
|
||||
After editing: backup old config (`cp file{,.bak-$(date +%F)}`), apply
|
||||
new config, `systemctl restart traefik`, check `/var/log/traefik/traefik.log`
|
||||
(not journald — Traefik logs to files on this host).
|
||||
|
||||
## InfluxDB (CT109 + CT134) — Next Migration Candidate
|
||||
|
||||
### Assessment Findings
|
||||
|
||||
| CT | Role | Effective Data | Allocated Disk | Status |
|
||||
|----|------|----------------|----------------|--------|
|
||||
| CT109 | Hot (90d retention) | 4.0 GB (76 shards, ~53 MB/day) | 30 GB | Active — HA writes continuously |
|
||||
| CT134 | Archive (5y retention) | 368 KB | 50 GB | **Dead** — bolt file last modified 12 days ago |
|
||||
|
||||
### Key Finding: CT134 Is Dead
|
||||
|
||||
CT134 (archive) receives no data. The InfluxDB bolt file was last
|
||||
modified July 2 (12 days ago). The `ha_archive_5y_5m` bucket has 368 KB
|
||||
of data. 50 GB disk allocated for 368 KB of data — pure waste.
|
||||
|
||||
### Storage Analysis Method
|
||||
|
||||
```bash
|
||||
# Find InfluxDB data path (v2 uses /var/lib/influxdb/, not /var/lib/influxdb2/)
|
||||
pct exec 109 -- bash -c "du -sh /var/lib/influxdb/"
|
||||
# Check shard count and sizes:
|
||||
pct exec 109 -- bash -c "du -sh /var/lib/influxdb/engine/data/BUCKET_ID/autogen/*/ | sort -rh | head -5"
|
||||
# Check bolt file mtime (staleness indicator):
|
||||
pct exec 134 -- bash -c "stat /var/lib/influxdb/influxd.bolt | grep Modify"
|
||||
# List buckets:
|
||||
pct exec 109 -- influx bucket list
|
||||
```
|
||||
|
||||
### Migration Recommendation
|
||||
|
||||
Consolidate CT109 + CT134 → single K8s InfluxDB with two buckets:
|
||||
- `ha_hot_90d` (90-day retention, ~4 GB)
|
||||
- `ha_archive_5y` (5-year retention, downsamples from hot)
|
||||
|
||||
10 Gi PVC sufficient (4 GB current + growth buffer). Frees 2 CTs on
|
||||
proxmox6 (the busiest node). Official Helm chart `influxdata/influxdb2`.
|
||||
Data migration via `influx backup` → `influx restore`.
|
||||
|
||||
### CT134 Action
|
||||
|
||||
CT134 can be deleted immediately — it receives no data and wastes 50 GB.
|
||||
Whether to reactivate the archive (configure InfluxDB downsampling task)
|
||||
or abandon archiving is a user decision.
|
||||
|
||||
## Old Docker Host (10.0.30.100) InfluxDB Investigation
|
||||
|
||||
User recalled "mehrere Jahre HomeAssistant Daten" on 10.0.30.100. Deep
|
||||
forensic search (see `references/docker-volume-forensics-2026-07.md` in
|
||||
docker-host-administration skill) found:
|
||||
|
||||
- **InfluxDB v1 container was removed years ago** — only a 15 KB skeleton
|
||||
volume remained (`influxd.bolt` + `influxd.sqlite` from Jan 2023)
|
||||
- **Shell history** showed `collectd` + `cadvisor` databases (2017-era
|
||||
system monitoring), NOT HomeAssistant data
|
||||
- **No `.tsm` files** anywhere on the host (root, ZFS pool, Docker volumes)
|
||||
- The old InfluxDB v1 was a **different service** (system monitoring via
|
||||
collectd/cadvisor) than the current InfluxDB v2 on CT109 (HA sensors)
|
||||
- HA data was never on 10.0.30.100 — the HA→InfluxDB v2 integration
|
||||
writes to CT109 directly
|
||||
|
||||
**Lesson:** "Running" ≠ "had the data you're looking for." The old
|
||||
InfluxDB v1 served a different purpose (system metrics) than the current
|
||||
v2 (HA sensor data). Cross-reference what databases existed historically
|
||||
(shell history, config files) before assuming data loss.
|
||||
Reference in New Issue
Block a user