feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
@@ -0,0 +1,274 @@
+# Gitea Backup CronJob — Session Detail (2026-07-14)
+
+## Context
+
+After deploying InfluxDB backup CronJob (§4.3i), user requested the
+same backup pattern for Gitea. Unlike InfluxDB (single PVC, single
+backup command), Gitea has two data stores:
+1. **PVC** (`/data`) — repos, LFS, config, attachments, avatars
+2. **External Galera DB** (10.0.30.70:3306) — users, issues, PRs, labels
+
+## Design Decisions
+
+### tar over `gitea dump`
+- `gitea dump` produces a zip but can miss LFS objects and repos outside
+  the default path
+- `tar czf /data` captures everything reliably
+- Trade-off: larger archive (143 MB), but complete
+
+### kubectl exec + kubectl cp for PVC access
+- PVC is RWO, mounted by the running `gitea-0` pod
+- Can't mount it in the backup pod (Multi-Attach error)
+- Solution: exec into the running pod to tar, then cp the archive out
+- Requires RBAC: ServiceAccount with pods/exec permission
+
+### Separate mysqldump initContainer
+- `mysql:8.0` image has `mysqldump`
+- Credentials from existing `gitea-db` K8s secret (already used by Gitea)
+- `--single-transaction` for consistent dump without locking
+
+### S3 credentials reuse
+- Same Norris S3 account as Postgres/InfluxDB backups
+- New ExternalSecret per namespace, same 1Password item keys
+- New bucket: `homelab-gitea-backup`
+
+## Manifest Files Created
+
+| File | Purpose |
+|------|---------|
+| `clusters/main/gitea/backup-rbac.yaml` | ServiceAccount + Role (pods/exec, pods/get) + RoleBinding |
+| `clusters/main/gitea/backup-s3-secret.yaml` | ExternalSecret → 1Password `postgres-s3-backup` creds |
+| `clusters/main/gitea/backup-cronjob.yaml` | CronJob: 2 initContainers (files + db) + 1 upload container |
+
+Commit: `1679786`
+
+## CronJob Spec
+
+- **Schedule:** `30 3 * * *` (03:30 UTC — 30 min after InfluxDB backup)
+- **initContainer 1 (gitea-files):** `bitnami/kubectl:1.31` — kubectl exec tar + kubectl cp
+- **initContainer 2 (gitea-db):** `mysql:8.0` — mysqldump against Galera
+- **Container (upload):** `amazon/aws-cli:2.35.22` — aws s3 cp to Norris S3
+- **Shared volume:** emptyDir (passed between all containers)
+- **ServiceAccount:** `gitea-backup` (with RBAC for kubectl exec/cp)
+
+## S3 Upload Paths
+
+```
+s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-data.tar.gz
+s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-db.sql.gz
+```
+
+## Execution Results
+
+### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`
+
+Both buckets (`homelab-influxdb-backup` + `homelab-gitea-backup`) were
+created successfully via one-off pods. **CRITICAL PITFALL**: `aws s3 mb`
+with `--region nsc-nbg` fails with `InvalidLocationConstraint: The
+nsc-nbg location constraint is not valid`. Must use `--region us-east-1`
+for bucket creation. The `nsc-nbg` region works for subsequent `aws s3 cp`
+operations but NOT for `s3 mb`.
+
+```bash
+# CORRECT — us-east-1 for bucket creation
+kubectl run aws-mb-$bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
+  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
+  --command -- sh -c "aws s3 mb s3://$bucket \
+    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
+```
+
+### InfluxDB Backup — VERIFIED SUCCESSFUL ✅
+
+InfluxDB backup CronJob ran successfully:
+- 77 shards backed up
+- `backup.tar.gz` = 25.4 MB
+- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
+- Total time: ~45 seconds
+
+### Gitea Backup — BLOCKED by Image Pull Issues
+
+The Gitea CronJob requires a kubectl image for the initContainer that
+runs `kubectl exec` + `kubectl cp`. Multiple images failed:
+
+1. **`bitnami/kubectl:1.31`** — tag does NOT exist on Docker Hub
+   (`failed to resolve reference: docker.io/bitnami/kubectl:1.31: not found`)
+2. **`bitnami/kubectl:1.31.4`** — same NotFound error
+3. **`bitnami/kubectl:latest`** — pulling but extremely slow (Docker Hub rate-limit)
+4. **`alpine/k8s:1.31.4`** — pulling but took 3+ minutes, still Pending
+
+The `amazon/aws-cli:2.35.22` and `mysql:8.0` images pull fine (already
+cached from the InfluxDB backup CronJob). Only the kubectl image is
+problematic.
+
+### kubectl cp Fails on Large Files
+
+Attempting `kubectl cp` of the 143 MB gitea-data.tar.gz from the
+mgmt-runner into an upload pod failed with:
+```
+write tcp 10.0.30.124:35976->10.0.30.51:6443: write: connection reset by peer
+websocket: close 1006 (abnormal closure): unexpected EOF
+```
+
+`kubectl cp` uses the K8s API websocket to stream file data. Large files
+(>~50 MB) are unreliable — the websocket connection resets before
+transfer completes. Small files (<1 MB) work fine.
+
+### ArgoCD Reports Success But Doesn't Create New Resources
+
+When new manifest files (backup-rbac.yaml, backup-s3-secret.yaml,
+backup-cronjob.yaml) were added to the existing `gitea-config` Application's
+directory (`clusters/main/gitea/`), ArgoCD showed:
+- Sync status: `Succeeded`
+- Operation phase: `Succeeded`
+- Message: `successfully synced (all tasks run)`
+- BUT: `kubectl get cronjob -n gitea` returned `No resources found`
+
+This is a known ArgoCD behavior with `ServerSideApply=true` — new files
+in an existing directory aren't always detected on the first sync cycle.
+The hard-refresh annotation was applied but still showed `OutOfSync`.
+
+**Fix that worked:** Transfer the manifest files to VM200 via base64
+encoding through SSH, then `kubectl apply -f` manually. The CronJob,
+RBAC, and ExternalSecret were all created successfully this way.
+
+```bash
+# Transfer files from Hermes host to VM200 via base64
+B64=$(base64 -w0 /path/to/manifest.yaml)
+ssh root@10.0.30.124 "echo '$B64' | base64 -d > /root/iac-homelab/path/to/manifest.yaml"
+ssh root@10.0.30.124 "KUBECONFIG=/root/.kube/config kubectl apply -f /root/iac-homelab/path/to/manifest.yaml"
+```
+
+### VM200 Repo Remote Not Updated After Gitea Migration
+
+The mgmt-runner's `/root/iac-homelab` repo still pointed at the old
+Gitea on CT108 (`http://...@10.0.30.105:3000/...`). After ArgoCD was
+switched to K8s Gitea, the VM200 repo became stale. New commits pushed
+to K8s Gitea were invisible to VM200.
+
+**Fix**: Update the remote URL on VM200 to point at K8s Gitea, or
+transfer files via base64 encoding through SSH.
+
+### Manual Backup from mgmt-runner — Successful Workaround
+
+When the CronJob pod approach is blocked by image issues, run the backup
+directly from the mgmt-runner (VM200, 10.0.30.124) which has kubectl +
+mysqldump already installed:
+
+```bash
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
+KUBECONFIG=/root/.kube/config
+set -e
+
+GITEA_POD=$(kubectl -n gitea get pod -l app.kubernetes.io/name=gitea \
+  -o jsonpath="{.items[0].metadata.name}")
+AKI=$(kubectl -n gitea get secret gitea-s3-backup \
+  -o jsonpath="{.data.aws_access_key_id}" | base64 -d)
+SAK=$(kubectl -n gitea get secret gitea-s3-backup \
+  -o jsonpath="{.data.aws_secret_access_key}" | base64 -d)
+DB_PASS=$(kubectl -n gitea get secret gitea-db \
+  -o jsonpath="{.data.password}" | base64 -d)
+DATE=$(date +%Y-%m-%d)
+
+# 1. tar /data from gitea pod
+kubectl -n gitea exec $GITEA_POD -- tar czf /tmp/gitea-data.tar.gz -C / data
+kubectl -n gitea cp $GITEA_POD:/tmp/gitea-data.tar.gz /tmp/gitea-data.tar.gz
+kubectl -n gitea exec $GITEA_POD -- rm -f /tmp/gitea-data.tar.gz
+
+# 2. mysqldump against Galera
+mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
+  --password="$DB_PASS" --single-transaction \
+  --routines --triggers gitea | gzip > /tmp/gitea-db.sql.gz
+
+# 3. Upload via one-off pod with emptyDir (avoid hostPath — pods run on
+#    different nodes!)
+#    See pitfall below about hostPath
+'
+```
+
+**Pitfall: hostPath volumes are node-local.** A pod scheduled on
+worker-01 cannot read files from the mgmt-runner's `/tmp` via hostPath.
+The pod lands on whichever node the scheduler chooses. Use `kubectl cp`
+to transfer files into a pod with emptyDir, or use `nodeName:` to pin
+the pod to a specific node (fragile — not recommended for production).
+
+**Pitfall: `kubectl cp` to upload pod fails for large files.** The
+143 MB gitea-data.tar.gz caused websocket connection resets. Workaround:
+split the upload into smaller chunks, or use an initContainer that
+generates the data directly (avoiding the cp step entirely).
+
+## Recommended Fix for CronJob
+
+The CronJob approach needs a kubectl image that can be reliably pulled.
+Options:
+
+1. **Pre-pull the image on all worker nodes** — `crictl pull alpine/k8s:1.31.4`
+   on each worker, then the CronJob pulls instantly from local cache.
+2. **Use a different kubectl image from a non-Docker-Hub registry** — e.g.
+   `registry.k8s.io/kubectl-sidecar` or an internally hosted image.
+3. **Restructure the CronJob to avoid kubectl entirely** — mount the PVC
+   read-only (requires PVC to be RWX, or stop the pod first).
+4. **Run backup as a script on the mgmt-runner** via systemd timer or
+   crontab — pragmatic but not GitOps-managed.
+
+Until the image issue is resolved, the manual mgmt-runner approach
+(above) produces correct backups.
+
+## Remaining Steps
+
+1. ~~Create S3 bucket `homelab-gitea-backup`~~ ✅ Done
+2. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
+3. ~~InfluxDB backup test~~ ✅ Verified (25.4 MB uploaded)
+4. **Fix Gitea CronJob kubectl image** — pre-pull or use alternative
+5. **Gitea backup test run** — verify both files in S3
+6. Monitor first scheduled runs (03:00 + 03:30 UTC)
+
+## Pitfalls Summary
+
+### `bitnami/kubectl:1.31` tag does NOT exist
+The bitnami/kubectl image uses full semver tags (`1.31.4`, not `1.31`).
+However, even with correct tags, Docker Hub rate-limiting causes
+extremely slow pulls (3+ minutes to Pending). Pre-pull on all workers
+or use an alternative registry.
+
+### S3 bucket creation: `--region us-east-1` not `nsc-nbg`
+`aws s3 mb` with `--region nsc-nbg` fails with `InvalidLocationConstraint`.
+Use `--region us-east-1` for bucket creation. The `nsc-nbg` region works
+for subsequent `aws s3 cp` operations.
+
+### `kubectl cp` unreliable for large files (>50 MB)
+WebSocket connection resets before transfer completes. Small files
+(<1 MB) work fine. For large file transfers, consider splitting or
+avoiding `kubectl cp` entirely.
+
+### VM200 repo remote may be stale after Gitea migration
+After switching ArgoCD from CT108 to K8s Gitea, the mgmt-runner's local
+repo remote still points at the old CT108 URL. New commits are invisible.
+Fix: `git remote set-url origin <new-url>` on VM200, or transfer files
+via base64 through SSH.
+
+### hostPath volumes are node-local
+A pod using `hostPath: /tmp` reads from the NODE it's scheduled on, not
+from the machine that created the files. If files are on the mgmt-runner
+but the pod runs on a worker, the files are invisible. Use emptyDir +
+`kubectl cp`, or pin the pod with `nodeName:` (fragile).
+
+### RBAC required for kubectl exec/cp
+Without explicit Role granting `pods/exec` + `pods/get`, the backup
+pod's kubectl commands fail with `forbidden`. Must create ServiceAccount,
+Role, and RoleBinding before the CronJob can function.
+
+### kubectl cp namespace syntax
+`kubectl cp` requires `namespace/pod:path` format. Missing the
+namespace prefix causes failures when the backup pod and target pod
+are in different namespaces (though typically same namespace here).
+
+## Pattern Reuse
+
+This pattern (PVC tar via kubectl exec + external DB dump + S3 upload)
+applies to any K8s service with both PVC data and an external database:
+- Gitea (PVC + Galera)
+- Future services with similar dual-data-store architecture
+
+For services with only PVC data (no external DB), skip the mysqldump
+initContainer. For services with only an external DB (no PVC), skip
+the kubectl exec initContainer.